---
abstract: 'Learning representations with diversified information remains as an open problem.  Towards learning diversified representations, a new approach, termed Information Competing Process (ICP), is proposed in this paper.  Aiming to enrich the information carried by feature representations, ICP separates a representation into two parts with different mutual information constraints.  The separated parts are forced to accomplish the downstream task independently in a competitive environment which prevents the two parts from learning what each other learned for the downstream task.  Such competing parts are then combined synergistically to complete the task.  By fusing representation parts learned competitively under different conditions, ICP facilitates obtaining diversified representations which contain rich information.  Experiments on image classification and image reconstruction tasks demonstrate the great potential of ICP to learn discriminative and disentangled representations in both supervised and self-supervised learning settings.' 
author:
- |
    Jie Hu$^{12}$,  Rongrong Ji$^{123}\thanks{Corresponding Author.}$,  ShengChuan Zhang$^{1}$,  Xiaoshuai Sun$^{1}$,\
    **Qixiang Ye$^{4}$,  Chia-Wen Lin$^{5}$,  Qi Tian$^{6}$.**\
    $^1$Media Analytics and Computing Lab, Department of Artificial Intelligence,\
    School of Informatics, Xiamen University.\
    $^2$National Institute for Data Science in Health and Medicine, Xiamen University.\
    $^3$Peng Cheng Laboratory.   $^4$University of Chinese Academy of Sciences.\
    $^5$National Tsing Hua University.   $^6$Noah’s Ark Lab, Huawei.\
bibliography:
- 'egbib.bib'
title: Information Competing Process for Learning Diversified Representations
---

Introduction
============

Representation learning aims to make the learned feature representations more effective on extracting useful information from input for downstream tasks [@bengio2013representation], which has been an active research topic in recent years and has become the foundation for many tasks [@radford2015unsupervised; @chen2016infogan; @hamilton2017inductive; @hu2018towards; @zhang2019freeanchor; @kolesnikov2019revisiting; @chen2019variational].  Notably, a majority of works about representation learning have been studied from the viewpoint of mutual information constraint.  For instance, the Information Bottleneck (IB) theory [@tishby2000information; @alemi2016deep] minimizes the information carried by representations to fit the target outputs, and the generative models such as $\beta$-VAE [@higgins2017beta; @burgess2018understanding] also rely on such information constraint to learn disentangled representations.  Some other works [@linsker1988self; @bell1995information; @oord2018representation; @hjelm2018learning] reveal the advantages of maximizing the mutual information for learning discriminative representations.  Despite the exciting progresses, learning diversified representations remains as an open problem.  Diversified representations are learned with different constraints encouraging representation parts to extract various information from inputs, which results in powerful features to represent the inputs.  In principle, a good representation learning approach is supposed to discriminate and disentangle the underlying explanatory factors hidden in the input [@bengio2013representation].  However, this goal is hard to realize as the existing methods typically resort to only one type of information constraint.  As a consequence, the information diversity of the learned representations is deteriorated.  In this paper we present a diversified representation learning scheme, termed Information Competing Process (ICP), which handles the above issues through a new information diversifying objective.  First, the separated representation parts learned with different constraints are forced to accomplish the downstream task competitively.  Then, the rival representation parts are combined to solve the downstream task synergistically.  A novel solution is further proposed to optimize the new objective in both supervised and self-supervised learning settings. 

We verify the effectiveness of the proposed ICP on both image classification and image reconstruction tasks, where neural networks are used as the feature extractors.  In the supervised image classification task, we integrate ICP with four different network architectures (*i.e.*, VGG [@simonyan2014very], GoogLeNet [@szegedy2015going], ResNet [@he2016deep], and DenseNet [@huang2017densely]) to demonstrate how the diversified representations boost classification accuracy.  In the self-supervised image reconstruction task, we implement ICP with $\beta$-VAE [@higgins2017beta] to investigate its ability of learning disentangled representations to reconstruct and manipulate the inputs.  Empirical evaluations suggest that ICP fits finer labeled dataset and disentangles fine-grained semantic information for representations. 

Related Work
============

**Representation Learning with Mutual Information.**  Mutual information has been a powerful tool in representation learning for a long time.  In the unsupervised setting, mutual information maximization is typically studied, which targets at adding specific information to the representation and forces the representation to be discriminative.  For instance, the InfoMax principle [@linsker1988self; @bell1995information] advocates maximizing mutual information between the inputs and the representations, which forms the basis of independent component analysis [@hyvarinen2000independent].  Contrastive Predictive Coding [@oord2018representation] and Deep InfoMax [@hjelm2018learning] maximize mutual information between global and local representation pairs, or the input and global/local representation pairs.  In the supervised or self-supervised settings, mutual information minimization is commonly utilized.  For instance, the Information Bottleneck (IB) theory [@tishby2000information] uses the information theoretic objective to constrain the mutual information between the input and the representation.  IB was then introduced to deep neural networks [@tishby2015deep; @shwartz2017opening; @saxe2018information], and Deep Variational Information Bottleneck (VIB) [@alemi2016deep] was recently proposed to refine IB with a variational approximation.  Another group of works in self-supervised setting adopt generative models to learn representations [@kingma2013auto; @rezende2014stochastic], in which the mutual information plays an important role in learning disentangled representations.  For instance, $\beta$-VAE [@higgins2017beta] is a variant of Variation Auto-Encoder [@kingma2013auto] that attempts to learn a disentangled representation by optimizing a heavily penalized objective with mutual information minimization.  Recent works in [@burgess2018understanding; @kim2018disentangling; @chen2018isolating] revise the objective of $\beta$-VAE by applying various constraints.  One special case is InfoGAN [@chen2016infogan], which maximizes the mutual information between representation and a factored Gaussian distribution.  Besides, Mutual Information Neural Estimation [@belghazi2018mine] estimates the mutual information of continuous variables.  Differing from the above schemes, the proposed ICP leverages both mutual information maximization and minimization to create competitive environment for learning diversified representations. ** Representation Collaboration.**  The idea of collaborating neural representations can be found in Neural Expectation Maximization [@greff2017neural] and Tagger [@greff2016tagger], which uses different representations to group and represent individual entities.  The Competitive Collaboration [@ranjan2018adversarial] method is the most relevant to our work.  It defines a three-player game with two competitors and a moderator, where the moderator takes the role of a critic and the two competitors collaborate to train the moderator.  Unlike Competitive Collaboration, the proposed ICP enforces two (or more) representation parts to be complementary through different mutual information constraints for the same downstream task by a competitive environment, which endows the capability of learning more discriminative and disentangled representations. 

![  The proposed Information Competing Process.  In the competitive step, the rival representation parts are forced to accomplish the downstream task solely by preventing both parts from knowing what each other learned under different constraints for the task.  In the synergetic step, these representation parts are combined to complete the downstream task synthetically.  ICP can be generalized to arbitrary number of constrained parts, and in this paper we make an example of two. []{ data-label="fig1"}](./Fig1.pdf){width="4.5in"}

Information Competing Process
=============================

The key idea of ICP is depicted in Fig.  \[fig1\], in which different representation parts compete and collaborate with each other to diversify the information.  In this section, we first unify supervised and self-supervised objectives for acheving the target tasks.  Then, the information competing objective for learning diversified representations is proposed. 

Unifying Supervised and Self-Supervised Objectives
--------------------------------------------------

The information constraining objective in supervised setting has the same form as that of self-supervised setting except the target outputs.  We therefore unify these two objectives by using $t$ as the output of the downstream tasks.  In supervised setting, $t$ represents the label of input $x$. In self-supervised setting, $t$ represents the input $x$ itself.  This leads to the unified objective function linking the representation $r$ of input $x$ and target $t$ as: $$\begin{split}
\label{eq1}
\max\big[\mathcal{I}(r,t)\big],
\end{split}$$ where $\mathcal{I}(\cdot,\cdot)$ stands for the mutual information.  This unified objective describes a constraint with the goal of maximizing the mutual information between the representation $r$ and the target $t$.

Separating and Diversifying Representations
-------------------------------------------

To explicitly diversify the information on representations, we directly separate the representation $r$ into two parts $[z,y]$ with different constraints, and encourage representations to learn discrepant information from the input $x$. Specifically, we constrain the information capacity of representation part $z$ while increasing the information capacity of representation part $y$. To that effect, we have the following objective function: $$\begin{split}
\label{eq2}
\max\big[\mathcal{I}(r,t) + \alpha\mathcal{I}(y,x) - \beta\mathcal{I}(z,x)\big],
\end{split}$$ where $\alpha>0$ and $\beta>0$ are the regularization factors. 

Competition of Representation Parts
-----------------------------------

To prevent any one of the representation parts from dominating the downstream task, we let $z$ and $y$ to accomplish the downstream task $t$ solely by utlizing the mutual information constraints $\mathcal{I}(z, t)$ and $\mathcal{I}(y, t)$. Additionally, for ensuring the representations catch diversified information through different constraints, ICP prevents $z$ and $y$ from knowing what each other learned for the downstream task, which is realized by enforcing $z$ and $y$ independent of each other.  These constraints result in a competitive environment to enrich the information carried by representations.  Correspondingly, the objective of ICP is concluded as: $$\begin{split}
\label{eq3}
\max\big[\underbrace{\mathcal{I}(r,t)}_{\text{\textcircled{1} Synergy}}
+
\underbrace{\alpha\mathcal{I}(y,x)}_{\text{\textcircled{2} Maximization}}
-
\underbrace{\beta\mathcal{I}(z,x)}_{\text{\textcircled{3} Minimization}}
+
\underbrace{\mathcal{I}(z,t) + \mathcal{I}(y,t) - \gamma\mathcal{I}(z,y)}_{\text{\textcircled{4} Competition}} \big],
\end{split}$$ where $\gamma>0$ is the regularization factor. 

Optimizing the Objective of ICP
===============================

In this section, we derive a solution to optimize the objective of ICP.  Although all terms of this objective have the same formulation that calculates the mutual information between two variables, they need to be optimized using different methods due to their different aims.  We therefore classify these terms as the mutual information minimization term $\mathcal{I}(z,x)$, the mutual information maximization term $\mathcal{I}(y,x)$, the inference terms $\mathcal{I}(z,t), \mathcal{I}(y,t), \mathcal{I}(r, t)$ and the predictability minimization term $\mathcal{I}(z,y)$ to find the solution. 

Mutual Information Minimization Term
------------------------------------

To minimize the mutual information between $x$ and $z$, we can find out a tractable upper bound for the intractable $\mathcal{I}(z, x)$. In the existing works [@kingma2013auto; @alemi2016deep], $\mathcal{I}(z, x)$ is usually defined under the joint distribution of inputs and their encoding distribution, as it is the constraint between the inputs and the representations.  Concretely, the formulation is derived as: $$\begin{split}
\label{eq4}
\mathcal{I}(z, x) &= \int\int P(z, x)\log\frac{P(z,x)}{P(z)P(x)}dxdz = \int\int P(z, x)\log \frac{P(z|x)}{P(z)}dxdz\\
&=\int\int P(z,x)\log P(z|x)dxdz - \int\int P(x|z)P(z)\log P(z)dxdz \\
&=\int\int P(z,x)\log P(z|x)dxdz - \int P(z)\log P(z)dz. 
\end{split}$$ Let $Q(z)$ be a variational approximation of $P(z)$, we have: $$\begin{split}
\label{eq5}
KL\big[P(z)||Q(z)\big] \ge 0 \Rightarrow \int P(z)\log P(z)dz \ge \int P(z)\log Q(z)dz. 
\end{split}$$ According to Eq.  \[eq5\], the trackable upper bound after applying the variational approximation is: $$\begin{split}
\label{eq6}
\mathcal{I}(z,x) \le \int\int P(z|x)P(x)\log \frac{P(z|x)}{Q(x)}dxdz =\mathbb{E}_{x\sim P(x)}\Big[KL\big[P(z|x)||Q(z)\big]\Big],
\end{split}$$ which enforces the extracted $z$ conditioned on $x$ to a predefined distribution $Q(z)$ such as a standard Gaussian distribution. 

Mutual Information Maximization Term
------------------------------------

To maximize the mutual information between $x$ and $y$, we deduce a tractable alternate for the intractable $\mathcal{I}(y, x)$. Specifically, like the above minimization term, the mutual information should also be defined as the joint distribution of inputs and their encoding distribution.  As it is hard to derive a tractable lower bound for this term, we expand the mutual information as: $$\begin{split}
\label{eq7}
\mathcal{I}(y, x) = \int\int P(y, x)\log \frac{P(y,x)}{P(y)P(x)}dxdy = KL\big[P(y|x)P(x)||P(y)P(x)\big]. 
\end{split}$$ Since Eq.  \[eq7\] means that maximizing the mutual information is equal to enlarging the Kullback-Leibler (KL) divergence between distributions $P(y|x)P(x)$ and $P(y)P(x)$, and the maximization of KL divergence is divergent.  We instead maximize the Jensen-Shannon (JS) divergence as an alternative which approximates the maximization of KL divergence but is convergent.  As [@nowozin2016f], a tractable variational estimation of JS divergence can be defined as: $$\begin{split}
\label{eq8}
JS\big[P(y|x)P(x)||P(y)P(x)\big] =\max\Big[&\mathbb{E}_{(y,x)\sim P(y|x)P(x)}\big[\log D(y,x)\big] \\
+&\mathbb{E}_{(\hat{y},x)\sim P(y)P(x)}\big[\log\big(1-D(\hat{y},x)\big)\big]\Big],
\end{split}$$ where $D$ is a discriminator that estimates the probability of the input pair, $(y,x)$ is the positive pair sampled from $P(y|x)P(x)$, and $(\hat{y}, x)$ is the negative pair sampled from $P(y)P(x)$. As $\hat{y}$ shoule be the representation conditioned on $x$, we disorganize $y$ in the positive pair $(x,y)$ to obtain the negative pair $(x,\hat{y})$.

Inference Term
--------------

The inference terms in Eq.  \[eq3\] should be defined as the joint distribution of representation and the output distribution of downstream task solver.  We take $\mathcal{I}(r, t)$ as an example, and $\mathcal{I}(z, t), \mathcal{I}(y, t)$ have the same formulation with $\mathcal{I}(r, t)$. We expand this mutual information term as: $$\begin{split}
\label{eq9}
\mathcal{I}(r, t) &= \int\int P(r,t)\log \frac{P(t|r)}{P(t)}drdt \\
&= \int\int P(r,t)\log P(t|r)dtdr - \int P(t)\log P(t)dt \\
&= \int\int P(r,t)\log P(t|r)dtdr + \mathcal{H}(t) \\
&\ge \int\int P(t|r)p(r) \log P(t|r)dtdr,
\end{split}$$ where $\mathcal{H}(t) \ge 0$ is the information entropy of $t$. Let $Q(t|r)$ be a variational approximation of $P(t|r)$, we have: $$\begin{split}
\label{eq10}
KL\big[P(t|r) || Q(t|r)\big] \ge 0 \Rightarrow \int P(t|r)\log P(t|r)dt \ge \int P(t|r)\log Q(t|r)dt. 
\end{split}$$ By applying the variational approximation, the trackable lower bound of the mutual information between $r$ and $t$ is: $$\begin{split}
\label{eq11}
\mathcal{I}(r,t)\ge\int\int P(r,t) \log Q(t|r) dtdr. 
\end{split}$$ Based on the above formulation, we derive different objectives for the supervised and self-supervised settings in what follows. ** Supervised Setting.**  In the supervised setting, $t$ represents the known target labels.  By assuming that the representation $r$ is not dependent on the label $t$, *i.e.*, $P(r|x, t) = P(r|x)$, we have: $$\begin{split}
\label{eq12}
P(x,r,t)=P(r|x,t)P(t|x)P(x) =P(r|x)P(t|x)P(x). 
\end{split}$$ Accordingly, the joint distribution of $r$ and $t$ can be written as: $$\begin{split}
\label{eq13}
P(r,t)=\int P(x,r,t)dx =\int P(r|x)P(t|x)P(x)dx. 
\end{split}$$ Combining Eq.  \[eq11\] with Eq.  \[eq13\], we get the lower bound of the inference term in the supervised setting: $$\begin{split}
\label{eq14}
\mathcal{I}(r,t)&\ge\int\int\int P(x)P(r|x)P(t|x)\log Q(t|r)dtdrdx \\
&=\mathbb{E}_{x\sim P(x)}\Big[\mathbb{E}_{r\sim P(r|x)}\big[\int P(t|x)\log Q(t|r)dt\big]\Big]. 
\end{split}$$ Since the conditional probability $P(t|x)$ represents the distribution of labels in the supervised setting, Eq.  \[eq14\] is actually the cross entropy loss for classification. ** Self-supervised Setting.**  In the self-supervised setting, $t$ is the input $x$ itself.  Therefore, Eq \[eq11\] can be directly derived as: $$\begin{split}
\label{eq15}
\mathcal{I}(r,x) \ge \int\int P(r|x)P(x)\log Q(x|r)dxdt =\mathbb{E}_{x\sim P(x)}\Big[\mathbb{E}_{r\sim P(r|x)}\big[\log Q(x|r)\big]\Big]. 
\end{split}$$ Assuming $Q(t|r)$ as a Gaussian distribution, Eq.  \[eq15\] can be expanded as the L2 reconstruction loss for the input $x$.

Predictability Minimization Term
--------------------------------

To diversify the information and prevent the dominance of one representation part, we constrain the mutual information between $z$ and $y$, which equals to make $z$ and $y$ be independent with each other.  Inspired by [@schmidhuber1992learning], we introduce a predictor $H$ to fulfill this goal.  Concretely, we let $H$ predict $y$ conditioned on $z$, and prevent the extractor from producing $z$ which can predict $y$. The same operation is conducted on $y$ to $z$. The corresponding objective is: $$\begin{split}
\label{eq16}
\min\max\Big[\mathbb{E}_{z\sim P(z|x)}\big[H(y|z)\big]+\mathbb{E}_{y\sim P(y|x)}\big[H(z|y)\big]\Big]. 
\end{split}$$ So far, we have all the tractable bounds and alternatives for optimizing the information diversifying objective of ICP.  The optimization process is summarized in Alg.  \[alg1\]. 

\[0.85\]

                                 VGG16 [@simonyan2014very]    GoogLeNet [@szegedy2015going]     ResNet20 [@he2016deep]      DenseNet40 [@huang2017densely]
  ---------------------------- ----------------------------- ------------------------------- ----------------------------- --------------------------------
            Baseline                       6.67                           4.92                           7.63                            5.83
      VIB [@alemi2016deep]         6.81$^{\uparrow0.14}$          5.09$^{\uparrow0.17}$         6.95$^{\downarrow0.68}$        5.72$^{\downarrow0.11}$
   DIM\* [@hjelm2018learning]     6.54$^{\downarrow0.13}$        4.65$^{\downarrow0.27}$        7.61$^{\downarrow0.02}$         6.15$^{\uparrow0.32}$
              VIB                  6.86$^{\uparrow0.19}$         4.88$^{\downarrow0.04}$        6.85$^{\downarrow0.78}$         6.36$^{\uparrow0.53}$
             DIM\*                 7.24$^{\uparrow0.57}$          4.95$^{\uparrow0.03}$         7.46$^{\downarrow0.17}$        5.60$^{\downarrow0.23}$
              ICP                  6.97$^{\uparrow0.30}$         4.76$^{\downarrow0.16}$        6.47$^{\downarrow1.16}$         6.13$^{\uparrow0.30}$
              ICP                 6.59$^{\downarrow0.08}$        4.67$^{\downarrow0.25}$        7.33$^{\downarrow0.30}$        5.63$^{\downarrow0.20}$
            **ICP**             **6.10**$^{\downarrow0.57}$    **4.26**$^{\downarrow0.66}$    **6.01**$^{\downarrow1.62}$    **4.99**$^{\downarrow0.84}$

  : Classification error rates (%) on CIFAR-10 test set.[]{data-label="tab1"}

\[0.85\]

                                 VGG16 [@simonyan2014very]     GoogLeNet [@szegedy2015going]      ResNet20 [@he2016deep]      DenseNet40 [@huang2017densely]
  ---------------------------- ------------------------------ ------------------------------- ------------------------------ --------------------------------
            Baseline                       26.41                           20.68                          31.91                           27.55
      VIB [@alemi2016deep]         26.56$^{\uparrow0.15}$         20.93$^{\uparrow0.25}$         30.84$^{\downarrow1.07}$        26.37$^{\downarrow1.18}$
   DIM\* [@hjelm2018learning]      26.74$^{\uparrow0.33}$         20.94$^{\uparrow0.26}$          32.62$^{\uparrow0.71}$         27.51$^{\downarrow0.04}$
              VIB                 26.08$^{\downarrow0.33}$        22.09$^{\uparrow1.41}$         29.74$^{\downarrow2.17}$         29.33$^{\uparrow1.78}$
             DIM\*                25.72$^{\downarrow0.69}$        21.74$^{\uparrow1.06}$         30.16$^{\downarrow1.75}$        27.15$^{\downarrow0.40}$
              ICP                  26.73$^{\uparrow0.32}$         20.90$^{\uparrow0.22}$         28.35$^{\downarrow3.56}$        27.51$^{\downarrow0.04}$
              ICP                 26.37$^{\downarrow0.04}$        20.81$^{\uparrow0.13}$          32.76$^{\uparrow0.85}$         26.85$^{\downarrow0.70}$
            **ICP**             **24.54**$^{\downarrow1.87}$   **18.55**$^{\downarrow2.13}$    **28.13**$^{\downarrow3.78}$    **24.52**$^{\downarrow3.03}$

  : Classification error rates (%) on CIFAR-100 test set.[]{data-label="tab2"}

Experiments
===========

In experiments, all the probabilistic feature extractors, task solvers, predictor and discriminator are implemented by neural networks.  We suppose $Q(z), Q(t|r), Q(t|z), Q(t|y)$ are standard Gaussian distributions and use reparameterization trick by following VAE [@kingma2013auto].  The objectives are differentiable and trained using backpropagation.  In the classification task (supervised setting), we use one fully-connected layer as classifier.  In the reconstruction task (self-supervised setting), multiple deconvolution layers are used as the decoder to reconstruct the inputs.  The implementation details and the experimental logs are all avaliable at our source code page. 

Supervised Setting: Classification Tasks
----------------------------------------

### Datasets

CIFAR-10 and CIFAR-100 [@krizhevsky2009learning] are used to evaluate the performance of ICP in the image classification task.  These datasets contain natural images belonging to 10 and 100 classes respectively.  CIFAR-100 comes with finer labels than CIFAR-10.  The raw images are with 32$\times$32 pixels and we normalize them using the channel means and standard deviations.  Standard data augmentation by random cropping and mirroring is applied to the training set. 

### Classification Performance and Ablation Study

We utilize four architectures including VGGNet [@simonyan2014very], GoogLeNet [@szegedy2015going; @szegedy2016rethinking], ResNet [@he2016deep], and DenseNet [@huang2017densely] to test the general applicability of ICP and to study the diversified representations learned by ICP.  We use the classification results of original network architectures as our baselines.  The deep Variational Information Bottleneck (VIB) [@alemi2016deep] and global version of Deep InfoMax with one additional mutual maximization term (DIM\*) [@hjelm2018learning] are used as references, in which VIB is optimized by maximizing $\mathcal{I}(z,t) - \beta\mathcal{I}(z,x)$ , and DIM\* is optimized by maximizing $\mathcal{I}(y,t) + \alpha\mathcal{I}(y,x)$. To make a fair comparison, we expand the representation dimension of both methods to the same size of ICP’s (denoted as VIB, and DIM\*).  The VIB, DIM\*, VIB and DIM\* are the methods that only use one type of representation constraints in ICP, which can also be regarded as ablation study for ICP with single information constraint and without the information diversifying objective.  For further ablation study, we optimize ICP without all the information diversifying and competing constraints (*i.e.*, optimize Eq.  \[eq1\]), which is denoted as ICP.  We also optimize ICP with the information diversifying objective but without the information competing objective (*i.e.*, optimize Eq.  \[eq2\]), which is denoted as ICP. 

The classification results on CIFAR-10 and CIFAR-100 are shown in Tables \[tab1\] and \[tab2\].  We find that VIB, DIM\*, VIB and DIM\* achieve sub-optimal results due to the limited diversification of representations.  ICP do not work well as the large model capacity overfits the training set, and ICP fails because of the dominance of one type of representations.  These results show that expanding models with sole constraint or removing one constraint from the objective decreases the performance.  Only ICP generalizes to all these architectures and reports the best performance.  In addition, the results on different datasets (*i.e.*, CIFAR-10 and CIFAR-100) suggest that ICP works better on the finer labeled dataset (*i.e.*, CIFAR-100).  We attribute the success to the diversified representations that capture more detailed information of inputs. 

### Interpretability of The Diversified Representations

To explain the intuitive idea and the superior results of ICP, we study the learned classification models to explore why ICP works and provide some insights about the interpretability of the learned representations.  In the following, we make an example of VGGNet on CIFAR-10 and visualize the normalized absolute value of the classifier’s weights.  As shown in Fig.  \[fig2\_a\], the classification dependency is fused in ICP, which means combining two representations directly without any constraints does not diversify the representation.  The first green bounding box shows that the classification relies on both parts.  The second and the third green bounding boxes show that the classification relies more on the first part or the second part.  On the contrary, as shown in Fig.  \[fig2\_b\], the classification dependency can be separated into two parts.  As the mutual information minimization makes the representation carry more general information of input while the maximization makes the representation carry more specific information of input, a small number of dimensions are sufficient for inference (*i.e.*, the left bounding box of Fig.  \[fig2\_b\]), while a large number of dimensions are required for inference (*i.e.*, the right bounding box of Fig.  \[fig2\_b\]).  This suggests that ICP learns diversified representations for classification. 

Self-supervised Setting: Reconstruction
---------------------------------------

### Datasets

We perform quantitative and qualitative disentanglement evaluations with the dataset of 2D shapes (dSprites) [@dsprites17] and the dataset of synthetic 3D Faces [@paysan20093d].  The ground truth factors of dSprites are scale(6), rotation(40), posX(32) and posY(32).  The ground truth factors of 3D Faces are azimuth(21), elevation(11) and lighting(11).  Parentheses contain number of quantized values for each factor.  The dSprites and 3D Faces contain 3 types of shapes and 50 identities, respectively, which are treated as noise during evaluation.  The images of both datasets are reshaped to 64$\times$64 pixels to compare with the baseline methods.  We also evaluate the reconstruction and manipulation performance on more challenging CelebA [@liu2015faceattributes] dataset which contains a large number of celebrity faces.  The images are reshaped to 128$\times$128 pixels for more detialed reconstruction instead of 64$\times$64 pixels. 

### Quantitative Evaluation

We evaluate the disentanglement performance quantitatively by the Mutual Information Gap (MIG) score [@chen2018isolating] with the 2D shapes (dSprites) [@dsprites17] dataset and 3D Faces [@paysan20093d] dataset.  MIG is a classifier-free information-theoretic disentanglement metric and is meaningful for any factorized latent distribution.  As shown in Table \[tab3\], ICP achieves the state-of-the-art performance on the quantitative evaluation of disentanglement.  We also conduct ablation studies as what we do in the supervised setting. 

[r]{}[6.7cm]{}

\[0.85\]

                                        dSprites [@dsprites17]   3D Faces [@paysan20093d]
  ------------------------------------ ------------------------ --------------------------
     $\beta$-VAE [@higgins2017beta]              0.22                      0.54
   $\beta$-TCVAE [@chen2018isolating]            0.38                      0.62
                  ICP                            0.33                      0.26
                  ICP                            0.20                      0.57
                **ICP**                        **0.48**                  **0.73**

From the results of ICP and ICP, we find disentanglement performance decreases without the information diversifying and competing process.  For the challenging CelebA [@liu2015faceattributes] dataset, we evaluate the reconstruction performance via the average Mean Square Error (MSE) and the Structural Similarity Index (SSIM) [@wang2004image].  The MSE of ICP is $8.5*10^{-3}$ compared with $9.2*10^{-3}$ of $\beta$-VAE [@higgins2017beta] and the SSIM of ICP is $0.62$ compared with $0.60$ of $\beta$-VAE [@higgins2017beta], which show ICP retains more information of input for reconstruction. 

### Qualitative Evaluation

For qualitative evaluation, we conduct the latent space traverse by traversing a single dimension of the learned representation over the range of \[-3, 3\] while keeping other dimensions fixed.  We manually pick the dimensions which have semantic meaning related to human concepts from the reconstruction results.  The qualitative disentanglement results are shown in Figs.  \[fig3\] and \[fig4\].  It can be seen that many fine-grained semantic attributes such as rotation on dSprites dataset, face width on 3D Face dataset and goatee on CelebA dataset are disentangled clearly by ICP with details. 

Conclusion
==========

We proposed a new approach named Information Competing Process (ICP) for learning diversified representations.  To enrich the information carried by representations, ICP separates a representation into two parts with different mutual information constraints, and prevents both parts from knowing what each other learned for the downstream task.  Such rival representations are then combined to accomplish the downstream task synthetically.  Experiments demonstrated the great potential of ICP in both supervised and self-supervised settings.  The nature behind the performance gain lies in that ICP has the ability to learn diversified representations, which provides fresh insights for the representation learning problem. 

Acknowledgments {#acknowledgments .unnumbered}
===============

This work is supported by the National Key R$\&$D Program (No.2017YFC0113000, and No.2016YFB1001503), Nature Science Foundation of China (No. U1705262, No.61772443, No.  61802324, No.61572410 and No.61702136), and Nature Science Foundation of Fujian Province, China (No.  2017J01125 and No.  2018J01106).