Title: AN INFORMATION THEORETIC PERSPECTIVE ON DIS-ENTANGLED REPRESENTATION LEARNING

Abstract: Existing works on disentangled representation learning usually lie on a common assumption: all factors in a disentangled representation should be independent. We argue that this assumption is not sufficient and another assumption is vital for disentangled representation learning: information contained in each factor of a disentangled representation is irrelevant to others, i.e. the containing information about data of factors is isolated. We formulate this assumption into two equivalent equations via mutual information, and theoretically show its relation with independence and conditional independence of factors in a representation. Meanwhile, we prove that conditional independence is satisfied in encoders of VAEs due to "no-sharing-parameter block" and reparameterization trick. To highlight the importance of the proposed assumption, we show in experiments that violating the assumption leads to decline of disentanglement. Based on this assumption, we further propose to split the deeper layers in encoder to ensure parameters in these layers are not shared for different factors. The proposed encoder, called Split Encoder, can be applied into other models and shows significant improvement in unsupervised learning of disentangled representations and reconstructions.

1. INTRODUCTION:

Learning disentangled representations has been considered as an important step towards interpretable and more effective machine learning (Bengio et al., 2013; Bengio, 2007; Lake et al., 2017; Shanmugam, 2018; Tschannen et al., 2018; Schmidhuber, 1992) . The disentangled representations are proved to be interpretable or semantically meaningful (Chen et al., 2016; Kumar et al., 2017) , robust to adversarial attacks (Alemi et al., 2017) , more generalizable (Steenbrugge et al., 2018) and correlated to fairness (Locatello et al., 2019) . They are also useful to many downstream tasks, including sequential data generating (Yingzhen & Mandt, 2018) , reinforcement learning (Higgins et al., 2017b; Nair et al., 2018) , robot learning (Laversanne-Finot et al., 2018 ), transfer (Liu et al., 2018) and few shot learning (Janzing et al., 2012; Bengio et al., 2013) , etc. Although there is no formal definition for a disentangled representation except some attempts (Higgins et al., 2018) , we adopt one from (Bengio et al., 2013) : a representation in which each factor corresponds to a single factor of variation in data, and meanwhile is invariant to other factors of variation.

There exist unsupervised approaches to learn disentangled representations, based on generative models including generative adversarial nets (GANs) (Goodfellow et al., 2014) and variational autoencoders (VAEs) (Kingma & Welling, 2014) . And the VAE-based models have become mainstream due to the stability of VAEs. Although different VAE-based models for learning disentangled representations are proposed from different motivations, such as limiting the bottleneck capacity (Higgins et al., 2017a; Burgess et al., 2018) , penalizing the total correlation (Kim & Mnih, 2018; Chen et al., 2018) and matching factorized priors (Kumar et al., 2017) , they can be attributed to factorizing the distribution of representations (Locatello et al., 2018a; b; Kim & Mnih, 2018; Chen et al., 2018) . Therefore, these models lie on a common assumption: the distribution of a disentangled representation is factorized, i.e. factors are independent. However, Locatello et al. (2018a) pointed out that this assumption cannot ensures disentanglement, and inductive biases are necessary.

In this work, we argue that the independence of factors in a representation is not sufficient for the disentanglement since it does not model the relation between representations and data, which should be the foundation of disentanglement. Thus we consider another assumption: the information about data contained in a single factor of a disentangled representation is irrelevant to other factors, or factors in a disentangled representation contain isolated information. This assumption is formulated into two equivalent equations in terms of mutual information. Furthermore, we theoretically build up its relation with independence and conditional independence of factors in the representation, showing that conditional independence is also an important property for disentangled representations. Then we divide the encoders in VAEs into "sharing-parameter block" and "no-sharing-parameter block", and prove that conditional independence originates from no-sharing-parameter block and reparameterization trick (Kingma & Welling, 2014) , which can be viewed as inductive biases for disentangled representation learning.

To verify our assumption, we perform experiments by violating the conditional independence of factors in representations and find it lead to decline of disentanglement, which demonstrates the importance of conditional independence and supports our assumption. Motivated by this, we propose the split encoder to improve model capacity of no-sharing-parameter block, which can help different factors learn isolated information, and thus facilitates disentanglement. Our experiments show that the split encoder can significantly improve disentanglement of representations learned by VAE and FactorVAE (Kim & Mnih, 2018) on dSprites (Higgins et al., 2017a) and Cars3D (Reed et al., 2015) datasets when the penalty for independence is strong, and also improves reconstructions.

The main contributions of this paper can be summarized as follows:

• We propose a novel and fundamental assumption for disentangled representation learning, and connect it with independence and conditional independence. We show the importance of both conditional independence and our assumption for disentanglement in experiments.

• We mathematically prove the sources of conditional independence are no-sharingparameter block and reparameterization trick, which can be viewed as inductive biases on encoders.

• Based on the above analysis, we then develop a simple and effective architecture called split encoder to improve disentanglement and reconstructions, which can be applied into other models to improve their performance.

• Experimental results on dSprites and Cars3D show our approach combined with vanilla VAE and FactorVAE can significantly improve performance on disentanglement when the penalty for independence is strong enough, and it also improves reconstructions.

2. THEORETICAL ANALYSIS:

In this section, we first discuss the intuition and common assumption of disentangled representation learning. Then we point out that the common assumption is not sufficient for disentanglement, and motivated by this we propose another assumption from an information theoretic perspective. Finally, we obtain two equivalent equations to formulate the proposed assumption, and connect it with independence and conditional independence.

Although there is no formal definition for disentangled representations, the key intuition is that a disentangled representation should isolate the distinct and semantic factors of variations in data. In other words, a single factor in a disentangled representation is only sensitive to the changes of a single underlying factor of variation in data, while being relatively invariant when other underlying factors change. This statement lies on the relation between factors in representations and data, and also indicates the independence of factors in a disentangled representation. Unfortunately, underlying factors of variation in data commonly have no explicit expression, and thus the statement is hard to be formulated into mathematical expression.

Most variational auto-encoders (VAEs) (Kingma & Welling, 2014) based models for unsupervised disentanglement learning lie on a common assumption: factors in a disentangled representation are independent. VAEs assume a factorized prior p(z) like standard normal distribution and utilize a generative network to generate data and parameterize p(x|z). Meanwhile, an inference network is used to learn representations from data distribution q(x) and parameterize p(z|x). The objective of vanilla VAE is equivalent to the KL divergence D KL (q(z, x) p(z, x)), thus the aggregate posterior q(z) is pushed to match p(z) and then becomes factorized -detailed discussion about this can be found in Appendix. Due to this property, most works focus on factorizing the aggregated posterior q(z) to improve disentanglement. Therefore, those proposed models based on VAEs (Higgins et al., 2017a; Burgess et al., 2018; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018) are attributed to penalizing the divergence between q(z) and j q(z j ), and thus enhancing independence of factors in representations, which coincides with the common assumption.

However, the common assumption is not sufficient for disentanglement of a representation. Because it is only related to the inner property of a representation. While as pointed out above, the relation between factors of a representation and data is also the foundation of disentanglement. Moreover Locatello et al. (2018a) prove the impossibility to learn disentangled representations without inductive biases in an unsupervised manner by simply ensuring the independence. To model the relation between factors of a disentangled representation with data, more assumptions should be considered.

2.1. THE PROPOSED ASSUMPTION:

Our key insight is that the relation between factors in a representation with data can be described from the perspective of containing information. Since each factor in a disentangled representation corresponds to a single factor of variation in data, each factor only contains the information of the corresponding factor of variation in data. In other words, different factors in a disentangled representation contain absolutely different information about data, and the containing information about data in a single factor is irrelevant to other factors. Motivated by this insight, we aim at proposing an information-based assumption to describe the relation between factors in a disentangled representation with factors of variation in data. For mathematical analysis of the information terms, we adopt mutual information to measure the information of a variable contained by another:

EQUATION

where entropy H(z) = -E q(z) [log q(z)] and conditional entropy H(z|x) = -E q(z,x) [log q(z|x)] are measures of uncertainty. Hence mutual information measures the uncertainty reduction of one variable when another is given. The mutual information satisfies the symmetry I(x; z) = I(z; x).

Therefore, for a single factor z j in a disentangled representation z, its containing information about the corresponding underlying factor of variation in data x can be expressed as I(z; x). To represent the irrelevance of z j to other information in data, we involve conditional mutual information:

EQUATION

where j ∈ {1, 2, . . . , J}, S -j is any subset of {1, . . . , j -1, j + 1, . . . , J} and z i∈S-j denotes all factors with subscript index in S -j . The conditional mutual information is the difference of two conditional entropy, representing the containing information of z j on the corresponding underlying factor of variation in data x when other factors z i∈S-j are given. Considering the corresponding factors of variation in data of z i∈S-j , they are fixed when z i∈S-j are given. Hence the conditional mutual information term above can describe the their relation with z j . Now we give more strict formulation. Since z j only contains information of the corresponding factor of variation in data, denoted by I(z j ; x), its containing information should be irrelevant to other factors of variation. Therefore, the conditional mutual information term I(z j ; x|z i∈S-j ) should be equal to I(z j ; x). To conclude, we propose an assumption as follows:

Assumption. Suppose z ∈ R J is a disentangled representation of data x and I(z; x) > 0, then for any single factor z j , its containing information in data is irrelevant to other factors z i∈S-j , i.e.

I(z:

EQUATION

where j ∈ {1, 2, . . . , J} and S -j is any subset of {1, . . . , j -1, j + 1, . . . , J}.

For further investigation on our proposed assumption, we consider turning Eq. 3 into a more intuitive one. Since the assumption is motivated by our insight that different factors in a disentangled representation contain different information in data, the Eq. 3 should intuitively reflect the separation of those information terms. Using the chain rule of mutual information, we prove that Eq. 3 is equivalent to the decomposition of mutual information as follows: Lemma 1. I(z j ; x|z i∈S-j ) = I(z j ; x) for any j is equivalent to the following equation:

EQUATION

where S = {j} ∪ S -j is any subset of {1, 2, . . . , J}, j is any single element in S.

Based on our assumption and the lemma, we can draw an intuitive conclusion that in a disentangled representation z, the containing information of z j is different from those of z i∈S-j , and thus their corresponding factors of variation in data are distinct. Furthermore, by iteratively using Eq. 1, we can decompose the mutual information I(z i∈S ; x) into the sum of I(z i ; x), which shows that the containing information of different factors in a disentangled representation are different and coincide with our insight. Hence our assumption can be summarized into an equivalent statement: Assumption. Suppose z ∈ R J is a disentangled representation of data x and I(z; x) > 0, then for any subset S of {1, 2, . . . , J}, the containing information of z i ∈ S in data is isolated, i.e.

I(z i∈S ; x) = i∈S I(z i ; x) (5)

2.2. RELATION WITH INDEPENDENCE AND CONDITIONAL INDEPENDENCE:

From the containing information perspective, an intuitive assumption of disentangled representations has been proposed to describe the relation between factors in a disentangled representation and data. Two equivalent expressions Eq. 3 and Eq. 5 are also derived. Although the two equations have clear physical meanings, the structured relation between a disentangled representation and data in the proposed assumption is still implicit. Hence in this subsection, we aim at investigating the structure of the proposed assumption from Eq. 5.

To analyze Eq. 5, we investigate the conditions of establishing Eq. 5 by considering I(z i∈S ; x) -i∈S I(z i ; x). We find that this term is tightly related to independence and conditional independence of factors in a representation. Specifically, we build up the following lemma: Lemma 2. For any subset S of {1, 2, . . . , J}, we have:

I(z i∈S ; x) - i∈S I(z i ; x) = E q(x) [D KL (q(z i∈S |x) i∈S q(z i |x))] -D KL (q(z i∈S ) i∈S q(z i )) (6)

In the lemma above, the left term is equal to the difference of two KL divergences. The first divergence measures the level of conditional independence for factors z i∈S conditioned on data x, and the second one measures the level of independence for z i∈S . Specifically, when the conditional distribution q(z i∈S |x) is more factorized, the first divergence is closer to zero, which indicates higher level of conditional independence. Similarly, when the distribution q(z i∈S ) is more factorized, the second divergence is more closer to zero, and hence the level of independence is higher.

Eq. 6 shows the importance of conditional independence and independence in our assumption: when the factors in a representation are independent and conditionally independent, our assumption is satisfied and thus the representation is oriented towards disentanglement. Note that conditional independence and independence are sufficient but not necessary to our assumption. This shows that our assumption is different and independent to the common assumption.

It is important to note that our assumption is necessary but not sufficient for learning meaningful disentangled representations. For example, if q(z|x) = q(z), then the two divergences in Eq. 6 are equal, and thus Eq. 5 holds and our assumption is satisfied, but in this case z does not contain any information about data x. To prevent this meaningless case, we should pay attention to the reconstruction error, which reflects the containing information of a representation about data. Lower reconstruction error indicates that more information about data is contained in representation.

As mentioned above, the common assumption highlights the independence of factors in a disentangled representation, which is enhanced via the objective in VAEs. Given the establishment of the common assumption, our proposed assumption is equivalent to conditional independence, which is exactly the structured relation between a disentangled representation and data. Motivated by this consideration, we investigate the inductive biases on model about conditional independence, and successfully find the sources of conditional independence in encoders of VAEs.

3. INDUCTIVE BIASES ON ENCODERS:

On condition of independence, the proposed assumption is equivalent to conditional independence. In this section we analyze the inductive biases about conditional independence, and then propose a simple architecture to improve disentanglement based on our assumption and the inductive biases. 

3.1. SOURCES OF CONDITIONAL INDEPENDENCE:

Motivated by our proposed assumption, we find that there are some inductive biases on encoders of VAEs to ensure conditional independence. Considering q(z) is pushed to match a factorized prior p(z) in VAE-based models and thus encourages independence and the common assumption is satisfied, this discovery shows that our assumption is also satisfied in these models.

In VAEs, the encoders encode data into a Gaussian distribution of representation q(z|x) via reparameterization trick. Specifically, the data is encoded into a mean µ(x) and a variance σ(x), then the representation is sampled from N (z; µ(x), σ 2 (x)I) with reparameterization trick:

EQUATION

For analyzing the outputs of those encoders, we divide them into two blocks: if the parameters of several layers are shared for different factors in a representation, the block consists of these layers is called "sharing-parameter block". Conversely, those layers without shared parameters for different factors form "no-sharing-parameter block". In vanilla VAE, the last layers in encoders are fullyconnected nets, which are no-sharing-parameter block (see Fig. 1 ). Obviously, factors in the output of no-sharing-parameter block are independent conditioned on the input, thus factors in µ(x) and σ(x) are conditionally independent. Considering the independence of factors in , we have:

EQUATION

From these equations, we can obtain the conditional independence of factors in a representation:

EQUATION

To conclude, conditional independence originates from no-sharing-parameter block and the independence of noise in reparameterization trick, which can be regarded as inductive biases on model. These two inductive biases are very important for disentangled representation learning. We will see in experiments that without no-sharing-parameter block or independent noise, the learned representations are more entangled, which indicates that our proposed assumption is vital for disentanglement.

3.2. SPLIT ENCODER:

In this subsection, we aim at devising a simple and effective architecture to enhance disentanglement in encoders based on our assumption and the inductive biases about conditional independence.

According to the proposed assumption, factors in a disentangled representation contain isolated information about data. Intuitively no-sharing-parameter block is beneficial to learn isolated information from input, but in previous works, the no-sharing-parameter block is a single fully-connected layer, which limits the model ability to learn isolated information. A limited no-sharing-parameter block may encourage the conditional distribution q(z|x) to match q(z) without capturing any information about data, which violates the essence of our assumption.

To tackle this problem, we propose split encoder, which improves model capacity of the no-sharingparameter block without increasing the depth of encoder. As Fig. 1 shows, in split encoder more convolution layers and fully-connected layers are created for each single factor in representation without parameter sharing, then the outputs are concatenated to calculate a representation with reparameterization trick. In experiments, we only create one convolution layer and two fully-connected layers for each factor due to the increase of parameters.

From the perspective of containing information, split encoder coincides with the essence of the proposed assumption: factors in a disentangled representation contain isolated information. But in the case of q(z|x) = q(z), the containing information of factors are zero. This trivial case is excluded to ensure our assumption meaningful. To conclude, the split encoder is to prevent the trivial case, rather than improve conditional independence.

The split encoder can be viewed as an architecture to improve the flexibility for z. Improving flexibility of representation in split encoder is beneficial to independence of factors and thus improves disentanglement when the penalty for independence is strong enough. But when the penalty is too weak, flexibility for factors might lead to smaller probability to be independent, and thus causes less disentanglement. As will be shown in the experiment part, these phenomena described from the two perspective above are observed in experiments of learning disentangled representations. In addition, the relation of split encoder KL term in objective of VAEs is discussed in Appendix A.2.

4. RELATED WORKS:

There are many works related to disentangled representation with different objectives. A line of works focus on learning disentangled representations from typical types of data (Yingzhen & Mandt, 2018) , some pay attention to supervised learning (Mathieu et al., 2016) or semi-supervised learning of disentangled representations (Spurr et al., 2017; Siddharth et al., 2017) , others explore other objectives (Rubenstein et al., 2018; Zhao et al., 2019) . There are also works on the evaluation of disentanglement (Eastwood & Williams, 2018) . Since our assumption and analysis lie on the foundation of disentangled representation learning, in this section we focus on those related to the understanding of disentanglement or basic unsupervised models for disentangled representation learning.

Some works on unsupervised learning of disentangled representations are proposed based on early generative models. Schmidhuber (1992) proposes a variant of auto-encoder to learn disentangled representations by minimizing the predictablity of one factor in representation when other factors are fixed, this model obviously is motivated by the independence of factors, i.e. the common assumption. Desjardins et al. (2012) and Reed et al. (2014) propose variants of (Restricted) Boltzmann Machine in which interactions act to entangle the factors.

Recent works on unsupervised learning of disentangled representations are mainly based on GANs and VAEs. In line with GANs, InfoGAN (Chen et al., 2016) penalizes the mutual information of representations, and qualitatively shows that different factors in a representation correspond to different visual concepts. Brakel & Bengio (2018) propose to penalize the Jensen-Shannon divergence between the distribution of representations and its factorized distribution with a discriminator, based on Independent Component Analysis.

The mainstream models of disentangled representation learning are variants of VAE due to its stability. β-VAE (Higgins et al., 2017a) introduces to encourage the encoder to learn a disentangled representation by penalizing the KL term in the objective of vanilla VAE. AnneledVAE (Burgess et al., 2018) proposes to progressively increase the bottleneck capacity of VAE to encourage the encoder to learn different factors of variation when capacity grows. FactorVAE (Kim & Mnih, 2018) uses a discriminator to penalize the KL divergance between the distribution of representations and its factorized distribution (total correlation) via ratio trick to enhance independence of factors in representations. DIP-VAE (Kumar et al., 2017) matches the distribution of representations with dis- In subfigures, the first violin graph is the baseline, the second one is a vanilla encoder with correlated noise (σ = 0.9), and the third one is a sharing-parameter encoder with uncorrelated noise.

entangled priors. In TC-VAE (Chen et al., 2018) , the authors decompose the objective of VAE and argue that the total correlation term is the source of disentanglement, then they derive a mini-batch estimator for the total correlation term and penalize it to enhance disentanglement. Most VAE-based models can be attributed to enhance independence, which coincides with the common assumption.

5. EXPERIMENTS:

The objective in this section is not only to show our trick can improve unsupervised learning of disentangled representations, but also to verify the importance of the proposed assumption by investigating the change of disentanglement when conditional independence is violated.

Datasets: We use dSprites (Higgins et al., 2017a ) and a more complex dataset Cars3D (Reed et al., 2015) . DSprites is a set of 737280 64*64 images in black and white, generated from fives independent latent factors. Cars3D consists of 199 colorful 3D car models in shape of 128*128*3*24*4.

Models::

We select two models as baselines: vanilla VAE and FactorVAE. Since most VAE-based models are attributed to enhance independence of factors in representations, it might be redundant to consider more VAE-based models. Vanilla VAE and FactorVAE are weakly and strongly penalize the total correlation respectively, hence we believe that this choice is sufficient for showing the importance of our assumption and the power of our proposed split encoder. Additional experiment results of β-VAE, which does not explicitly penalize total correlation, are reported in Appendix.

Metrics: Following (Watters et al., 2019) , we use Mutual Information Gap (MIG) (Chen et al., 2018) to evaluate disentanglement for its usefulness and rationality. MIG is defined by first computing the normalized mutual information of each factor in representations and each ground truth factor, then computing the gap between the highest two normalized mutual information values along factors in representations, and finally returning the gap averaged along ground truth factors. We also use reconstruction error to reflect the quality of the containing information for representations.

Lower reconstruction error means more meaningful information are contained in representations.

Additional results of FactorVAE Score (Kim & Mnih, 2018) are reported in Appendix.

5.1. VIOLATING CONDITIONAL INDEPENDENCE:

When independence holds, our assumption is equivalent to conditional independence. According to our analysis, conditional independence originates from no-sharing-parameter block and the independence of noise in reparameterization trick. Therefore, for verifying the importance and necessity of our assumption, in this section we adopt two ways to violate conditional independence and compare the disentanglement of learned representations in terms of MIG: involving a encoder without no-sharing-parameter block, and a correlated noise for reparameterization. To show the necessity of no-sharing-parameter block, we remove it and develop a sharing-parameter encoder: the inputs are extracted features by a sequence of convolution neural networks, then the features are straightly flattened into representations. We compare the disentangling performance of sharing-parameter encoder and the vanilla encoder on dSprites.

To show the importance of the independence of noise in reparameterization trick, we introduce a correlated noise for comparison: ∼ N (0, Σ), Σ = (1 -σ)I + σ11 T where 1 is an identity column vector, and σ ∈ [0, 1) is correlation weight. Larger σ corresponds to higher correlation. We set σ = 0.9.

The two designs are applied into vanilla VAE and FactorVAE, and compare with the original models in terms of MIG metric. The results are shown in Fig. 2 , from which we can see that both using a correlated noise and removing the no-sharing-parameter block lead to the significant decrease of MIG metric in both vanilla VAE and FactorVAE. These results show that without uncorrelated noise and no-sharing-parameter block the learned representations will become more entangled, i.e. conditional independence and our assumption are vital for disentanglement. And rarely ensuring independence like FactorVAE is not sufficient for learning disentangled representations.

5.2. UNSUPERVISED LEARNING OF DISENTANGLED REPRESENTATIONS:

In this subsection, we compare the performance of vanilla VAE and FactorVAE with ones combined with split encoder on dSprites and Cars3D. We not only focus on the MIG metric to reflect the performance of disentanglement, but also concern the reconstruction loss which indicates the quality of containing information in representations, and meanwhile observe the quality of reconstructed images and traversals for intuitive understanding. Traversals are generated from a data point, which is first encoded into a representation, then for each row only a single factor in the representation is changed in [-2, 2] and then reconstruct the data point. For the fairness of comparison, we show the reconstruction images and traversals from output model with the highest MIG score for each case.

5.2.1. PERFORMANCE ON DSPRITE:

As shown in Fig. 3 , combing split encoder with both vanilla VAE and FactorVAE significantly improve disentanglement in terms of MIG on dSprites. This result indicates that dSprites is simple and the penalty for independence of representation in vanilla VAE is strong enough, so that improving flexibility of representations by involving split encoder can significantly improve disentanglement.

And Fig. 4 shows that involving split encoder leads to smaller reconstruction error. This result means that split encoder is beneficial to learn meaningful information from data, which coincides with our assumption. Note that the KL term in vanilla VAE with split encoder is smaller, while in FactorVAE with split encoder it is larger. This phenomenon prove that the improvement of disentanglement in split encoder cannot be explained as the results of implicitly penalizing the KL term, which does not coincide with the analysis for β-VAE (Higgins et al., 2017a; Burgess et al., 2018) .

The reconstructions and traversals are shown in Fig. 5 . We can see the four models work well in dSprites, and the learned representations are well disentangled except some small flaws.

5.2.2. PERFORMANCE ON CARS3D:

The results on Car3D are different from those on dSprites. As shown in Fig. 6 , vanilla VAE combined with split encoder leads to lower MIG, while FactorVE combined with split encoder outperforms FactorVAE. This result is not surprising, which means that Car3D is too complex and the penalty of independence in vanilla VAE is not strong enough, and involving split encoder leads to flexibility for representation and smaller probability of being disentangled. While for FactorVAE, the penalty of independence is strong enough, and thus involving split encoder leads to better performance.

As shown in Fig. 7 , involving split encoder can lead to smaller reconstruction error, which indicates the containing information in representations is more meaningful. While the KL terms are larger in both models with split encoder. This suggests the effect of split encoder is not penalizing the KL terms, but increasing the quality of information contained in representations and flexibility.

The reconstructions and traversals results on the four models are different, as shown in Fig. 8 Proof: This conclusion is a straight corollary of chain rule of mutual information. For further understanding, here we derive it from scratch using Bayes rule: I(z j ; x|z i∈S-j ) = E q(z i∈S ,x) [log q(z j , x|z i∈S-j ) q(z j |z i∈S-j )q(x|z i∈S-j ) ]

= E q(z i∈S ,x) [log q(x|z i∈S ) q(x|z i∈S-j ) ]

= E q(z i∈S ,x) [log q(x, z i∈S )q(z i∈S-j ) q(z i∈S )q(x, z i∈S-j ) ]

= E q(z i∈S ,x) [log q(x, z i∈S ) q(z i∈S )q(x) -log q(x, z i∈S-j ) q(z i∈S-j )q(x) ]

= I(z i∈S ; x) -I(z i∈S-j ; x) Hence we have: I(z j ; x|z i∈S-j ) = I(z j ; x) ⇔ I(z i∈S ; x) = I(z j ; x) + I(z i∈S-j ; x)

Lemma 2. For any subset S of {1, 2, . . . , J}, we have:

I(z i∈S ; x) - i∈S I(z i ; x) = E q(x) [D KL (q(z i∈S |x)

i∈S q(z i |x))] -D KL (q(z i∈S ) i∈S q(z i ))

Proof:

I(z i∈S ; x) -i∈S I(z i ; x) = E q(z i∈S ,x) [log q(z i∈S , x) q(z i∈S )q(x) ] -i∈S E q(zi,x) [log q(z i , x) q(z i )q(x) ]

= E q(z i∈S ,x) [log q(z i∈S , x) q(z i∈S )q(x) -i∈S log q(z i , x) q(z i )q(x) ] = E q(z i∈S ,x) [log q(z i∈S |x) q(z i∈S ) -i∈S log q(z i |x) q(z i ) ]

= E q(z i∈S ,x) [log q(z i∈S |x) i∈S q(z i ) q(z i∈S ) i∈S q(z i |x) ]

= E q(z i∈S ,x) [log q(z i∈S |x)

i∈S q(z i |x)

-log q(z i∈S )

i∈S q(z i ) ] = E q(x) [D KL (q(z i∈S |x) i∈S q(z i |x))] -D KL (q(z i∈S ) i∈S q(z i ))

C MODELS PERFORMANCE IN TERMS OF FACTORVAE SCORE:

There are several metrics for evaluating disentanglement, including Mutual Information Gap (MIG) (Chen et al., 2018) , DCI Disentanglement (Ridgeway & Mozer, 2018) , BetaVAE metric (Higgins et al., 2017a ), FactorVAE Score (Kim & Mnih, 2018) , SAP Score (Kumar et al., 2018) and Modularity (Ridgeway & Mozer, 2018) . Locatello et al. (2018a) report that MIG and DCI Disentanglement strongly correlated with each other, and so do BetaVAE Score and FactorVAE Score, but SAP and Modularity are not correlated with other metrics. This result indicates that not all metrics are suitable for evaluating disentanglement. We believe that MIG is a suitable metric due to its usefulness and rationality, so we mainly report experimental results in terms of MIG. However, for a more complete comparison, here we report models performance in terms of evaluation accuracy in FactorVAE score as shown in Fig. 9 and Fig. 10 , which is not well correlated with MIG.

D EXPERIMENTAL RESULTS OF BETA-VAE ON DSPRITES:

In Fig. 11 , we can see that split encoder significantly improves the disentanglement performance of β-VAE in terms of MIG. And the KL term in β-VAE is larger while the reconstruction error is smaller. These results supports our analysis on the effect of split encoder in Section A. 

:

CONCLUSIONWe argue that an assumption from the perspective of information is vital for disentanglement: factors in a disentangled representation should contain isolated information about data. We formulate this assumption into mathematical equation and connect it with independence and conditional independence of factors. Then we find two inductive biases the key to conditional independence: no-sharing-parameter block and uncorrelated noise in reparameterization trick. Inspired by this, we propose split encoder to improve model capacity of no-sharing-parameter block and thus improve disentanglement. Experimental results demonstrate the importance of our assumption and the power of split encoder for improving disentanglement and reconstructions.

Figures and Tables:

Figure 1: Structure of vanilla encoder and split encoder. The blue and short-dash box denotes sharing-parameter block, and the red and long-dash box represents no-sharing-parameter block.

Figure 2: MIG scores of verifying experiments in vanilla VAE and FactorVAE on dSprite.In subfigures, the first violin graph is the baseline, the second one is a vanilla encoder with correlated noise (σ = 0.9), and the third one is a sharing-parameter encoder with uncorrelated noise.

Figure 3: MIG scores of vanilla VAE and FactorVAE (+ split encoder) on dSprites.

Figure 4: KL vs. reconstruction error of vanilla VAE, FactorVAE (+ split encoder) on dSprites.

Figure 5: Reconstructions and traversals of VAE and FactorVAE (+ split encoder) on dSprites.

Figure 6: MIG scores of vanilla VAE and FactorVAE (+ split encoder) on Cars3D.

Figure 7: KL vs. reconstruction error of vanilla VAE, FactorVAE (+ split encoder) on Cars3D.

Figure 8: Reconstructions and traversals of VAE and FactorVAE (+ split encoder) on Cars3D.

Figure 9: FactorVAE Score of vanilla VAE and FactorVAE (+ split encoder) on dSprites.

Figure 10: FactorVAE Score of vanilla VAE and FactorVAE (+ split encoder) on Cars3D.

. Models with split encoder obviously have better reconstructions. And traversals in FactorVAE with split encoder are more disentangled than FactorVAE. Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. international conference on machine learning, pp. 5656-5665, 2018. Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. pp. 5885-5892, 2019.

Table content: <table/>

Peer review (iclr2020):
On its surface structure, the paper looks well organized and existing literature on disentangled representations is well reviewed. However, the paper is extremely difficult to read and it is hard to follow the reasoning of the authors. I hope that the authors can clarify misunderstandings during rebuttal. “Existing works on disentangled representation learning usually lie on a common assumption: all factors in a disentangled representation should be independent. We argue that this assumption is not sufficient and another assumption is vital for disentangled representation learning: information contained in each factor of a disentangled representation is irrelevant to others, i.e. the containing information about data of factors is isolated. “ -How can the first sentence be claimed if there is not a consensus about the formal definition of disentangled representations as stated in the introduction? - Assuming that ‘.. [Independence] is not sufficient for disentangled representation learning’. Then you propose an additional assumption that seems to be the central contribution of the present paper: ‘information contained in each factor of a disentangled representation is irrelevant to others, i.e. the containing information about data of factors is isolated’. - I don’t understand this notion of information that must be different than independence, namely that ‘information contained in each factor [...] is irrelevant to other [factors] ’ and ‘the containing information about data of factors is isolated’. I guess the notions of ‘data of factors’, and ‘information being isolated’, ‘irrelevant information’ need to be clarified. The text in the introduction is unfortunately also not very helpful “... Since each factor [...] corresponds to a single factor of variation in data, each factor only contains the information of the corresponding factor of variation in data.” I think there is a collection of ‘factors’ and a collection of ‘factors of variation in data’, and between these two objects there is a correspondence. The statement ‘each factor only contains the information of the corresponding factor of variation in data’ implies that a ‘factor of variation in data’ is an object that posses information that is contained in the factor. At the end of the introduction, the authors state ‘conditional independence originates from no-sharing-parameter block and the independence’ of noise in reparameterization trick, which can be regarded as inductive biases on model. This statement is alarming as it hints at a clash of terminology, the concept of independence in the information theoretic/probabilistic sense -- that is the density of two random variables can be written exactly as the product of marginals -- with ‘the lack of a functional dependence’. (Consider the Box-Muller method for generating independent Gaussian random variables x and y, where x = r cos(2pi u) and y = r sin(2 pi u) where u is uniform on the unit interval and r is exponential with rate 2. x and y are independent random variables but they are both functions of r and u. Hence parameter splitting is not necessary for independence.) Then, we encounter (9) and the following statement *To conclude, conditional independence originates from no-sharing-parameter block and the independence of noise in reparameterization trick, which can be regarded as inductive biases on model.* Unfortunately, this conclusion is not correct. (9) is the defining property of an encoder with a factorized Gaussian -- it does not follow from *non-sharing* of parameters. To see this, assume each mu_j = f(x), i.e. all means are the same, all parameters are shared. This is not a very flexible encoder distribution but still enjoys the same conditional independence structure, because parameter sharing and independence are different concepts. (8) is misleading as mu and sigma are not random variables in a VAE. I suspect there may be some interesting ideas here but the paper clearly needs further iterations for clarification of concepts so it is accessible for the general technical audience.