RTC-VAE: HARNESSING THE PECULIARITY OF TOTAL CORRELATION  IN LEARNING DISENTANGLED REPRESENTATIONS
Authors: Ze Cheng
Juncheng B Li
Chenxu Wang
Jixuan Gu
Hao Xu
Xinjian Li
Florian Metze
Venue:  (2020)
Keywords: Total Correlation
VAEs
Disentanglement

Abstract: In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables.  Unfortunately, this well-motivated strategy often fail to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation.  We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the  mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of abounded total correlation.  However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy,  RTC-VAE, which rectifies the total correlation penalty.   Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g.,β-TCVAE and FactorVAE.

Under review as a conference paper at ICLR 2020
RTC- VAE: HARNESSING THE PECULIARITY OF TOTAL CORRELATION IN LEARNING DISENTANGLED REPRE- SENTATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari- ables. Unfortunately, this well-motivated strategy often fails to achieve disentan- glement due problematic difference between the sampled latent representation and its corresponding mean representation. We provide theoretical explanation that low total correlation of sample distribution cannot guarantee low total corre- lation of the mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions bounded total correlation. However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose remedy, RTC-VAE, which rectifies the total correlation penalty. Ex- periments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g., B-TCVAE and FactorVAR
latent variables of a
INTRODUCTION
VAEs (Variational AutoEncoders) Kingma & Welling (2013); Bengio et al. (2007) follow the com- mon assumption that the high-dimensional real world observations x can be re-generated by lower- dimension latent variable z which semantically meaningful. Recent works Kim & Mnih (2018); Chen et al. (2018); Kumar et al. (2017) suggest that decomposing the ELBO (Evidence Lower Bound) could lead to distinguishing the factor of disentanglement. In particular, recent works Kim & Mnih 20; Chen al. (2018) focused lief Chen et al. (2018) is that by adding weights to this term in objective function, a VAE model can learn disentangled representation. This approach appears to be promising since the total correlation of sampled representation should describe the level of factorising since total correlation is defined the KL-divergence between the joint distribution tributions II, g(2)). In this case, low value suggests a less entangled joint distribution. However, Locatello et al. (2018) pointed out that the total correlation of sampled distribution TC being low does not necessarily give rise low total correlation of the corresponding mean representa- tion TCmeen- Conventionally, the mean representation is used as the encoded latent variables, an unnoticed high TCmean is usually the culprit behind the undesirable entanglement. Moreover, Lo- catello et al. (2018) found that as regularization strength increases, the total correlation of sampled representation TC Taaampee and mean representation T TCCeea are actually negatively correlated.
term called total correlation (TC). The popular be-
g(2) and the product of marginal dis-
Locatello et al. (2018) put doubts on most methods of disentanglement including penalizing the total correlation term Kim Mnih (2018); Chen et al. (2018), and they concluded that "the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases".
Acknowledging the difficulty in learning disentangled representation, we provide a detailed ex- planation of the seemingly contradictory behaviors of the total correlations of sampled and mean representation in previous works on TC penalizing strategy. Moreover, we find that this problem de- scribed above can be remedied simply with additional penalty term on the variance of sampled representation.
Our contributions:

---

Under review as a conference paper at ICLR 2020
In Theorem 1, prove that for all mean representations, there exists large class of sample distributions with bounded total correlation. Particularly, a mean representation with arbitrarily large total correlation can have corresponding sample distribution with low total correlation. This implies that a low total correlation of sample distribution cannot guarantee low total correlation of the mean representation. (Section. 2)
Acknowledging the issue above, further delve into total correlation, and provide sim- ple remedy by adding an additional penalty term on the variance of sample distribution. The penalty term forces sampled representation to behave similar to the corresponding mean representation. Such penalty term is necessary for the strategy of penalizing 7 T'mmaa in the view of Theorem 1. (Section. 4)
We study several different methods of estimating total correlation. They compared and benchmarked against the ground truth value on the multivariate Gaussian distribution Lo- catello et al. (2018). We point out that the method of (minibatch) estimators suffers from the curse of dimensionality and other drawbacks, making their estimation accuracy decay significantly with the increase of the dimension of the latent space, and some strong corre- lated distributions can be falsely estimated to have low total correlation. (Section. 5)
2 THE PECULIARITY OF TOTAL CORRELATION
In information theory, total correlation is one of the generalizations of mutual information, which measures the difference between the joint distribution of multiple random variables and the product of their marginal distributions. A high value means the joint distribution is far from an independent distribution, and hence suggests high entanglement among these random variables. Definition 1. Total correlation of random variable x,
p(x)
TC(x) == KT. p(x))] ( p(az)) = Bp(c) log
II, p(x5)
Naturally, people seek the solution of disentanglement in the form of low total correlation of the latent variables, e.g. Kim & Mnih (2018); Chen et al. (2018). However, there can be large differ- ence between the total correlations of sample representation and mean representation. Forcing the former to be small does not guarantee the latter being small. In fact, given a mean representation of arbitrarily large total correlation, can construct family of distribution of sample representation that have a bounded total correlation, where the bound does not rely on the total correlation of the mean.
Theorem 1. Let μa NOO, E) and oj be the standard deviation of μtj. j j,.... 1,..., D, and maxj Jj = co. For a fixed μs, let z ~ N(u, E'))), where E' E'()) is diagonal and satisfies that for some R > 0,
22 (1) > 0,
i|u < R
C3 (H)
|nll for some I 1, if |||| > R.
for some constants C1 C2 C3 Ca. Then TC(z)= I C for some C C(R, co.
.. C4, 0.
The details of the proof are presented in Appendix A.3. Here's 's another way to interpret Theorem 1: with and parameters R, cu. ... C1. fixed, one can make TC(μ) arbitrarily large, since TC(μ) depends only on the correlation matrix of fa (see Proposition 1).
Theorem 1 provides an explanation to the contradiction observed by Locatello et al. (2018) that TC(Z) is low does not mean TC(μ) is low (actually much higher than TC(2)). Since there exist such a large class of distributions of that all have bounded [C')z). When the objective function only penalizes (u,),, neural networks are flexible to easily find distribution with low TC(z), and total correlation estimators like MSS can encourage shutting down latent dimensions (see Sec- tion 5.3), which together cause the disparity of TC(u) and TC(z). This fact not noticed until Locatello et al. (2018), and our investigation gives an explanation of the peculiar property of total correlation. Hence, Theorem leads to the necessity of regularizer of the difference between the
2

---

Under review as a conference paper at ICLR 2020
distributions of ja and when penalizing TC sample. In Section. 4, we propose simple regularizer that serves this goal.
It is interesting question whether there exists distribution of (M, (()) with arbitrarily small (z) given p. If not, what's the lower bound TC(2)? These questions remain open to us for now, and we leave them to future work.
RELATED WORKS
In the study of disentanglement, Higgins et al. (2017) proposed modification of the VAE framework and introduced an adjustable hyperparameter 3 that balances latent channel capacity and indepen- dence constraints with reconstruction accuracy. One drawback of 3-VAE is the trade-off between the reconstruction quality and disentanglement. Motivated to alleviate this trade-off of 3-VAE, Kim & Mnih (2018) proposed FactorVAE which decomposes the evidence lower bound and penalize a term measuring the total correlation between latent variables. Around the same time, Chen et al. (2018) proposed a similar ELBO decomposition method called B-TCVAF. The major difference between Factor VAE and 3-TCVAE lies in their different strategies of estimating total correlation. Chen et al. (2018) used formulated estimators while Kim & Mnih (2018) utilized the density-ratio trick which requires auxiliary discriminator network and an inner optimization loop. We will discuss these two strategies more in details in Section.
The works above belong to representative learning without inductive biases. There also works about representative learning with inductive biases, see Rolinek et al. (2019) and references therein.
As for the disentanglement metric, this will be discussed in Section. 6.
4 RECTIFIED-TCVAE
To simplify notation, let p(n) = p(xn), (((2,,))))) ((2/n)). Recall the average evidence lower bound (ELBO),
ELBO == Pp))) (Fa(el))) [Lg (o]=((n]=)) ([((()))))))]]
Chen al. (2018) and independently by Kim & Mnih (2018) introduced objective function that penalizes total correlation, which can be formulated as
C3-TC = ELBO - (TC()).
(3)
This approach unfortunately has a drawback. It turns out that instead of being able obtain disen- tangled representation, we often find a sample representation appears to be disentangled while the mean representation still entangled. In fact, when we are maximizing Læ-TC, we could totally end up learning distribution of that makes TC(z) goes low, while the total correlation of its mean μ is still high. To resolve this, we define RTC-VAE,
CRIC == L3-TC n tr(Ep(n) (o)g(s)m)))))
(4)
where
tr (T((E Comgte)))))))))))) = Σ
Our penalty originates from the first term of the law of total covariance [ = EptnnCoUu(ssto) + Coup(n) (Bg(()nn)))) Factorized representation' indicates diagonal covari- ance matrix [=] Motivated by this, Kumar et al. (2017) penalizes the off-diagonal terms in since it is diagonal. Their penalty term leads to
the second term, while ignores Spon) vanishing μ4, which is the mean representation. The remedy to this is add another penalty term on the distance between jtss and 1. DIP-VAE Kumar et al. (2017) employs this remedy, however, DIP- VAE does not outperform other VAE's when measured by various disentanglement metries, e.g., FactorVAE score, see Fig. 3 & 14 in Locatello et al. (2018). This is actually not surprising since the
Factorized representation each latent dimension is independent.
3

---

Under review as a conference paper at ICLR 2020
two penalty terms in LIP-VAE contribute in opposite directions, with one leading to vanishing p's and another fighting against it. This formulation can easily get the model stuck in saddle points.
Our objective, the other hand, does penalize directly μ.. Instead, it penalizes o, the standard deviation of the distribution {zz|n). This may seem little counter-intuitive at first sight, since penalizing diagonal component of covariance Cou(z] (o))(=)(=) [2] seems helpful to factorising. However, in the view of Theorem I, our objective will force the distribution of z to be
similar the distribution of μ.. Hence, it pushes away from the situation of large TC(μ) and low TC(z). Consequently, by minimizing (Z( we get a model that has low TC(μ), disentangled mean representation.
ESTIMATION OF TOTAL CORRELATION
The naive Monte Carlo method comes with an intrinsic issue of underestimating total correlation. To avoid or resolve this, Kim & Mnih (2018) proposed discriminator network with the help of density-ratio trick (see equation (3) and Appendix D). of Kim & Mnih (2018)). In Chen et al. (2018), two kinds of estimator total correlation are proposed, Minibatch Weighted Sampling (MWS) and Minibatch Stratified Sampling (MSS) (see Appendix C.1 and C.2 in Chen al. (2018)).
5.1 METHOD OF MINIBATCH ESTIMATORS
For instance, MSS can be described as followed. For a minibatch of sample, Bas+1 = (nq....,MM+I) RM+1).
M+1
(u(=,))logg M +1
log f(a,, ni, BM4+ \ (mit),
(5)
2here?
M-1
N M
Bas+1 {n*}) = N (2/2"")) + M m=1
Σ ((( + NM ((2(mm)).
(6)
For the convenience of readers, MWS is listed in Appendix A.I.
5.2 METHOD OF DENSITY-RATIO TRICK AND DISCRIMINATOR
Density-ratio trick Nguyen et al. (2010); Sugiyama et (2012) can be used to estimate KL- divergence,
9[z)) Bg(e)) log ((z)
TC(z) KL(g(=)]
I1, ((²)
((x)
2s (xy(s)
(8)
I D(2)
where D is discriminator that classifies being sampled from g(z) or II, 9(2,). For detail imple- mentation, please refer to section 3 in Kim & Mnih (2018).
5.3 COMPARISON OF THE TWO METHODS
For multivariate normal distribution, the total correlation can be explicitly calculated which can be used ground truth for comparison. To be specific, Proposition 1. Let x ~ No0, E), then
TC(x) 2 (log) diag Σ)] log] -0
"There a small part of the implementation of MSS Chen et al.'s code that not quite clear to us, specifically, the computation of log importance weight matrix in equation 6. In our experiment, we implement MSS with our understanding and denote it as MSS1, and we denote Chen et al.'s implementation MSSp See Appendix A.2

---

Under review as a conference paper at ICLR 2020
It's difficult track the exact reference of Proposition 1 since it is fundamental property in infor- mation theory. Locatello et al. (2018) used this proposition to approximate the total correlation of the mean representation in latent space. In appendix, we provide simple proof for the convenience of the readers.
We compared the performance of each method, MWS, MSSo and MSS; the estimation of total correlation. For ~ (0, I), and zIju ~ [M,, Σ) where Σ = diag(o²) and o = 0.1. We choose o small that the distribution of z be approximated by normal distribution. Results are presented in Figure 1.
2r22
Dedd
Dn1
D-110
(443
Figure I: Different estimators of total correlation vs. ground truth on latent space of dimension low to high.
From Figure. we can summarize the following observations: MWS tends to underestimate total correlation in general; 2. For latent space of dimension < 4, MSS, and MSS) are quite accurate; For latent space of high dimension, both MSS, and MSS; tend overestimate total correlation when the actual value of total correlation is small; 4. Overall MSS; estimates closer to ground truth than MMS, does.
In the following analysis of the above observations, we use a less formal way of analyzing, which can be formalized be rigorous, in order to convey our idea directly.
To interpret the third observation, let NNO0, Id) and zIμ ~ NO E), where Id is identity matrix and Σ diag(o?). Then TC(p) = and TC(z) small if small. Consider q(>b (6) mn()) where (1, k) are indices of cube (minibatch,, minábalth,, dimension) with size M M and n(() sample drawn in minibatch and 2(0) ((n())). We claim this: when the ground truth total correlation of is low (the off-diagnal values of correlation matrix is small), only the elements on the diagonal surface of the cube, namely those elements of index (i, k), take some bounded values (1), and all the other elements very small o11) (since 0.1).
To see the claim, let's first consider 1-D case, where μ ~ NN, 1), z/p ~ o(). When or is small, z can be approximately treated as NO,,1) 1). 2(0) and u(lr are independent for i * j. hence 2(0) v(5) ~0 2), and
P([0() (1100~1001 110 <t)
t 0
(10)
See proof in Appendix A.5. Then for D-dimension, the probability O(P Now, only if takes value as small as a, q(>6 (n) is not small. For example, = 0.1 and = 10 and the chance of such case to happen is 0(10-19 Compared to batchsize, usually O(10)), the amount of such cases can be ignored.
-,00 () would be
5

---

Under review as a conference paper at ICLR 2020
Hence,
((2)
TC(z) = By(z)) log
II [(x))
E (( Dog [(=)] B)) Dog [[{E))]
log M
II
g(2 ( (2) - log II
Σ g( a(
M
log
Σ
(()) log II
[ou)
M
M j=i
j=i
1 log O1
1
log 0
M
M
MD
(((D 1) log M).
Assigning weights to elements q(>x (0) (r(00) such as MSS does not make essential change to the analysis above.
In addition, 3-TCVAE (trained with MSS in our experiments and with MWS in Locatello et al. (2018)) seems to have an increasing total correlation mean representation as regularization strength increases (higher F's), as observed by Locatello et al. (2018). Here, we provide ex- planation to the cause of this problem:
First, MSS and MWS prefer to shut down latent dimensions, meaning that distributions with fewer active dimensions can score lower estimated total correlation. Consider that Hu ~ N(, 0.01), Po- (0, Id) and also z/p (0, E), where Σ diag(o"). Then all elements of index (1, 0), i.e. q(20) (1) (n( are not small, say O(1), for the same reason as previous in dimension 1 Ej 7(20 latent space. Thus, M
9(=00) (n(0)
E, O(1) x 0(1, and
TC(z) = Ey(x) log ((=)
II {(x))
M Σ, II a(= (n()
log
M
I E, g( (n(i))
(7)
M Σ, (1(2) =0 (3)
(n()))
?= (n(0)))
log
M
I 1
M E, ((=b In())
0(1)
log
M
1s 77 0(1)
log (MM
O(D 2) log M):
compared to O((D - 1) log M) when (O, Id) in previous analysis.
Now, consider any strongly correlated z's (e.g.. (2),22) NO E). where Σ 0.01 0.1 ). see Figure 2). Then the Gaussian (ground truth) total correlation is arbitrar- -0.1
ily large (TC(z), 22) oo). This kind of distribution can score a relatively low TC value (for instance lower than z) with estimators such as MSS and MWS by the analysis above. Hence, as B increases, VAE trained with these estimators will be encouraged to obtain some dimensions of very low variance, and these dimensions are casily trapped in a strong correlation with other dimensions (like 2 and 22).

---

Under review as a conference paper at ICLR 2020
-1
sample
-3 mean
2
Figure 2: One dimension of mean has low variance (shutting down), and the distribution is strongly correlated (It appears to be almost flat due to small scale of the shutdown dimension). sampled distribution (e.g. z/ı (p, 1)) has very low TC.
Shutting down dimension is preferable because latent dimensions should not be fewer than ground truth. Moreover, considering datasets such as dSprites, though shape is labelled as a single dimension, models can learn represent complex geometry with multiple dimensions, hence more active dimensions are learned than the number of ground-truth dimensions.
Based on the reasons above, opt for the method of discriminators (density-ratio trick) in our implementation. Experiment shows that density-ratio trick provides a more stable estimation of total correlation when training VAE.
EXPERIMENTS
The datasets we use include dSprites Matthey et al. (2017), Shapes3D Burgess Kim (2018) and 3D faces Paysan al. (2009). At the time of writing, the scale of experiments is limited, but there are already some evidence to deliver our arguments. We have scheduled further experiments and tests on larger scale in future works.
For every model, we trained with 10 different initialization. Hyperparameter B, also the regulariza- tion strength, takes 2, 4,..... 10. For hyperparameter 7), we fix 1/ 10 for simple reason. Since the variance term in equation 4 becomes small (close to 0) shortly after training begins, this term will contribute much compared with Lbeta-TC term. So, 17 10 enough to strengthen the penalty at the beginning of training. In our experiments, higher value of cannot bring further improvement since close to f already. And lower values may not guarantee z being close to J..
From experiments, we observe that RAC-VAE has much lower TCmeon with different regulariza- tion strength than FactorVAE does (Figure 4). And on different datasets, this also the case (see Appendix). The TCCmea behaves almost identically as TCsample in RTC-VAE (see Figure 3 (b)). The problem contradictory behaviors TCmean and TCsample is evidently remedied by RTC- VAE. In addition, the ELBO of RTC-VAE seems converge faster than FactorVAE as byproduct (see Figure 5 and Figure 6). Examining the distributions of latent dimensions (mean representation), Factor tends have some strongly correlated latent dimensions (see Figure 8), and RTC-VAR shows well factorized latent distributions (see Figure 7).
6.1 METRICS OF DISENTANGLEMENT
We wish there were a widely accepted metric of disentanglement to compare our model RTCVAE with other models. Unfortunately, it is still open question, how can measure disentanglement. Various attempts have been made so far, but Locatello et al. (2018) challenged most of them, in- dicating that the score under any metric varies due to different initializations and datasets. Here,
7

---

Under review as a conference paper at ICLR 2020
FactoVes
dSprites-RTCC RTCVAE
0.045
0.45
0.040
412
0035
T_mean
m.maan
310
0.30
0.000
PaSaaplee
0.000
0.020
K.5ample
0.15
K_meen
8
10
10
regularzatio strength
regulariztion strengt
(a)
(b)
Figure 3: The shaded region indicates 90% confidence interval. (a) The total correlation of sample and mean representation of FactorVAE dSprite with 3 2, 6, 10. There is a large difference in scales sample and meann (b) The total correlation of sample and mean representation of RICVAE on dSprite with 3 2, 6, 10 and 7 10. There is almost no difference between I'Csample and menn due to the variance penalty term in equation 4.
dSprites
1-
FocEwVAE ATOVAE
10
repulatiaaion strungth
Figure 4: Direct comparison between the TC of FactorVAB and RTCVAE.
analyse several important metrics and attempt point out blind spots that have not being considered by these metrics.
Higgins al. (2017) proposed using classifier measure each dimension of latent space and each ground truth factor, e.g. (xr, y) coordinates, scale, rotation, etc. Kim & Mnih (2018) 8) revised this
dSprites beta-
betaits
dSprites beta-10
-13
-15-
-22
-50
-90
-90
100
100
110
1120
-120
GeliNE RTCHEE
-123
-130
1
10
2
22
d2
Etraton
Iterdd o1
ferss
(a)
(b)
(c)
Figure 5: Training ELBO of RTCVAE and Factor VAE.
8

---

Under review as a conference paper at ICLR 2020
S0upe:3
Shape:3 bela=6
Slupe3D
3300
3930
-350
-350
-3550
-100
3730
-300
-300
3530
-350
-3000
bata KTOME
-1900
bctoiii KTCHE
-3930
-3500
-310
RTOME
10
hembon
tenttoo
Ianttoo
(a)
(b)
(c)
Figure Training ELBO of RTCVAE and Factor on Shapes3D.
Figure 7: The pairplot of 10 latent dimensions of RTCVAE on Shapes3D, No dimensions show strong correlation.
approach yet kept the idea. Chen et al. (2018) introduced mutual information gap (MIG), which estimates the mutual information between each latent dimension and each ground truth factor. Note that this is classifier-free metric. Eastwood & Williams (2018) proposed framework of disentan-

---

Under review as a conference paper at ICLR 2020
Figure 8: The pairplot of 10 latent dimensions of FactorVAE on Shapes3D. Some dimensions show strong correlation, e.g. dim 1&3, dim 1&4, dim 3&4, dim 3&6.
glement metric that considers modularity, compactness and explicimesss Then Ridgeway & Mozer (2018) made analysis on compaciness, and compactness mean that each ground truth factor asso- ciates with only one few latent dimensions. They pointed out that in some situation perfectly disentangled representation may not be compact (see Section. 3 in Ridgeway & Mozer (2018)).
Here we argue that modularity also should be reconsidered. A modular representation means that each dimension of latent space conveys information of at most one ground truth factor. This is exactly the goal attempted by Higgins al. (2017); Kim Mnih (2018); Chen et al. (2018), etc. However, multiple latent dimensions meanwhile these latent dimensions are disentangled. For instance, I and y coordinates can be represented by and 0 in polar coordinate system (or any coordinate system under rotation, i.e., (x', s/)"' A(x, y) where A is any orthogonal matrix). These coordinate systems are perfectly disentangled but (or x') conveys information of both x and y.
work together represent multiple ground truth factors
CONCLUSION
In this work, we demonstrated that our RTC-VAE, which rectifies the total correlation penalty can remedy its peculiar properties (disparity between total correlation of the samples and the mean rep- resentations). Our experiments show that model has a more reasonable distribution of the mean representation compared with baseline models including 3-TCVAE and FactorVAE. We also pro- vide several theoretical proofs which could help diagnose several specific symptoms of entangle-
10

---

Under review as a conference paper at ICLR 2020
ment. Hopefully, our contributions could add to the explainability of the unsupervised learning of disentangled representations.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http:///gghhoooooooocccm/m/////s 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in vaes. arXiv preprint arXiv: 049422 2018.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5);6, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05900 2018.
Diederik P Kingma and Max Welling, arXiv:13/2.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint 17Xiv:1711.00843 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv: 2018.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://gginbb.ooomcceppa/////ssp
2017.
XuanLoong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. JEEE Transactions on Information Theory, 56(11):5847-5861, 2010.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. 2009 Sixth IEEE International Conference Advanced Video and Signal Based Surveillance, pp. 296-301. Ieee, 2009.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances in Neural Information Processing Systems, pp. 185-194, 2018.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12406-12415, 2019.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the breg- man divergence: a unified framework of density-ratis estimation. Annals of the Institute of Sta- tistical Mathematics, 1409):1009 2012.
A APPENDIX
A.1
MINIBATCH WEIGHTED SAMPLING (MWS)
See Chen et al. (2018),
1 M
M
Eg (x
[ogg(2)]
Σ
log
((((n)))1))- - log(NMM)
(1I)
M
1=1
12

---

Under review as a conference paper at ICLR 2020
A.2 MMS AND 1MS1
There is no mathematical difference between MSSa and MSS; (same formulation as equation 5), only a difference in implementation. We replace this chunk of code (https: /ttg//gttti github. com/ htt.chhen/bttttttcaaa///////aaaa
pyp.ant..tt.111.1 199999-20011 to
for i in range batchh _sze
Wii il 1/N W[ (1i.(1+1 batch..siizzee
seigh
A.3 PROOF OF THEOREM 1
In the following proof, we use the convention mathematical analysis that the meaning C can change through lines to eliminate some redundant work of tracking.
Theorem (Theorem restated). Let μ ~ NO, Σ) and oj be the standard deviation of μj. j = 1,..- D. and max, J Co. For fixed μ., let ~ Nu,, E' (4)), where E' is diagonal and satisfies that for some R > 0,
C2 (1( 2 0, 03 c3>of(1)
i| < R.
(12)
|n|" for some 1 12 1, | R.
for some constants C1 C2 Cz C4. Then TC(z) & C for some C = C(R, co. ... ",c,,) > 0.
Proof. Let
= (= p(2) 2 I(z))) S_ = E (/v(()) II
then
TC(z) 1 + = =T((z) + TC(Z)_
Since KL--dierreecee is non-negative, if TC(2)+ is bounded, then TC(z) must be bounded. In the following, we work on S4, i.c., we assume p(z) 2 II, p(zz).
For |2| < R, p(z) Epp(a [p(2|1)]
= f p(l)
1
(x-))(E)) (-)))()))((())))))))) (a-p))
dp
(( E
1
(22x) p(u)d4
< n
and
(3--0 20
I >
II p(u))
dp
((2
Bn(0)
C 28a
13

---

Under review as a conference paper at ICLR 2020
For |2| > 2R.
1 JB((0) B (1 (0)(BR(0)
+ $
p(=) =
(0)
4
C
1al R1
1/
121
Lef2
p(() e =)(=)))(=))
2(x z)²
<
26
+ 0
+
dp
B (0)
v ((2)) >>
1al Al
1/
DI
1-12
14101 Pt
+
C
+
p(u)
/22x) du
Big (0)
C
C
1=
20
+
2
dil E
+
|||
cf
cf
cf
C
< 2
DDDD
2--D-2-
2e
1
where ro max 02) and ri min(ci, c4), and since 12 1 and D 2 DI D 2 For |x| E (R,, 281), it is easy to see that p(z) < C. And for | > R
1
I 2
II
plu;)
2c
DPj
(2
1
j
Boo
(0)
C
3112
7
D
2
Hence,
p(z)
TC(z) Ep(z)
log
[I p(2))
< /
p(2) log C
D
2n
dz
+
p(z)log( log( ( 2 |2|
D1
21212 1n|2 of
2 )dz + C
BB(())
9
(0)
pp
< Doog
2
+ +
2R
C + (DI + D 2) log |2| +
2/
|
da + C
C1
(5)
7
2r
2R
02
< Dog
+
+
C.
C1
A.4 PROOF OF PROPOSITION
Proposition (Proposition restated). Let NO, E). then
1 TC(x) 2 (log diag Σ)] log|E]]
(13)
Proof. First, recall that the KL-divergence between two distributions IP and Q is defined
KL Q Ey flog
Also, the density function for a multivariate Gaussian distribution N E) is
1 pxz)= (2n) 1/2 exp( exp(( ---00 (0- T 1)).
14

---

Under review as a conference paper at ICLR 2020
Now, for two multivariate Gaussian P1 and P2, we have KL(PI
P2)
Er, [logIP] 1 log F2]
log det E
+ E (=)(t))((z= (x (1) (1) (I (x ( (x -22)
det 21
det 2et22
log der E
+ L
pp)())) -t [ (x -1)
- m1)²) + tr(20 (x - -2) - 122)
2
det Σn
= 2
log E1
tr(
(Ei E1) +
Bp Bpi(a) [tr [5g (x ((xx" -2m12 + w21l221
2
2
det
1
log del En
pp)(t) (2ng ((x ( Mi + m1)(2 -i + p1)² 2012 + p2l2 ))
E
2
der E2
1
=
log Ei
+
E
(xr(2 ((x - 11)(2 11)" + 2(x ui)l + +u2set - 2x12 + k211 ))
[tr Eg
2
2
2
B
2
log
+
=
ir
1 (Ei + (12 mn m1)(22 - )1)
2 det Σ
2
det
(log detE,
+ tr " Σn) (س² - 21) EE, (22 11))
Let P be multivariate Gaussian Ni,, En), and then the product of the marginal distribution Il px(x) is also Gaussian ((u, E2), where E₂ = iia Thus, the total correlation of multi- variate Gaussian distribution is
TC(x) = (p())l)) pi(x))
del Σg
(log di
n + tr(2i) ' + (a - 11)² E3²2211 (11 - 14)
2
1
(log detE2
n +
2
Eet
1
2 (log/diag( (naliag({))]]) log) En
A.5
PROOF OF EQUATION 10
Proof.
P([2() - -1000 <t)= -P(x] <t) where N(0.2)
e
dt
1 4
1
22
dr
dy
V
4
1
2241²
drdy
4m
2T
& drd0
re
4T
1
t
0(2)
2
A.6 EXPERIMENTS
15

---

Under review as a conference paper at ICLR 2020
Shape3D-FactorVAE
Shape3D-RTCVAE
175
014
0.60
8063
150
012
0.500
0050
125
100
310
0.455
0.000
0040
0.05
0:035
0.5
2.06
t. sample
0.000
0.000
ta.ample
2.04
--- 1. meen
0025
0.025
080
reguibrizatio strenghh
regulrizztion strength
(a)
Figure 9: The shaded region indicates 90% confidence interval. (a) The total correlation of sample
and mean representation of Factor VAE on Shapes3D with = 2, 6, 10. There is large difference
in scales of sample and T'mmaan- (b) The total correlation of sample and mean representation
of RTCVAE on Shapes3D with = 2, 6, 10 and = 10. There is almost no difference between TC sample and TCmeon due the variance penalty term in equation 4.
Shape3D
175
150
125
1.00
0.5
025
000
Figure 10: Direct comparison between the TCmean of FactorVAE and RTCVAE.
16


Reviews:
[{"review_id": "ryxVVIjaFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the extensions of variational autoencoders (VAEs), which take into account the total correlation of sampled distribution of latent variables. Proving a theorem that a family of distributions of sample representations with a bounded total correlation can have a mean representation of arbitrarily large total correlation, the authors propose RTC-VAE, which additionally penalizes total covariance of sampled latent variables. The authors demonstrate that RTC-VAE produces less correlated distributions of mean representation compared with baselines.\n\nThe proposed method, RTC-VAE, is based on a simple idea and its performance in experiments is promising. However, its derivation is somewhat ad-hoc and the experiments are not so comprehensive enough to provide the evidence for its good performance.\n\nThe covariance term in RTC-VAE of Eq. (4), is indirectly related to mean representation. Is it difficult to penalize TC(\\mu), directly?\n\nThe arguments in Section 5 are somewhat superficial. It isn’t explained how Eq. (5) is derived. Without the explicit definition of D(z), Eq. (8) is not so informative.\n\nSection 6: It is not discussed why the regularization parameter \\eta is fixed to 10 and how RTC-VAE is sensitive to \\eta.\n\nAre there any other performance measures to indicate the good overall performance of RTC-VAE other than the training ELBO in Fig. 5?\n\nMinor comments:\np.3, l.21 from the bottom: [z] should not be in the subscript.\np.4, l.3: n_{m+1} -> n_{M+1}\np.6, l.17 from the bottom: ``\"that Factor VAE does\" -> ``\"than Factor VAE does\"\n\n\nThe authors' responses answered my questions. However, I still think that the paper  has room for improvement in justifying the method, explaining the choice of hyperparameter and so on.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skxbij0pYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to address a known problem for unsupervised disentangling methods that penalises total correlation, namely that while the total correlation of the samples from q(z) (denoted TC(z)) are encouraged to be small, the total correlation of the means of q(z|x) (denoted TC(mu)), used as the disentangled representation in practice, is not necessarily small and can increase with regularisation strength.\n\nIn the introduction, I think that the statement “they concluded pessimistically that it is fundamentally impossible to learn a disentangled representation in an unsupervised setting” is a wrong interpretation of Locatello et al. They show that optimising marginal likelihood in a generative model (such as a VAE) cannot achieve disentangling without any inductive biases in the model. But there inductive biases in the models used by disentangling methods, along with the loss (that is a variant of the ELBO and not the marginal likelihood), that allow disentangling in practice. There are also theoretical works such as [1] that explain this behaviour.\n\nThe theoretical contribution of the paper is Theorem 1, that claims to show the existence of distributions with arbitrarily large TC(mu) but with small, bounded TC(z). Indeed the proof shows that TC(z) is bounded by C, but looking at Appendix A it seems as though C is a function of c1,c2 and R that is used to define p(z|mu), and is lower bounded by (2pi)^{-D/2} (a constant, which confusingly, is also denoted by C in the appendix). It appears necessary to have another line that mentions how small C can be chosen to be via choice of c1,c2,R. Also it seems as though the proof can be largely simplified by having sigma’_j(mu)=c_1 if |mu|<R and sigma’_j(mu)=c_4/|mu| if |mu|>R, removing free parameters l,c_2,c_3.\n\nThe methodological contribution of the paper is to propose an extra regularisation term that penalises the variances of q(z|x). While this does introduce another hyperparameter to tune, it has the advantage of being simple to implement and having an intuitive explanation of how it can address the problem; if q(z|x) is encouraged to have smaller variance, the distribution of z will be encouraged to be closer to the distribution of mu, hence helping to address the disparity between TC(z) TC(mu). \n\nI like the simplicity of the idea, however the analysis is lacking in rigour. First of all, when comparing the different methods of estimating TC(z), it’s not clear what the mathematical difference of MSS_0 and MSS_1 is. This should be explicitly stated so that one can understand the results in Figure 1. Regarding the following analysis, it’s not clear why the off diagonal elements of the cube (i.e. q(z^(i)_k|n^(j)) when i != j ) should be very small compared to the diagonal elements. For example, it could be the case that z^(i) and z^(j) are close to one another, in which case the (i,j)th entries will be non-negligible compared to the diagonal. Hence the analysis is difficult to accept. Also the claim that MSS and MWS prefer to shut down latent dims should be verified by experiments. Further, it’s unclear why this is undesirable from a disentangling point of view. Of course we don’t want to use fewer number of latent dimensions than the number of ground truth factors, but also we don’t want to use more latent dimensions. Also the at the bottom of page 5 is a Gaussian with a correlated covariance matrix, and it’s claimed that its TC can be arbitrarily large, but surely this is fixed? Finally the authors list lots of reasons why not to use MSS, but there is no discussion about the density-ratio trick method for estimating TC. For example, it is known that this method suffers underestimates of TC (c.f. Kim & Mnih) - it has its own issues, that can be arguably more severe than MSS. This analysis needs a lot more explanation and rigour.\n\nThe experimental results are very weak and sparse, that is nowhere near enough to give a convincing case for the newly proposed method. The method has only been trained on dsprites and 3d shapes, with three choices of beta and a single value of eta, and only estimates of TC(mu) and TC(z) are reported, with no evaluation of disentanglement performance. There is some evidence in the paper that the regularisation makes both TC(mu) and TC(z) close to 0 with eta=10, but it’s unclear how this affects disentangling performance, and whether smaller values of eta can give a sweetspot. The experiments should cover a larger range of datasets, with evaluation on how different disentanglement metrics, TC(mu) and TC(z) change for different values of beta and eta, along with a comparison with other disentangling methods, especially DIP-VAE-1, that directly penalises correlation in mu (for an open source library that facilitates this, see e.g. github.com/google-research/disentanglement_lib). Even the authors acknowledge that “the scale of our experiments is limited”, and it is clear that the paper is not yet ready for publication.\n\nOverall, the proposed idea is simple and easy to implement, which is the main advantage of the paper, but it is evident that the analysis and evaluation lacks rigour, hence the paper will need to undergo significant revision to be in a publishable state.\n\n[1] Rolinek, M., Zietlow, D. and Martius, G., Variational Autoencoders Pursue PCA Directions (by Accident). CVPR 2019.\n\nMinor typos/comments:\nEqn(4): the ||.||_1 should be replaced by trace, since the term inside ||.||_1 is a covariance matrix (although it’s diagonal). If A is a matrix, ||A||_1 is the maximum absolute column sum, which is different to what is meant by the paper, the trace (sum of diagonals).\np3: Can MWS also be stated in the paper (or at least in the appendix) to make it self-contained? \np4: followed <- follow, n_{m+1} <- n_{M+1}\np6-7: Section 6.1 should be in a related work section, and not under the Experiment section. The point about modularity is a fair but known issue, closely related to the issue of axis alignment/unidentifiability (see e.g. Rolinek et al)\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyePr6MAKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe novelty of this paper is adding an extra regularization term to the objective of beta-TCVAE (a VAE that regularizes total correlation), based on the discovery that low TC(z) does not necessarily mean low TC(mu). The added term enforces sample and mean representations stay close. \n\nThe authors' idea is understandable at a coarse resolution. However, the authors explain the mathematics poorly. Explanations of lots of variables and notions are missing. For example in Theorem 1, what is \"j\"? what is \\sigma_j? In Section 4, the simplification of notations lead to more difficulties to understand the formulas. In \"x_n\", is n the index of a sample or a dimension? The notations of variables are also confusing. Boldface lowercase letters should be used for vectors, and plain letter should be used for scalars. In Equation 4, what are D and k?    \n\nIt is nice to see, in the given experimental results,  that latent representations of RTCVAE are less correlated in comparison with FactorVAE in Figures 6 and 7. However, the authors should show some generated examples through latent variable traversal to qualitatively demonstrate the potential advantages of the proposed improvement.\n\nMinors: Section. X -> Section X\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper highlights the problem of penalizing the total correlation of sampled latent variables for unsupervised learning of disentangled representations. Authors prove a theorem on how sample representations with bounded total correlation may have arbitrarily large total correlation when computed with the underlying mean. As a fix, the authors propose RTC-VAE method that penalizes total covariance of sampled latent variables.

R2 appreciated the simplicity of the idea, making it easy to understand and implement, but raises serious concerns on empirical evaluation of the method. Specifically, very limited datasets (initially dsprites and 3d shapes) and with no evaluation of disentanglement performance and no comparison against other disentangling methods like DIP-VAE-1. While the authors added another dataset (3d face) in their revised versions, the concerns about disentanglement performance evaluation and its comparison against baselines remained as before, and R2 was not convinced to raise the initial score.

Similarly, while R1 and R3 appreciate author's response, they believe the response was not convincing enough for them, and maintained their initial ratings.

Overall, the submission has room for improvement toward a clear evaluation of the proposed method against related baselines.
Decision: Reject