Year: 2019

Abstract:
In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent variables. Unfortunately, this well-motivated strategy often fails to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation. We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of a bounded total correlation. However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy, RTC-VAE, which rectifies the total correlation penalty. Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g., β-TCVAE and FactorVAE.

heading: 1 INTRODUCTION
text: VAEs (Variational AutoEncoders) Kingma & Welling (2013); Bengio et al. (2007) follow the common assumption that the high-dimensional real world observations x can be re-generated by a lowerdimension latent variable z which is semantically meaningful. Recent works Kim & Mnih (2018); Chen et al. (2018); Kumar et al. (2017) suggest that decomposing the ELBO (Evidence Lower Bound) could lead to distinguishing the factor of disentanglement. In particular, recent works Kim & Mnih (2018); Chen et al. (2018) focused on a term called total correlation (TC). The popular belief Chen et al. (2018) is that by adding weights to this term in objective function, a VAE model can learn a disentangled representation. This approach appears to be promising since the total correlation of a sampled representation should describe the level of factorising since total correlation is defined to be the KL-divergence between the joint distribution z ∼ q(z) and the product of marginal distributions ∏ j q(zj). In this case, a low value suggests a less entangled joint distribution. However, Locatello et al. (2018) pointed out that the total correlation of sampled distribution TCsample being low does not necessarily give rise to a low total correlation of the corresponding mean representation TCmean. Conventionally, the mean representation is used as the encoded latent variables, an unnoticed high TCmean is usually the culprit behind the undesirable entanglement. Moreover, Locatello et al. (2018) found that as regularization strength increases, the total correlation of sampled representation TCsample and mean representation TCmean are actually negatively correlated.
Locatello et al. (2018) put doubts on most methods of disentanglement including penalizing the total correlation term Kim & Mnih (2018); Chen et al. (2018), and they concluded that ”the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases”.
Acknowledging the difficulty in learning disentangled representation, we provide a detailed explanation of the seemingly contradictory behaviors of the total correlations of sampled and mean representation in previous works on TC penalizing strategy. Moreover, we find that this problem described above can be remedied simply with an additional penalty term on the variance of a sampled representation.
Our contributions:
• In Theorem 1, we prove that for all mean representations, there exists a large class of sample distributions with bounded total correlation. Particularly, a mean representation with arbitrarily large total correlation can have a corresponding sample distribution with low total correlation. This implies that a low total correlation of sample distribution cannot guarantee a low total correlation of the mean representation. (Section. 2) • Acknowledging the issue above, we further delve into total correlation, and provide a sim-
ple remedy by adding an additional penalty term on the variance of sample distribution. The penalty term forces a sampled representation to behave similar to the corresponding mean representation. Such penalty term is necessary for the strategy of penalizing TCmean in the view of Theorem 1. (Section. 4) • We study several different methods of estimating total correlation. They are compared and
benchmarked against the ground truth value on the multivariate Gaussian distribution Locatello et al. (2018). We point out that the method of (minibatch) estimators suffers from the curse of dimensionality and other drawbacks, making their estimation accuracy decay significantly with the increase of the dimension of the latent space, and some strong correlated distributions can be falsely estimated to have low total correlation. (Section. 5)

heading: 2 THE PECULIARITY OF TOTAL CORRELATION
text: In information theory, total correlation is one of the generalizations of mutual information, which measures the difference between the joint distribution of multiple random variables and the product of their marginal distributions. A high value means the joint distribution is far from an independent distribution, and hence it suggests high entanglement among these random variables. Definition 1. Total correlation of random variable x,
TC(x) := KL p(x)||∏ j p(xj)  = Ep(x) [ log p(x)∏ j p(xj) ] .
Naturally, people seek the solution of disentanglement in the form of low total correlation of the latent variables, e.g. Kim & Mnih (2018); Chen et al. (2018). However, there can be large difference between the total correlations of sample representation and mean representation. Forcing the former to be small does not guarantee the latter being small. In fact, given a mean representation of arbitrarily large total correlation, we can construct a family of distribution of sample representation that have a bounded total correlation, where the bound does not rely on the total correlation of the mean. Theorem 1. Let µ ∼ N (0,Σ) and σj be the standard deviation of µj , j = 1, · · · , D, and maxj σj = c0. For a fixed µ, let z ∼ N (µ,Σ′(µ)), where Σ′(µ) is diagonal and satisfies that for some R > 0,  c2 > σ ′ j(µ) > c1 > 0, if |µ| < R,
c3 > σ ′ j(µ) > c4 |µ|l , for some l ≥ 1, if |µ| > R. (1)
for some constants c1, c2, c3, c4. Then TC(z) ≤ C for some C = C(R, c0, · · · , c4, l) > 0.
The details of the proof are presented in Appendix A.3. Here’s another way to interpret Theorem 1: with C and parameters R, c0, · · · , c4, l fixed, one can make TC(µ) arbitrarily large, since TC(µ) depends only on the correlation matrix of µ (see Proposition 1).
Theorem 1 provides an explanation to the contradiction observed by Locatello et al. (2018) that TC(z) is low does not mean TC(µ) is low (actually much higher than TC(z)). Since there exist such a large class of distributions of z that all have bounded TC(z). When the objective function only penalizes TC(µ), neural networks are so flexible to easily find a distribution with low TC(z), and total correlation estimators like MSS can encourage shutting down latent dimensions (see Section 5.3), which together cause the disparity of TC(µ) and TC(z). This fact was not noticed until Locatello et al. (2018), and our investigation gives an explanation of the peculiar property of total correlation. Hence, Theorem 1 leads to the necessity of a regularizer of the difference between the
distributions of µ and z when penalizing TCsample. In Section. 4, we propose a simple regularizer that serves this goal.
It is an interesting question whether there exists a distribution of z N (µ,Σ′(µ)) with arbitrarily small TC(z) given µ. If not, what’s the lower bound of TC(z)? These questions remain open to us for now, and we leave them to future work.

heading: 3 RELATED WORKS
text: In the study of disentanglement, Higgins et al. (2017) proposed a modification of the VAE framework and introduced an adjustable hyperparameter β that balances latent channel capacity and independence constraints with reconstruction accuracy. One drawback of β-VAE is the trade-off between the reconstruction quality and disentanglement. Motivated to alleviate this trade-off of β-VAE, Kim & Mnih (2018) proposed FactorVAE which decomposes the evidence lower bound and penalize a term measuring the total correlation between latent variables. Around the same time, Chen et al. (2018) proposed a similar ELBO decomposition method called β-TCVAE. The major difference between FactorVAE and β-TCVAE lies in their different strategies of estimating total correlation. Chen et al. (2018) used formulated estimators while Kim & Mnih (2018) utilized the density-ratio trick which requires an auxiliary discriminator network and an inner optimization loop. We will discuss these two strategies more in details in Section. 5.
The works above belong to representative learning without inductive biases. There are also works about representative learning with inductive biases, see Rolinek et al. (2019) and references therein.
As for the disentanglement metric, this will be discussed in Section. 6.

heading: 4 RECTIFIED-TCVAE
text: To simplify notation, let p(n) = p(xn), q(z|xn) = q(z|n). Recall the average evidence lower bound (ELBO),
ELBO := Ep(n) [ Eq(z|n)[log p(n|z)]−KL(q(z|n)‖p(z)) ] . (2)
Chen et al. (2018) and independently by Kim & Mnih (2018) introduced objective function that penalizes total correlation, which can be formulated as
Lβ−TC := ELBO− βTC(z). (3)
This approach unfortunately has a drawback. It turns out that instead of being able to obtain disentangled representation, we often find a sample representation appears to be disentangled while the mean representation is still entangled. In fact, when we are maximizing Lβ−TC, we could totally end up learning a distribution of z that makes TC(z) goes low, while the total correlation of its mean µ is still high. To resolve this, we define RTC-VAE,
LRTC := Lβ−TC − η · tr(Ep(n)Covq(z|n)[z]), (4)
where
tr(Ep(n)Covq(z|n)[z]) = D∑ k Ep(n)[σ2k(n)].
Our penalty originates from the first term of the law of total covariance Covq(z)[z] = Ep(n)Covq(z|n)[z] + Covp(n)(Eq(z|n)[z]). Factorized representation1 indicates a diagonal covariance matrix Covq(z)[z]. Motivated by this, Kumar et al. (2017) penalizes the off-diagonal terms in the second term, while ignores Ep(n)Covq(z|n)[z] since it is diagonal. Their penalty term leads to a vanishing µ, which is the mean representation. The remedy to this is to add another penalty term on the distance between µ’s and 1. DIP-VAE Kumar et al. (2017) employs this remedy, however, DIPVAE does not outperform other VAE’s when measured by various disentanglement metrics, e.g., FactorVAE score, see Fig. 3 & 14 in Locatello et al. (2018). This is actually not surprising since the
1Factorized representation means each latent dimension is independent.
two penalty terms in DIP-VAE contribute in opposite directions, with one leading to vanishing µ’s and another fighting against it. This formulation can easily get the model stuck in saddle points.
Our objective, on the other hand, does not penalize directly on µ. Instead, it penalizes on σ, the standard deviation of the distribution q(z|n). This may seem little counter-intuitive at first sight, since penalizing a diagonal component of covariance Cov[z] = Covq(z)[z] seems not helpful to factorising. However, in the view of Theorem 1, our objective will force the distribution of z to be similar to the distribution of µ. Hence, it pushes us away from the situation of large TC(µ) and low TC(z). Consequently, by minimizing TC(z) we get a model that has low TC(µ), a disentangled mean representation.

heading: 5 ESTIMATION OF TOTAL CORRELATION
text: The naive Monte Carlo method comes with an intrinsic issue of underestimating total correlation. To avoid or resolve this, Kim & Mnih (2018) proposed a discriminator network with the help of density-ratio trick (see equation (3) and Appendix D. of Kim & Mnih (2018)). In Chen et al. (2018), two kinds of estimator of total correlation are proposed, Minibatch Weighted Sampling (MWS) and Minibatch Stratified Sampling (MSS) (see Appendix C.1 and C.2 in Chen et al. (2018)).

heading: 5.1 METHOD OF MINIBATCH ESTIMATORS
text: For instance, MSS can be described as followed. For a minibatch of sample, BM+1 = {n1, . . . , nM+1},
Eq(z,n)[log q(z)] ≈ 1
M + 1 M+1∑ i=1 log f(zi, ni, BM+1 \ {ni}), (5)
where2
f(z, n∗, BM+1 \ {n∗}) = 1 N q(z|n∗) + 1 M M−1∑ m=1 q(z|nm) + N −M NM q(z|nm). (6)
For the convenience of readers, MWS is listed in Appendix A.1.

heading: 5.2 METHOD OF DENSITY-RATIO TRICK AND DISCRIMINATOR
text: Density-ratio trick Nguyen et al. (2010); Sugiyama et al. (2012) can be used to estimate KLdivergence,
TC(z) = KL(q(z)‖ ∏ j q(zj)) = Eq(z)
[ log
q(z)∏ j q(zj)
] (7)
≈ Eq(z) [ D(z)
1−D(z)
] , (8)
where D is discriminator that classifies z being sampled from q(z) or ∏ j q(zj). For detail implementation, please refer to section 3 in Kim & Mnih (2018).

heading: 5.3 COMPARISON OF THE TWO METHODS
text: For multivariate normal distribution, the total correlation can be explicitly calculated which can be used as a ground truth for our comparison. To be specific, Proposition 1. Let x ∼ N (0,Σ), then
TC(x) = 1
2 (log|diag(Σ)| − log|Σ|) . (9)
2There is a small part of the implementation of MSS in Chen et al.’s code that is not quite clear to us, specifically, the computation of log importance weight matrix in equation 6. In our experiment, we implement MSS with our understanding and denote it as MSS1, and we denote Chen et al.’s implementation MSS0. See Appendix A.2
It’s difficult to track the exact reference of Proposition 1 since it is a fundamental property in information theory. Locatello et al. (2018) used this proposition to approximate the total correlation of the mean representation in latent space. In appendix, we provide a simple proof for the convenience of the readers.
We compared the performance of each method, MWS, MSS0 and MSS1 on the estimation of total correlation. For µ ∼ N (0, I), and z|µ ∼ N (µ,Σ) where Σ = diag(σ2) and σ = 0.1. We choose σ small so that the distribution of z can be approximated by normal distribution. Results are presented in Figure 1.
From Figure. 1, we can summarize the following observations: 1. MWS tends to underestimate total correlation in general; 2. For latent space of dimension ≤ 4, MSS0 and MSS1 are quite accurate; 3. For latent space of high dimension, both MSS0 and MSS1 tend to overestimate total correlation when the actual value of total correlation is small; 4. Overall MSS1 estimates closer to ground truth than MMS0 does.
In the following analysis of the above observations, we use a less formal way of analyzing, which can be formalized to be rigorous, in order to convey our idea directly.
To interpret the third observation, let µ ∼ N (0, Id) and z|µ ∼ N (0,Σ), where Id is identity matrix and Σ = diag(σ2). Then TC(µ) = 0 and TC(z) small if σ small. Consider q(z(i)k |n(j)) where (i, j, k) are indices of a cube (minibatch,minibatch, dimension) with size M ×M ×D and n(j) is a sample drawn in a minibatch and z(i) = z(n(i)). We claim this: when the ground truth total correlation of z is low (the off-diagnal values of correlation matrix is small), only the elements on the diagonal surface of the cube, namely those elements of index (i, i, k), take some bounded values O(1), and all the other elements are very small o(1) (since σ = 0.1).
To see the claim, let’s first consider 1-D case, where µ ∼ N (0, 1), z|µ ∼ N (0, σ2). When σ is small, z can be approximately treated as N(0, 1). z(i) and µ(j) are independent for i 6= j, hence z(i) − µ(j) ∼ N (0, 2), and
P (|z(i) − µ(j)| < t) = t 2 +O(t2) (10)
See a proof in Appendix A.5. Then for D-dimension, the probability P (|z(i) − µ(j)| < t) would be O(tD). Now, only if t takes value as small as σ, q(z(i)k |n(j)) is not small. For example, σ = 0.1 and D = 10 and the chance of such case to happen is O(10−10). Compared to batchsize, usually O(103), the amount of such cases can be ignored.
Hence,
TC(z) = Eq(z) [ log
q(z)∏ k q(zk) ] = Eq(z,n)[log q(z)]− Eq(z,n)[log
∏ k q(zk)]
≈ 1 M ∑ i log 1 M ∑ j ∏ k q(z (i) k |n (j))− log ∏ k 1 M ∑ j q(z (i) k |n (j))  ≈ 1 M ∑ i log 1 M ∑ j=i O(1)− log ∏ k 1 M ∑ j=i O(1)
 ≈ 1 M ∑ i ( logO( 1 M )− logO( 1 MD )
) ≈ O((D − 1) logM).
Assigning weights to elements q(z(i)k |n(j)) such as MSS does not make essential change to the analysis above.
In addition, β-TCVAE (trained with MSS in our experiments and with MWS in Locatello et al. (2018)) seems to have an increasing total correlation of mean representation as regularization strength increases (higher β’s), as observed by Locatello et al. (2018). Here, we provide an explanation to the cause of this problem:
First, MSS and MWS prefer to shut down latent dimensions, meaning that distributions with fewer active dimensions can score lower estimated total correlation. Consider that µ0 ∼ N (0, 0.01), µ0− ∼ N (0, Id) and also z|µ ∼ N (0,Σ), where Σ = diag(σ2). Then all elements of index (i, j, 0), i.e., q(z(i)0 |n(j)) are not small, sayO(1), for the same reason as previous inD−1 dimension latent space. Thus, 1M ∑ j q(z (i) 0 |n(j)) ≈ 1M ∑ j O(1) ≈ O(1), and
TC(z) = Eq(z) [ log
q(z)∏ k q(zk)
]
≈ 1 M ∑ i log 1 M
∑ j ∏ k q(z
(i) k |n(j))∏
k 1 M ∑ j q(z (i) k |n(j))
≈ 1 M ∑ i log 1 M
∑ j(q(z (i) 0 |n(j)) · ∏ k>0 q(z
(i) k |n(j)))∏
k>0 1 M ∑ j q(z (i) k |n(j))
≈ 1 M ∑ i log 1 M ·O(1)∏ k>0 1 M ·O(1) ≈ logO(MD−2) ≈ O((D − 2) logM),
compared to O((D − 1) logM) when µ ∼ N (0, Id) in previous analysis. Now, consider any strongly correlated z’s (e.g., (z1, z2) ∼ N (0,Σ), where Σ =(
0.01 −0.1 −0.1 1
) , see Figure 2). Then the Gaussian (ground truth) total correlation is arbitrar-
ily large (TC(z1, z2) = ∞). This kind of distribution can score a relatively low TC value (for instance lower than z) with estimators such as MSS and MWS by the analysis above. Hence, as β increases, VAE trained with these estimators will be encouraged to obtain some dimensions of very low variance, and these dimensions are easily trapped in a strong correlation with other dimensions (like z1 and z2).
Shutting down dimension is not preferable because latent dimensions should not be fewer than ground truth. Moreover, considering datasets such as dSprites, though shape is labelled as a single dimension, models can learn to represent complex geometry with multiple dimensions, hence more active dimensions are learned than the number of ground-truth dimensions.
Based on the reasons above, we opt for the method of discriminators (density-ratio trick) in our implementation. Experiment shows that density-ratio trick provides a more stable estimation of total correlation when training VAE.

heading: 6 EXPERIMENTS
text: The datasets we use include dSprites Matthey et al. (2017), Shapes3D Burgess & Kim (2018) and 3D faces Paysan et al. (2009). At the time of writing, the scale of our experiments is limited, but there are already some evidence to deliver our arguments. We have scheduled further experiments and tests on larger scale in future works.
For every model, we trained with 10 different initialization. Hyperparameter β, also the regularization strength, takes 2, 4, · · · , 10. For hyperparameter η, we fix η = 10 for a simple reason. Since the variance term in equation 4 becomes small (close to 0) shortly after training begins, this term will not contribute much compared with Lbeta−TC term. So, η = 10 is enough to strengthen the penalty at the beginning of training. In our experiments, higher value of η cannot bring further improvement since z is close to µ already. And lower values may not guarantee z being close to µ.
From experiments, we observe that RTC-VAE has much lower TCmean with different regularization strength than FactorVAE does (Figure 4). And on different datasets, this is also the case (see Appendix). The TCmean behaves almost identically as TCsample in RTC-VAE (see Figure 3 (b)). The problem of contradictory behaviors of TCmean and TCsample is evidently remedied by RTCVAE. In addition, the ELBO of RTC-VAE seems to converge faster than FactorVAE as a byproduct (see Figure 5 and Figure 6). Examining the distributions of latent dimensions (mean representation), FactorVAE tends to have some strongly correlated latent dimensions (see Figure 8), and RTC-VAE shows well factorized latent distributions (see Figure 7).

heading: 6.1 METRICS OF DISENTANGLEMENT
text: We wish there were a widely accepted metric of disentanglement to compare our model RTCVAE with other models. Unfortunately, it is still an open question, how we can measure disentanglement. Various attempts have been made so far, but Locatello et al. (2018) challenged most of them, indicating that the score under any metric varies due to different initializations and datasets. Here,
we analyse several important metrics and attempt to point out some blind spots that have not being considered by these metrics.
Higgins et al. (2017) proposed using a classifier to measure each dimension of latent space and each ground truth factor, e.g. (x, y) coordinates, scale, rotation, etc. Kim & Mnih (2018) revised this
approach yet kept the idea. Chen et al. (2018) introduced mutual information gap (MIG), which estimates the mutual information between each latent dimension and each ground truth factor. Note that this is a classifier-free metric. Eastwood & Williams (2018) proposed a framework of disentan-
glement metric that considers modularity, compactness and explicitness. Then Ridgeway & Mozer (2018) made analysis on compactness, and compactness mean that each ground truth factor associates with only one or a few latent dimensions. They pointed out that in some situation a perfectly disentangled representation may not be compact (see Section. 3 in Ridgeway & Mozer (2018)).
Here we argue that modularity also should be reconsidered. A modular representation means that each dimension of latent space conveys information of at most one ground truth factor. This is exactly the goal attempted by Higgins et al. (2017); Kim & Mnih (2018); Chen et al. (2018), etc. However, multiple latent dimensions can work together to represent multiple ground truth factors meanwhile these latent dimensions are disentangled. For instance, x and y coordinates can be represented by r and θ in polar coordinate system (or any coordinate system under rotation, i.e., (x′, y′)T = A(x, y)T where A is any orthogonal matrix). These coordinate systems are perfectly disentangled but r (or x′) conveys information of both x and y.

heading: 7 CONCLUSION
text: In this work, we demonstrated that our RTC-VAE, which rectifies the total correlation penalty can remedy its peculiar properties (disparity between total correlation of the samples and the mean representations). Our experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models including β-TCVAE and FactorVAE. We also provide several theoretical proofs which could help diagnose several specific symptoms of entangle-
ment. Hopefully, our contributions could add to the explainability of the unsupervised learning of disentangled representations.

heading: A APPENDIX
text: A.1 MINIBATCH WEIGHTED SAMPLING (MWS)
See Chen et al. (2018),
Eq(z)[log q(z)] ≈ 1
M M∑ i=1 log M∑ j=1 q(z(ni)|nj)− log(NM)  (11)
A.2 MMS0 AND MMS1
There is no mathematical difference between MSS0 and MSS1 (same formulation as equation 5), only a difference in implementation. We replace this chunk of code (https://github.com/ rtqichen/beta-tcvae/blob/master/vae_quant.py#L199-L201) to
f o r i i n r a n g e ( b a t c h s i z e ) : W[ i , i ] = 1 /N W[ i , ( 1 + i )% b a t c h s i z e ] = s t r a t w e i g h t
A.3 PROOF OF THEOREM 1
In the following proof, we use the convention of mathematical analysis that the meaning of C can change through lines to eliminate some redundant work of tracking.
Theorem (Theorem 1 restated). Let µ ∼ N (0,Σ) and σj be the standard deviation of µj , j = 1, · · · , D, and maxj σj = c0. For a fixed µ, let z ∼ N (µ,Σ′(µ)), where Σ′(µ) is diagonal and satisfies that for some R > 0, c2 > σ ′ j(µ) > c1 > 0, if |µ| < R,
c3 > σ ′ j(µ) > c4 |µ|l , for some l ≥ 1, if |µ| > R. (12)
for some constants c1, c2, c3, c4. Then TC(z) ≤ C for some C = C(R, c0, · · · , c4, l) > 0.
Proof. Let S+ = {z ∈ RD|p(z) ≥ ∏ j p(zj)}, S− = {z ∈ RD|p(z) < ∏ j p(zj)},
then
TC(z) = ∫ S+ + ∫ S− = TC(z)+ +TC(z)−.
Since KL-divergence is non-negative, if TC(z)+ is bounded, then TC(z) must be bounded. In the following, we work on S+, i.e., we assume p(z) ≥ ∏ j p(zj).
For |z| < R,
p(z) = Ep(µ)[p(z|µ)]
= ∫ p(µ)
1√ (2π)D|Σ′| e− 1 2 (z−µ)(Σ ′)−1(z−µ)T dµ
≤ 1√ (2π)Dc2D1
∫ p(µ)dµ
≤ C cD1 ,
and
∏ j p(zj) ≥ ∏ j ∫ BR(0) p(µj) 1√ (2π)Dc22 e − (zj−µj) 2 2c21 dµj
≥ C cD2 e − 2R2 c21 .
For |z| > 2R,
p(z) = ∫ BR(0) + ∫ B |z|
2
(0)\BR(0) + ∫ Bc|z|
2
(0)
≤ C cD1 e − ||z|−R|
2
2c22 + C |z|Dl cD4 e − |z|
2
8c22 + ∫ Bc|z|
2
(0)
p(µ) e− 1 2 (µ−z)(Σ ′)−1(µ−z)T√ (2π)d|Σ′| dµ
≤ C cD1 e − ||z|−R|
2
2c22 + C |z|Dl cD4 e − |z|
2
8c22 + ∫ Bc|z|
2
(0) p(µ) |µ|Dl√ (2π)DcD4 dµ
≤ C cD1 e − ||z|−R|
2
2c22 + C
cD4 |z|dle
− |z| 2
8c22 + C
cD4 |z|Dl+D−2e
− |z| 2
8c20
≤ C rD1 |z|Dl+D−2e
− |z| 2
8r20 ,
where r0 = max(c0, c2) and r1 = min(c1, c4), and since l ≥ 1 and D ≥ 1, Dl +D − 2 ≥ 0. For |z| ∈ (R1, 2R1), it is easy to see that p(z) < C. And for |z| > R,
∏ j p(zj) ≥ ∏ j (∫ Bc0 (0) p(µj) 1√ (2π)Dc22 e − (zj−µj) 2 2c21 Dµj )
≥ C cD2 e − 2|z|
2
c21 .
Hence,
TC(z) = Ep(z)
[ log
p(z)∏ j p(zj)
]
≤ ∫ BR(0) p(z) logC cD2 cd1 e 2R2 c21 dz + ∫ Bc2R(0) p(z) log(C cD2 rD1 |z|Dl+D−2e 2|z|2 c21 − |z| 2 2r20 )dz + C
≤ D log c2 c1 + 2R2 c21 + ∫ Bc2R(0) e − |z| 2 8r20 [C + (Dl +D − 2) log |z|+ 2|z| 2 c21 − |z| 2 2r20 ]dz + C
≤ D log c2 c1 + 2R2 c21 + C.
A.4 PROOF OF PROPOSITION 1
Proposition (Proposition 1 restated). Let x ∼ N (0,Σ), then
TC(x) = 1
2 (log|diag(Σ)| − log|Σ|) . (13)
Proof. First, recall that the KL-divergence between two distributions P and Q is defined as
KL(P||Q) = EP[log P Q ]
Also, the density function for a multivariate Gaussian distribution N (µ,Σ) is
p(x) = 1 (2π)n/2det(Σ)1/2 exp(−1 2 (x− µ)TΣ−1(x− µ)).
Now, for two multivariate Gaussian P1 and P2, we have KL(P1||P2) = EP1 [logP1 − logP2]
= 1
2 log detΣ2 detΣ1 + 1 2 Ep1(x)[−(x− µ1) TΣ−11 (x− µ1) + (x− µ2)TΣ −1 2 (x− µ2)]
= 1
2 log detΣ2 detΣ1 + 1 2 Ep1(x)[−tr(Σ −1 1 (x− µ1)(x− µ1)T ) + tr(Σ −1 2 (x− µ2)(x− µ2)T )]
= 1
2 log detΣ2 detΣ1 − 1 2 tr(Σ−11 Σ1) + 1 2 Ep1(x)[tr(Σ −1 2 ((xx T − 2xµT2 + µ2µT2 ))]
= 1
2 log detΣ2 detΣ1 − n 2 + 1 2 Ep1(x)[tr(Σ −1 2 ((x− µ1 + µ1)(x− µ1 + µ1)T − 2xµT2 + µ2µT2 ))]
= 1
2 log detΣ2 detΣ1 − n 2 + 1 2 Ep1(x)[tr(Σ −1 2 ((x− µ1)(x− µ1)T + 2(x− µ1)µ1︸ ︷︷ ︸
Ep1(x)(x)=µ1
+µ1µ T 1 − 2xµT2 + µ2µT2 ))]
= 1
2 log detΣ2 detΣ1 − 1 2 n+ 1 2 tr(Σ−12 (Σ1 + (µ2 − µ1)(µ2 − µ1)T ))
= 1
2 (log detΣ2 detΣ1 − n+ tr(Σ−12 Σ1) + (µ2 − µ1)TΣ −1 2 (µ2 − µ1))
Let P be a multivariate Gaussian N (µ,Σ1), and then the product of the marginal distribution∏ i pi(x) is also Gaussian N (µ,Σ2), where Σ2 = diag(Σ1). Thus, the total correlation of multivariate Gaussian distribution is TC(x) = DKL(p(x)||
∏ i pi(x))
= 1
2 (log detΣ2 detΣ1 − n+ tr(Σ−12 Σ1) + (µ− µ)TΣ −1 2 (µ− µ))
= 1
2 (log detΣ2 detΣ1 − n+ n)
= 1
2 (log|diag(Σ1)| − log|Σ1|)
A.5 PROOF OF EQUATION 10
Proof. P (|z(i) − µ(j)| < t) = P (|x| < t) where x ∼ N (0, 2)
= ∫ t −t 1√ 4π e− x2 4 dt
= √∫ t −t 1√ 4π e− x2 4 dx ∫ t −t 1√ 4π e− y2 4 dy
= √∫ t −t ∫ t −t 1 4π e− x2+y2 4 dxdy
= √∫ 2π 0 ∫ t 0 1 4π re− r2 4 drdθ
= √ 1− e− t 2 4
= t
2 +O(t2)
A.6 EXPERIMENTS

Review summaries:
This paper considers the extensions of variational autoencoders (VAEs), which take into account the total correlation of sampled distribution of latent variables. Proving a theorem that a family of distributions of sample representations with a bounded total correlation can have a mean representation of arbitrarily large total correlation, the authors propose RTC-VAE, which additionally penalizes total covariance of sampled latent variables. The authors demonstrate that RTC-VAE produces less correlated distributions of mean representation compared with baselines.