Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
The novelty of this paper is adding an extra regularization term to the objective of beta-TCVAE (a VAE that regularizes total correlation), based on the discovery that low TC(z) does not necessarily mean low TC(mu). The added term enforces sample and mean representations stay close. 

The authors' idea is understandable at a coarse resolution. However, the authors explain the mathematics poorly. Explanations of lots of variables and notions are missing. For example in Theorem 1, what is "j"? what is \sigma_j? In Section 4, the simplification of notations lead to more difficulties to understand the formulas. In "x_n", is n the index of a sample or a dimension? The notations of variables are also confusing. Boldface lowercase letters should be used for vectors, and plain letter should be used for scalars. In Equation 4, what are D and k?    

It is nice to see, in the given experimental results,  that latent representations of RTCVAE are less correlated in comparison with FactorVAE in Figures 6 and 7. However, the authors should show some generated examples through latent variable traversal to qualitatively demonstrate the potential advantages of the proposed improvement.

Minors: Section. X -> Section X


review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes a method to address a known problem for unsupervised disentangling methods that penalises total correlation, namely that while the total correlation of the samples from q(z) (denoted TC(z)) are encouraged to be small, the total correlation of the means of q(z|x) (denoted TC(mu)), used as the disentangled representation in practice, is not necessarily small and can increase with regularisation strength.

In the introduction, I think that the statement “they concluded pessimistically that it is fundamentally impossible to learn a disentangled representation in an unsupervised setting” is a wrong interpretation of Locatello et al. They show that optimising marginal likelihood in a generative model (such as a VAE) cannot achieve disentangling without any inductive biases in the model. But there inductive biases in the models used by disentangling methods, along with the loss (that is a variant of the ELBO and not the marginal likelihood), that allow disentangling in practice. There are also theoretical works such as [1] that explain this behaviour.

The theoretical contribution of the paper is Theorem 1, that claims to show the existence of distributions with arbitrarily large TC(mu) but with small, bounded TC(z). Indeed the proof shows that TC(z) is bounded by C, but looking at Appendix A it seems as though C is a function of c1,c2 and R that is used to define p(z|mu), and is lower bounded by (2pi)^{-D/2} (a constant, which confusingly, is also denoted by C in the appendix). It appears necessary to have another line that mentions how small C can be chosen to be via choice of c1,c2,R. Also it seems as though the proof can be largely simplified by having sigma’_j(mu)=c_1 if |mu|<R and sigma’_j(mu)=c_4/|mu| if |mu|>R, removing free parameters l,c_2,c_3.

The methodological contribution of the paper is to propose an extra regularisation term that penalises the variances of q(z|x). While this does introduce another hyperparameter to tune, it has the advantage of being simple to implement and having an intuitive explanation of how it can address the problem; if q(z|x) is encouraged to have smaller variance, the distribution of z will be encouraged to be closer to the distribution of mu, hence helping to address the disparity between TC(z) TC(mu). 

I like the simplicity of the idea, however the analysis is lacking in rigour. First of all, when comparing the different methods of estimating TC(z), it’s not clear what the mathematical difference of MSS_0 and MSS_1 is. This should be explicitly stated so that one can understand the results in Figure 1. Regarding the following analysis, it’s not clear why the off diagonal elements of the cube (i.e. q(z^(i)_k|n^(j)) when i != j ) should be very small compared to the diagonal elements. For example, it could be the case that z^(i) and z^(j) are close to one another, in which case the (i,j)th entries will be non-negligible compared to the diagonal. Hence the analysis is difficult to accept. Also the claim that MSS and MWS prefer to shut down latent dims should be verified by experiments. Further, it’s unclear why this is undesirable from a disentangling point of view. Of course we don’t want to use fewer number of latent dimensions than the number of ground truth factors, but also we don’t want to use more latent dimensions. Also the at the bottom of page 5 is a Gaussian with a correlated covariance matrix, and it’s claimed that its TC can be arbitrarily large, but surely this is fixed? Finally the authors list lots of reasons why not to use MSS, but there is no discussion about the density-ratio trick method for estimating TC. For example, it is known that this method suffers underestimates of TC (c.f. Kim & Mnih) - it has its own issues, that can be arguably more severe than MSS. This analysis needs a lot more explanation and rigour.

The experimental results are very weak and sparse, that is nowhere near enough to give a convincing case for the newly proposed method. The method has only been trained on dsprites and 3d shapes, with three choices of beta and a single value of eta, and only estimates of TC(mu) and TC(z) are reported, with no evaluation of disentanglement performance. There is some evidence in the paper that the regularisation makes both TC(mu) and TC(z) close to 0 with eta=10, but it’s unclear how this affects disentangling performance, and whether smaller values of eta can give a sweetspot. The experiments should cover a larger range of datasets, with evaluation on how different disentanglement metrics, TC(mu) and TC(z) change for different values of beta and eta, along with a comparison with other disentangling methods, especially DIP-VAE-1, that directly penalises correlation in mu (for an open source library that facilitates this, see e.g. github.com/google-research/disentanglement_lib). Even the authors acknowledge that “the scale of our experiments is limited”, and it is clear that the paper is not yet ready for publication.

Overall, the proposed idea is simple and easy to implement, which is the main advantage of the paper, but it is evident that the analysis and evaluation lacks rigour, hence the paper will need to undergo significant revision to be in a publishable state.

[1] Rolinek, M., Zietlow, D. and Martius, G., Variational Autoencoders Pursue PCA Directions (by Accident). CVPR 2019.

Minor typos/comments:
Eqn(4): the ||.||_1 should be replaced by trace, since the term inside ||.||_1 is a covariance matrix (although it’s diagonal). If A is a matrix, ||A||_1 is the maximum absolute column sum, which is different to what is meant by the paper, the trace (sum of diagonals).
p3: Can MWS also be stated in the paper (or at least in the appendix) to make it self-contained? 
p4: followed <- follow, n_{m+1} <- n_{M+1}
p6-7: Section 6.1 should be in a related work section, and not under the Experiment section. The point about modularity is a fair but known issue, closely related to the issue of axis alignment/unidentifiability (see e.g. Rolinek et al)

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper considers the extensions of variational autoencoders (VAEs), which take into account the total correlation of sampled distribution of latent variables. Proving a theorem that a family of distributions of sample representations with a bounded total correlation can have a mean representation of arbitrarily large total correlation, the authors propose RTC-VAE, which additionally penalizes total covariance of sampled latent variables. The authors demonstrate that RTC-VAE produces less correlated distributions of mean representation compared with baselines.

The proposed method, RTC-VAE, is based on a simple idea and its performance in experiments is promising. However, its derivation is somewhat ad-hoc and the experiments are not so comprehensive enough to provide the evidence for its good performance.

The covariance term in RTC-VAE of Eq. (4), is indirectly related to mean representation. Is it difficult to penalize TC(\mu), directly?

The arguments in Section 5 are somewhat superficial. It isn’t explained how Eq. (5) is derived. Without the explicit definition of D(z), Eq. (8) is not so informative.

Section 6: It is not discussed why the regularization parameter \eta is fixed to 10 and how RTC-VAE is sensitive to \eta.

Are there any other performance measures to indicate the good overall performance of RTC-VAE other than the training ELBO in Fig. 5?

Minor comments:
p.3, l.21 from the bottom: [z] should not be in the subscript.
p.4, l.3: n_{m+1} -> n_{M+1}
p.6, l.17 from the bottom: ``"that Factor VAE does" -> ``"than Factor VAE does"


The authors' responses answered my questions. However, I still think that the paper  has room for improvement in justifying the method, explaining the choice of hyperparameter and so on.

review_title: Official Blind Review #3

Meta-review: This paper highlights the problem of penalizing the total correlation of sampled latent variables for unsupervised learning of disentangled representations. Authors prove a theorem on how sample representations with bounded total correlation may have arbitrarily large total correlation when computed with the underlying mean. As a fix, the authors propose RTC-VAE method that penalizes total covariance of sampled latent variables.

R2 appreciated the simplicity of the idea, making it easy to understand and implement, but raises serious concerns on empirical evaluation of the method. Specifically, very limited datasets (initially dsprites and 3d shapes) and with no evaluation of disentanglement performance and no comparison against other disentangling methods like DIP-VAE-1. While the authors added another dataset (3d face) in their revised versions, the concerns about disentanglement performance evaluation and its comparison against baselines remained as before, and R2 was not convinced to raise the initial score.

Similarly, while R1 and R3 appreciate author's response, they believe the response was not convincing enough for them, and maintained their initial ratings.

Overall, the submission has room for improvement toward a clear evaluation of the proposed method against related baselines.
Decision: Reject