Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: (v,w) categorizes the latent representations into disentangled dimensions and non-disentangled dimensions. Is it possible to consider group-wise disentangling? This is more practical for complex factors, such as human identity.

The measurement proposed in Section 3 is intuitively to measure if a certain dimension (or certain dimensions) has lower variance when a certain factor is fixed. Compared to more basic measurements, such as variance, what are the benefits of using a linear classifier? 

The presentation of Section 3 is confusing. Is it possible to make the intuition clearer in the main text?

In Section 4, was the measurement in Section 3 maximized to determine beta? (I have not read all details in Section 4, so I might just miss this)



review_title: About the assumption and measurement

final_score: rating: null
review_content: * I'm having trouble understanding the proposed disentanglement metric. Here's my attempt at summarizing the metric

1. Choose some factor k.
2. Sample two sets of latent representations, A and B, one with perturbed v_k.
3. Decode the re-encode each latent representation.
4. Use the difference between re-encoded latent representations to predict k and measure the accuracy of this predictor.

If the latent representation v_k is disentangled then step 4 will be easy, otherwise it will be hard. Thus, the performance of the predictor measures something like disentanglement.

Is that right? Could section 3 be clarified in the paper?

* Supposing my understanding of the metric is accurate, if v_k is correlated with other latent factors then it's impossible to distinguish between v_k and the correlated factors using a simple classifier. This makes the prediction task in step 4 hard. Otherwise, when v_k is not correlated with other factors, it's possible that z_diff can distinguish between v_ks, so step 4 could be easy. If this is the right idea, then is the proposed metric simply saying that a representation is disentangled when its components are not correlated with each other?
review_title: Questions about the disentanglement metric

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper proposes the beta-VAE, which is a reasonable but also straightforward generalization of the standard VAE. In particular, a weighting factor beta is added for the KL-divergence term to balance the likelihood and KL-divergence. Experimental results show that tuning this weighting factor is important for learning disentangled representations. A linear-classifier based protocol is proposed for measuring the quality of disentanglement. Impressive illustrations on manipulating latent variables are shown in the paper. 

Learning disentangled representations without supervision is an important topic. Showing the effectiveness of VAE for this task is interesting. Generalizing VAE with a weighting factor is straightforward (though reformulating VAE is also interesting), the main contribution of this paper is on the empirical side. 

The proposed protocol for measuring disentangling quality is reasonable. Establishing protocol is one important methodology contribution of this paper, but the presentation of Section 3 is still not good. Little motivation is provided at the beginning of Section 3. Figure 2 is a summary of the algorithm, which is helpful, but it still necessary to intuitively explain the motivation at the first place (e.g., what you expect if a factor is disentangled, and why the performance of a classifier can reflect such an expectation). Moreover, 1) z_diff appeared without any definition in the main text. 2) Use “decoding” for x~Sim(v,w) may make people confuse the ground truth sampling procedure w ith the trained decoder. 

The illustrative figures on traversing the disentangled factor are impressive, though image generation quality is not as good as InfoGAN (not the main point of this paper). However, 1) it will be helpful to discuss if the good disentangling quality only attribute to the beta factor and VAE framework. For example, the training data in this paper seems to be densely sampled for the visualized factors. Does the sampling density play a critical role? 2) Not too many qualitative results are provided for each experiment? Adding more figures (e.g., in appendix) to cover more factors and seeding images can strength the conclusions drawn in this paper. 3) Another detailed question related to the generalizability of the model: are the seeding image for visualizing faces from unseen subjects or subjects in the training set? (maybe I missed something here.)

Quantitative results are only presented for the synthesized 2D shape. What hinders this paper from reporting quantitative numbers on real data (e.g., the 2D and 3D face data)? One possible reason is that not all factors can be disentangled for real data, but it is still feasible to pick up some well-defined factor to measure the quantitative performance. 

Quantitative performance is only measured by the proposed protocol. Since the effectiveness of the protocol is something the paper need to justify, reporting quantitative results using simpler protocol is helpful both for demonstrating the disentangling quality and for justifying the proposed protocol (consistency with other measurement). A simple experiment is facial identity recognition and pose estimation using disentangled features on a standard test set (like in Reed et al, ICML 2014). 

In Figure 6 (left), why ICA is worse than PCA for disentanglement? Is it due to the limitation of the ICA algorithm or some other reasons? 

In Figure 6 (right), what is “factor change accuracy”? According to Appendix A.4 (which is not referred to in the main text), it is the “Disentanglement metric score”. Is that right?
If so Figure 6 (right) shows the reconstruction results for the best disentanglement metric score. Then, 1) how about random generation or traversing along a disentangled factor? 2) more importantly, how is the reconstruction/generation results when the disentanglement metric score is suboptimal. 

Overall, the results presented in this paper are very interesting, but there are many details to be clarified. Moreover, more quantitative results are also needed. I hope at least some of the above concerns can be addressed. 




review_title: Very interesting results, but more details and more quantitative results are needed

final_score: rating: 7: Good paper, accept
review_content: The paper is attacking a classical and very hard problem: non-linear PCA i.e. learning the principal components a.k.a independent factors, a.k.a. orthogonal geodesics in the highly non-linear latent manifold of a given data-set. Moreover, the paper is hoping for these factors to be humanly-interpretable. The problem is so hard that is likely unsolvable in unsupervised fashion for real-life datasets. After all, even we humans are able to zero-in the the type of, say, legs of chairs (Fig 3 in the paper), only after we have identified the object as a chair and have rotated and translated it. The importance of this problem is hard to overstate, so we hope the paper is accepted, on the grounds of asking so fundamental a question.

The paper correctly points out the inability of VAE to disentangle important factors even on toy data-sets. This of course has been known for awhile, e.g., Fig 4. on reference [1], from almost 2 years ago. On the back of so much hope and expectation built-up in the introduction, the solution put forward in the paper strikes us as a curious but a hardly useful toy. Slapping a large multiplicative factor on the generative error term of a VAE is not going to work for any real-life datasets. Generation without reconstruction leads to schizophrenia, as every respectable psychiatrist will  testify. Physicists have attacked this problem considerably earlier than DeepMind and their scalable and conceptual solutions have been discussed at length in references [1], section 1.6, section 4, and most of reference [2]. In short, for spatial, color, time symmetries, symmetry statistics are produced by smaller specialized nets like spatial transformers and used to augment the latent variables, to aid the decoder. As demonstrated in reference [2], Figures 3,4, this is indispensable in order to handle distortion, e.g., spatial transformation. Note that this approach is completely unsupervised. 

This of course does not handle complex factors like "type of legs" but that is a hell of an ambitious goal, and while we hope to be proven wrong, probably way beyond reach of machines and even many humans (for real-life data-sets that is!)

By complete luck, the authors may have hit upon something else of fundamental importance: the letter "beta" for their multiplicative coefficient is of course reserved in statistical physics for the inverse of the temperature. This requires the separation of generative "energy" and "entropy" and is  partially addressed in section 2.9 of reference [1]. The correct definition of "generative temperature" is not published yet, but used extensively in experiments and can be privately communicated upon request. When worked out correctly, the beta does not multiply the generative error, as in this paper, so it would be very interesting to repeat the toy experiments here with the "correct" physical model instead.

A question to the authors: creating a new metric ("disentanglement") as u do is commendable but after hundred years of PCA, ICA, etc , is there no some proxy that can be used instead? Also, could you not for example simply distort the dataset along  different factors and then look for the quality of the reconstructed images (as in the Introduction of reference [2])?

[1] https://arxiv.org/pdf/1508.06585v5.pdf
[2] https://arxiv.org/pdf/1511.02841v3.pdf
review_title: Asking very important questions of non-linear PCA (learning latent manifolds of datasets), often way beyond even human capabilities. Offering a toy solution, partly superseded by existing approaches. Non-scalable to real-life, non-toy data-sets.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes beta-VAE which strengthen the KL divergence between the recognition model and the prior to limit the capacity of latent variables while sacrificing the reconstruction error. This allows the VAE model to learn more disentangled representation. 

The main concern is that the paper didn't present any quantitative result on log likelihood estimation. On the quality of generated samples, although the beta-VAE learns disentangled representation, the generated samples are not as realistic as those based on generative adversarial network, e.g., InfoGAN. Beta-VAE learns some interpretable factors of variation, but it still remains unclear why it is a better (or more efficient) representation than that of standard VAE.

In experiment, what is the criteria for cross-validation on hyperparameter \beta?

There also exists other ways to limit the capacity of the model. The simplest way is to reduce the latent variable dimension. I am wondering how the proposed beta-VAE is a better model than the VAE with reduced, or optimal latent variable dimension.

review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary
===

This paper presents Beta-VAE, an augmented Variational Auto-Encoder which
learns disentangled representations. The VAE objective is derived
as an approximate relaxation of a constrained optimization problem where
the constraint matches the latent code of the encoder to a prior.
When KKT multiplier beta on this constraint is set to 1 the result is the
original VAE objective, but when beta > 1 we obtain Beta-VAE, which simply
increases the penalty on the KL divergence term. This encourages the model to
learn a more efficient representation because the capacity of the latent
representation is more limited by beta. The distribution of the latent
representation is rewarded more when factors are independent because
the prior (an isotropic Gaussian) encourages independent factors, so the
representation should also be disentangled.

A new metric is proposed to evaluate the degree of disentanglement. Given
a setting in which some disentangled latent factors are known, many examples
are generated which differ in all of these factors except one. These examples
are encoded into the learned latent representation and a simple classifier
is used to predict which latent factor was kept constant. If the learned
representation does not disentangle the constant factor then the classifier
will more easily confuse factors and its accuracy will be lower. This
accuracy is the final number reported.

A synthetic dataset of 2D shapes with known latent factors is created to
test the proposed metric and Beta-VAE outperforms a number of baselines
(notably InfoGAN and the semi-supervised DC-IGN).

Qualitative results show that Beta-VAE learns disentangled factors
on the 3D chairs dataset, a dataset of 3D faces, and the celebA dataset
of face images. The effect of varying Beta is also evaluated using the proposed
metric and the latent factors learned on the 2D shapes dataset are explored
in detail.


Strengths
===
* Beta-VAE is simple and effective.

* The proposed metric is a novel way of testing whether ground truth factors
of variation have been identified.

* There is extensive comparison to relevant baselines.


Weaknesses
===

* Section 3 describes the proposed disentanglement metric, however I feel
I need to read the caption of the associated figure (I thank for adding
that) and Appendix 4 to understand the metric intuitively or in detail.
It would be easier to read this section if a clear intuition preceeded
a detailed description and I think more space should be devoted to this
in the paper.

* Appendix 4: Why was the bottom 50% of the resulting scores discarded?

* As indicated in pre-review comments, the disentanglement metric is similar
to a measure of correlation between latent features. Could the proposed metric
be compared to a direct measure of cross-correlation between latent factors
estimated over the 2D shapes dataset?


* The end of section 4.2 observes that high beta values result in low
disentanglement, which suggests the most efficient representation is not
disentangled. This seems to disagree with the intuition from the approach
section that more efficient representations should be disentangled. It would
be nice to see discussion of potential reasons for this disagreement.

* The writing is somewhat dense.


Overall Evaluation
===
The core idea is novel, simple and extensive tests show that it is effective.
The proposed evaluation metric is novel might come into broader use.
The main downside to the current version of this paper is the presentation,
which provides sufficient detail but could be more clear.
review_title: Simple and effective

Meta-review: This paper proposes a modification of the variational ELBO in encourage 'disentangled' representations, and proposes a measure of disentanglement. The main idea and is presented clearly enough and explored through experiments. This whole area still seems a little bit conceptually confused, but by proposing concrete metrics and methods, this paper makes several original contributions.
Decision: Accept (Poster)