Title: beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Authors: Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner
Venue: ICLR 2017

Abstract: Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial intelligence that is able to learn and reason in the same way that humans do. We introduce beta-VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. Our approach is a modification of the variational autoencoder (VAE) framework. We introduce an adjustable hyperparameter beta that balances latent channel capacity and independence constraints with reconstruction accuracy. We demonstrate that beta-VAE with appropriately tuned beta > 1 qualitatively outperforms VAE (beta = 1), as well as state of the art unsupervised (InfoGAN) and semi-supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs). Furthermore, we devise a protocol to quantitatively compare the degree of disentanglement learnt by different models, and show that our approach also significantly outperforms all baselines quantitatively. Unlike InfoGAN, beta-VAE is stable to train, makes few assumptions about the data and relies on tuning a single hyperparameter, which can be directly optimised through a hyper parameter search using weakly labelled data or through heuristic visual inspection for purely unsupervised data.

TL;DR: We introduce beta-VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner.

Review discussion:
(v,w) categorizes the latent representations into disentangled dimensions and non-disentangled dimensions. Is it possible to consider group-wise disentangling? This is more practical for complex factors, such as human identity.
The measurement proposed in Section 3 is intuitively to measure if a certain dimension (or certain dimensions) has lower variance when a certain factor is fixed. Compared to more basic measurements, such as variance, what are the benefits of using a linear classifier?
The presentation of Section 3 is confusing. Is it possible to make the intuition clearer in the main text?
In Section 4, was the measurement in Section 3 maximized to determine beta? (I have not read all details in Section 4, so I might just miss this)
What do you mean by ‘group-wise disentangling’? Do you mean conditionally dependent factors (i.e. given identity X, factors such as face shape, eye colour, hair style etc can be explained away)? If this is the case, we believe that identity would be a ‘complex’ factor consisting of a particular combination of more ‘simple’ factors of face shape, eye colour etc. Learning such ‘complex’ factors would require learning hierarchical latent representations, which is something we do not address in the current work but leave for future efforts. The present paper only attempts to learn a set of ‘simple’ data generative factors that lie at the bottom of such potential hierarchy.
Our model can only learn to disentangle factors that are statistically independent in the data. If in the data certain features are strongly correlated (e.g. full lips often co-occur with wider noses due to the two features being encoded in the same genes), then our model will learn a factor that encodes both of these features, resulting in ‘group-wise disentangling’ of a sort. An example of such coding can be found in CelebA, where it appears that the majority of young people are female and the majority of older people are male.
Hence our model learns to disentangle out a ‘group-wise’ factor of age/gender (as seen in Figure 4).
Linear classifier helps our disentanglement metric have a single interpretable score that we can use to better compare different models in a way that reflects disentanglement. Calculating the average variance per factor per X batches would be a worse measure, because we are interested in factor k having lower variance on average compared to other factors i != k. In absolute terms such average variance is not informative with regards to the level of disentanglement achieved. We believe that using a linear classifier is one of the easier ways to quantify whether factor k has lower average variance.
We have added Figure 2 in Section 3 to add an intuitive clarification of our proposed metric.
We did not explicitly try to maximise the disentanglement metric to determine beta, because this metric works in a semi-supervised case and we wanted to demonstrate that beta-VAE works in a completely unsupervised setting. We did show, however, that the best values of betas chosen through visual inspection heuristics match well the optimal betas found through calculating the disentanglement metric (see Fig. 5 right)
* I'm having trouble understanding the proposed disentanglement metric. Here's my attempt at summarizing the metric
1. Choose some factor k.
2. Sample two sets of latent representations, A and B, one with perturbed v_k.
3. Decode the re-encode each latent representation.
4. Use the difference between re-encoded latent representations to predict k and measure the accuracy of this predictor.
If the latent representation v_k is disentangled then step 4 will be easy, otherwise it will be hard.
Thus, the performance of the predictor measures something like disentanglement.
Is that right? Could section 3 be clarified in the paper?
* Supposing my understanding of the metric is accurate, if v_k is correlated with other latent factors then it's impossible to distinguish between v_k and the correlated factors using a simple classifier. This makes the prediction task in step 4 hard.
Otherwise, when v_k is not correlated with other factors, it's possible that z_diff can distinguish between v_ks, so step 4 could be easy. If this is the right idea, then is the proposed metric simply saying that a representation is disentangled when its components are not correlated with each other?
The intuition is almost correct. In the terms suggested by the reviewer, the correct intuition would be:
1. Choose some factor k.
2. For a batch of N samples:
a. Sample two sets of latent representations, A and B, where v_k^A = v_k^B (and all other v_i may change).
b. Decode then re-encode each latent representation.
c. Calculate the difference z_diff between re-encoded latent representations
3. Use the average z_diff over N samples to predict k and measure the accuracy of this predictor.
We have modified Section 3 to add an intuitive clarification.
Your intuition is correct - the metric does say that a representation is disentangled when its components are not correlated, whereby p(z) = ∏_i ∫_x p(z_i | x) . The components may be conditionally dependent given a particular image though: p(z_x) != ∏_i p(z_i | x). The metric can deal with such conditional dependencies due to its batch averaging of z_diff.
Also the model can only learn to disentangle factors that are statistically independent in the data.
For example, if in CelebA the majority of young people are female and the majority of older people are male, the best one can hope for is for the model to disentangle out a joint factor of age/gender (as seen in Figure 4).
This paper proposes the beta-VAE, which is a reasonable but also straightforward generalization of the standard VAE.
In particular, a weighting factor beta is added for the KL-divergence term to balance the likelihood and KL-divergence. Experimental results show that tuning this weighting factor is important for learning disentangled representations. A linear-classifier based protocol is proposed for measuring the quality of disentanglement. Impressive illustrations on manipulating latent variables are shown in the paper.
Learning disentangled representations without supervision is an important topic. Showing the effectiveness of VAE for this task is interesting. Generalizing VAE with a weighting factor is straightforward (though reformulating VAE is also interesting), the main contribution of this paper is on the empirical side.
The proposed protocol for measuring disentangling quality is reasonable. Establishing protocol is one important methodology contribution of this paper, but the presentation of Section 3 is still not good. Little motivation is provided at the beginning of Section 3. Figure 2 is a summary of the algorithm, which is helpful, but it still necessary to intuitively explain the motivation at the first place (e.g., what you expect if a factor is disentangled, and why the performance of a classifier can reflect such an expectation).
Moreover, 1) z_diff appeared without any definition in the main text. 2) Use “decoding” for x~Sim(v,w) may make people confuse the ground truth sampling procedure w ith the trained decoder.
The illustrative figures on traversing the disentangled factor are impressive, though image generation quality is not as good as InfoGAN (not the main point of this paper).
However, 1) it will be helpful to discuss if the good disentangling quality only attribute to the beta factor and VAE framework.
For example, the training data in this paper seems to be densely sampled for the visualized factors. Does the sampling density play a critical role? 2) Not too many qualitative results are provided for each experiment? Adding more figures (e.g., in appendix) to cover more factors and seeding images can strength the conclusions drawn in this paper. 3) Another detailed question related to the generalizability of the model: are the seeding image for visualizing faces from unseen subjects or subjects in the training set? (maybe I missed something here.)
Quantitative results are only presented for the synthesized 2D shape. What hinders this paper from reporting quantitative numbers on real data (e.g., the 2D and 3D face data)? One possible reason is that not all factors can be disentangled for real data, but it is still feasible to pick up some well-defined factor to measure the quantitative performance.
Quantitative performance is only measured by the proposed protocol. Since the effectiveness of the protocol is something the paper need to justify, reporting quantitative results using simpler protocol is helpful both for demonstrating the disentangling quality and for justifying the proposed protocol (consistency with other measurement). A simple experiment is facial identity recognition and pose estimation using disentangled features on a standard test set (like in Reed et al, ICML 2014).
In Figure 6 (left), why ICA is worse than PCA for disentanglement? Is it due to the limitation of the ICA algorithm or some other reasons?
In Figure 6 (right), what is “factor change accuracy”? According to Appendix A.4 (which is not referred to in the main text), it is the “Disentanglement metric score”. Is that right?
If so Figure 6 (right) shows the reconstruction results for the best disentanglement metric score.
Then, 1) how about random generation or traversing along a disentangled factor? 2) more importantly, how is the reconstruction/generation results when the disentanglement metric score is suboptimal.
Overall, the results presented in this paper are very interesting, but there are many details to be clarified.
Moreover, more quantitative results are also needed. I hope at least some of the above concerns can be addressed.
We are very grateful to the reviewer for their very thorough assessment of our work and the numerous suggestions. We have made a number of changes to fully address them.
1.
About our presentation of our disentanglement metric, this was a concern shared with Reviewer 3. We have now addressed this by introducing three new paragraphs at the start of Section 3 to provide an intuitive explanation of our disentanglement metric. We also rewrote the end of Section 3 to simplify the presentation.
2. Z_diff is now properly defined, sorry for this oversight.
3. We changed ‘decode’ to ‘simulate’ and ‘re-encode’ to ‘infer’ in order to avoid confusion between our disentanglement metric and the trained decoder, thank you for mentioning it.
4. We have added a sentence in Conclusions with a reference to Section A.8 in the Appendix. This section discusses the effects of data generative factor sample density on the learnt representations of beta-VAE, with empirical results.
5. We have added extra traversal plots in the appendix, with more seeding images. These include all latents that learnt to be informative when presented with 3D chairs or CelebA.
6.
About the generalizability of our model: currently all seeding images are from the same full training set. As done in this literature [e.g. InfoGAN], we did not hold out any test set. This would involve handling a domain adaptation scenario, which we are interested in tackling but decided to leave for future work.
7.
On the topic of running the disentanglement metric on real datasets: we explored the possibility on the CelebA dataset, but did not report a score for two main reasons.
First, as the reviewer suggested, many of the labels in the dataset are not those that could be reasonably expected to be disentangled, and indeed may not correspond to ground truth factors of variation.
For example, ‘attractiveness’ is such a label.
Secondly, even though there were a (small) subset of labels that could potentially disentangle well, these labels do not necessarily correspond to factors that control for large visual changes in the subject’s faces. Visually, we feel that the primary factors of variation are “background colour”, “azimuth/rotation”, “facial identity”, and “hair colour”. Of these, only hair colour was present as an available label (albeit with quite poor labelling, e.g. many subjects did not have a particular hair colour specified).
In future work we would hope to demonstrate the efficacy of the metric on better labeled datasets, such as Caltech-UCSD Birds (http://www.vision.caltech.edu/visipedia/CUB-200.html) or Oxford Flowers (http://www.robots.ox.ac.uk/~vgg/data/flowers/).
8. The reviewer mentions using another simpler protocol to assess our disentanglement metric, this is indeed a great suggestion.
We have added a new section in Appendix A.5, with a new Table 2, to describe the performance of a simple classifier trying to predict the data generative factor values of the 2D shapes dataset using different latent representations.
We find that the results of this classification test match those of our proposed disentanglement metric, thus providing further validation for it as the reviewer was suggesting.
9. We are not entirely sure why PCA representations are more disentangled than ICA representations according to our metric. Both approaches score similarly well in terms of cross correlations between the latents, as would be expected (as mentioned to AnonReviewer1). According to our new Table 2 in Appendix A.5, the independent components learnt by ICA tend to be very well suited for object identity identification, while those learnt by PCA are more representative of the object location.
10. Thank you for finding the mention of the “factor change accuracy” term, this indeed was a typo. We updated Fig.6 to use “Disentanglement Metric Score” as expected.
11. We also now added a reference to Appendix A.4 in the main text, sorry for the oversight.
12.
Finally about your points following the mention of Figure 6 (right):
We have included random samples from both disentangled and entangled models in Appendix A.9, Figure 9.
As for latent traversal visualisations for both conditions, this is actually what can already be seen in Fig. 7, panels A and B, if we understood the question correctly?
We hope this addresses the reviewer’s concerns and welcome any further feedback.
Thank you for the authors' detailed reply. The descriptions of the evaluation metric are much better.
The discussion in A.8 is very helpful. In the meanwhile, learning disentangled features with beta-VAE on relatively discontinuous data also seems to be difficult.
For Table 2, the results are fine, but also a bit mixed (compared to PCA).
Moreover, in order to show the degree of disentanglement, it is also important to show the classification rate using features mismatched with the tasks (e.g., id recognition using rotation features). I understand it may be cumbersome to report all possible combinations, but focusing on one or two recognition tasks (e.g., the id recognition) is definitely doable.
In general, I recommend acceptance for this paper, because 1) disentangling factors of variations in an unsupervised way is important; 2) the visualization results are interesting. In the meanwhile, the quantitive results and practical significance are not so strong as expected (I am not saying they are too weak).
Summary
This paper presents Beta-VAE, an augmented Variational Auto-Encoder which
learns disentangled representations. The VAE objective is derived
as an approximate relaxation of a constrained optimization problem where
the constraint matches the latent code of the encoder to a prior.
When KKT multiplier beta on this constraint is set to 1 the result is the
original VAE objective, but when beta > 1 we obtain Beta-VAE, which simply
increases the penalty on the KL divergence term. This encourages the model to
learn a more efficient representation because the capacity of the latent
representation is more limited by beta. The distribution of the latent
representation is rewarded more when factors are independent because
the prior (an isotropic Gaussian) encourages independent factors, so the
representation should also be disentangled.
A new metric is proposed to evaluate the degree of disentanglement. Given
a setting in which some disentangled latent factors are known, many examples
are generated which differ in all of these factors except one. These examples
are encoded into the learned latent representation and a simple classifier
is used to predict which latent factor was kept constant. If the learned
representation does not disentangle the constant factor then the classifier
will more easily confuse factors and its accuracy will be lower. This
accuracy is the final number reported.
A synthetic dataset of 2D shapes with known latent factors is created to
test the proposed metric and Beta-VAE outperforms a number of baselines
(notably InfoGAN and the semi-supervised DC-IGN).
Qualitative results show that Beta-VAE learns disentangled factors
on the 3D chairs dataset, a dataset of 3D faces, and the celebA dataset
of face images. The effect of varying Beta is also evaluated using the proposed
metric and the latent factors learned on the 2D shapes dataset are explored
in detail.
Strengths
* Beta-VAE is simple and effective.
* The proposed metric is a novel way of testing whether ground truth factors
of variation have been identified.
* There is extensive comparison to relevant baselines.
Weaknesses
* Section 3 describes the proposed disentanglement metric, however I feel
I need to read the caption of the associated figure (I thank for adding
that) and Appendix 4 to understand the metric intuitively or in detail.
It would be easier to read this section if a clear intuition preceeded
a detailed description and I think more space should be devoted to this
in the paper.
* Appendix 4: Why was the bottom 50% of the resulting scores discarded?
* As indicated in pre-review comments, the disentanglement metric is similar
to a measure of correlation between latent features. Could the proposed metric
be compared to a direct measure of cross-correlation between latent factors
estimated over the 2D shapes dataset?
* The end of section 4.2 observes that high beta values result in low
disentanglement, which suggests the most efficient representation is not
disentangled. This seems to disagree with the intuition from the approach
section that more efficient representations should be disentangled. It would
be nice to see discussion of potential reasons for this disagreement.
* The writing is somewhat dense.
Overall Evaluation
The core idea is novel, simple and extensive tests show that it is effective.
The proposed evaluation metric is novel might come into broader use.
The main downside to the current version of this paper is the presentation,
which provides sufficient detail but could be more clear.
We’d like to thank the reviewer for their thorough review and helpful suggestions.
1. We acknowledge the importance of clearly explaining our disentangled metric and we agree that our initial attempt can be improved. In order to address that, we added three new paragraphs at the beginning of Section 3 to provide an intuitive explanation for our disentanglement metric. We also rewrote the end of that section to simplify the formal presentation of the metric. We hope this provides enough background and support for why we devised it that way and what exactly it is assessing.
2.
With respect to the bottom 50% question: we have added a sentence at the end of the second paragraph of Appendix A.4 to address it. The bottom 50% of the resulting scores were discarded to control for the outlier results from the few experiments that diverged during training.
3. Thank you for the suggestion. The cross-correlation fails to fully capture the properties we would look for in a disentanglement metric, as we now discuss in the new first 3 paragraphs of Section 3.
We did calculate it though (average absolute values of upper triangular terms in correlation matrix, estimated over the entire dataset), but unfortunately found that the cross-correlation does not seem to provide an appropriate measure (we’re also concerned with the large standard deviations).
Here are the values we obtained:
PCA: 4.7e-9 (std 5e-9) // ICA: 1.15e-6 (std 9e-7) // DC-IGN: 0.15 (std 0.16) //
InfoGAN: 0.09 (std 0.08) // VAE 0.039 (std 0.04) // beta-VAE: 0.057 (std: 0.053)
As expected, PCA and ICA produce the least correlated latents, since these approaches directly optimise for reducing cross-correlation between latents. The other approaches, including the semi-supervised DC-IGN, produce more correlated latents.
However, these are nevertheless disentangled according to a human interpretation of what disentangled factors should look like.
This suggests that humanly interpretable disentangled factors of variation are still somewhat correlated in the real world, which suggests that PCA and ICA may be optimising the wrong objective for interpretable disentangling.
Hence we think that our metric provides a better test for interpretable disentangled representations, which a simple cross-correlation fails to capture.
4. We have added a sentence in the relevant paragraph in Section 4.2. If $\beta$ is too high and the resulting capacity of the latent channel is lower than the number of data generative factors, then the learnt representation necessarily has to be entangled (as we are compressing more than what a factorial representation would require).
We hope this addresses the reviewer’s concerns and welcome any further feedback.
[EDIT: sorry, we initially entered wrong cross-correlation values for VAE and beta-VAE, but have now corrected them]
Sorry, we initially entered wrong cross-correlation values for VAE and beta-VAE in the original response, but have now corrected them.
The paper proposes beta-VAE which strengthen the KL divergence between the recognition model and the prior to limit the capacity of latent variables while sacrificing the reconstruction error. This allows the VAE model to learn more disentangled representation.
The main concern is that the paper didn't present any quantitative result on log likelihood estimation. On the quality of generated samples, although the beta-VAE learns disentangled representation, the generated samples are not as realistic as those based on generative adversarial network, e.g., InfoGAN. Beta-VAE learns some interpretable factors of variation, but it still remains unclear why it is a better (or more efficient) representation than that of standard VAE.
In experiment, what is the criteria for cross-validation on hyperparameter \beta?
There also exists other ways to limit the capacity of the model. The simplest way is to reduce the latent variable dimension. I am wondering how the proposed beta-VAE is a better model than the VAE with reduced, or optimal latent variable dimension.
We thank the reviewer for his work, but given their review we are afraid they may have misunderstood the point of our paper and did not provide a fair assessment of our contribution.
We hope our responses below and the comments of the other reviewers may help clarify the scope of our work and its significance.
1. We do not present any quantitative result on log-likelihood estimation, because it is not the main point of our paper (as also pointed out by AnonReviewer3). This also relates to the reviewer’s comment about the quality of beta-VAE samples and how they compare to those by InfoGAN. The quality of reconstructions of beta-VAE and InfoGAN are in line with the base methods the two approaches are built upon - the VAE and GAN.
Hence our reconstructions are blurrier than those of InfoGAN due to the limitations of VAEs. Saying this, InfoGAN suffers from the shortcomings of GANS, such as training instability and mode jumping. We stress again: we do not strive in this paper for good quality reconstructions;
the main point of our paper is the quality of the learned representations i.e. the disentangling capability of our approach, and in this regard we clearly demonstrate that our method outperforms InfoGAN, which is the current state of the art fully unsupervised method for disentangled factor learning.
In addition, we address the other main shortcoming of the InfoGAN paper (and indeed, most other unsupervised representation-learning papers): a lack of justification for the conclusion apart from weak empirical evidence, which we address by our quantitative disentanglement metric.
2. The benefits of the representations learnt by beta-VAE over those learnt by a standard VAE come from their disentangled property. The benefits of disentangled representations have been argued for by many, including Bengio (2013) and most recently in a review article on representation learning that has just come out on arxiv (https://arxiv.org/abs/1612.05299). These benefits include novel example generation through interpretable factor recombination (problems that arise from recombining entangled factors can be seen in the reduced quality of samples generated by an entangled VAE vs a disentangled beta-VAE as shown in Figure 9, which we have now added to the appendix), compression (according to the minimum description length principle), and performance in zero-shot inference, transfer and domain adaptation scenarios. We have amended our introduction to present these benefits more clearly.
Our current manuscript describes a new state of the art framework for disentangled factor learning. We leave the demonstration of benefits that arise from learning such disentangled representations to future work. We have, however, added a new section in Appendix A.5, as well as Table 2 to describe the performance of a simple classifier trying to predict the data generative factor values using different latent representations. We find that such a classifier performs significantly better using the disentangled representation of beta-VAE compared to the entangled representations of VAE.
3. We did not cross-validate the parameter \beta, as this doesn’t really make sense in an unsupervised learning setting.
Indeed, as is usually the case in the literature (see InfoGAN paper for example), we train on the full dataset and assess characteristics of the representation obtained.
As we mention in the text, in this current work we chose \beta using visual assessments of the latent traversals.
Alternatively, we also mention that our disentanglement metric score could be used for this purpose. This is validated by the fact that \beta values found to be optimal through the visual assessment heuristic on the dataset of 2D shapes were also found to be within the optimal range of \beta’s as assessed with the disentanglement metric. We have now also included Appendix A.5 as further validation; in this Appendix, we use the \beta-VAE representation to directly predict the values of the ground truth factors on the 2D shapes dataset with a linear classifier.
Although this does not measure disentangling directly (and hence is not used as our preferred disentangling metric), it further backs up the efficacy of \beta-VAE relative to other models in the literature.
Additionally, we always train multiple networks with random initialisations of weights for every value of \beta we tried (as can be seen in Figure 6). We did not see any strong variations in their behaviour, and hence we believe that \beta is not a particularly sensitive parameter.
4. We have trained VAEs with different latent sizes (z_size \in [2, 10]) and found that reducing z_size was linearly *reducing* the disentangling metric score.
In particular, a VAE with the “optimal” z_size (e.g. z_size = 5 for our 2D shape dataset, which has 5 ground truth factors of variation) was not able to learn latents corresponding to the ground truth generative factors, in contrast with the beta-VAE (as shown in Fig 7 and by the disentangling metric score).
Hence beta-VAE is indeed different from a VAE with a reduced/optimal latent channel capacity.
We hope this addresses the reviewer’s concerns and welcome any further feedback.

Meta-review:
role: PC
data: comment: This paper proposes a modification of the variational ELBO in encourage 'disentangled' representations, and proposes a measure of disentanglement. The main idea and is presented clearly enough and explored through experiments. This whole area still seems a little bit conceptually confused, but by proposing concrete metrics and methods, this paper makes several original contributions.

Decision: Accept (Poster)