Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ### Summary:
This submission proposes an ensemble framework to improve learning disentangled representations with Variational Autoencoders (VAEs). The approach builds on the assumption that entangled latent representations learned by VAEs show some “uniqueness” in their latent space structure, while disentangled representations exhibit some “similarity”; an assumption corroborated by recent studies. On that basis, a VAE ensemble approach is proposed where several VAEs are connected through linear mappings between the individual latent spaces to encourage alignment of latent representations and thus disentanglement. A formal derivation of the framework is provided and the formal validity of the underlying assumption demonstrated. Furthermore, empirical evaluation of the proposed approach in comparison to the standard VAE, beta-VAE and FactorVAE on the datasets dSprites (main results, main text) and CelebA (appendix) is performed, yielding improved results on the FactorVAE disentanglement metric (all baseline methods considered) as well as the Distance to Orthogonality (DtO) metric (only standard VAE considered).

### Strengths:
- Significance / Novelty: The proposed approach builds on recent work by Rolinek et al. and Duan et al., which show PCA-like behaviour in VAEs and leverage these results to develop disentanglement scores for model selection. This submission uses these insights for training an ensemble of VAEs in order to improve learning of disentangled representations. The claim is validated both formally as well as empirically on a benchmark dataset (dSprites) and state-of-the-art methods like FactorVAE, where the proposed framework performs favourably. To my knowledge the proposed idea is novel and simple yet potentially quite powerful. This approach could be relevant for other disentanglement methods and a wider audience employing VAE approaches.
- Technical Quality: An important contribution of this paper is the thorough formal derivation and theoretical justification of the approach which to me appears sound. The experimental evaluation is well-designed and mostly succeeds in justifying the claims, with some exceptions outlined below. I believe that all the relevant details to reproduce the results are provided.
- In particular, the results that the DtO comes close to 0 (fig. 2) for the ensemble approach illustrate that the latent representations of the different VAE in the ensemble converge (question 1), i.e. the linear transformations between latent space converge to (signed) permutations. This means that it should not matter which latent representations in the ensemble is studied (in the paper the first model in the ensemble is chosen; lines 274-275). However, I am curious whether the authors considered the results (“polarisation” and FactorVAE scores) for other latent representations (i.e. not the first model) and how much the results agreed?
- Clarity: I consider this paper well-written and well-structured. Relevant details and formal justifications are provided in an appropriate manner resulting in a self-contained paper.

### Weaknesses:
- The ensemble approach comes at a cost which is probably the reason why only up to 5 parallel models were used. Can the authors comment on the running time and memory requirements compared to the competing methods? I think the quality of the paper could be improved if these details and the restrictions of the ensemble approach were provided.
- The results in table 1 (comparison of baseline methods and ensemble approach w.r.t. FactorVAE metric) show that an ensemble of size >=3 can outperform state-of-the-art methods like FactorVAE on the considered FactorVAE metric. However, they also show that it might not always be beneficial to put more weight onto enforcing aligned latent representations for the same ensemble size (gamma > 1). This is a bit at odds with the premise of the paper. As the discussion points out (question 3, lines 285-289), this could be due to balancing different contributions in the more extensive objective function. However, this could also hint at potential optimisation problems for more challenging tasks. 
- The examples for the latent traversal (in the appendix) are slightly less convincing and a comparison is only done w.r.t. a standard VAE. However, it would be much more insightful to compare the ensemble approach to beta-VAE and FactorVAE latent traversal results.
- Similar to the last point, in figure 2, it would be quite insightful to see the DtO results for the beta-VAE and especially the FactorVAE. In my opinion, this is a crucial aspect which so far is missing and could justify the approach even more. Isn’t the whole motivation that beta-VAE and FactorVAE should perform slightly worse w.r.t DtO?

### Additional Feedback:
- Figure 1: I like the illustration, however I do not understand the bar plot (“VAE, BetaVAE, FactorVAE, VAE Ensemble”). Maybe an additional annotation could help?
- Line 8: *“sometime”* -> *”sometimes”*
- Line 24: *”state-of-the-arts”* -> *”state-of-the-art”*
- Line 25: *”[…] deploy Variational Autoencoder […]”* -> *”[…] deploy the Variational Autoencoder […]”* or *”[…] deploy Variational Autoencoders […]”*
- Line 37, line 190, line 221 : *”On contrary, […]”* -> *”On the contrary, […]”*
- Line 74: *”[…] closely approximate prior […]”* -> *”[…] closely approximate the prior […]”*
- Line 127: *”[…] models […]”* -> *”[…] model […]”*
- Line 164: *”[…] decomposition L2 term […]”* -> *”[…] decomposition, the L2 term […]”*
- Line 224: *”Such gap […]”* -> *”Such a gap […]”*
- Line 225: *”[…] such case […]”* -> *”[…] such a case […]”*
- Line 233: *”Does VAE ensemble improves […]”* -> *”Does the VAE ensemble improve […]”*

### Recommendation:
This submission was an enjoyable read, it provides some new insights and I believe this paper can pose an important contribution in areas which are concerned with learning disentangled representations and VAE models. In my opinion, the claims of the paper are justified both theoretically and empirically. However, there are certain aspects and concerns outlined above which need to be addressed adequately to warrant a publication. At the moment, I am inclined to accept the paper, but I would like the authors to clarify the concerns and questions above.

### Post-Rebuttal:
I would like to thank the authors for the insightful rebuttal! The authors were able to address my concerns adequately and I believe that the revision improved the quality of the paper quite a bit. Therefore, I stand with my initial recommendation and due to the reasons stated above, I endorse accepting this paper. 


### References: 
- Rolinek et al., “Variational autoencoders pursue pca directions (by accident)”, CVPR 2019.
- Duan et al., “Unsupervised model selection for variational disentangled representation learning”, ICLR 2019.
review_title: Interesting VAE ensemble approach for improving disentangled representations with theoretical as well as experimental validation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a simple and effective technique to improve disentanglement by coupling the latent spaces of different VAE models. It builds on Duan et al. (2019)’s proposed method to rank the representations of different models. By learning a VAE ensemble with linear transformations between the latent spaces and an additional “cross-model” reconstruction loss, the authors show that they can achieve significantly better disentangling.

Strengths:
- The theoretical justification seems reasonable and builds on previous work.
- The experiments are organized to answer three meaningful questions. The results do suggest the VAE ensemble learns better latent representations which can be converted between models with simple, orthogonal linear transformations.

Questions:
- Regarding the last term of the loss in equation (2): for a fixed i and j, the loss is E_{q(z_ij|x)} ||z_jj - z_ij|| = E_{q(z_ij|x)}||z_jj - M_ji z_ii||. This loss term can be optimized by tuning the parameters of VAE i, VAE j, and M_ji. Do you backprop through all these? Or is there a stopgradient on z_ii when used in computing this loss term (i.e. no gradients through VAE i from this loss term)?
- What would be the effect of training the VAE models in two stages: independently first and then jointly in the ensemble? Would it help or hurt disentangling?
- How would you express the total information cost of representing an image across the VAEs in the ensemble (say if you wanted to to compare the information rate to a single VAE)? It doesn't make sense to add up the KL costs linearly.

Suggestions:
- It would help enormously to strengthen the findings and assertions regarding the effect of ensemble size and the gamma hyperparameter.
- Consider adding another disentanglement metric e.g. MIG.
- Figure 5 in the Appendix shows a larger effect on DtO of the number of dims than the gamma hyperparameter. This result (and other results on CelebA) are perhaps worth describing in the main paper. 

Minor:
- In Figure 2(a) I assume the curves are overlapping? Does it help to use a log scale for the y-axis?
- How are the latent dimensions sorted in Figure 3?
- Are the scores in Table 2 across different training runs?
review_title: Simple ensembling technique to improve disentanglement

final_score: rating: 3: Clear rejection
review_content: # Summary

The authors introduce a novel VAE-based approach for unsupervised learning of disentangled representations of image data.  The approach trains an ensemble of VAEs along with pair-wise linear transformations between their latent spaces.  The objective includes the ELBO objectives for each VAE as well as two additional pressures:  (i) An L2 similarity objective that pressures samples from each VAE latent space to match under linear transformations samples from the other VAE latent spaces, and (ii) A cross-model decoding objective that encourages decoding accuracy of the linearly transformed latent samples.  The authors provide a theoretical argument that the linear transformations should learn to be orthogonal, and show some experimental results indicating that their model performs well compared to baselines when evaluated with an established disentangling metric.

# Pros

* The theoretical analysis in section 4.1 is clear and provides good mathematical intuition for the authors’ results.
* The introduction and related work sections are clear and include a thorough set of references.


# Cons:

* The authors’ baseline results give unexpectedly low metric scores.  The authors report FactorVAE metric values of 0.665 for beta-VAE and 0.764 for FactorVAE on the dSprites dataset.  However, the values reported in the FactorVAE paper (and corroborated by others) on the same dataset are significantly higher.  This makes me suspicious that something went wrong with the authors’ training --- perhaps they didn’t train those baseline models to completion or something else went wrong.  Having baseline results that are inconsistent with the existing literature makes me uneasy.
* The traversals in Figure 8 from the authors’ model are much less disentangled than other models in the literature.  For example, they are much less disentangled than the traversals shown in the beta-VAE paper and the FactorVAE paper on the same dataset.  Thus from these traversals, it seems that the authors’ model is performing worse than existing models in the literature (the authors’ metrics indicate the opposite, but as mentioned above I’m uncertain about the validity of those metric results).  Figure 3-A also suggests that the authors’ model is using too many informative latents, i.e. not disentangling well.
* I am not convinced by the authors’ intuitive justification in lines 216-225 (and appendix C) that the cross-model objective encourages entangled models to align to disentangled models.  Specifically, in that argument the authors seem to assume that orthogonal linear transformations are orthonormal.  However, there is nothing to enforce normality of the transformations in the model, hence the cross-model encoding variance from an entangled to a disentangled model could be quite small.
* The purpose of the cross-model reconstructions is not clear, particularly given that I’m not convinced by the authors’ intuitive justification of them.  The L2 regularization between the transformed encodings should pressure the cross-model reconstructions to be good, so I do not see the reason to include them in the model objective.  It would be good if the authors could do an ablation study without the cross-model reconstructions.
* The authors do not mention the computational complexity of their model, yet computational complexity seems to be a significant drawback of it.  Ensemble training is very computationally expensive, so the authors should include some discussion about it as well as runtimes and memory requirements for their model.  Furthermore, with the cross-model reconstructions the computational complexity of the authors’ model scales with the square of the number of ensemble elements, which is quite a steep scaling.
* The authors only compare to a couple (relatively old) baselines, betaVAE and FactorVAE, which are no longer state-of-the-art.  However, more recently a number of other VAE models have been published that perform better.  In order to support their claims about state-of-the-art performance, the authors should compare to newer baselines.  Here are a few examples:
DIP-VAE  (Variational inference of disentangled latent concepts from unlabeled observations.  Kumar et al., 2017)
TCVAE (Isolating sources of disentanglement in variational autoencoders. Chen et al., 2018)
Spatial Broadcast VAE  (Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs.  Watters et al., 2019)
* The authors also don’t include many metrics or datasets.  dSprites and CelebA were used in the original betaVAE paper, but more recently it has become the norm to test on a larger set of datasets and with a number of different metrics to convincingly show disentangling.  By the way, a number of models, datasets, and metrics have been open-sourced in DistLib (https://github.com/google-research/disentanglement_lib), which may be useful for comparing to more models with more metrics on more datasets.

# Summary

I do not recommend accepting this paper.  Baseline results are inconsistent with prior work, the model seems to disentangle less well than existing methods, and the authors don’t do ablation experiments to justify the high computational complexity of the model.

review_title: Recommend to Reject

Meta-review: This paper proposes to use an ensemble of VAEs to learn better disentangled representations by aligning their representations through additional losses. This training method is based on recent work by Rolinek et al (2019) and Duan et al (2020), which suggests that VAEs tend to approximate PCA-like behaviour when they are trained to disentangle. The method is well justified from the theoretical perspective, and the quantitative results are good. Saying this, the reviewers raised concerns about the qualitative nature of the learnt representations, which do not look as disentangled as the quantitative measures might suggest. There was a large range of scores given to this paper by the reviewers, which has generated a long discussion. I have also personally looked at the paper. Unfortunately I have to agree that the latent traversal plots do not look as disentangled as the metric scores would suggest, and as one might hope to see on such toy datasets as dSprites. The traversals are certainly subpar to even the most basic approaches to disentanglement, like beta-VAE. For this reason, and given the reviewer scores, I unfortunately have to recommend to reject the paper this time around, however I hope that the authors are able to address the reviewers' concerns and find the source of disagreement between their qualitative and quantitative results for the future revisions of this work.
Decision: Reject