Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper studies the problem of learning disentangled representations while maintaining good data reconstruction. As common modeling, the latent representation is decomposed into disentangled representation C and correlated representation Z. Then a hierachical generative process is proposed, where the first stage is to reconstruct a preliminary version of the data given the disentangled representation C, and the second step is to reconstruct a full version of the data given C and correlated representation Z. The two stages are learned separately, with the first stage using the previous β-TCVAE model to learn C, and the second stage using the Feature-wise Linear Modulation (FiLM) technique.

Experiments show the approach can learn disentangled representation C as well as the previous β-TCVAE model, but improves in terms of the reconstruction quality. It is also shown that the second stage can properly maintain the conditioning on C (instead of conditioning only on Z). 

As above, overall, the proposed modeling is simple and reasonably sound, and the experimental results / analyses are interesting and promising. 

A potential shortcoming is that the two stage components are learned separately instead of jointly, which could results in suboptimial learning. 

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This article introduces a method for learning high-quality generative model with disentangled latent representation by splitting the learning process into two steps. The first step consists in learning a generative model on the data using a method with strong disentanglement constraints, producing a low-quality generation. As a second step, a conditional generative model is trained to turn this low-quality sample into an high quality one. This intermediate generation acts as an observed variable, and thus separates the latent spaces of the two models, effectively preventing disruptive interference in the learning of the "independent factors" on the one hand and the "dependent factors" on the other, as has been previously observed as a difficulty in the literature. The authors provide detailed empirical analysis of the performance of the model.

Overall I would say that this is an interesting article, and a decent contribution to the conference. The design of the model is explained in detail, and its position among the literature is explained in detail. The method in itself is pretty general, and the authors provide explanation on how to apply it in non-VAE contexts (Flows & GANs mostly) in appendix.

However, I have a few questions, and I think these points should be clarified in the text of the article:

1. The use of FiLM structure of the conditional VAE is motivated by the risk that the model would ignore Y and solely rely on Z to generate X. However, this seems rather counter-intuitive from a theoretical point of view: the training objective of the VAE rewards storing as little information as possible in the latent space. If the model can reliably use information present in Y, we should thus expect it to do so. The previously observed issue of models ignoring their disentangled latent and storing everything in the correlated one does not applies as, this is the point of your model, Y is here treated as if it was part of the dataset. So, is this ignoring of Y something you observed experimentally before deciding to use FiLM structures?

2. As a follow-up of the previous point, it would seem natural to expect that an abstract value for Y would potentially be more easily used by the conditional model than a low-quality reconstruction. Going further with that, one could consider using directly C as input to the conditional model: training β-TCVAE as you describe, and only keeping its encoder to produce a dataset of (X, C), then using this dataset to train the second stage as a conditional generative model p(X | C, Z) (with an encoder q(Z | X, C)). As per the previous argument, this second stage should be using C as much as possible. How do you think that would compare to MS-VAE?

3. Regarding the simple pendulum example cited in the text, it is unclear how Y0 and Y1 are produced. Given how precisely they are defined in the appendix, I'm inferring that they are ground truth data produced by the simulator. If that is the case, it seems to me that the structure presented here is no different from a regular latent variable model with multiple observed variables, as the main defining component of your approach (learning Y with a disentanglement-inducing method) is not present at all. If on the other hand Y0 and Y1 are indeed directly learned from θ, then it should be clarified.

-----------------

Minor remarks and typos:

- the legend for figure (2d) is mislabeled (c)
- the legend for figure (3e) seems to invoke the wrong dataset (the images look like from Cars3D, not SmallNORB)
- bottom of page 7, the text invokes figure (3h), which does not exist
review_title: An interesting article and decent contribution, but a few points need clarifying

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a multi-stage approach to learning disentangled representation that refines the low quality reconstructions of a state of the art architecture using a second deep generative model that learns how to modify these reconstructions with additional correlated latent variables.

The paper is well written and easy to follow. The introduced idea is quite straightforward, I would not be surprised at all if it was introduced before, but I do not recall seeing it in any other work so I will regard it as novel (unless the other reviewers point out similar works). Despite its simplicity, this method makes a noticeable difference in the experiments, so it could be very useful trick for other researchers working in the area of learning disentangled representations.

My main concern for this paper is the motivation of the proposed approach. The need for a multi-stage approach is justified using d-separation, saying that in this way independent and correlated latent variables (c and z) are completely separated during training. This is true, but is the multi-stage approach the best/only way to obtain this effect?
It would be useful to understand whether a similar effect could be obtained instead in an end to end fashion with the right architecture or training procedure, since I believe that such a procedure would have a much bigger impact for other researchers.
An interesting ablation study that would allow to better understand the mechanics of this technique would be for example using the same split of c and z and the same FiLM + AdaIN decoder, but training end to end placing the beta factor as in the beta-VAE only on the c latent variables, with for example a schedule for beta in which independent latent variables are more important at the beginning of training, or some alternating updates.
review_title: Good paper, but lacks some details on the motivation of the proposed approach

final_score: rating: 6: Marginally above acceptance threshold
review_content: =======================================================================================

Summary : 

Disentangled representation (DR) of data is useful in downstream tasks. However, VAE-based DR fundamentally suffers from a trade-off between high-quality reconstruction images and disentangling. To overcome this point, the paper approaches VAE from the multi-stage modeling (so-called MS-VAE). The proposed method starts from the other standard DR method which learns low-quality image(Y) and then, improves the quality of the image via training additional encoded representation(Z). The widely-used techniques in style transfer (FILM and AdaIN) are used to adopt 'Z' into 'Y' for a high-quality reconstructed image. The authors evaluated the proposed method through FID(high-quality image) and MIG(disentanglement). At the similar scale of complexity, the proposed method obtained high FID score and low MIG score than the baselines.

=======================================================================================

Reasons for score: 

Overall, I vote for rejection. I'm not fully persuaded by the results from the novelty of this paper.
I left my concerns (see cons). Hopefully, the authors cover my concerns in the rebuttal period. 

=======================================================================================

Strong points:

(1) Paper is well written.

(2) Authors provided the justification of the proposed method via a graphical model.

=======================================================================================

Cons :

The insight in the MS-VAE is quite overlapped in CascadeVAE [1]. CascadeVAE learns the continuous representation sequentially and shows that this sequential learning of representation helps disentanglement representation without special technique provided in FactorVAE, TC-\betaVAE, and etc. I suggest the authors provide comparison experiments with this kind of sequential learning in representation. 

Also, an ablation study on the proposed method without AdaIN and FILM would increase the persuasion of readers and provide justification for the proposed method. At a high level, the overall method is to learn the representation of an original image which helps to transform a low-quality reconstructed image, which is trained in VAE, to a high-quality reconstructed image. Also, the experiment results in figure 4 show that disentanglement was not improved and only the quality of the image was improved. 

Finally, if the ultimate motive of this paper is to learn the representation which can reconstruct the original image fairly well, the comparison with the VAE-based Method([2], [3])  which generated a high-quality image should be added.
For the disentangled representation which is used for future downstream tasks, there is no big difference with \beta-TC VAE in the best hyper parameter setting as shown in Figure 6. 

[1] Learning discrete and continuous factors of data via alternating disentanglement, ICML2019.
[2] IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis, NIPS2018. 
[3] NVAE: A Deep Hierarchical Variational Autoencoder, NIPS 2020.

=========================================================================================================

After rebuttal (and a final question):


Thank you for your responses and the experiments I requested. I hope the authors would add discussions with M5 and quantitative results in the final version. Before I finalize my ratings, I have a final question after seeing supplementary material L. I believe M5 is enough to achieve the goal of this paper since $[Z, C]$ captures disentangled factors in $Z$ and $[Z, C]$ improves the reconstruction quality. Then, what is the advantage of MSVAE over M5?


=========================================================================================================

After the final question : 

Thank you for your reply. I misunderstood the experiment M2 and propose M5. I believe M5 is just a variant of M2. 
The authors should add quantitative results (FID and disentanglement score) comparing the baselines (in supplementary material L) in their final version. I would raise my score to 6. 

Minor : M2-M4 denote mutual information (paper) and baselines (supp). Please use different notations.

review_title: Official Review 4 : I'm not fully persuaded by the proposed method

Meta-review: This paper presents a two-step approach to achieve disentangled representation and good reconstruction at the same time in deep generative models: the first step focuses on good disentanglement (e.g., with beta-TCVAE) while possibly sacrificing reconstruction reconstruction, while the second step focuses on high-quality reconstruction, conditioned on the low-quality reconstruction from disentangled representation. In this paper, each step uses an existing method: beta-TCVAE is used for the first step and AdaIN is used for the second, so the paper presents an intuitive combination of two existing methods to achieve both goals. Some useful ablation studies are provided to empirically justify the specific method choices. The concern is whether the two step approach is necessary to achieve both; the authors' argument is that models learning only one set of latent variables may not have the capacity to achieve both goals, and methods jointly learning two set of variables ("disentangled" and "correlated" variables) can not guarantee they represent disjoint structures of data. Some of these statements seem somewhat handwavy (including the d-separation argument, I am not very sure if it applies when the variables are learned separately), and shall be made rigorous and justified (theoretically and/or empirically).

The reviewers rate this paper to be borderline.
Decision: Reject