Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an unsupervised learning model for disentangled latent representation learning. Specifically, the model works on a combined latent space including both entangled variable and separable variable. The most impressive part is the one-at-a-time (OAT) factor learning approach that iteratively uncovers disentangled dimension and learned from reconstructed samples. The OAT approach generally sounds. 
main_review: The paper is well motivated and the OAT technique is kind of novel and impressive. However, paper clarity can be improved, which has impeded reader's understanding and rating of the paper.

Contribution wise, this paper is not the first ones to use an "entangled+disentangled" latent space (eg paper named "Toward Controlled Generation of Text" in 2017), but the OAT learning approach is interesting. 

The presentation can be improved, given so many typos, improper subscripts, equation-naming (Eq 6 or Eq 3.4.2), etc. Moreover, the central equation (Eq.6) is hard to follow, say how q(x) and its KL divergence is evaluated is unknown.

In Eq 5, whether or not the p(z_1) in last KL term is also factorizable. 

Since the intervention is key idea, it's a miss to see it not shown in Figure 1 plot.

The paper needs more explanation on how to avoid posterior collapse and avoid the collapse of disentangled latent variables.    

The table column in Table 1 is hard to follow. From Table 1, it seems OAT model has not outperformed other baselines, right?
summary_of_the_review: The paper is well motivated and the idea of OAT technique generally sounds. However, the paper reads like a manuscript in a rush: we see many typos, unclear subscripts. Moreover, we see unclear equations, such as Eq. 6, which is hard to follow, but the most important equation for the paper. Weird Table 1 columns and not strong experimental metrics. Overall, the paper needs a polish to improve its clarity and readiness to publish.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This study proposes a disentangled representation learning method called "one at a time", or OAT factor learning which is a VAE-GAN network to generate high resolution samples and to learn variational factors in an unsupervised manner, without knowing the number of ground-truth factors a priori. The authors formulate the latent space as a union of two disjoint sets, $z_1$ and $z_2$, where $z_1$ corresponds to the variational (disentangled) factors, and $z_2$ denotes either noise or sample-specific factors. To train the proposed network, the authors use a two-step training approach in which they first train the VAE network with only $z_2$ latent factors. Then, they start to learn $z_{1_k}$, one dimension at the time. To enforce $z_1$ to encode the disentangled factors, they use an intervention-based approach, in which the value of one dimension in $z_1$ is changed, while the remaining $z_1$ factors are kept unchanged. The authors reported their experimental results for two datasets, i.e., dSprites and CelebA.
main_review: Originality: 
I think this work lacks notable originality and novelty. The VAE-GAN networks (proposed by Larsen et al., 2016) have been studied in many papers. The proposed formulation of latent space that splits latent factors to two disjoint sets is also not novel. The only novel part is assuming an upper bound for variational factor, where it is not clear how the model can find the true number of factors, while $|z_1|$ is gradually increasing up to $K$ over iterations of training.

Quality: 
The motivation of the work is good; learning a disentanglement representation is quite a challenging problem. However, the paper could not appropriately address this challenge and the authors’ contribution is quite limited. The paper does not offer any theoretical analysis or profound experimental studies. The reported results for dSprites and CelebA do not showcase the superiority of the proposed method, neither in terms of existing disentanglement scores nor quality of the reconstructed images. 


Limitations: 
- It is not clear how the suggested formulation for the latent space is enhancing the disentitlement.  I cannot see why we need to encode $z_2$ while it is expected to encode noise or sample-specific variations, which is not desirable in the latent representation learning. 

- While the authors claim that the model does not need to know $|z_1|$, it is not clear how the model learns the true $|z_1|$. If I am not wrong the model iteratively unmasks the dimensions of $z_1$, without any stopping criterion. 

- The experimental study is limited to two datasets, without sufficient ablation studies on each dimension of $z_1$ and $z_2$. It is critical to explore the impact of $z_2$ vs. $z_1$ and quantify the correlation between these factors. 

- Additionally, I think the recent study by Träuble et al. ICML2021, and its discussion on the correlated latent space is quite relevant to this work. The authors might need to refine the discussion on the casual factors and disentailment studies, based on the findings of this paper.

Minors comment:
- Page 7: “The value of $\beta$ is increased linearly during training to encourage the model to encode information in $z_1$ instead of $z_2$.” $\beta$ or $\gamma$? I got confused. 
- There are some typos in the text, e.g., “/mse”, “a a” at page 3.
- The table and Figures are not cited in the manuscript. 

summary_of_the_review: Learning a disentangled and interpretable representations is an important topic in the machine learning. The proposed method tried to address this challenge. However, I think the contribution is not notable and the paper lacks theoretical and empirical justifications to support its claim.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new method to learn a disentangled representation. The proposed method is Variational Autoencoder (VAE) based. The latent variables consist of nuisance factors and disentangled factors that form the $k$ generative factors per dataset. To help learn the disentangled factors of variation the authors propose to use an additional discriminator and interventions on the disentangled factors.
The authors demonstrate the performance of the proposed model on the dSprites and CelebA dataset.
main_review: ### Strengths
- Interesting approach to combine the strengths of VAE for disentangled representation with the power of a discriminator.

### Weaknesses
- Not really convincing experimental results in Table 1
- Lack of clarity and details regarding the training procedure (see questions)
- Lack of ablations with respect to the number of learned disentangled factors $k$, which is a central part of the proposed pipeline

### Questions
- The authors of the ID-GAN paper (Lee et al: High-fidelity synthesis with disentangled representation, 2020) write about the trade-off between generative quality and disentangled representation. Did the authors also experience that? Or was the generative quality not part of their evaluation?
- What about model selection? The authors do not write much about how they selected their models. The training procedure involves multiple steps and to me it is not straightforward how to select the model in an unsupervised manner for every training step out of a range of multiple possible hyper-parameters. It would be interesting to also see an ablation study regarding the most important hyper-parameters.
- Table 1 is a bit confusing. What is the scheme of making some entries bold? It does not seem the highest value per column. In general, I would have appreciated a more detailed description of the results presented in Table 1. Also, I cannot find a reference to table 1 in the text.
- There is no ablation on the number of learned factors of variations $k$. To me, it is not obvious that the right number of disentangled factors can be found. And for table 1 it is not even disclosed what the respective k is, nor compared to the latent space size of previous works.
- What are the motivation for another researcher to use your method? It does not seem to be able to outperform previous methods, nor does it seem to have a simple training procedure.
- In section 3.4.2., step 1 the authors write that the $\gamma_i$ are set to zero and, hence the corresponding latent factors are not trained. Although it is true, the regularisation for these factors is not part of the objective, in my opinion,  information flows through this part of the network nevertheless and the weights will be updated through their contribution to the reconstruction loss. I would appreciate the authors' thoughts on that.
- Also in section 3.4.2, step 1, the authors describe their one-at-a-time training approach by switch the $\gamma_i$ from 0 to 1. I cannot find any information on how this is scheduled during training nor how it is stopped. I understood that the number of disentangled factors $k$ is not a hyper-parameter (only upper-bounded) as it is learned during training. I would appreciate a bit more details on that.
- Is it a goal of the proposed method that no information correlated to the disentangled factors $z_1$ is encoded in $z_2$ (low mutual information between the two representation)? If so, how is this achieved? Did the authors explore that direction?

summary_of_the_review: Although this work presents an interesting approach to learning disentangled representations, in my opinion it is not ready to be published at ICLR at this stage. It lacks convincing arguments why a researcher in this area should choose the proposed method. In my opinion what is needed to improve the paper is more clarity and details regarding the training procedure, e.g. model selection criteria for the different training stages, ablation studies on the learned number of disentangled factors $k$ and a better description of their central and only results table.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach for training disentangled generative models. On top of VAE, it separates the latents into two disjoint groups: disentangled and entangled ones, and progressively increases the size of the disentangled group one at a time. To encourage disentanglement, it changes one random dimension of disentangled latent and pushes it to decode and then encode into the same one, and has a GAN loss to make sure that the changed generated distribution matches the ground-truth distribution. The paper shows the performance of the proposed approach on dSprites and CelebA datasets.
main_review: The paper addresses an important problem. However, it has several issues, including inaccurate claims, insufficient experiments, and many typos. See below for more details.

*** Inaccurate claims ***

* Page 2: 'Current state of the art methods make the assumption that there are a ﬁxed number of independent factors for all the data points in the dataset. However in real datasets, in addition to the independent factors common to all points in the dataset, there might also be some correlated, nuisance or noisy factors pertinent to speciﬁcally only certain data points'. Note that many GAN-based approaches (e.g., InfoGAN) only regularize a small part of latents; they can indeed model 'correlated, nuisance or noisy factors' (using the non-regularized latents) besides the 'independent factors' (using the regularized latents). Please fix this statement. (I see that you do admit it later at the end of Section 3.2.)

* The paper mentions in many places that the idea of progressively learning one factor at a time is new. However, a similar idea has already been proposed in 'Robust Disentanglement of a Few Factors at a Time' at NeurIPS 2020.

*** Insufficient experiments and results ***

* From table 1 it seems like the proposed approach performs much worse than the state-of-the-arts like FactorVAE and InfoGAN-CR. 

* If the main selling point of the paper is the ability to disentangle real-world datasets with **unknown number of factors**, then the paper should conduct experiments to demonstrate that (i.e., on the settings when the approaches that set a pre-defined number of factors fail, and show how your approach can improve that)

*** Typos ***

* Page 2: 'related Work'

* Page 3: '/mse'

* Page 5: missing a period after 'factors are integrated out'

* Figure 1 caption: 'so we ﬁrst group the correlated latents into one space, z_1'. Here z_1 should be z_2?

* Page 7: 'z_2.For brevity' -> 'z_2. For brevity'

* Page 7: '\gamma_ii = 1:K' -> '\gamma_i i = 1:K'

* Page 7: '\gamma_1=1 and \gamma_i1=2: k=0'

* Page 7: sometimes you use k to denote the number of learned disentangled latents (e.g., the line before Eq. 5), and sometimes you use K instead (e.g., Eq. 5 and 'We then uniformly select a dimension k ∈ [K]').

* Page 8: 'Kim & Mnih (2019)use' -> 'Kim & Mnih (2019) use'

*** Other suggestions ***

* Page 2: You could probably add refs to the corresponding sections for 'related work' and 'preliminaries'

* Eq. 6: I understand from the text that you are matching the intervened latents, but this equation does not express it precisely.

* Figure 2, 3: You could label the factors besides the images to make them easier to read.

summary_of_the_review: In summary, (1) the main selling idea of 'disentangling one factor at a time' is not new, (2) the results are worse than the state-of-the-arts, and are not sufficient to show the benefit of 'disentangling one factor at a time'. Therefore, I have to give a negative score.
review_title: null

Meta-review: This paper presents a method for unsupervised learning of disentangled representations by first training a VAE with a tangled set of latents, and then sequentially learning disentangled latent variables one at a time from the entangled initial VAE latent space. On several toy disentanglement benchmarks, the method is shown to perform competitively with previous VAE and GAN approaches. 

There were several concerns from reviewers around the clarity and description of the proposed one-factor-at a time (OAT) training procedure. While the updated draft addressed several typos and some clarity issues, multiple reviewers continued to find the method description problematic. There were additional concerns around the viability of the method on real-world datasets where the number of factors are not known, and as the authors stated the proposed method can also result in one factor of variation encoded into mulitple latent variables, which hurts on many of the disentanglement metrics.  The addition of CelebA downstream task evaluation begins to address this concern of real-world data, but more rigorous experiments (including more description of how models were selected) and discussion of the limtiations of the proposed method are needed. There is also no theoretical motivation as to why the proposed intervention-based factor learning algorithm should recover the ground truth factors.

Given the concerns over experimental results, clarity, and lack of theoretical motivation, I suggest rejecting this paper in the current form.
Decision: Reject