Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method called DeVAE for learning disentangled representations based on beta-VAE. Specifically, the proposed method builds a hierarchical latent space with disentanglement-invariant transformation between them and decreases information bottleneck layer-by-layer to balance disentanglement and reconstruction fidelity. Experimental results show that DeVAE performs comparably as previous works. DeVAE seems to perform better than other methods when the latent dimension is high.
strength_and_weaknesses: Strengths
- The paper focuses on the balance between disentanglement and reconstruction fidelity, an important problem in the field of disentangled representation learning.
- The proposed method is technically sound.
- The proposed method performs better than beta-VAE and beta-TCVAE on dSprites and shapes3D datasets.
- Ablation studies show the importance of each proposed component and selection of hyperparameters.


Weaknesses:

- The paper has low readability. A lot of the issues are certainly fixable but in its current form, it is confusing enough to distract from evaluating the technical contributions of the paper. Certain examples are:
    - “DeVAE surpasses 2% for β-TCVAE and 9% for β-VAE.” In what terms exactly?
    - Many instances in the introduction talking about spreading the conflict of disentanglement and reconstruction over time and space is not easy to follow and understand.
    - “However, in this work, we get rid of calculating TC by leveraging the narrow information bottleneck (Tishby et al., 1999; Burgess et al., 2018) to find efficient codes for representing the data, which promotes disentanglement.” This is pretty confusing right where it is in the introduction and only becomes somewhat clear after reading the method section.
- The authors claim that DynamicVAE suffers from Information Diffusion problems. If that is the case, wouldn’t that result in low disentanglement scores or at least high variance across different seeds for DynamicVAE? But, that’s not the case in Figure 2.
- The experiment for high-dimensional latent space is weak. 1024 dimensional latent space for dSprites seems unrealistic, it ideally should be for a dataset that requires high-dimensional latent space. And, there are no quantitative numbers. Why is DeVAE worse for low dimensions? A more high-level question would here is — what exactly in DeVAE makes it more compatible for handling high-dimensional latent space?
- A lot of design decisions are unexplained. How are the hyperparameters for other methods chosen? Are the beta values chosen for DeVAE selected based on test performance or on a validation set? 
- What are the layer embeddings? They seem to be an important component of the method but are never explained.
- How much is the computational overhead because of the hierarchical latent space and how does it compare to other methods that are compared within the paper?
- Why is DeVAE not compared with FactorVAE and Cascade-VAE, they both seem highly relevant as well.
clarity,_quality,_novelty_and_reproducibility: Clarity - The paper is not very clearly written and is difficult to understand at various instances. Most of the design decisions are not supported with justifications, some of them are mentioned in weaknesses above.

Quality - The proposed method is technically sound but the quality of experiments is lower than what I would expect from an ICLR paper. 

Novelty - The contributions of the paper seem to be novel. 

Reproducibility - The paper provides many experimental details and a pseudocode, although many design choices are unexplained and/or maybe based on test results instead of on a validation set.
summary_of_the_review: The paper focuses on the important problem of the balance between disentanglement and reconstruction fidelity in disentangled representation learning research. The proposed method is technically sound. However, there are significant concerns with the experiments in the paper and the paper clarity is low. Hence, I would recommend the paper for rejection in its current form.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a hierarchy of latent representations weighted by a series of monotonically increasing hyper-parameters, which compose an information bottleneck. The authors empirically demonstrate their idea on dSprites and Shapes3D to show the proposed method is able to learn disentangled representations while preserving reconstruction fidelity.
strength_and_weaknesses: **Strength**

The proposed method is not complex and easy to understand. The experiments are comprehensive and sufficient ablation studies are provided to support the claims.

**Weakness**

The method part is not properly stated; some mathematical expressions might be wrong (at least they cannot correspond to either Fig 1. or Alg. 1). I am confused after reading it:

- According to Fig 1. and Alg. 1, the latent variables depend on previous ones, i.e., $z_1$ is dependent to $z_0$, $z_2$ depends on $z_1$. However, Equation 2 seems to show $z_i$ is only dependent to $x$ or $z_0$, and $z_1, z_2, ..., z_K$ are independent like K VAEs. Equation 3, also seems to have $z_i$ marginalized out, while the code in Alg. 1 does not have such an operation. The authors need to clarify this part.

- The notation v is in Fig. 1 is not used in the mathematical definition.

- What is j in Equation 5-6? 

The paper lacks theoretical analysis. The choice of $\beta$ does not have clear guidance. The authors do not show whether the modified ELBO is still valid and whether it is a tight bound.

According to the results in Table 2, by adding HiS and DiT, the reconstruction fidelity seems to get worse, which does not support the claim of the paper: "optimize reconstruction and keep disentangled representation". 

In the case when the latent space is stacked with redundant variables, a non-trivial study could be whether there are some variables that are more important while some variables can be distilled. 

The granularity of the disentanglement of the proposed method is correlated to the parameter size and training efficiency. In other words, if we would like to refine the granularity of the information bottleneck (have more choices of $\beta_i$ like done in Table 3), the parameter size and required training iterations also increase. 

From the perspective of practical use, the improvement is a little bit marginal, and the experiments are conducted on small-scale data. It is still unclear whether this method could be useful for larger data and models. 

**Minors**

Equation 4: some 'i's are bold while some are not.
Equation 5: same issue.

Below Equation 7: $p\left(\boldsymbol{z}_i\right)=(1-s) s^i /\left(1-s^{i+1}\right), s \neq 1 ; \frac{1}{K}, s=1$ is not a rigorous math expression. 

Section 4. Paragraph1-Line7: one citation is missing.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The method part of the paper is unclear while the experiment part is  overall clear. Please see the weakness part.

**Novelty**

The proposed method has some overlaps in previous methods that propose hierarchical latent variables, such as ladderVAE, but is novel in using the diagonal transformation matrix. 

**Quality**

The paper provides reasonable empirical studies but lacks theoretical analysis. 

**Reproducibility**

The authors include details of the algorithm and model parameterization. Although the method formulation is ambiguous, it should be reproducible after the clarification.
summary_of_the_review: The disentanglement of representation has been an important topic in the study of representation learning and has been studied for a long time along the development of information bottleneck and variational autoencoders. The proposed method has its contributions in improving the disentanglement of the latent variables while preserving the reconstruction quality. Both the quality and clarity of the paper still need improvement. I am inclined to consider this paper below the acceptance threshold given the current version.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a novel hierarchical latent space structure and regularisation strategy (beta-pressure) to improve the balance of representation and information retention, or reconstruction quality, in VAEs. The latent space is built to have a Markov chain-like structure, where, in the encoder, the first latent space is conditioned on inputs q(z_0|x), just like standard VAEs, and all subsequent encodings are conditioned on the previous one, i.e., q(z_i+1|q_z_i). The decoder reconstructs the input from each latent space in parallel. Although I am not entirely clear on the whole mechanism (see below).
strength_and_weaknesses: Strengths: 

The idea to have parallel latent spaces with different betas and using them all to obtain "the best of both worlds" is interesting and, to the best of my knowledge, sufficiently novel in this form.


Weaknesses:

I have several concerns regarding the technical description and experiments.

Technical Section:
- The description lacks clarity in some places. For example, it is not clear what is the role of the vectors(?) v_i shown in figure 1, as they are never mentioned in the text and do not appear in the ELBO of equation 2 or the transformation tau_i of equations 3-6. Therefore, I could not understand how the decoder works; does it individually decode each z_i with the same weights and add up the reconstruction costs?
- According to theorem 1, the disentanglement of z_0 is preserved in all subsequent latent spaces. This is inconsistent with the aim of the method, where, as phrased in the paper "the sequential layers aim to disentangle factors of variation by setting narrow bottlenecks". The representations' disentanglement cannot be changed by scaling. I may have got this wrong somewhere, but if this is the case, this is a critical flaw.

Experiments:
The experimental section is rich, but in most cases hard to interpret. My main concerns are the following:
-  The results of figure 2 have several issues (at least in their presentation): 1) the plots show 3 disentanglement metrics and 1 reconstruction metrics. As per the objective of the method, what counts is their tradeoff, so a much better way to show performance would be to do something more similar to figure 5 and plot each of the 3 disentanglement metrics vs. reconstruction error. 2) It is not clear where better is higher and where better is lower. Clearly lower rec. error is better, but do we want high or low MIG for example? From the descriptions below it is not clear. 3) In the text it is stated that the DeVAE is the best, but in this plots it is often somewhere in the middle, so it is not clear how the reader is meant to interpret these plots as proving superior performance of the proposed method.
- Figure 3 is not interpretable at all. I think the columns (x-axis) are the 3 random samplings, but what is the 2-6 numbers above? Where is the -2 to +2 positions? what is the y-axis? There is no comparison with any other method, so the "disentangled perfectly" mentioned in the text is not very meaningful.
- Figure 4 has a similar problem to figure 2; The information diffusion shows how well the model is disentangling, but we need also to see the reconstruction. As stated also in this paper, it is their relationship that counts.
- Figure 5 shows, in my opinion, the relevant information; reconstruction vs. a measure of disentanglement. However, these results raise some doubts. Assuming MIG is better when higher, for high dimensions the proposed method is the best, but for low dimensions, it is outperformed significantly in both MIG and reconstruction error by the DynamicVAE. Is this right?

clarity,_quality,_novelty_and_reproducibility: Novelty: The idea is, to the best of my knowledge, novel and it is interesting. Using different latent spaces in parallel to address both reconstruction and representations is a general strategy that could be viable for addressing this very real problem in representation learning/generative models.

Clarity: The method is not clear in my opinion. In particular, the fact that variables are shown in the main architecture figure (fig 1), but never mentioned or introduced in equations is a major clarity flaw. Some mechanisms about the decoding are also just explain in-text and are difficult to follow.

Quality: The quality is a bit lacking in the experiments. Most experiments do not clearly show the important features claimed in the abstract intro and technical section, i.e., the better tradeoff between reconstruction and disentanglement. The lack of axis in a lot of the graphs is also a major quality flaw.

Reproducibility: The experiments are reproducible, aside of the confusions about the architecture mentioned in the quality section.
summary_of_the_review: The novelty of the paper is sufficient in my opinion, but the clarity and quality do not meet the bar. The technical section is not sufficiently clear and the experiments are difficult to read and not sufficiently conclusive.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a hierarchy of latent representations to gradually promote disentangled representations, where the loss for each latent layer has a KL-term and the weight monotonically increases with depth. The authors empirically demonstrate their idea on standard synthetic datasets, to show the proposed method is able to learn disentangled representations while preserving reconstruction fidelity, outperforming several previous methods.

Strength

The proposed method is intuitive, resembling prior work on increasing the KL weight (\beta) during training, but the authors propose to have multiple layers with increasing weight so the paper has some novelty. The experiments are somewhat comprehensive.

Weakness

The reviewers had severe concerns about the presentation of the paper.  On the other hand, while the intuitions are reasonable, there is no strong theoretical justification. Given there are multiple layers, the choice of \beta's may become tricky and the paper does not yet provide clear guidance on how to set them.
justification_for_why_not_higher_score: The issues of this papers are clear, and reviewer ratings are unanimously low.
Decision: Reject