Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is about learning disentanglement using scalable non-parametric density estimation. The author proposes a dual total correlation (DTC) metric for disentangling that divides the joint distribution into many low-dimensional conditional distributions. The authors conduct extensive experiments to show the superiority of their approach against state-of-the-art baselines.

strength_and_weaknesses: Strength

1) The analysis is shown in section 3, which derives the DTC and its comparison against total correlation. DTC in eqn 1. removes the requirement to compute p(z). DTC requires computing the conditional entropies for disentanglement.
2) Propose a new "Gaussian Channel Autoencoder" (GCAE). As the name suggests, gaussian noise with variance sigma^2 is added to smooth the representations. And in Sec 2, the authors have detailed how they use the discriminator-based method, which applies the density-ratio trick and the Radon-Nikodym theorem to estimate the density of samples from an unknown distribution.
3) The authors conduct experiments on Beamsynthesis (time series dataset) and dsprites (synthetic images) to show GCAE is better than previous baselines. Fig 3a and 3b show the scatter plot of GCAE for summed information loss against mean squared error (MSE) and mutual information gap (MIG). GCAE performs better than baseline models for Beamsynthesis, while in the case of dsprites it does better only for a small range of lambda (hyperparameter of summed information loss).



Weakness

1) The novelty of the paper is limited. The authors claim that using DTC  removes the constraint of estimating p(z), which is difficult compared to the conditional distribution. However, computing "m" conditional distributions needs "m" discriminators, which are sensitive to hyperparameters. It also requires a lot of data and time.
2) The authors use only the MIG metric to show disentanglement. The authors should have compared against other metrics such as DCI, SAP, etc. Please refer to "Measuring Disentanglement: A Review of Metrics" to understand why there is not one disentangling metric and each metric explores different aspects.
3) The datasets used for the experiments are limited. The results work well for Beamsynthesis, not for dsprites. It can be seen in figure 4a and 4b. The scatter plot in figure 3b for beam synthesis shows a negative correlation coefficient of -0.823. There is no mention of dsprites.
4) No qualitative results were shown for dsprites and no results on the latent traversal on these z axes. Since the authors chose m=10 as the latent dimension, and dsprites has five factors of variation, the qualitative result should have helped understand how the disentanglement happened.


clarity,_quality,_novelty_and_reproducibility: The work needs more finetuning and structuring. 
Example:
1) Page 6: "The tightly grouped samples in the lower right of the plot correspond with lambda = 0, and incorporating any lambda > 0 leads to a decrease in L_sumI and increase in MSE. As lambda is increased further the MSE increases only slightly as the average L_sumI decreases significantly." This line is unclear in context with lambda and figure 3a.
2) Page6: "Figure 3b plots the relationship between end-of-training L_ΣI values with MIG evaluation scores for both Beamsynthesis and dSprites". But in that figure square box representing dsprites as shown in fig 3a is missing.
3) In eq1, the authors have mentioned DTC(z) = DTC_i(z); I think a few lines describing why that is necessary for the readers.

summary_of_the_review: 
Overall, the paper seems to be a work in progress, and there are many areas where it needs further improvement. Currently, the work has limited novelty and needs more experiments with results to justify the made claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To address the reliability issue of learning independent representations, the paper proposes to regularize a noisy autoencoder with a new regularization term based on dual total correlation. This results in a new Gaussian Channel Autoencoder, which does not need to estimate the joint distribution of the latent representation and achieves higher disentanglement scores.
strength_and_weaknesses: Strength:

+ It is interesting to use the dual total correlation instead of total correlation as the metric for disentangle representation learning. The metric circumvents the curse of dimensionality of estimating the joint distribution of latent representations.

+ The paper is well organized and easy to follow. The motivation is clear.

Weakness

- The paper claims the method can achieve reliable and high-performing disentanglement outcomes. It seems that only the "reliable" part is supported by higher disentanglement scores from the experiment. What is the definition of "high-performing" and how it is supported by the paper?

- The experimental results are not convincing. As stated in the paper that "there currently aren’t good unsupervised indicators of disentanglement", why only the Mutual Information Gap (MIG) is used as the disentanglement score? How about Z-Diff score [1], SAP score [2], and Factor score [3]?

[1] beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017

[2]  Variational inference of disentangled latent concepts from unlabeled observations. In ICLR, 2018

[3]  Disentangling by factorising. In ICML, 2018.

- Some notations are confusing. For example, $DTC(z) = DTC_i(z)$ in Eq. (1). What do $z_{\forall j\neq i}$ and $z_{k\neq j}$ mean and what are their difference? I don't follow  the discussion from $I(z_i; z_{\forall j\neq I})$ in Eq.(2) to $I(z_i;z_j)$.

- The motivation for the major loss term in Eq. (3) is not clear. What is the definition of feature-scale dependent term and why it is helpful?

clarity,_quality,_novelty_and_reproducibility: Clarity: Not very clear. The paper has confusing notations. The proposed method is not well explained.

Quality: Some claims are not supported. Experiments are not convincing.

Novelty: The paper contributes some new ideas.

Reproducibility: Good. Key details of the experiments are provided.
summary_of_the_review: The paper contributes some new ideas to disentangled representation learning. However, some claims are not well supported and the experiments are weak. The writing could also be improved by clearly defining the notations and clarifying some confusing terms.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose learning disentangled representation via regularizing Gaussian Channel Autoencoder (GCAE) with Excess Entropy Power (EEP) loss. The authors claim their proposed method can achieve more reliable and high-performing disentanglement.
Their methods are largely motivated by the curse of dimensionality difficulty issue of the FactorVAE, one of the leading VAE-based disentangling approaches.  FactorVAE modifies beta-VAE objective by specifically penalizing the dependencies between the latent dimensions towards benefiting the Total Correlation (TC). The authors (of this paper) propose to use Dual Total Correlation (DTC) towards avoiding the curse of dimensionality.

The authors examined their proposed approaches on two widely used datasets using one metric – Mutual Information Gap (MIG). Experiments show that their approaches achieve (in average) higher MIG compared with other three leading VAE-based disentanglement approaches. 

strength_and_weaknesses: The strengths include but not limited to:

-- The proposed methods are well-motivated. The authors performed preliminary experiments to show the effect of curse of dimensionality for non-parametric density estimation, which is illustrated in Figure one. Based on this fact, the authors analyze the TC and DTC, and show the merits of DTC.

-- The proposed method does achieve higher MIG compared with a few leading VAE-based approaches.


The weaknesses include:

-- There are a lot of quantitative and qualitative ways for evaluating disentangled representations. For quantitative evaluation, there are a few widely used metrics besides MIG, including explicitness (Ridgeway & Mozer, 2018), beta score (Higgins et al., 2017), SAP (Kumar et al., 2018) and DCI disentanglement (Eastwood & Williams 2018).  Typically, previous works (on VAE-based disentanglement) would try to show their methods are consistently better using multiple metrics.

-- Besides the three leading approaches (by 2018) mentioned by the authors, there are also some important works missed by the authors. For example, DIP-VAE (Kumar et al., 2017) and CCI-VAE (Burgress et al., 2017). More importantly, latest works on VAE-based disentangled representation learning has already realized the importance of using inductive biases. A recent work published in ICML (Mita et al., 2021) employs a particular form of factorized prior (also conditionally depends on auxiliary variables) towards learning an identifiable model with theoretical guarantees on disentanglement. Te authors could try to compare with some more (especially recent ones) important works using more datasets.

-- The preliminary experiments do show that higher m may cause difficulty in training/optimization for other works (especially Factor VAE), but the whole set of experiments in the paper are done under a single dimension m=10. It would be interesting to conduct more ablation studies to show the benefits of the proposed methods in terms of curse of dimensionality compared with FactorVAE and Beta VAE.

-- Some comments on ablation studies: In Figure 3, Figure 4 and Figure 5, the best performance is achieved using sigma=0.3 (at least for Beamsythesis), did you further try sigma larger than 0.3? Also, in page 5, you mentioned the importance of adding Gaussian noise – it ensures that p(z) is continuous and finite with respect to uniform density. Did you try the scenario where sigma=0?

clarity,_quality,_novelty_and_reproducibility: Overall, the paper’s idea is interesting, while descriptions can be less clear at places

— In the bottom of page 4, the authors wrote ‘to extract a compressed, noise-resistant representation’. Can you elaborate why the representation is noise-resistant?

— Figure two and page 4 descriptions are not aligned. For example, in the figure 2, the encoder with parameter phi (lower case) would generate representation z_phi; In the main text description, you are using uppercase phi. More importantly, u is missing in Figure 2, and the figure 2 is not so informative. 

—It’s unclear to me if m<<n (in page 4) is always realistic.

--Figure 3 caption is missing.

summary_of_the_review: Overall, the method is novel and demonstrates its effectiveness on learning disentangled representations. However, I do have some concerns/comments as mentioned in my review. I’m slightly not inclined at this moment, but am willing to discuss with authors and change my views accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of disentangling representations, where we aim to learn representations that have independent features. Authors start with the common learning objective that encourages disentanglement, the Total Correlation (TC), and hypothesize that an important disadvantage of this objective is that it requires an estimate of the latent density. This density is difficult to compute for high-dimensional representations due to the curse of dimensionality. Authors propose an alternative objective, Excess Entropy Power (EEP), that combines TC with Dual Total Correlation (DTC), and can be computed using only the estimates of _conditional_ densities in the latent space. Such conditional densities are often easier to estimate, as also shown by authors in a toy experiment. Authors run experiments on two datasets with known latent factors, observing that EEP improves upon strong VAE-based baselines.
strength_and_weaknesses: Strengths:
- The observation that TC depends on the quality of the full density estimate is a keen one, and the solution based on combining TC with DTC is non-trivial and novel.
- The proposed method is introduced in great detail. The mathematical notation is clear and consistent.
- Experiments use strong baselines: $\beta$-VAE and its variants. The proposed method consistently outperforms these baselines.
- Empirical studies of the role of hyperparameters $\sigma$ and $\lambda$.
- A good critical discussion of the results, as well as the limitations of the method.

Weaknesses:
- Related work limited to a single paragraph in the introduction. I would've liked to see a broader discussion of disentanglement methods, representation learning methods, and density estimation methods. For density estimation, especially methods that model the density as a sequence of conditional distributions (autoregressive density estimators).
- The KL divergence values are sometimes _negative_ in Figure 1(b): how is this possible?
- Several choices in the paper are not clearly motivated, including the density ratio estimation as the density estimation method, the loss weighting scheme in equation 3, and the autoencoder with Gaussian noise as the representation learning method.
- Density ratio estimation is characterized several times as "non-parametric", even though we're parametrizing the density (albeit implicitly) via the discriminator neural net.
- I am not convinced we're comparing apples to apples in Figure 1: in (a) we're estimating an $m$-dimensional density, while in (b) we're estimating a 1-dimensional density. Estimating the $m$-dimensional density as a series of conditional densities in (b) would be more convincing.
- Authors do discuss that the method is computationally expensive (need to train $m$ discriminators), but there are no runtime metrics reported in the experiments section.

Questions:
- Have you tried using GCAE as a generative model (as you suggest in the discussion)?
- Do you have intuition for why in Figure 4, (b) part especially, the MIG is going down for $\alpha > 0.2$? Is it because we're "over-disentangling", and the model struggles to capture any info about the true factors?

Minor:
- Citation style: consider making use of `\citep` and `\citet`.
- Define $m$ in the caption of Figure 1.
- A figure on page 6 has no caption. It is also not clear what the different dots that have the same color represent.
- Define $h$ in the first equation of section 3.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the manuscript could be improved (see weaknesses above). To the best of my knowledge, the method is novel, albeit it could be related better to prior work. Reproducibility is good: authors provide the details of their training procedure.
summary_of_the_review: The proposed method solves an important problem, is novel, and is evaluated well. However, several choices in the paper are not very well motivated, and the clarity and quality of the manuscript could be improved. As it stands, I consider the paper to be below the bar for acceptance, but would be willing to reconsider if the authors addressed the points raised above.

---

Authors have responded to some of my points, especially in regards to motivating the choices in the paper, and I have increased my score as a result. I believe the paper can be made stronger still by better connecting to methods in density estimation and studying the pros and cons of autoregressive density estimation in this context, but I'd now support it getting accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strengths:
- better results on benchmark data sets than SOTA (strong baselines) 3
- proposed method circumvents curse of dimensionality
- well-motivated methods
- well written paper

Weakness:
- limited scope of comparison (metrics) 3 - added metrics in revision
- limited novelty in density estimation
- No qualitative results demonstrating disentanglement
- limited discussion of related work in density estimation (not addressed in rebuttal)
justification_for_why_not_higher_score: The paper is a borderline accept and thus on the lower end of the score.
justification_for_why_not_lower_score: Due to the significant additional experiments that the authors have performed during the rebuttal period the reviewers could be swayed to increase their scores wards an accept.
Decision: Accept: poster