Unpicking Data at the Seams: VAEs, Disentanglement and Independent Components

Authors: Carl Allen
Primary area: generative models
Keywords: VAE, variational autoencoder, beta-VAE, disentanglement

Abstract:
Disentanglement, or identifying statistically independent salient factors of the data, is of interest in many aspects of machine learning and statistics, having potential to improve generation of synthetic data with controlled properties, robust classification of features, parsimonious encoding, and greater understanding of the generative process behind the data. Disentanglement arises in various generative paradigms, including Variational Autoencoders (VAEs), GANs and diffusion models, and particular progress has recently been made in understanding the former. That line of research shows that the choice of diagonal posterior covariance matrices in a VAE promotes mutual orthogonality between columns of the decoder's Jacobian. We continue this thread to show how such *linear* independence translates to *statistical* independence, completing the chain in understanding how the VAE objective leads to the identification of independent components of the data, i.e. disentanglement.

Decision: reject
Reviewer scores: 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730196787803

thread_type:
official_review

<thread_object>
creator: reviewer_kwdn

<content>
summary:
The paper connects orthogonality of the decoder Jacobian in VAEs to disentangling statistically independent latent factors.

strengths:
The paper is
- generally well-written,
- the figures are high quality,
- the structure is logical and easy to follow.

weaknesses:
Despite the neat structure and the obvious effort the authors put into this work, **I honestly struggled to discern what the added value of the contribution of the paper is, compared to the literature** (while knowing the field and having worked on related projects). I will try to make the points that confused me clear below. Please let me know if my understanding is incorrect.

## Major issues
- The added value of the contribution is unclear to me. A potential statement for this could be the _wrong_ statement, made in L468: _"Reizinger et al. (2022) relate the VAE objective to independent causal mechanisms (Gresele et al., 2021) which consider non-statistically independent sources that contribute to a mixing function by orthogonal columns of the Jacobian. This clearly relates to the orthogonal Jacobian bias of VAEs, but differs to our approach that identifies statistically independent components/sources"_. Namely, **Reizinger et al. (2022) assume a standard VAE with statistically independent sources.** It seems to me that the authors themselves differentiate their work from Reizinger et al. (2022) only by the presumed (but not true) dependent/independent sources divide. As this is not the case (unless my understanding is incorrect), I cannot see the added value of the paper. This holds for the theorems, and also for the elucidation of the role of $\beta,$ as Reizinger et al. (2022) also relate it to the decoder variance in their Appx. A.3.
- The paper is a bit too heavy on notation in the main text. While I understand that the proofs require precision, communicating (the intuition behind) the results do not. Please rework the main text to include only the necessary notation. Also, please always explain what your notation means (e.g., the caption of Figure 1 is not self-contained)

## Minor points
- L167: expand of what you are trying to do with the two losses
- Eq (9): define $s_i$
- L248: why is the image manifold parallel to the $U$-basis?
- Eq (12): it should include the log absolute determinant of teh Jacobian, and not $W$ as this is about the nonlinear case, right?

questions:
- What do the two types of bold D's in Eq. (6) denote?
- L156-160: isn't statistical independence of latent factors assumed in VAEs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642581510

thread_type:
official_review

<thread_object>
creator: reviewer_yxof

<content>
summary:
The paper provides a theoretical understanding of disentanglement within Variational Autoencoders when the columns of the jacobian of the decoder are orthogonal (theorem 2). 

Starting from this hypothesis, namely the linear independence of the columns of the decoder Jacobian, the author shows that it is possible to link statistically independent components of the output distribution to independent latent factors of variation.

strengths:
The authors have clearly defined the problem and the goal of the paper;

The analysis extends existing work, offering theoretical proof of why an architectural constraint such as diagonal posterior covariance induces an effective disentanglement in the VAE latent space.

weaknesses:
The paper lacks an experimental part. 
Even if the contribution is theoretical, a small experiment on a synthetic dataset corroborates the claims and strengthens the contribution.

Splitting the deductive reasoning between ll 210 to 259 in paragraphs highlighting the key logical steps can also be helpful for the reader.

Additional figures or clarification of the only existing one (Fig 1) can help the reader understand the sequence of logical steps more easily.

questions:
How sensitive is the disentanglement effect to minor violations of the assumptions (orthogonality of the decoder Jacobian)?

in 059: do you mean encoder or decoder Jacobian?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718741847

thread_type:
official_review

<thread_object>
creator: reviewer_fhcg

<content>
summary:
This paper aims to make progress on understanding why VAEs are able to learn disentangled representations. To this end, the authors connect orthogonality of the decoder Jacobian to disentanglement by showing that the VAE learns statistically independent sub-manifolds in the observed space. Further, the paper provides additional insights into the VAE objective such as on the role of beta in a beta-VAE.

strengths:
* The paper addresses an important problem. Namely, developing a principled understanding of how and why disentanglement is possible in deep generative models.

* The paper is well written and the presentation is well structured.

* The theoretical results and insights are presented in an intuitive and accessible way, with nice visual intuition from the figures.

weaknesses:
My main issue with this paper is that I do not think the contributions presented in this paper offer sufficient novelty relative to prior work to merit acceptance, and, moreover, I do not think the authors adequately compare their contribution to prior work. I discuss these points in detail below with specific examples.

**Identifiability of Latent Factors in VAEs**

The authors build on several prior works which try to explain why VAEs are able to identify the ground-truth latent factors by showing that the VAE objective promotes the decoder to have an orthogonal Jacobian. The authors perform a similar analysis and present a result claiming to be an identifiability result for VAEs.

Understanding the identifiability of VAEs, however, has been analyzed in detail in prior works. Specifically, the work of [1], rigorously showed that the VAE objective with vanishing decoder variance is equivalent to maximum likelihood under an independent Gaussian prior plus a regularization term enforcing that the Jacobian has orthogonal columns. The identifiability of such models with independent latents and orthogonal Jacobians was studied in depth by [2], who proved a certain form of identifiability for such models and showed the theoretical challenges in recovering a complete identifiability result.

The authors do not mention the work of [2] and wrt [1], they state that their analysis differs because [1] assumes statistically dependent factors such that there result is novel relative to this work. This statement is wrong. The work in [1] does not assume statistically dependent factors. To this end, I think the works of [1] and [2] conduct a more rigorous and comprehensive analysis of the VAE objective and its identifiability than the current work, such that I do not feel this works adds sufficient novelty.

Additionally, the analysis of the VAE objective and the identifiability analysis conducted in this work are not as rigorous as these prior works in [1, 2], and are closer to the results in [3, 4]. From this standpoint, however, I am also not sure what the novelty is of the authors SVD based identifiability argument, as similar ideas were presented in [3] as I understand. Perhaps, the authors can clarify this as well.

**Understanding the Role of Beta in Beta-VAEs**

Another stated contribution of this work is the authors' analysis of the role of beta in a beta-VAE (Section 3.3). The authors state that beta can be interpreted "as scaling the variance of the likelihood distribution,". From what I can tell, this result also seems very similar to the result in [1] presented in Appendix A.3 on the role of beta. I am curious if the authors can comment on the novelty of their result relative to this prior work.


**Theorem Statements**

As an additional point, I did not see proofs for Theorems 1 and 3, and am thus curious if I am missing something or if there are proofs for these results.

**Bibliography**

1. Embrace the Gap: VAEs Perform Independent Mechanism Analysis
 (https://arxiv.org/abs/2206.02416)

2. Function Classes for Identifiable Nonlinear Independent Component Analysis
(https://arxiv.org/abs/2208.06406)

3. Variational Autoencoders Pursue PCA Directions (by Accident)
(https://arxiv.org/abs/1812.06775)

4. On Implicit Regularization in β-VAEs (https://arxiv.org/abs/2002.00041)

questions:
**1.** Can the authors comment on the novelty of their results relative to the prior results discussed above?

**2.** Where are the proofs for Theorems 1 and 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731698685689

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response to All

comment:
Dear reviewers, many thanks for your time and detailed consideration of our work. We believe we can address all concerns raised and look forward to discussing any that remain. While we hope the scores will improve in due course, we are delighted that our work is considered *well written/clearly set out* (all), *accessible/easy to follow* (fhCg, KWdN) and *addresses an important problem* (fhCg). We reply separately to each reviewer in detail and provide here a general response to all reviewers on key points.

1) **Prior work [1]**: Reviewers **fhCg** and **KWdN** raise similar concerns regarding novelty vs [1]. We chose not to critique this work in our paper, but we must now address it.
    * [1] appears to contain **material errors** in the proof of Theorem 1 (their main result), e.g.:
        * (p.33) the reverse triangle inequality is used with *square norms*, which [does not hold in general](https://math.stackexchange.com/questions/1671911/squared-reverse-triangle-inequality)
        * (p.33) the penultimate inequality drops unbounded cross terms without explanation
        * (p.34) without explanation, the 1st equation drops the K term (from p.33) and thus all second and higher order derivatives, which are unbounded.
    * [1] makes **strong assumptions** that do not hold in general and are not made in our work or similar works [2,3], e.g.
        * the latent space and data space must have the same dimension
        * Var$[x|z] \to 0$, whereas disentanglement is seen to improve as variance (i.e. $\beta$) *increases* [5]
    * The key mathematical arguments in [1] mirror those in [3] (an earlier work, summarised in our paper) and it is unclear that the differences are valid (as above)

    We will reconsider how [1] is referenced in our work but hope this sufficiently addresses the concerns raised.

2) **Novelty**: independently of the above, we emphasise the novelty of our work (note that to explain disentanglement from the ELBO requires steps (**A**)  from ELBO to orthogonality; and (**B**) from orthogonality to disentanglement. [1,2,3] largely address A, we address B.):
    * we provide a first *constructive* proof of how disentanglement follows from column-orthgonality, with insight by analogy to the linear case (**B**)
    * we generalise the connection between $\beta$ and Var$[x|z]$ for Gaussian VAEs [4] to *exponential family* VAEs, for which disentanglement is observed [5].
    * to further increase novelty over prior work, we include the *precise* relationship (new Eq 8) between posterior covariance and log-likelihood Hessian, previously approximated [1,2,3] (**A**).
    * In summary, to the best of our knowledge we now provide the most complete constructive proof of **A** and **B**

3) **Changes to the paper**: 
    * we have made minor or stylistic edits throughout based on feedback (**YxoF0**, **KWdN**), in particular we hope the main theoretical section is now less terse and easier to follow. We agree that this is important to aid the reader.
    * new aspects are highlighted in teal.
    * to simplify and make space for adjustments we have removed Section 4, which was incidental to the main contribution. 


[1] [Embrace the Gap: VAEs Perform Independent Mechanism Analysis (2022)](https://arxiv.org/abs/2206.02416)

[2] [Variational Autoencoders Pursue PCA Directions (by Accident) (2019)](https://arxiv.org/abs/1812.06775)

[3] [On Implicit Regularization in β-VAEs (2020)](https://arxiv.org/abs/2002.00041)

[4] [Don’t Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019)](https://arxiv.org/abs/1911.02469)

[5] [$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework](https://www.cs.toronto.edu/~bonner/courses/2022s/csc2547/papers/generative/disentangled-representations/beta-vae,-higgins,-iclr2017.pdf)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732220062910

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Phase ending

comment:
Dear Reviewers, we write as the discussion phase is shortly due to end (26/11). 

We are grateful for your feedback, which has helped improve the paper. We hope to have now addressed all points raised and kindly ask if you could confirm, or to let us know of anything further we can clarify.  As perhaps the key points, we hope to have:
* clarified the significant differences and resolved any concerns regarding [1] (**fhCg**, **KWdN**) ; and
* sufficiently justified why we do not provide further empirical support given the extensive amount present in the literature. (Related works evidence Jacobian-orthogonality and we effectively solidify/fill gaps in previous theoretical justifications) (**YxoF**) 

Our work justifies disentanglement in ($\beta$-)VAEs, a highly intriguing, domain-spanning phenomenon, known of for almost a decade yet still very much of interest. We also give mathematical meaning to $\beta$ and justify why it mitigates posterior collapse (in *non-Gaussian* settings). We believe our work improves theoretical understanding and could be of interest to many in the ML community. We very much hope that, given the updates made over the course of the review period, you might now reconsider it together with your scores.

Many thanks, Paper Authors

[1] [Embrace the Gap: VAEs Perform Independent Mechanism Analysis (2022)](https://arxiv.org/abs/2206.02416)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper analyzes theoretically why variational autoencoders tend to "disentangle" the data in their latent representation, encoding statistically independent factors.  The authors show that the training promotes column orthogonality in the decoder's Jacobian.  They then go on to show that this translates to statistical independence.  The reviewers overall found the paper well written, technically correct and convincing.  One reviewer noted that the figures are "super high quality".  Note, that of the three reviews received, one seems low confidence.  However, the reviewers voiced concerns about lack of experiments, some clarity (too heavy notation) and primarily novelty.  Specifically, the reviewers pointed out that the paper has a similar narrative to [1] (Embrace the Gap: VAEs Perform Independent Mechanism Analysis (https://arxiv.org/abs/2206.02416)), which also has the column orthogonality result demonstrating disentanglement.  There was significant discussion regarding this connection between the reviewers and the authors, and the AC that led to disagreement.  I will elaborate below.  

In the related work, this paper [1] was cited but only very briefly: "Reizinger et al. (2022) relate the VAE objective to independent mechanism analysis (Gresele et al., 2021)".  I would agree that this is not properly attributing prior highly-related work.  The authors subsequently added "which encourages column-orthogonality in the mixing function of ICA, similarly to that induced implicitly in the decoder of a VAE.", but the reviewers contend that this mischaracterizes the work.  A primary argument from the authors is that [1] has a technical flaw in a central theorem that renders the paper incorrect, and thus should not be be cited at all.  The reviewers do not seem to agree with this assessment, i.e. regarding correctness or whether the paper is discussed sufficiently.

While I sympathize with the authors, I disagree with their argument regarding novelty.  Even if there was a technical flaw in a major proof (not suggesting this is true, but hypothetically speaking...), one should not pretend as if the paper didn't exist.  It sets a precedent in ideas, methods, etc. that should be discussed constructively.  The papers are so similar, in motivation, claims and end result that there must be an in-depth discussion *in the paper* of what this work adds, or does better.  

The reviewers all voted to reject the paper.  After careful deliberation, discussion with the reviewers and input from the authors, I do not believe it would be appropriate to override the reviews.  Therefore I would recommend reject.  

In addition to more detailed discussion of and comparison to previous literature, I asked the reviewers what the authors' could do to change their recommendation to an accept.  I'll paste that below (edited) and hopefully it will be helpful to the authors to improve the manuscript for a future submission.

- Elaborating on the factorization in data space, what that means, and how realistic that is (complex data distributions from the real world do not have this property?) - including experimental validation of this claim
- The authors refer to independent submanifolds, but (statistical) independence is defined for random variables, so this needs a formal treatment.
- An in-depth discussion of the literature, particularly the work by Buchholz et. al, 2022, titled "Function classes for identifiable nonlinear independent component analysis"
- Correction of the claim of providing the first precise result (the updated submission refers to "Gaussian posterior approximations" and "average Laplace approximation" around L149, which is not precise, at least calling it precise without defining the exact notion of precision can be confusing)
- A self-critical assessment of the assumptions, including the ones omitted, compared to prior works