Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper analyzes the gap between ELBO and exact log-likelihood under the self-consistency assumption that the encoder is the maximizer of ELBO with the decoder parameters being fixed. The authors theoretically show that the gap can be characterized with independent mechanism analysis (IMA) in a near-deterministic regime. Experiments empirically support the theoretical results.
strengths_and_weaknesses: ### Strong points
Understanding the gap between ELBO and exact log-likelihood is important. The analysis/interpretation of the gap from the perspective of IMA is new and interesting. The analysis provides new insight into disentanglement in (usual) VAE. In addition, the empirical results successfully support the theoretical parts.

### Weak points
- In the experiments, $\gamma$ is always fixed. Since $\gamma$ is optimized as well as encoder and decoder in the current VAEs, I would like to have seen a case with trainable $\gamma$.
- How large the gap between ELBO optimized in the usual way and ELBO^* is not clear.
questions: - What does the limitation that dimensionality of $x$ and $z$ should be the same come from? It is unclear to the reviewer whether $c_\text{IMA}$ in Section 4.3 is valid for the theory part.
limitations: Limitations are addressed, and no negative societal impact is observed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proves that for Guassian VAEs with isotropic Gaussian encoders, when the support of the data $\mathbf{x}\in\mathbb{R}^d$ has full dimensionality $d$, 

- The ELBO optima has decoder variance tend to zero ("self-consistency", Sec 3.1), and that
- ELBO converges to a regularized likelihood objective where the regularization encourages the decoder Jacobian to have orthogonoal columns (Sec 3.2).

The latter is connected to the recent mechanism independence formulation for causal representation learning, and the authors provide experiments demonstrating the recovery of the true latents when the mechanism independence assumptions are met.
strengths_and_weaknesses: The established connection of ELBO-trained VAEs to mechanism independence is interesting, and the experimental results can be of value to the causal representation learning community (I'm not deeply familiar with that literature).  The theoretical results are new, in the sense that they have not been established in this exact setting.

That being said, I feel that the significance of the results, in the broader context of understanding representation learning in ELBO trained VAE, is overstated.  It has been known that 

- Linear-Gaussian VAE trained with ELBO recover the principal components, and thus their decoder (Jacobian) have orthogonal columns ([4, 60]; Dai et al, 2018);
- Nonlinear Gaussian VAE trained with ELBO are regularized towards having decoders with orthogonoal Jacobian columns ([43, Section 5, 5.1]; Nakagawa et al, 2021);

Dai and Wipf (2019) has also shown that near-deterministic VAEs constitute a family of global optima of the Gaussian VAE ELBO, and empirically verified that it is recovered by common VAE architectures.

It is true that no results have been provided in the precise setting of this work, but I think this has more to do with the restrictiveness of the setting here: this work assumes the data distribution has full support (by assuming the optimal ELBO is finite), and it also uses an isotropic encoder with output dimensionality $d_z = \mathrm{dim}\\;\mathbf{x}$. The former is inappropriate for image or other structured inputs which are typically supported on low-dimensional manifolds embedded in $\mathbb{R}^{d_x}$, while the latter may prevent the automatic adaptation to the correct latent dimensionality (Dai and Wipf, 2019).  Relaxation of either restriction introduces technical challenges, so I don't think readers familiar with previous work on VAE inductive bias will find the results here surprising.  The related works are only sparingly discussed in the main text, which does not help to put the contributions of this work in context.

I would also note that self-consistency may cease to hold beyond the $d_z = \mathrm{dim}\\:\mathbf{x}$ regime.  Suppose for simplicity that $\mathrm{dim}\\:\mathbf{x}=1$.  Then any input $x$ can be encoded up to precision $O(2^{-d_z})$ by setting $q(z_i | x) = \mathcal{N}(C s_i(x), 1)$, where $C>1$ is a moderate constant, and $s_i(x)\in\\{1, -1\\}$. Then we have $\mathrm{sgn}(z_i) = s_i(x)$ with high probability, for $z\sim q(z|x)$, and the sign decoder can be easily approximated with NN decoders.  More generally, we can always take an input dimensionality $i\in [\mathrm{dim}\\:\mathbf{x}]$ and approximate it with precision $\gamma$, using up to $O(\log \gamma)$ latent dimensions which are *never deterministic in the limit*.  While this issue does not prevent the use of ELBO-trained VAEs for the recovery of independent mechanisms, thus the contribution of this work in this context, it does make it less interesting in the broader context of understanding VAEs (which are often overparamterized in a similar nature).

## References

- Nakagawa et al (2021): Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding, in ICML.
- Dai and Wipf (2019): Diagnosing and Enhancing VAE Models, in ICLR.
- Dai et al (2018): Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models, in JMLR.

questions: I think the paper should be revised to better position the existing results, or to improve them in the aforementioned directions.  Some of the discussions in Appendix D could also be moved to the main text.
limitations: Some limitations, e.g., the dimensionality issue, are mentioned, but their implications not fully discussed (see above).  Others are adequately discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper asks why VAEs learn useful representations despite the non-identifiability of the latent variables (in a nonlinear data generating process) using unregularized maximum likelihood estimation. The paper's main contribution is to show that VAEs with near-deterministic decoders do not converge to the marginal likelihood but to an IMA-regularized likelihood. The IMA-regularization (based on previous work) enforces a column-orthogonality condition on the Jacobian of the decoder, which can be beneficial for recovering the true latent factors when the data generating process satisfies the IMA assumption. Experimental results on synthetic numerical data corroborate that 

- the encoder approximately inverts the decoder; 
- the ELBO converges to the IMA-regularized likelihood if the true decoder is used, even when the generating process violates IMA; 
- the encoder recovers the true latent factors if the data generating process satisfies the IMA assumption. The recovery is further corroborated by experiments on synthetic image data (Sprites).

The analysis focuses on near-deterministic decoders, for which the paper shows that the optimal encoder approximately inverts the decoder. This result is interesting in its own right as it formalizes an existing hypothesis and is necessary for deriving the convergence of the ELBO to the IMA-regularized likelihood.
strengths_and_weaknesses: **Strengths**

- Formalization of the mechanism by which VAEs can learn disentangled representations and comprehensive theoretical analysis with a detailed explanation.
- Thorough empirical analysis with controlled synthetic experiments.
- Clear and transparent contextualization of previous work, positioning of the paper, and contribution statement.
- Overall, a solid contribution that advances the understanding of representation learning with VAEs.

**Weaknesses**

- The paper does not discuss the practical significance of their results sufficiently well. For example: How much IMA violation would be acceptable to still recover the true factors in more tangible settings (e.g., in more complex variants of the Sprites dataset)? Can practitioners control or validate the degree of IMA violation when their goal is to recover the true but unknown latent factors? Even though the paper presents basic research, more experiments in practical settings could help readers better understand the practical significance of the theoretical results.
- The paper is dense and has a lengthy appendix. Subjectively speaking, the paper might be easier to parse and understand if it focused more on one part of the story.
questions: The abstract calls it a paradox that VAEs seem to learn "useful" representations despite not inverting the data generating process. As the usefulness of a representation depends on the downstream task, one could argue that a representation can be useful, even if it does not invert the data generating process. Maybe you could clarify this part in the abstract?

You assume that the variational posterior has a diagonal covariance matrix by design (e.g., in Figure 1 and Section 3.2).  However, is it safe to assume that the _empirical_ covariance is diagonal? Would a non-diagonal empirical covariance affect the IMA regularization? Similarly, what if the empirical covariance of the decoder is not isotropic?

In Definition 1, did you mean "mean decoder" instead of "mean encoder"?  In Definition 1 and Assumption 1, you assume that the mean decoder $f^{\theta}(z)$ is invertible. Is this assumption justified? I find it hard to imagine that the assumption is fulfilled at initialization time, even if the number of latent dimensions is equal to the number of input dimensions. But maybe there is an argument to be made for a model that reconstructs the inputs sufficiently well?

In section 4.1, to test the convergence of $\mu^{\hat{\phi}}(x)$ to $g^{\theta}(x)$, why do you not measure the MCC? Why is the comparison in the right subplot of Figure 2 more suitable?

For the numerical experiments you use different types of weight matrices (orthogonal, square upper-triangular, Möbius transform). Could you please clarify if the choice of weight matrix influences whether the mixing is in the IMA class?

Minor points:

- For the experiment in Section 4.1, you say that you use a uniform prior, but Table 1 lists a Gaussian prior.
- In line 79, you introduce the subscript $k$, but do not explain its purpose.
limitations: Overall, the paper is very transparent about the limitations.

One point for which I would have appreciated a bit more detail is the distinction between $\beta$ and $\gamma$. It is not immediately clear why self-consistency is harder to prove for $\beta$-VAEs, especially since you write that $\beta$ and $\frac{1}{\gamma^2}$ play a similar role in the objective (Section A.3).

The second point which could be addressed is the practical significance of the results (see Strengths and Weaknesses).
review_title: null

Meta-review: metareview: ## MetaReview

**Summary**: The submission examins why VAEs learn useful representations despite the non-identifiability of the latent variables (in a nonlinear data generating process). To do so they analyze the amortization gap from the perspective of independent mechanism analysis (IMA). The main contribution is to show that VAEs with near-deterministic decoders do not converge to the marginal likelihood but to an IMA-regularized likelihood. Previous work has shown that this regularization enforces a column-orthogonality condition on the Jacobian of the decoder, which aids in recovering the true latent factors when the data generating process satisfies the IMA assumption. The analysis focuses on near-deterministic decoders, for which the paper shows that the optimal encoder approximately inverts the decoder. This result  formalizes an existing hypothesis and is necessary for deriving the convergence of the ELBO to the IMA-regularized likelihood. Synthetic experiments empirically support theoretical results.

**Strengths**: Reviewers [tFG1] and [UUBT] appreciate the  analysis of the amortization gap from the perspective of independent mechanism analysis, finding it both new and interesting. Reviewer [8jNM] appreciates the comprehensive theoretical analysis and detailed explanation, as well as the clear and transparent contextualization relative to related work (although reviewer [UUBT] finds that related work is only sparingly discussed in main text) 

**Weaknesses**: While the reviewers were overall appreciative of this submission, they also expressed concerns. 

Reviewer [tFG1] notes that γ is always fixed in experiments, whereas it is typically optimized in VAEs. A comparison to a case with trainable γ would therefore be desirable. The reviewer also notes that it is unclear how big the gap is between ELBO and ELBO*. 

Reviewer [UUBT] expresses concern that significance of the results is somewhat overstated. In particular it has been known that a linear-Gaussian VAE recovers the principal components and that the decoder therefore has orthogonal Jacobian columns (Dai et al 2018, Dai and Wipf 2019), and that nonlinear Guassian VAEs are also regularized towards decoders with orthogonal Jacobian columns. While no results for precise setting of work have been presented, this may reflect restrictiveness of setting, which assumed that data distribution has full support, along with an isotropic encoder (note: the authors clarified in their response that the encoder is in fact not isotropic). 

Reviewer [8jNM] finds there is insufficient discussion of practical significance. In this context it would be helpful to understand IMA violation would be acceptable to still recover true factors. The reviewer further finds that the paper is dense and the appendix is long. The paper might be clearer if it focused more on one part of the story.

**Author Reviewer Discussion**: The authors provided detailed responses to all reviewers, including additional analyis of the role of the γ hyperparameter in response to [tFG1], additional discussion of the work by (Dai and Wipf 2019) in response to [UUBT], and provided various clarifications to reviewer [8jNM]. Reviewer [UUBT] raised their score 5->6 and reviewer [tFG1] raised their scored 6->7. 

**Reviewer AC Discussion**: Reviewers [8jNM] and [tFG1] affirmed that they are happy for this paper to appear.

**Overall Recommendation**: The AC is satisfied with the level of examination and discussion that has taken place and will follow the recommendation of the reviewers. This is a relatively clear accept.  
confidence: Certain
Decision: Accept