Embrace the Gap: VAEs Perform Independent Mechanism Analysis
Authors: Patrik Reizinger
Luigi Gresele
Jack Brady
Julius Von Kügelgen
Dominik Zietlow
Bernhard Schölkopf
Georg Martius
Wieland Brendel
Michel Besserve
Venue: NeurIPS 2022 Accept (2022)
Keywords: variational autoencoder
ELBO
representation learning
independent mechanism analysis
variational inference

Abstract: Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood. While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder---a commonly used but unproven conjecture---which we refer to as self-consistency. Leveraging self-consistency, we show that the ELBO converges to a regularized log-likelihood. This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds an inductive bias towards decoders with column-orthogonal Jacobians, which helps recovering the true latent factors. The gap between ELBO and log-likelihood is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning. In experiments on synthetic and image data, we show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.

Embrace the Gap: VAEs Perform
Independent Mechanism Analysis
Patrik Reizinger"! Luigi Gresele"? Jack Brady"!, Juliuus Juliuus von Kügelgen Dominik Zietlow2,, Bernhard Schölkopf", Georg Martius,, Wieland Brendel', and Michel Besserve
University of Tübingen, Germany
Max Planck Institute for Intelligent Systems, Tübingen, Germany University of Cambridge, Cambridge, United Kingdom
Amazon Web Services, Tübingen, Germany
{patrik. reizinger, jack. brady, wieland Jrrendell@uiittuueieee (luigi grrsiigeeeee,,,e jvk, bs, gmartius, besserve) &tue. mpg. de, zietld@amazon.dd
jack.brady
Abstract
Variational autoencoders (VAEs) a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of gap the exact (log-) mmrrinall likelihood. While VAES are commonly used for disentangled representation learning, unclear why ELBO maximization would yield such representations, since unregularized maximum likelihood estimation generally cannot invert the data-generating process without additional assumptions. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs the limit of near-deteriinistic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder a commonly used but unproven conjecture- swhcch which we refer to self-consistency. Leveraging self-consistency. we show that the ELBO converges to regularized log- ligeliiel.oood. This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds inductive bias towards decoders with column-ortie oothhgonal Jacobians, which helps recovering the true latent factors. The gap between ELBO and ligelikeli is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning. In experiments on synthetic and image data, show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.
Introduction
Latent Variable Models (LVMs) allow effectively approximate a complex data distribution and to sample from it [3, 48]. Deep LVMs employ neural network (the decoder or generator) to parameterize the conditional distribution of the observations given latent variables, which are typically assumed to be independent. However, Maximum Likelihood Estimation (MLF) of the model parameters computationally intractable. Variational Autoencoders (VAEs) [35, 56), the exact log-like liglliellhoodd/ is substituted with tractable lower bound, the evidence lower bound (ELBO). This objective introduces an approximate posterior of the latents given the observations (the encoder) from suitable variational distribution whose mean and covariance are parametrized by neural networks. The encoder introduced to efficiently train a deep LVM: however, not explicitly designed to extract useful representations [17, 58).
Equal contribution. Code available at: github. honn//patrrik9/imaa--a
1 Senior author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Decoder
X' = =2d
Z R
35" Ozs
(x)
(z)
5 (z0)
Vol (a)
Encoder
(x)
ar D2g
(z)
2 == 9o (2x)
Cov diag (x)),,.. .... .....,,(aa. (
1 = go (20)
Cov
(s(2 (.22 (x)
722 go
9 Ipo (z/|2) 22] pe po =)
Figure 1: Modeling choices in VAËS promote Independent Mechanism Analysis (IMA) [23]. We assume a Gaussian VAE (3), and prove that in the near-deterministcc regime the mean encoder approximatetely inverts the mean decoder, 9º f0 (self-consistency. Prop. 1). Bottom: Closing the gap requires matching the covariances of the variational (I.HS, (2(xx)) and the true posterior (RHS, approximated by o [po [=), cf. $ 3.2 for details). Under self-consisteney, an encoder with diagonal covariance enforces a row-orthogonal encoder Jacobian J (x))-or equivalently, column-orthogonal decoder Jacobian J (z). This regularization was termed Independent Mechanism Analysis (IMA) [23] and shown be beneficial for learning the true latent factors. The connection elucidates unintended benefits of using the ELBO for representation learning.
Nonetheless, VAEs and their variants are widely used in representation learning [25, 1J, where they often recover semantically meaningful representations [39, 8, 34, 5]. Our understanding of this empirical success is still incomplete, since (deep) LVMs with independent latents nonidentifiable from i.i.d. data [29, 42]; different models fitting the data equally well may yield arbitrarily different representations, thus making the recovery of ground truth generative model impossible. While auxiliary variables, weak supervision [28, 31, 21, 43, 72, 19], or specific model constraints [29, 67, 68, 26, 23] help identifiability the mechanism through which the ELBO may enforce a useful inductive bias remains unclear, despite recent efforts [5, 57, 38, 15, 71].
In this work, we investigate the benefits of optimizing the ELBO for representation learning by analyzing VAES in a near-deterministic limit for the conditional distribution parametrized by the nonlinear decoder. Our first result concerns the encoder's optimality in this regime. Previous works relied on the intuitive assumption that the encoder inverts the decoder in the optimum [50. 38, 71]; we formalize this self- coffistesst assumption and prove its validity for the optimal variational posterior in the metr-deterministic nonlinear regime.
Using self- eeff-conistteny, we show that the ELBO tends to regularized log- logllikel rather than to the exact one as conjectured previous work [50]. The regularization term allows VAEs to perform what has been termed Independent Mechanism Analysis (IMA) [23]: it encourages column orthogo- nality of the decoder's Jacobian. This generalizes previous findings based on linearizations or approxi- mations of the ELBO [57, 44, 38), and allows us characterize the gap w.r.t. the log-likelihood in the deterministic limit. Our results elucidate the gap between ELBO and exact log-likelihood as a possible mechanism through which the ELBO implements useful inductive bias. Unlike the unregularized log- ligeeio,,,,,d, the IMA-regularized objective can help invert the data generating process under suitable assumptions [23]. We verify this by training VAEs in experiments synthetic and image data, showing that they can recover the ground truth factors when the IMA assumptions are met.
The contributions of this paper can be summarized as follows:
we characterize and prove self- reff-consistency of VAEs in the near-detent sear-determnnittic regime (i.e., when the
decoder variance tends zero), justifying usage in previous works ($ 3.1):
we show that under self-con self-consistency, the ELBO converges to regularized log- ligeiiilldood ($ 3.2).
and discuss its possible role as a useful inductive bias in representation learning;
we test the applicability of our theoretical results in experiments on synthetic and image data, and show that VAEs recover the true latent factors when the IMA assumptions are met ($ 4).
2 Background
We will connect two unsupervised learning objectives: the ELBO in VAES and the IMA-regularized log-likelihood. Both stem from LVMs with latent variables z distributed according to prior po(z), and mapping from to observations given by conditional generative model pz(x))).

---

Variational Autoencoders. Optimizing the data likelihood pe(x) in deep LVMs-i.e finding decoder parameters 0 maximizing Po( objectives are required. Variational approximations [63] replace the true posterior po (2(2))) by an approximate one, called the variational posterior qs [ (z), which is stochastic mapping x + z with parameters Φ). This allows to evaluate tractable evidence lower bound (ELBO) [35, 56] of the model's log-likoe oog-likelihood that can be defined as
(=) (zzd)dzii intractable in general, so approximate
(1) The two terms in (I) are sometimes interpreted as a reconstruction term measuring the sample quality of the decoder and a regularizer- the Kullback-Leibler Divergence (KL) between the prior and the encoder [36]. The variational approximation trades off computational efficiency with a difference w.r.t. the exact log-likelihood, which is expressed alternatively as (see [17, 36] and Appx. A)
ELBO( 0 Φ) = (n m) [log pe [x) - KL (9)(
ELBO 0, Φ) log po (z) KL (g (z/ (=o
where the KL between variational and true posteriors characterizes the gap: if the variational family of qo (x (z)) does not include po (z|z)) the ELBO will be strictly smaller than log po))..
VAEs [35] rely on the variational approximation in (1) to train deep LVMs where neural networks parametrize the encoder 9 ( (z|zz) and the decoder (a( x). A common modeling choice constrains the variational family of qo (z ax) to a factorized Gaussian with posterior means n. (a+) and variances
of (z)² for the htth factor z4 x., and with diagonal covariance ΣΦ =//=> and the decoder to factorized
Gaussian, conditional on with mean fd (z) and an isotropic covariance in d dimensions,
Zx x ~ (N(u(( of (xx)) x|2~~N
~ N (fo (z) (z),
(3)
The deterministic limit of VAEs. The stochasticity of VAEs makes it nontrivial to relate them to generative models with deterministic decoders such as Independent Component Analysis (see paragraph below), though postulating deterministic regime (where the decoder precision y becomes infinite) is possible. Interestingly, Nielsen et al. [50] explored this deterministic limit and argued that deterministic VAEs optimize exact log-likelihood, similar to normalizing flows [55, 51]. Normalizing flows model arbitrarily complex distributions using a simple base distribution po(=) and nonlinear, deterministic and invertible transformations f". Through a change of variables," 3 the likelihood of the original variables becomes
log pe (x) = lo - log م (z)l-
(4)
The comparison is nontrivial, since VAES contain an encoder and decoder, whereas normalizing flows consist of a single architecture. Nielsen et al. [50] made this analogy by resorting to what we call a self-consistenccy assumption, stating that the VAE encoder inverts the decoder. We define self-consistency in the near-deterministte regime: as the decoder variance goes to zero, i.c. + too.
Definition 1 ENNear-deterministic self seff-coonsstency). For a fixed f assume that mean decoder fo is in- vertible with inverse g and that a map associates each choice of decoder parameters and observation (0,7, x) to an encoder parameter (0, Y, x) + 2(0, x), we say the VAE is self-consistent whenever
_ (x) g g0(x) (xx) and o (xx² 0 7 400
(5)
The encoder parameter map @ reflects the choice of particular encoder model for each (0,7) pair:* in $ 1, we study this problem by introducing and justifying a particular choice for Φ (see also $ 5). This self-consistency assumption appears central to deterministic claims [50, 38], but has not yet been proven. In particular, Nielsen et al. [50] assume that taking the deterministic limit is well-behaved. However, VAEs" near-deterministic properties have not been investigated analytically.
Identifiability, ICA, and IMA. Independent Component Analysis (ICA) [9, 30] models observations as the mixing of latent vector x with independent components through a deterministic function f. i.c., (z). (z) II. Po (2.).5 In ICA the focus is on defining conditions under which the original
"nte that in normalizing flows the change of variables is usually expressed in terms of g - 'both the BLOO and & depends on the decoder precision ry: we will omit this in the following for simplicity the conditional distribution p( x) is therefore degenerate
3

---

latent variables can be recovered from observations i..., the model is "identifiable by design" [31]. The goal is learn an unmixing g" such that the recovered components y = g" (a: are estimates of the true ones up to some ambiguities (e.g., permutation and element-wise nonlinear transformations). Unfortunately, the nonlinear problem is nonidentifiable without further constraints [16, 29]: any two observationally equivalent models can yield components which are arbitrarily entangled, thus making recovery of the ground truth factors impossible. This is typically shown by suitably constructed counterexamples [29, 42], and it was argued to imply impossibility statements for unsupervised disentanglement [42, 65]. Identifiability can be recovered when auxiliary variables [31, 21, 33, 19] available, exploiting temporal structure in the data [28, 24].
Restrictions on the mixing function class (e.g., linear [9]) are another possibility to recover iden- tifiability [29, 67]. Recently, Gresele et al. [23] proposed restricting the function class by taking inspiration from the principle of independent causal mechanisms [52], in an approach termed In- dependent Mechanism Analysis (IMA). IMA postulates that the latent components influence the observations "independently". where influences correspond to the partial derivatives 05 /8x., and their non-statistical independence amounts to an orthogonality condition. While full identifiability has not been proved for this model class, it was shown to rule out classical families of spurious solutions used counterexamples to identifiability of unconstrained non-linear ICA [23, 4]. Mo- roever, Buchholz et al. [4] further demonstrated local identifiability of this function class. Also, IMA constraints were empirically shown [23, 62] to help recover the ground truth through regularization of the likelihodo in (4) with objective LIMA(f (f" 2) log (z()) - CIMA(f", x), where À >> 0 and the regularization term CIMA (f", z) and its expectation (IMA .P)) are given by
CIMA
GM (f,,
log
as" dzz
(=) - log J (z)l; Cf (po))))p)=
-po)= = Epo(x) (CIMa(F) 2)].
,=), (6)
and termed local (resp. global) IMA contrast. When 8d is in the IMA function class (i.e., (f po) vanishes), the objective is equal to the log-likelihood; otherwise, it lower bounds it.
3 Theory
Our theoretical analysis assumes that all the model's defining densities (Po(z), q (2/2x) and pe (z)2)) are factorized. We also assume a Gaussian decoder, matching common modeling practice VAEs.
Assumption 1 (Factorized VAE class with isotropic Gaussian decoder and log-concave prior). We are given a fixed latent prior and three parameterized classes of ERd - Idd mappings: the mean decoder class 0 + fo and the mean and standard deviation encoder classes, do + 4 and db + の s.I.
(i) pu(z) ~ Ih m(zk), with m being smooth and fully supported on R, having bounded non-positivy second-order, and bounded third-order logarithmic derivatives; (ii) the encoder and decoder are of the form in (3), with isotropic decoder covariance 1/~2Ta;
(iil) the variational mean and variance encoder classes are universal approximators: (iv) for all 0, f RRd > Red is a bijection with inverse g and both are C2 with bounded first and second order derivatives.
Crucially, both the mean encoder and the mean decoder can be nonlinear. Moreover, the family of log-concave priors contains the commonly-usee Gaussian distribution as special case. We study the near-deterministic decoder regime of such models, where 7 + +oc This regime is expected to model data generating processes with vanishing observation noise well in line with the typical ICA setting- and is commonly considered in thcoretical analyses of VAEs, e.g., in [50] (which additionally assumes quesi-deterministie encoders), and in [44, 38). Unlike Nielsen et al. [50], we consider a large but finite 7, not at the limit y = 00, where the decoder is fully deterministic. In fact, for any large but finite y, the objective is well-behaved and amenable to theoretical analysis, while the KL-divergence is undefined in the deterministic setting. The requirement in assumption (iv) deviates from common practice in Vhes-where observations are typically higher-dimenssonal-buu but it allows to connect VAΣs and exact likelihood methods such as normalizing flows [50] (see also $ 5).
Due to considering Y + +oo, results are stated in the following "big-O" notation for an integer p:
f(x, g(x,7) 7) O 0,++00 + 7 (1) g(x, m) is bounded as 7 + +00.

---

3.1 Self-consistency
In this section, will prove self-consistency result in the near-deterministic regime. This rests on characterizing optimal variational posteriors (i.e., those minimizing the ELBO gap w.r.t. the likelihood) for a particular point and fixed decoder parameters O. Based on (2), any associated optimal choice of encoder parameters satisfies
お ar 0) arg max ELBO( 0, Φ) arg min KL (0=(2
We call self selfsconsistent ELBO the resulting achieved value, denoted as
ELBO" (a:;
(a:; 0) = ELBO( 0, Φ(x 0))
(8)
The expression in (7) corresponds problem of information projection [10, 48] of po (=)ar) onto the set of factorized Gaussian distributions. This means that given a variational family, we search for the optimal qs (z(2)) to minimize the KL (=)xzx) While such information projection problems are well studied for closed convex sets where they yield a unique minimizer [11]. the set projected onto in our case is not convex (convex combinations of arbitrary Gaussians are not Gaussian), making this problem of independent interest. After establishing upper and lower bounds on the KI. divergence (exposed in Prop. 7-8 in Appx. C.2), we obtain the following self-consistency result.
Proposition 1. (Self-consistency of heta-determiniisicc VAEs] Under Assumption 1, for all æ, 0, as ? - +oc, there exists at least one global minimum solution of These solutions satisfy
~ (x) = 9o (x) + +(/7) and
0(1/2)))))))))))))))()))) for all k.
(9)
Prop. 1 states that minimizing the ELBO gap (equivalently, maximizing the ELBO) w.r.t. the encoder parameters & implies in the limit of large y that the encoder's mean (x)) tends to g (x:). the image of by the inverse decoder. We can interpret this as the decoder "inverting" the encoder. Additionally, the variances of the encoder will converge to zero.
Let us now consider the relevance of this result for training VAEs, i.e., maximizing the expectation of the ELBO for an observed distribution p(x).. While maximization only w.r.t. Φ (7) does not match common wractice-wiich -which is learning 0 and Φ jointly it models this process in the limit of large-capacity encoders. Indeed, in this case, (7) can be solved for each æ as separate learning problem, which entails that the following inequality is satisfied for any parameter choice
E Bm=p(m) EEBOO 0 0)] = (LLOO) 0, 0.
((a:)ELBO(rr 0, क(. 0))da= =: Ba-p(a) [ELBO* (a::: 0)]
(10)
The joint optimization of encoder and decoder parameters thus reduces to optimizing the subset of pairs (0, (xx, ))), and is equivalent to optimizing the expected self-consistent ELBO, that is
maximize Em=p(ar) EEBOO(z; 0, Φ)] < maximize Ex--p(=)) ELBO"
(11)
0.6
This problem reduction is aligned with the original purpose of the ELBO: building a tractable but optimal likelihood approximation. Namely, (i) ELBO* depends on the same parameters the likelihood (x, 7 and 0), (ii) its gap KL [go ((=x))p is minimal. The problem reduction of (1I) allows us compare the optimality of different decoders and Prop. helps addressing the case of near-deterministic decoders.
3.2 Self-consistent ELBO, IMA-regularized log-likelihood and identifiability of VAEs
We want investigate how the choice of q (2(1) and (0(=)) implicitly regularizes the Jacobians of their means (x(z) and 8d (z) in the dearrdetermiiistic regime. Exploiting self-consistency, we are able to precisely characterize how this happens: we formalize this in Thm. 1.
Theorem /VAEs with a near-deterministic decoder approximate the IMA objective] Under Assumption 1, the variational posterior satisfies
o (x))² (-
d'log log Po
(of (x:) + x2 1J (99(x))) (x)) + +0(177),
(12)
d=y
and the self-consister ELBO (10) approximates the IMA IMA-raaulaizzed log-likelihood (6):
ELBO* (x; 0) logpe(x)) CIMA (f g (x)) + O (1//2).
(13)
5

---

logio t
log RHS
logull fl f (12 (z))
-3
1og
1ogno LHS
log ?
Figure 2: Self-consistency (Prop. 1) in VAE training, on log-log plot, cf. 4.1 for details. Left: convergence of o (a)² to O; Center: connecting (x)), .32?.. 7² and the column norms the decoder Jacobian via LHS and RHS of (12); Right: convergence of f to go (x)
Proof is in Appx. B. Below, we provide qualitative argument the interplay between distributional assumptions in the VAE and implicit constraints on the decoder's Jacobian and its inverse.
Modeling assumptions implicitly regularize the mean decoder class fe under self-consistency. In the near deterministic regime, (x) gets close to the pushforward distribution of the prior by the mean decoder f [po[=)] which can be used show that the true posterior Po(z [z(2 [x) = (= 22) (=))
is approximately the pushforward through the inverse mean de- coder g". [po (a) []] (see Appx. A for more details). If we select given latent zy and denote its image by f (zo). then we can locally linearize 9 by its Jacobian Jg" = -go (J (zo)). yielding Gaussian for the pushforward distribution g"~ [po |2] with covariance "/>AgoJJ' sufficient statistics of a Gaussian are given by its mean and covariance, the structure of the posterior covariance E 22/= (which is by design diagonal, cf. (3)) is crucial for minimizing the gap in (2). Practi- cally, this implies that in the zero gap limit, the covariances of (0 and po(z(1)) should match, go
As the
i.e. 1/>2J Jy will be diagonal with entries of (x)² 2 and therefore J go has orthogonal rows. We can express the decoder Jacobian via the inverse function theorem as J la (zo) Jge (S" (=o)) As the inverse of a row-orthogonal matrix has orthogonal columns, S satisfies the IMA principle. Additionally, we can relate the variational posterior's variances to the column-norms of J S as of (x)² = 1/>*ll [J (zo)] ik 11~ as predicted by (12).
Our argument indicates that minimizing the gap between the ELBO and the log-likelihood encourages column-orthogonalln
in J by matching the covariances of q x)) and g" [Po x x=]] When (z x) = Po (z )x) the gap is closed; this is only possible if the decoder is in the IMA class, for which CIMA vanishes and the ELBO tends to an exact fog-likelihood. To the best our knowledge, we are the first to prove this for nonlinear functions, extending related work for linear VAEs [44].
Implications for identifiability of VAEs. While previous works argued that the VAE objective favors decoders with a column-orthogonal Jacobian [57, 38), they did not exactly characterize how: result shows that the self-consistent ELBO tends regularized lieeliie,,,,,d, where the regularization term CIMA explicitly enforces this (soft) constraint. Thus, possibly explains why VAEs are successful in learning disentangled representations: namely, the IMA function class provably rules out certain spurious solutions for nonlinear ICA [23], and the IMA-regularized log-likelihood was empirically shown be beneticial in recovering the latent factors. Thus, we speak about embracing the gap, as its functional form equips VAES with useful inductive bias. While the IMA function class has not yet been shown be identifiable the classical sense such results exist for special cases such as conformal maps (d = [29], generalized by the very recent work [4]). isometries [26] and for closely-related unsupervised nonlinear ICA models [69]. Moreover, Buchholz et al. [4] demonstrate local form of identifiability for the IMA function class. In the following, we empirically corroborate that VAFs: 1) recover the ground truth sources when the mixing satisfies IMA, and thereby 2) achieve unsupervised disentanglement.

---

1.0
0.9
0.8
lo OMA
0.8
MCC
0.6
MCC
MCC
luga: RMMA
0:7
0.4
0.6
0.2
MCC
0.5
5
[-0.40;
-0.16) < 0.00
< 0.34
< 0.58
0.83
log10" 7
log 9MA
Figure 3: Left: CIMA and Mean Correlation Coefficient (MCC) for 3-dimensional Möbius mixings
Right: MCC depending on the volume-presereee linear map's CIMA (7² 1c5)
4 Experiments
Our experiments serve three purposes: 1) demonstrating that self-consistency holds in practice ($ 4.1); 2) showing the relationship of the self-consistent ELBO", the IMA-regularized and unregularized log- ligllikelihood objectives ($ 4.2); and 3) providing empirical evidence that the connection to the IMA function class in VAEs can lead success learning disentangled representations ($ 4.3). More details are provided in Appx. F.
4.1 Self-consistency in practical conditions
Experimental setup. We use a 3-layer Multi-Layyr Perceptron (MLP) with smooth Leaky ReLU nonlinearities [22] and orthogonal weight matrices- intentionally does not belong the IMA class, as our results are more general. The 60,000 source samples are drawn from a standard normal distribution and fed into VAE composed of 3-layer MLP encoder and decoder with a Gaussian prior. We use 20 seeds for each y2 E {1e1: 1c2; 1c3; 1e4; 1c5].
Results. Fig. 2 summarizes our results, featuring the logarithms on each axes. The left plot shows that the posterior variances of (a)) converge zero with a 1/7² rate, as predicted by (9). The center plot shows that the expression for of (x)² corresponds to (12) in the optimum of the ELBO by comparing both sides of the equation. The right plot shows approximate convergence of the mean encodings u (x) to g (z) with a 1/7 rate (see 5). As f is not guaranteed to be invertible, we use instead the optimal encoder and decoder parameters compare fd (44 (z)) æ.
4.2 Relationship between ELBO", IMA-regularized, and unregularized ligliioel
Experimental setup. We use an MLP od with square upper-triangular weight matrices and invertible clement-wist nonlinearities to construct mixing not in the IMA class [23] and fix the VAE de- coder the ground truth such that (4) gives the true data lig-liielihood. This way, we ensure that the unregularized and IMA-regularized log-likelihoods differ and make the claim of Nielsen et al. [50] com- parable ours. With a fixed decoder, the ELBO* depends only on Φ, therefore we only train the en- coder with 72 values from [le1; 1e5] seeds each). Results. Fig. 4 compares the difference of the
12
EyselCuus ELRO")
Eyseloepulet ELBO"
10
Caua
ELBO Gap
logn 7
estimate of ELBO* and the unregularized/IMA regularized ligllikess after convergence over the whole dataset. As the decoder and the data are fixed, log (æ()) and CIMA will not change during training, only the ELBO* does. The figure shows that as 00, ELBO" approaches LIMA z). predicted by Thm. 1, and not log pe(x), stated in [50]- the difference is CIMA-
Figure 4: Comparison of the ELBO', the IMA- regularized and unregularized log-likelihooee over different 72 Error bars are omitted as they are orders of magnitudes smaller
7

---

logie/ie
MCC
kipori MOC
-1.
0.5
Epoch
CIMA and MCC for Sprites [66] during training (-²= 1); Center: true and estimated latent factors for the best trained VAE on Sprites: Right: the corresponding latent interpolations and MCC values (from top to bottom): y- (0.989), ar-posiionn (0.996), scale (0.933), and color (0.989)
Figure 5: Leftt:
4.3 Connecting the IMA principle, 7², and disentanglement
Experimental setup (synthetic). We use 3-dimensional conformal mixings (i.e., the Möbius trans- form [53]) from the IMA class with uniform ground-truth and prior distributions. Our results quantify the relationship of the decoder Jacobian's IMA-contrast and identifiability with MCC [27] and show how this translates to disentanglementt
note that MCC was already used to quantify disentangle- ment [72, 37]. To determine whether a mixing from the IMA class is beneficial for disentanglement, we apply a volume- volume-preserring preserving linear map after the Möbius transform (using 100 seeds) to make CIMA # 0. Other parameters are the same as 4. with the exception of picking the best 72 1e5. Results (synthetic). The left of Fig. 3 empirically demonstrates the benefits of optimizing the IMA-regularized log- lieliiklloood. By increasing y² MCC increases, while CIMA decreases, suggesting that VAEs in the near -etermmirmiiiit regime encourage disentanglement by enforcing the IMA principle. The right plot shows that when the mixing is outside the IMA class, MCC decreases, corroborating the benefits of IMA class mixings for disentanglement,
Experimental setup (image). We train a VAE (not 3-VA with a factorized Gaussian posterior and Beta prior Sprites image dataset generated using the spriteworld renderer [66] with a Beta ground truth distribution. Similar to [32], we use four latent factors, namely, x- and y-position, color and size, and omit factors that can be problematic, such shape (as it is discrete) and rotation (due to symmetries) [57, 37J. Our choice is motivated by [26, 18] showing that this data- getta-genrating process may approximately satisfy the IMA principle.
Results (image). The left of Fig. 5 indicates that VAES can learn the true latent factors and MCC anticorrelated with CIMA, reinforcing the hypothesis that the data-generating process belongs the IMA class. The center plot compares estimated and true latent factors from the best model (scaling and permutation indeterminacies are removed), whereas the right plot shows the corresponding latent interpolations- connecting identifiability (measured by MCC) to disentanglement.
5 Limitations
The near-deterministic regime. Our theory relies on oo; this is the regime where posterior collapse may be avoided [44], and where calculating the reconstruction loss may be possible even without sampling [38]. However, in practice it may be unclear when y2 is large enough. This seems to be pooblem-dependee Moreover, large values of 72 may be harder to optimize due to exploding reconstruction term in (1). This may be one explanation for the slight deviation of Fig. 2, right from our theory's predictions: while convergence of u (z:) to go matches the prediction in Prop. 1, its rate is not precisely the one predicted for the self-consistent ELBO (10). Another cause could be the encoder's 's finite capacity. Nonetheless, have experimentally shown that for realistic hyperparameters, VAEs' behavior matches the predictions of our theory for the near-deterministic regime.
[57, 44], and possibly tied the covariance of the observations [60, 59].
Dimensionality. The setting in 3 requires equal dimensionality for observations x and latents 2, in line with work on normalizing flows [51] and nonlinear ICA [28, 31, 24] (but see, e.g., [33]). For -igh--imensional images, however, it is often assumed that æ lives on a lower-dimensional manifold embedded a higher-dimensional space, where the dimensionality of is greater than z [13]. While our theoretical results do not cover this case, we observe empirically in Fig. 5 that the predictions of our theory remain accurate when observations are high-dimensional images. Extending our theory to this setting could leverage ideas explored in, e.g., [13, 12. 7] and is left for future work.
8

---

The ELBO, the self-consistent ELBO, and amortized inference. There are in principle multiple ways to obtain self-consistency (Defn. 1). Notably, one could simply force the variational mean and variance encoder maps to behave this way; unlike [38], we model the actual behavior of VA trained under EI.BO maximization, and obtain self-consistency as a result. For this, we assume that the optimal encoder, which minimizes the gap between BLBO and log- aigllikeiihood, can be learned. This is not guaranteed in general, since it requires universal approximation capability of the encoder. On the other hand, (10) requires unamortized inference to introduce ELBO', which does not depend As in practice amortized inference may be used to efficiently estimate a single set of & for all æ [61], it can lead to a suboptimal gap to the log- likelieeeioood and discrepancies with our theoretical predictions.
Φ.
Discussion
On disentanglement in unsupervised VAEs. is widely believed that unsupervised VAEs cannot learn disentangled representations [42, 33], motivating work on models with, c.g., conditional priors [33] or sparse decoding [47]. We show that under certain assumptions, ELBO optimization can implement useful inductive biases for representation learning, yielding disentangled representations in unsupervised VAEs. However, while our results are formulated for VAEs, some of the most successful models at disentanglement are modifications thereof-e.g. 3-VAEs [25, 5], with an additional parameter 33 multiplying the KL. in (I). While they deviate from the information projection setting considered in $ 3.1, their objectives are equivalent to the ELBO in a sense described in Appx. A.3, which allows us to derive convergence to the IMA-regularized likelihood objective for 7/V3 +oo. This encompasses the deterministic limit, and also the setting 8 - 0 with constant y described in [38]. Whether this theoretical regime matches common practice remains an open question. Overall, we stress that we uncover one possible mechanism through which VAEs may achieve disentanglement. By connecting to IMA [23], we discuss implications on recovering the ground truth under suitable assumptions, extending uniqueness results presented in [38]. We speculate that our success in disentanglement is probably due to selecting data sets where the mixing in the IMA class (cf. [26. 18), which presumably was not the case in [42].
Characterizing the ELBO gap for nonlinear models. Thm. characterizes the gap between ELBO and true log-likelihne [44] and the results of Dai et al. [14] in the affine case; we also empirically characterize the gap in the deterministic limit in $ 4.2. An unanticipated consequence of this result is consissentt with [44]-VAEs optimize the IMA-regularieed log- -illicclioood in the dearidetermiiistic limit, and not the unregularized one, stated in [50].
for nonlinear VAEs, and extends the linear analysis of Lucas et al.
Extensions to related work. Several papers discuss the (near- deear-ddetemiiiittic regime [50, 57, 38, 13). For example, Nielsen et al. [50] postulate a deterministic VAE with the encoder inverting the decoder. Also Kumar and Poole [38] work in that regime, but without justifying the relationship between the encoder and decoder. Although they show that the choice of po(2) and yo g (=/zz) influences uniqueness (by, e.g., ruling out rotations), this does not imply recovering the true latents. Our approach formalizes (Defn. 1). proves (Prop. 1), and demonstrates the practical fcasibility of ($ 4) the hear-deterministic regime. To the best of our knowledge, all previous work relied on the linear case [44] or a (lincar) approximation and the evaluation of the ELBO around a point to show the inductive bias the decoder Jacobian. However, our main result (Thm. 1) yields a nonlinear equation where the decoder Jacobian can be evaluated at any point and is equipped with a convergence bound. Moreover, the consistency of VAE estimation for identifiable models [33] requires guarantees on q4 [z) [z)z;;; our result helps proving these. Dai and Wipf [13] use non-factorized Gaussian variational posterior and prove in their Thm. 2 (including the dim æ = dim z case) that in the deterministic limit their s-simple VAE can fit perfectly arbitrary observed data (barring few assumptions), while the ELBO gap tends to zero. In contrast, we use a factorized variational posterior; this prevents the BLB gap to vanish in the deterministic limit, except in the special case of a decoder mean in the IMA class fitting the data perfectly. Dai and Wipf [13] take the limit of + +oc (here using 7 as the square root of the decoder precision and not the decoder variance as used in [13)) to relate encoder and decoder properties in this limit in their Thm. 5, similarly to Prop. 1. In contrast to our nonlinear analysis, this derived when optimizing w.r.t. both encoder and decoder parameters, and with a mon-factor encoder assumption, leading to fundamentally different behavior of the solutions in the deterministic limit. The work done by Sliwa et al. [62], simultaneously to ours, showcases an extensive empirical study highlighting that the IMA contrast allows distinguishing true and spurious solutions for broad range of cases and outperforms standard regularizers such as weight decay. We discuss extended connections to the literature in Appx. D and Appx. E.
9

---

Covariance structure and IMA. We have shown that specific choices for encoder and decoder covariances regularize the decoder Jacobian, such that closing the ELBO gap constrains the decoder to
belong to the IMA class. Following our intuition (Fig. 1), assuming factorized gø (2 (2) and isotropic
(o 22, IMA holds only for the decoder; since in the other direction the pushforward of qΦ (z (z)
through f has covariance J 5 (z) Σ go (z)", which cannot be used to make row orthogonality statements on J y (z) in the general case. Additionally, we conjecture that assuming an isotropic encoder would constrain IMA hold in both encoding and decoding directions (as both J y0 (z) and such that the resulting decoder mean is constrained to have
J (z) need to be column-orthogone orthogonal columns of equal norms, which is a defining property of conformal maps [4]. On the other hand, conjecture that if the observation model is not isotropic, but the encoder model is, IMA would only tend to be enforced for the mean encoder Jacobian, converging to the inverse decoder mean in the deterministic limit.
Implications for recovering the true latent factors using unsupervised VAEs. Convergence of the ELBO to the rMA-eauulrizzed log- likelihood suggests that unsupervised VAEs may recover the true factors of variation according to current identifiability results of the IMA class [4]. This is based on the following reasoning: If the ground truth generative model belongs to the IMA class, unsupervised learning of the model with an infinite capacity VAE will, in the deterministic limit, ensure a solution that perfectly fits the data and whose decoder mean is also in the IMA class (by joint optimization of both the likelihood and the regularization term). Identifiability of the IMA class implies that the VAE will learn the true decoder (up to acceptable ambiguities)) then, since self-consistency guarantees that the encoder inverts the decoder, the encoder infers the ground truth generative factors associated to observations. Although striet identifiability for all functions in the IMA class remains to be proven, three concurrent papers provide guarantees that go towards identifiability: Leemann et al. [41] proves identifiability for a subset of the IMA class in the context of concept discovery; Zheng et al. [70] shows identifiability of nonlinear ICA by assuming a specific sparsity structure of the decoder Jacobian (called structural sparsity); whereas Buchholz et al. [4] introduce the concept of local identifiability and proves that IMA is locally identifiable.
Moreover, mentioned in the above paragraph, we suspect that closing the ELBO gap with an isotropic encoder (while the encoder in Thm. is only constrained to have diagonal covariance) constrains the decoder to be a conformal map. This is an interesting constraint, as nonlinear ICA with conformal mixings are identifiable: the two-dimensional case was first addressed with some additional constraints in [29]. while the general case (in arbitrary dimension) was shown to rule out certain spurious solutions for conformal mixings [23], and finally proven to be identifiable by Buchholz et al. [4] in concurrent work. Hence, we conjecture that given a ground truth generative model with a conformal map from latent to observation space, and an unsupervised VAEs with isotropic Gaussian encoders and decoders, the true latent factors can be recovered.
Conclusion. We provide a theoretical justification for VAEs' widely-used self-consistency assumption in the determmeeiiiiiticc regime of small decoder variance. Using this result, show that the self-consistent ELBO converges to the IMA-regularized log-likelihood, and not to the unregularized one. Thus, can characterize the gap between ELBO and true logllikelihood and reason about its role as an inductive bias for representation learning in nonlinear VAES. We characterize a set of assumptions under which unsupervised VAEs can be expected to disentangle and we demonstrate this behavior in experiments on synthetic and image data.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their suggestions. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039A & 011S L1IS18, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Dominik Zietlow and Patrik Reizinger. Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program.
10

---

References
[1] Alexander A. Alemi, Ben Poole, lan Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. Fixing broken ELBO. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmåsssan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 159-168. PMLR, 2018. 2
[2] Khaled Alyani, Marco Congedo, and Maher Moakher. Diagonality measures of hermitian positive-definit matrices with application to the approximate joint diagonalization problem. Linear Algebra Appl., 528:290-333 320, September 2017. ISSN 0024-3795. doi: 10. 10.1016/j.laz 2016.08.031. URL https://doii org/ httg/10.1166.. 1016/j 2016. .8. 031. 23, 24, 42
[3] Christopher M Bishop. Pattern Recognition and Machine Learning, volume 4. Springer New York, 2006. doi: 10.10077978.00333--55522
URL https://doi http:///o..rr//////// org/ 1007/
978-0- 387-44528-00
[4] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function Classes for Identifiable Nonlinear Independent Component Analysis. Advances in Neural Information Processing Systems, 36, 2022. 4, 6, 10
[5] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint 1804: 03599, 2018. 2,9
[6] Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? J. ACM, 58(3,, jun 2011. ISSN 0004-541 1. doi: 10. 1155/197092219903399 URL https://doi. org/10. 1145/1970392 1970395. 41
[7] Anthony L Caterini, Gabriel Loaiza-Ganent Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34, 2021.
[8] Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating Sources of Disentangiement in Variational Autoencoders. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2615 2625, 2018. 2. 21
[9] Pierre Comon Independent component analysis, a new concept? Signal processing, 36(3): 287-314, 1994. 3,
[10] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley series in telecom- munications. John Wiley & Sons, Inc., New York, 1991. ISBN 0471062596, 0471 0471200611. doi: 10.1002/04712206111 URL https:t////// //toi//oi/// org/ 10. 1 5
[11] I. Csiszar and F. Matus. Information projections revisited. IEEE Trans. Inf. Theory, 49 (6):1474-1490, June 2003. ISSN 0018-9448. doi: 10.1109911.2003.81000 URL https:
wtt.:///....t........... 0.0038806333 5
[12] Edmond Cunningham and Madalina Fiterau. A change of variables method for rectangular matrix- products. In International Conference Artificial Intelligence and Statistics, pages 2755-2763. PMLR, 2021. 8
[13] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2018. 8, 9, 21, 41
[14] Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with robust PCA and the role of emergent sparsity in variational autoencoder models. The Journal of Machine Learning Research, (9())1573-1614, 2018.
[15] Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 1 119, pages 2313-2322. PMLR, 13-18 Jul 2020. URL httph:ttp//t hattps://proceedingg mlr, press/v119/dail dat//ai22ccc html. 2
11

---

[16] George Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, page
231, 1951. 4
[17] Carl Doersch. Tutorial on Variational Autoencoders. ArXiv preprint, abs/ abs/1606.0590% 2016. 1, 3, 22
[18) David I. Donoho and Carrie Grimes. Image Manifolds which are Isometric to Euclidean Space. J. Math. Imaging Vis., 23(1):5-24, July 2005. ISSN 0924-9907, 1573-7683, doi: 10. 10.10077/10851--005440000 9.46
URL nttps://doi. http://00..rr1111100000110011000 000-1099--0000--0------0000 8,
[19] Elisabeth Gassiat, Sylvain Le Corff, and Luc Lchéricy. Deconvolution with unknown noise distribution is possible for multivariate signals. Ann. Statiist. 50(1). February 2022. ISSN 0090-5364. doi: 10.121421--as2211 106. URL https: org/ 10. 11..1.1//1--1011/
2,
[20] Partha Ghosh, Mehdi S. M. Sujjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
21, 41
[21] Luigi Gresele, Paul K. Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA. In Proceedings of the 35th Conference Uncertainty in Artificial Intelligence (UAI), volume 115 of Proceedings of Machine Learning Research, pages 217-227. PMLR, July 2019. URL https: http://oooeeedn.....eeee mlr. preseepesaaeee.ee //1 wwtl///greeeeee00000 html. 2,
[22] Luigi Gresele, Giancarlo Fissore, Adrián Javaloy, Bernhard Schölkopf, and Aapo Hyvärinen. Relative gradient optimization of the Jacobian term in unsupervised deep learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 6-, 2020, virtual, 2020. 7, 43, 44, 45
[23] Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanisms analysis, a new concept?
Advances Neural Information Processing Systems 34 (NeurIPS 2021), pages 28233-28248. Curran Asso- ciates, Inc., December 2021. URL https://proceeddngg wwahggg.neeer newrips...... cc/paper/0021/ Paper pdf. 2, 4. 6. 7, 10, 23, 43
[24] Hermanni Hälvä and Aapo Hyvärinen. Hidden Markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of the Thirty-Sixth Conference Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124 of Proceedings of Machine Learning Research, pages 939-948.
AUAI Press, 2020. 4,
(25) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. B-VAE: Learning Basic Visual Concepts with a Con- strained Variational Framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. 2, 9, 21
[26] Daniella Horan, Eitan Richardson, and Yair Weiss. When unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34, 2021. 2, 6, 8, 9, 46
[27] Aapo Hyvärinen and Hiroshi Morioka. Unsupervised Feature Extraction by Time- Contrastive Learning and Nonlinear ICA. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3765-3773, 2016. 8
[28] Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort
12

---

Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pages 469. PMLR, 2017. 2, 8
[29] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429-439, April 1999. ISSN 0893-6080. doi: 10. (00006008800000088880000000
URL bttt:p::/ddoi.rrge 1ttp://000.00//00000000000000000
00140-3. 4, 6, 10, 24, 41
[30] Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent Component Analysis. John Wiley & Sons, Inc., New York, May 2001. ISBN 047 047140540X, 0471221311 doi: 10.1002/0471221311. URI. https: //o:::///ooooo otog 000.100000077221117.
[31] Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner Nonlinear ICA using auxiliary variables and generalized contrastive learning. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference
Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 859-868. PMLR, 2019. 2, 4,8
[32] Jack Brady and Geoffrey Roeder. iSprites: A Dataset for Identifiable Multi- Object representation Learning. In ICM1.2020: Workshop Object-Oriente Learn- ing, 2020. URL https:/ /ttpp:://ithub...... thub. wwt. hoom//oowworkkhooooolorrrrot master/pdf master/pat/O0L_225 /00L_25....2.. 2df.f. 8, 46
gitttgitthhh io/blob/
[33] llyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 2207-2215 PMLR, 2020. 4, 8,9
[34] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2654-2663 PMLR, 2018. 21
[35] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. 1, 3, 21
[36] Diederik P. Kingma and Max Welling. An Introduction to Variational Autoencoders. Foun- dations and Trends@ in Machine Learning. 12(4):307-39) 2019. ISSN 1935-8237, 1935- 8245. doi: 10. 10.1561/2200000056 URL https: //doi. org/10 15010000000//22000055 arXiv: 1906.02691. 3
[37] David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan M. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. 2021. 8, 46
[38] Abhishek Kumar and Ben Poole. On Implicit Regularization in 8 3-VAEs. In International Conference on Machine Learning, pages 5480-5490. PMLR, 2020. 2, 3, 4, 8, 9, 21, 23, 41, 46
[39] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[40] Abhishek Kumar, Ben Poole, and Kevin Murphy. Regularized Autoencoders via Relaxed Injec- tive Probability Flow. Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sieily, Itly]] volume 108 of Proceedings of Machine Learning Research, pages 4292-4301, PMLR, 2020. 41
13

---

[41] Tobias Leemann, Michael Kirchhof, Yao Rong, Enkelejda Kasneci, and Gjergji Kasneci. Dis- entangling Embedding Spaces with Minimal Distributional Assumptions. June 2022. URL wttp:i..rrgrbbb////eeeee
13872. wnXiv:2206.13877 [es, stat]. 10
[42] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch. Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 41 14 4124. PMLR, 2019. 2,
[43] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervisee disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6348 6359. PMLR, 2020. 2
[44] James Lucas, George Tucker, Roger B. Grosse, and Mohammad Norouzi. Don't t blame the ELBO! linear VAË perspective posterior collapse. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Gamett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9403 9413, 2019. 2, 4, 6, 8, 9, 19, 41, 42, 46
[45] Jerrold E. Marsden and Anthony Tromba. Vector calculus. W.H. Freeman and Company, New York, sixth edition edition, 2012. ISBN 978- 128-1-4292-500--4 39
[46] Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentangle- ment in variational autoencoders. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4402 4412. PMLR, 2019. 41
[47] Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable variational autoencoders via sparse decoding. arXiv preprint arXiv:21 10. 10804, 2021. 9
[48] Kevin P Murphy, Machine learning: A probabilistic perspective. MIT press, 2012. 1.
[49] Akira Nakagawa, Keizo Kato, and Taiji Suzuki. Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding, In Proceedings of the 38th International Conference on Machine Learning, pages 7916-7926. PMLR, July 2021. ISSN: 2640-3498. 40
[50] Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. SurVAE flows: Surjections to bridge the gap between VAES and flows. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Fiorina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. 3, 4, 7, 41
[51] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. ArXiv preprint, abs/1912.02762, 2019. 3,
[52] J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA. USA. 2017. 4
[53] Robert Phillips. Liouville's 's theorem. Pac. J. Math., 28(2):397 405,, February 1969, ISSN 0030-8730, 0030-8730. doi: 10.2140/pjm..966 pjm. 196.1999... 28. 397.997.4 8, 43, 45
http:ppm..19......o0 1969.28.397. URL https://doi. orw.oogg///o 2140/
[54] Lutz Prechelt. Early stopping but when? In Neural Networks: Tricks of the Trade, volume 1524 of LNCS, chapter 2, pages 55-69. Springer-Verlag, 1997. 43, 45, 46
14

---

[55] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-11 I July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1530-1538, JMLR.org, 2015.
[56] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 3 31th International Conference Machine Learning. ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1278-1286. JMLR.org. 2014. 1,3
[57] Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue PCA directions (by accident). In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12406- 12405-12415, IEEE, June 2019. doi: 10. 109/cvp::001..01229. URL https://doi. org/10.//// 10. 1109/ cvpr 2019 01269. 2, 6. 8, 9, 21, 40, 41, 46
[58] Paul K. Rubenstein. Variational Autoencoders are not autoencoders, Jan 2019. URL http: http:///ei...oo.uurraraaattaiaae www.kaaaaaiiinna.a..........a.a. autonccoorsss are-not- autoencoders/. o
// //pplluuensteennccmmmm
[59] Olch Rybkin, Kostas Danillidis, and Sergey Levine. Simple and effective VAE training with calibrated decoders. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9179-9189, PMLR, 2021. 22, 46
[60] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. 2021. 8, 46
[61] Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. Advances in Neural Information Processing Systems, 31, 2018. 9
[62] Joanna Sliwa, Shubhangi Ghosh, Vincent Stimper, Luigi Gresele, and Bernhard Schölkopf. Probing the Robustness Independent Mechanism Analysis for Representation Learning, July 2022. URL http: bttt:://axxiz org/ 2207 06137. ScXiv:2207.06133 [cs, stat]. 4,
[63] Michael Struwe. Variational Methods, volume 991. Springer Berlin Heidelberg, 2000. ISBN 9783662041963. 9783662041949, doi: 10.1007/978-3-66--0110 URL https://doi.
[64] Michael E. Tipping and Christopher M. Bishop. Probabilistic Principal Component Analysis. J. R. Stat. Soc. B, 61 (1))61111622) August 1999. ISSN 1369-7412, 1467-9868. doi: 10.1111/ 117-99988.0099 URL. htt.tttttt//i..ooo
1111/ 1467-4469888 00196.
[65] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Con- ference Learning Representationss ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[66] Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Lerchner. Spriteworld: A flexible, configurable reinforcement learning environment. https://github.com/// www./eemiin//prrreoor.....o 2019. URL https://githinn http///gttttutt..///g com/ wtw deepmind// spriteworld/ 8, 46
[67] Kun Zhang and Laiwan Chan. Minimal nonlinear distortion principle for nonlinear Independent Component Analysis. Journal of Machine Learning Research, (Nov):2455-22) 2287,2477,44 2008. 2, 4
[68] Kun Zhang and Aapo Hyvarinen. On the Identifiability of the Post-Nonlinear Causal Model. arXiv:1005.2599 [cs, stat), 2012. arXiv: 1205.2599 2
[69] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA with unconditional priors. In /CLR2002 Workshop on the Flements of Reasoning: Objects, Structure and Causality, 2022. 6
15

---

[70] Yujia Zheng, Ignavier Ng, and Kun Zhang, On the Identifiability of Nonlinear ICA: Sparsity and Beyond, June 2022. URL http:://arzta www.vvv.rg//aaaee htwgg/bb//200...oo 07751. arXiv:2206.07751 [es, stat]. 10
[71] Dominik Zietlow, Michal Rolinek, and Georg Martius. Demystifying Inductive Biases for (Beta-) VAE Based Architectures. In International Conference on Machine Learning, pages 12955-12954. PMLR, 2021. 40
[72] Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12979-12990., PMLR, 2021. 2,
Acronyms
ELBO evidence lower bound IMA Independent Mechanism Analysis
MLLE Maximum Likelihood Estimation MLP Multi-Layer Perceptron
MSE Mean Squared Error
i.idd independent and identically distributed ICA Independent Component Analysis
PCA Principal Component Analysis
PPCA Probabilistic Principal Component Analy- sis
KL Kullback-Leibler Divergence
LVM Latent Variable Model
SVD Singular Value Decomposition
MCC Mean Correlation Coefficient
VAE Variational Autoencoder
Nomenclature
Independent Mechanism Analysis
Cima global IMA contrast
scalar field
D general diagonal matrix
O orthogonal matrix
Y reconstructed sources
LIMA IMA Ioss function
CIMA local IMA contrast Variational Autoencoder
V weight matrix a linear encoder
W weight matrix of linear decoder
4 th (x) optimal mean of 9 (2 x>)
4 th (x) mean q g (x)x))
o parameters of the variational posterior (z(xxx
è (z)² optimal variance of (z(()
parameters of the decoder Pe (=(()z
square root of the precision of the VAE decoder
d covariance matrix of (z)()
Ca 3-VAE loss function
S decoder
g inverse decoder
Φ optimal parameters of the variational posterior (2(=)))
p(æ) data distribution
(=) latent prior distribution
po(x(x) true posterior distribution of the decoded samples of the VAE, mapping x + parametrized by 0
po(a:) marginal likelihood Po a z)) conditional distribution of the decoded samples of the VAF, mapping - m, parametrized
by 0
16

---

(0(z(z) xz) variational posterior of the VAE, mapping a + 2 parametrized by Φ
97 (2 x:) optimal variational posterior of the VAE, mapping æ parametrized by
M (x) optimal mean of (=(=)²) in dimension k
Mg (z) mean of (=(=))) in dimension
( optimal variance of ( (2)2a))) in dimension k:
(x)² variance of (z(xx) in dimension k
inverse decoder component
H Hessian matrix
I d- d-diimensional identity matrix
Jacobian matrix
Σ covariance matrix
2 observation vector
latent vector
X observation space
d dimensionality of the observation space X
latent single component
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See $ 3, $ 4, 6
(b) Did you describe the limitations of your work? [Yes] See 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appx. H
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We use no data about human subjects.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See 3 (b) Did you include complete proofs of all theoretical results? [Yes] See 3, Appx. B
3. If you experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL.)? [Yes] See the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $ 4, Appx. F
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See $
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appx.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See $ 4 and the supplementary material
(b) Did you mention the license of the assets? [Yes] See the supplementary material
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We included the code and the experiment logs.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
18


Reviews:
[{"review_id": "pBw5dfhFS76", "reviewer": "Reviewer_tFG1", "summary": "The paper analyzes the gap between ELBO and exact log-likelihood under the self-consistency assumption that the encoder is the maximizer of ELBO with the decoder parameters being fixed. The authors theoretically show that the gap can be characterized with independent mechanism analysis (IMA) in a near-deterministic regime. Experiments empirically support the theoretical results.", "questions": "- What does the limitation that dimensionality of $x$ and $z$ should be the same come from? It is unclear to the reviewer whether $c_\\text{IMA}$ in Section 4.3 is valid for the theory part.", "limitations": "Limitations are addressed, and no negative societal impact is observed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strong points\nUnderstanding the gap between ELBO and exact log-likelihood is important. The analysis/interpretation of the gap from the perspective of IMA is new and interesting. The analysis provides new insight into disentanglement in (usual) VAE. In addition, the empirical results successfully support the theoretical parts.\n\n### Weak points\n- In the experiments, $\\gamma$ is always fixed. Since $\\gamma$ is optimized as well as encoder and decoder in the current VAEs, I would like to have seen a case with trainable $\\gamma$.\n- How large the gap between ELBO optimized in the usual way and ELBO^* is not clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RAn8HdYVsbb", "reviewer": "Reviewer_UUBT", "summary": "This work proves that for Guassian VAEs with isotropic Gaussian encoders, when the support of the data $\\mathbf{x}\\in\\mathbb{R}^d$ has full dimensionality $d$, \n\n- The ELBO optima has decoder variance tend to zero (\"self-consistency\", Sec 3.1), and that\n- ELBO converges to a regularized likelihood objective where the regularization encourages the decoder Jacobian to have orthogonoal columns (Sec 3.2).\n\nThe latter is connected to the recent mechanism independence formulation for causal representation learning, and the authors provide experiments demonstrating the recovery of the true latents when the mechanism independence assumptions are met.", "questions": "I think the paper should be revised to better position the existing results, or to improve them in the aforementioned directions.  Some of the discussions in Appendix D could also be moved to the main text.", "limitations": "Some limitations, e.g., the dimensionality issue, are mentioned, but their implications not fully discussed (see above).  Others are adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The established connection of ELBO-trained VAEs to mechanism independence is interesting, and the experimental results can be of value to the causal representation learning community (I'm not deeply familiar with that literature).  The theoretical results are new, in the sense that they have not been established in this exact setting.\n\nThat being said, I feel that the significance of the results, in the broader context of understanding representation learning in ELBO trained VAE, is overstated.  It has been known that \n\n- Linear-Gaussian VAE trained with ELBO recover the principal components, and thus their decoder (Jacobian) have orthogonal columns ([4, 60]; Dai et al, 2018);\n- Nonlinear Gaussian VAE trained with ELBO are regularized towards having decoders with orthogonoal Jacobian columns ([43, Section 5, 5.1]; Nakagawa et al, 2021);\n\nDai and Wipf (2019) has also shown that near-deterministic VAEs constitute a family of global optima of the Gaussian VAE ELBO, and empirically verified that it is recovered by common VAE architectures.\n\nIt is true that no results have been provided in the precise setting of this work, but I think this has more to do with the restrictiveness of the setting here: this work assumes the data distribution has full support (by assuming the optimal ELBO is finite), and it also uses an isotropic encoder with output dimensionality $d_z = \\mathrm{dim}\\\\;\\mathbf{x}$. The former is inappropriate for image or other structured inputs which are typically supported on low-dimensional manifolds embedded in $\\mathbb{R}^{d_x}$, while the latter may prevent the automatic adaptation to the correct latent dimensionality (Dai and Wipf, 2019).  Relaxation of either restriction introduces technical challenges, so I don't think readers familiar with previous work on VAE inductive bias will find the results here surprising.  The related works are only sparingly discussed in the main text, which does not help to put the contributions of this work in context.\n\nI would also note that self-consistency may cease to hold beyond the $d_z = \\mathrm{dim}\\\\:\\mathbf{x}$ regime.  Suppose for simplicity that $\\mathrm{dim}\\\\:\\mathbf{x}=1$.  Then any input $x$ can be encoded up to precision $O(2^{-d_z})$ by setting $q(z_i | x) = \\mathcal{N}(C s_i(x), 1)$, where $C>1$ is a moderate constant, and $s_i(x)\\in\\\\{1, -1\\\\}$. Then we have $\\mathrm{sgn}(z_i) = s_i(x)$ with high probability, for $z\\sim q(z|x)$, and the sign decoder can be easily approximated with NN decoders.  More generally, we can always take an input dimensionality $i\\in [\\mathrm{dim}\\\\:\\mathbf{x}]$ and approximate it with precision $\\gamma$, using up to $O(\\log \\gamma)$ latent dimensions which are *never deterministic in the limit*.  While this issue does not prevent the use of ELBO-trained VAEs for the recovery of independent mechanisms, thus the contribution of this work in this context, it does make it less interesting in the broader context of understanding VAEs (which are often overparamterized in a similar nature).\n\n## References\n\n- Nakagawa et al (2021): Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding, in ICML.\n- Dai and Wipf (2019): Diagnosing and Enhancing VAE Models, in ICLR.\n- Dai et al (2018): Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models, in JMLR.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "17BhzYOaQ-6", "reviewer": "Reviewer_8jNM", "summary": "The paper asks why VAEs learn useful representations despite the non-identifiability of the latent variables (in a nonlinear data generating process) using unregularized maximum likelihood estimation. The paper's main contribution is to show that VAEs with near-deterministic decoders do not converge to the marginal likelihood but to an IMA-regularized likelihood. The IMA-regularization (based on previous work) enforces a column-orthogonality condition on the Jacobian of the decoder, which can be beneficial for recovering the true latent factors when the data generating process satisfies the IMA assumption. Experimental results on synthetic numerical data corroborate that \n\n- the encoder approximately inverts the decoder; \n- the ELBO converges to the IMA-regularized likelihood if the true decoder is used, even when the generating process violates IMA; \n- the encoder recovers the true latent factors if the data generating process satisfies the IMA assumption. The recovery is further corroborated by experiments on synthetic image data (Sprites).\n\nThe analysis focuses on near-deterministic decoders, for which the paper shows that the optimal encoder approximately inverts the decoder. This result is interesting in its own right as it formalizes an existing hypothesis and is necessary for deriving the convergence of the ELBO to the IMA-regularized likelihood.", "questions": "The abstract calls it a paradox that VAEs seem to learn \"useful\" representations despite not inverting the data generating process. As the usefulness of a representation depends on the downstream task, one could argue that a representation can be useful, even if it does not invert the data generating process. Maybe you could clarify this part in the abstract?\n\nYou assume that the variational posterior has a diagonal covariance matrix by design (e.g., in Figure 1 and Section 3.2).  However, is it safe to assume that the _empirical_ covariance is diagonal? Would a non-diagonal empirical covariance affect the IMA regularization? Similarly, what if the empirical covariance of the decoder is not isotropic?\n\nIn Definition 1, did you mean \"mean decoder\" instead of \"mean encoder\"?  In Definition 1 and Assumption 1, you assume that the mean decoder $f^{\\theta}(z)$ is invertible. Is this assumption justified? I find it hard to imagine that the assumption is fulfilled at initialization time, even if the number of latent dimensions is equal to the number of input dimensions. But maybe there is an argument to be made for a model that reconstructs the inputs sufficiently well?\n\nIn section 4.1, to test the convergence of $\\mu^{\\hat{\\phi}}(x)$ to $g^{\\theta}(x)$, why do you not measure the MCC? Why is the comparison in the right subplot of Figure 2 more suitable?\n\nFor the numerical experiments you use different types of weight matrices (orthogonal, square upper-triangular, Möbius transform). Could you please clarify if the choice of weight matrix influences whether the mixing is in the IMA class?\n\nMinor points:\n\n- For the experiment in Section 4.1, you say that you use a uniform prior, but Table 1 lists a Gaussian prior.\n- In line 79, you introduce the subscript $k$, but do not explain its purpose.", "limitations": "Overall, the paper is very transparent about the limitations.\n\nOne point for which I would have appreciated a bit more detail is the distinction between $\\beta$ and $\\gamma$. It is not immediately clear why self-consistency is harder to prove for $\\beta$-VAEs, especially since you write that $\\beta$ and $\\frac{1}{\\gamma^2}$ play a similar role in the objective (Section A.3).\n\nThe second point which could be addressed is the practical significance of the results (see Strengths and Weaknesses).", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Formalization of the mechanism by which VAEs can learn disentangled representations and comprehensive theoretical analysis with a detailed explanation.\n- Thorough empirical analysis with controlled synthetic experiments.\n- Clear and transparent contextualization of previous work, positioning of the paper, and contribution statement.\n- Overall, a solid contribution that advances the understanding of representation learning with VAEs.\n\n**Weaknesses**\n\n- The paper does not discuss the practical significance of their results sufficiently well. For example: How much IMA violation would be acceptable to still recover the true factors in more tangible settings (e.g., in more complex variants of the Sprites dataset)? Can practitioners control or validate the degree of IMA violation when their goal is to recover the true but unknown latent factors? Even though the paper presents basic research, more experiments in practical settings could help readers better understand the practical significance of the theoretical results.\n- The paper is dense and has a lengthy appendix. Subjectively speaking, the paper might be easier to parse and understand if it focused more on one part of the story.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept