Identifiability of deep generative models without auxiliary information
Authors: Bohdan Kivva
Goutham Rajendran
Pradeep Kumar Ravikumar
Bryon Aragam
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep generative models
variational autoencoders
identifiability
nonparametric statistics
latent variable models
representation learning

Abstract: We prove identifiability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, or conditioning in the latent space. Specifically, we show that for a broad class of generative (i.e. unsupervised) models with universal approximation capabilities, the side information $u$ is not necessary: We prove identifiability of the entire generative model where we do not observe $u$ and only observe the data $x$. The models we consider match autoencoder architectures used in practice that leverage mixture priors in the latent space and ReLU/leaky-ReLU activations in the encoder, such as VaDE and MFC-VAE. Our main result is an identifiability hierarchy that significantly generalizes previous work and exposes how different assumptions lead to different ``strengths'' of identifiability, and includes certain ``vanilla'' VAEs with isotropic Gaussian priors as a special case. For example, our weakest result establishes (unsupervised) identifiability up to an affine transformation, and thus partially resolves an open problem regarding model identifiability raised in prior work. These theoretical results are augmented with experiments on both simulated and real data.

Identifiability of deep generative models without auxiliary information
Bohdan Kivva" University of Chicago bki akivaabuchicago.o edu
Goutham Rajendran" University of Chicago
Pradeep Ravikumar Carnegie Mellon University pradeeprecs cmu, edu
goutham@uchicago edu
Bryon Aragam University of Chicago bryon@chicagoboothh. edu
Abstract
We prove identifiability of broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoen- coders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, conditioning in the latent space. Specifically, we show that for a broad class generative (i.e. unsupervised) models with universal approximation capabilities, the side information is not necessary: We prove identifiability of the entire generative model where we do not observe and only observe the data The models we consider match autoencoder architectures used in practice that leverage mixture priors in the latent space and ReL.U/leaky-ReLU activations in the encoder, such as VaDE and MFC-VAE. Our main result is an identifiability hierarchy that significantly generalizes previous work and exposes how different assumptions lead to different "strengths" of identi- fiability, and includes certain "vanilla" VAEs with isotropic Gaussian priors as a special case. For example, our weakest result establishes (unsupervised) identifia- bility up to an affine transformation, and thus partially resolves open problem regarding model identifiability raised prior work. These theoretical results are augmented with experiments on both simulated and real data.
Introduction
One of the key paradigm shifts in machine learning (ML) over the past decade has been the transition from handcrafted features to automated, data-driven representation learning, typically via deep neural networks. One complication of automating this step in the ML pipeline is that is difficult to provide guarantees on what features will (or won't) 1) be learned. As these methods are being used in high stakes settings such as medicine, health care, law, and finance where accountability and transparency are not just desirable but often legally required, it has become necessary to place representation learning on rigourous scientific footing. In order do this, crucial to be able discuss ideal, target features and the underlying representations that define these features. As result, the MI. literature has begun move beyond consideration solely of downstream tasks (e.g. classification, prediction, sampling, etc.) in order to better understand the structural foundations of deep models.
Deep generative models (DGMs) such variational autoencoders (VAEs) [37, 56] are prominent example of such a model, and are a powerful tool for unsupervised learning of latent representations, useful for variety of downstream tasks such as sampling, prediction, classification, and clustering.
Equaa contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite these successes, training DGMs is an intricate task: They are susceptible to posterior collapse and poor local minima [13, 24, 69, 72], and characterizing their latent space remains a difficult problem [e.g. 40, 67]. For example, does the latent space represent semantically meaningful or practically useful features? Are the learned representations stable, or are they simply artifacts of peculiar choices of hyperparameters? These questions have been the subject of numerous studies in recent years [e.g. 7, 12, 42, 45, 48, 60), and in order to better understand the behaviour of these models and address these questions, the machine learning literature has recently turned its attention to fundamental identifiability questions [14, 35, 69]. Identifiability is a crucial primitive in machine learning tasks that is useful for probing stability, consistency, and robustness. Without identifiability, the output of a model can be unstable and unreliable, in the sense that retraining under small perturbations of the data and/or hyperparameters may result in wildly different models. In the context of deep generative models, the model output of interest is the latent space and the associated representations induced by the model.
In this paper, we revisit the identifiability problem deep latent variable models and prove a surprising new result: Identifiability is possible under commonly adopted assumptions and without conditioning in the latent space, or equivalently, without weak supervision or side information in the form of auxiliary variables. This contrasts a recent line of work that has established fundamental new results regarding the identifiability of VAEs that requires conditioning on an auxiliary variable u that renders each latent dimension conditionally independent [35]. While this result has been generalized and relaxed in several directions [10, 22, 23, 36, 39, 43, 50, 61, 73]. fundamentally these results still crucially rely on the side information We show that this is in fact unnecessary-cone confirming existing empirical studies [e.g 18, 70]-and do so without sacrificing any representational capacity. What's more, the model we analyze is closely related to deep architectures that have been widely used in practice [16, 18, 32, 33, 41, 41, 44, 71]: We show that there is good reason for this, and provide new insight into the properties of these models and support for their continued use.
c
Overview More specifically, we consider the following generative model for observations æ:
f()) +E, = (II...., R", 2 (21,.... (21,..., 2m) E R
where the latent variable 2 follows Gaussian mixture model (GMM), 2 f: T Be is piecewise affine nonlinearity such as a ReLU network, and Æ E IR' is independent, random noise." We do not assume that the number of mixture components, nor the architecture of the ReLU network, are known in advance, nor do we assume that = has independent components. Both the mixture model and neural network may be arbitrarily complex, and we allow for the discrete hidden state that generates the latent mixture prior to be high-dimensional and dependent. This includes both vanilla VAEs (i.e. with standard isotropic Gaussian prior) and classical ICA models (i.c. for which the latent variables are mutually independent) as special cases. Since both and f are allowed to be arbitrarily complex, the model (1) has universal approximation capabilities, which is crucial for modern applications.
This model has been widely studied in the literature from a variety of different perspectives:
Nonlinear ICA. When the 24 are mutually independent, (1) recovers the standard nonlinear ICA model that has been extensively studied in the literature [1, 26-29, 74]. Although our most general results do not make independence assumptions, our results cover nonlinear ICA special case (see Section A.2 for more discussion).
VAE with mixture priors. When the prior over 2 is a mixture model (e.g. such as a GMM), the model (1) is closely related popular autoencoder architectures such as VaDE [32], SVAE [33], GMVAE [16]. DLGMM [52], VampPrior [66], MFC-VAE [18], etc. Although such VABs with mixture priors have been used extensively in applications, theoretical results are missing.
Warped mixtures. Another closely related model the warped mixture model of Iwata et al. [31], which is a Bayesian version of (I). Once again, theoretical guarantees for these models are lacking.
Formally, identifiability means the parametrization of the model is injective. See Section 2 for details. "See Remark 2.1 for extensions to more general mixture priors. OOu results include the noiseless case = 0 a special case.

---

Assumptions on
Assumptions on Z
Theoretical guarantees
Result
(PI)
(FI), (F2)
P(Z) identifiable up to an affine transformation
Theorems
3.9(a), 3.10(a)
(P1)
(FI), (F4)
P(Z) and f up to identifiable an affine transformation
Theorems
3.9(c), 3.10(d)
(P1), (P2)
(FI), (F4)
P(Z) and f identifiable up to permutation, scaling and translation
Theorems 3.9(b), 3.10(b)
(PI), (P2), (P3)
(FI), (F4)
PUU Z) and are identifiable up permutation, scaling and translation
Theorems 3.10(c), 3.10(d)
Table 1:
Summary of results in this paper. The strength of the assumptions increases in each successive row, as do the strength of the guarantees. See Section 3.3 for formal statements.
iVAE. Finally, (1) is also the basis of the iVAE model introduced by Khemakhem et al. [35], where identifiability (up to certain equivalences) proved when there is an additional auxiliary variable u that is observed such that IL Rj u.
Contributions Driven by this recent interest from both applied and theoretical perspectives,
main results (Theorems 3.9, 3.10) show that the model (1) is identifiable up to various linear equivalences, without conditioning or auxiliary information in the latent space. In fact, develop hierarchy of results under progressively stronger assumptions on the model, beginning with affine equivalence and ending up with much stronger equivalence up to permutations only. See Table for summary.
In order to develop this hierarchy, prove several technical results of independent interest:
1. First, we establish a novel identifiability result for nonparametric mixtures (Theorem C.2);
2. Second, show how to the mixture prior to strengthen existing identifiability results for nonlinear ICA (Theorem D.1);
3. Third, we extend existing results [38] on the recovery of structured multivariate discrete latent variable models recovery under an unknown affine transformation (Theorem F.2).
Our proof techniques-bassee based elementary tools from analytic function theory and mixture identifiability are new and depart from existing work this area. As a consequence, the analysis itself provides new insight into the structure and behaviour of deep generative models.
Related work This problem is widely studied, and has garnered significant recent interest, focus only on the most closely related work here.
Classical results on nonlinear ICA [28] establish the nonidentifiability of the general model (i.e. without restrictions on and f); see also Darmois [15], Jutten al. [34]. More recently, Khemakhem et al. [35] proved major breakthrough by showing that given side information u, identifiability of the entire generative model possible up to certain (nonlinear) equivalences. Since this pathbreaking
work, many generalizations have been proposed [10, 22, 23, 36, 39, 43, 50, 61, 73], all of which require some form of auxiliary information. Other approaches to identifiability include various forms of weak supervision such contrastive learning [75], group- based disentanglement [46], and independent mechanisms [20]. Non-idenne Non-identifiability has also been singled out as a contributing factor to practical issues such as posterior collapse in VAEs [69, 72].
Our approach is avoid additional forms of supervision altogether, and enforce identifiability in purely unsupervised fashion. Recent work along these lines includes Wang et al. [69], who propose to use Brenier maps and input convex neural networks, and Moran et al. [51] who leverage sparsity and anchor feature assumption. Aside from different assumptions, the main difference between this line of work and our work is that their work only identifies the latent space P(Z), whereas our focus is jointly identifying both P(Z) and f. In fact, we provide a decoupled set of assumptions

---

that allow f or P(Z) or both to be identified. Thus, we partially resolve in the affirmative an open problem regarding model identifiability raised by the authors in their discussion.
Another distinction between this line of work and the current work is our focus on architectures and modeling assumptions that are standard in the deep generative modeling literature, specifically ReLU nonlinearities and mixture priors. As noted above, there is a recent tradition of training variational autoencoders with mixture priors [16, 18, 32, 33, 41, 41, 44, 71J. Our work builds upon this empirical literature, showing that there is good reason to study such models: Not only have they been shown to be more effective compared to vanilla VAEs, we show that they have appealing theoretical properties as well. In fact, recent work [18, 70] has observed precisely the identifiability phenomena studied in our paper, however, this work lacks rigourous theoretical results to explain these observations.
Another related line of work studies identification in graphical models with latent variables, albeit without any explicit connection to deep generative models [17, 38, 49, 54].
Finally, since key step in our proof involves the analysis of nonparametric mixture model (see Appendix C for details), it is worth reviewing previous work in mixture models. See Allman et al. [2] for an overview. Of particular use for the present work are Teicher [63] and Barndorff-Nielsen [8], wherein the identifiability of Gaussian and exponential family mixtures, respectively, are proved. Specifically for nonparametric mixtures, existing results consider product mixtures [21, 64], grouped observations [57, 68], symmetric measures [9, 25], and separation conditions [4]. For context,
note here that a discrete VAE can be interpreted as a mixture model in disguise: This is perspective
that we leverage in our proofs. We are not aware of previous work in the deep generative modeling
literature that exploits this connection to prove identifiability results.
2 Preliminaries
We first introduce the main generative model that we study and its properties, and then proceed with a brief review of identifiaiility in deep generative models.
Generative model The observations x E R" are realizations of a random vector X, and are generated according to the generative model (1), where T represents realizations of an unobserved random vector Z. We make the following assumptions on and f:
(P1) P(Z) is (possibly degenerate) Gaussian mixture model with unknown number of components 2 i.c.
x22) = Σ Agp(2510, Ej), j=1
2 Σ1= 0,
(2)
where p(z) is the density of P(Z) with respect to some base measure, and y22; Myy 2ny) is the gaussian density with mean H4 and covariance Σj.
(FI) f is a piecewise affine function, such as multilayer perceptron with ReLU (or leaky ReLU) activations.
Recall that an affine function is function + Ar + for some matrix A. As already discussed, special cases of this model have been extensively studied in both applications and theory, and both (PI)-(FI) are quite standard in the literature on deep generative models and represent useful model that is widely used in practice [c.g. 16, 18, 32, 33, 41, 41, 44, 71]. In particular, when / = =1 this is simply a classical VAE with an isotropic Gaussian prior (see Section A.2 for more discussion).
Remark 2. I. The assumption that P(Z) a GMM can replaced with more general exponential family mixtures [8] as long (a) the resulting mixture prior p(z) is an analytic function and (b) the exponential family is closed under affine transformations.
Universal approximation Under assumptions (PI)-(FI), the model (1) has universal approxima- tion capabilities. fact, any distribution can be approximated by mixture model (2) with sufficiently
'I the sequel, we will use (P#) to index assumptions on the prior P(Z), and (F#) to index assumptions on the decoder f.

---

many components J [e.g. 53]. Alternatively, when J is bounded, by taking f to be a sufficiently deep and/or wide ReLU network, any distribution can be approximated by f(2) [e.g. 47, 65]. even if is invertible [30]. Thus, there is no loss in representational capacity in (PI)-(F1). To the best of our knowledge, our results are the first to establish identifiability of both the latent space and decoder for deep generative models without conditioning in the latent space or weak supervision. We note that Wang et al. [69] and Moran et al. [51] also propose deep architectures that identify the latent space, but not the decoder.
Identifiability A statistical model is specified by (possibly infiniteddimensionall as in our setting) parameter space θ family of distributions P. and mapping x - - P; i.e. r0) E P for each 0 E O. In more conventional notation, we define P = {po 0 E @). in which case Pe = =(). A statistical model is called identifiable if the parameter mapping applications, the strict definition of identifiability is too strong, and relaxed notions of identifiability sufficient. Classical examples include identifiability up permutation, re-scaling, orthogonal transformation. More generally, a statistical model is identifiable up to an equivalence relation ~ defined on θ if (0) (6") = 0 ~ 9'. For more details the different notions of identifiability in deep generative models, see [35, 36, 58].
one-to-one (injective). In practical
More precisely, we use the following definition. Let Je denote the pushforward measure of P by J. Definition 2.2. Let P be a family of probability distributions on Te and F be family of functions f: 1
R"
1. For (P.f) E P F we say that the prior P is identifiable (from ftP) up to an affine transformation if for any (P',f"= E P x F such that fyP = f/P' there exists an invertible affine map h : : + IR" such that P = h2P (i.c., P' is the pushforward measure of P by h).
2. For (P,J) E F we say that the pair (P, is identifiable (from P2P) up to an affine transformation if for any (P", J') E P x such that ftP = f4P' there exists an invertible affine map h: Rm - IR' such that I' = fo o h and P = hyP.
If the noise E has a known distribution, then f is identifiable from the convolution (f. Hence, this definition can be automatically extended to the setup with known noise. This definition also can be extended transformations besides affine transformations (e.g. permutations, translations, etc.) in the obvious way.
Identifiability is a crucial property for a statistical model: Without identifiability, different training runs may lead to very different parameters, making training unpredictable and replication difficult. The failure of identifiability, also known as underspecification and ill- ullpooeedness, has recently been flagged in the ML literature as a root cause of many failure modes that arise in practice [14, 69, 72]. As a result, there has been a growing emphasis on identification in the deep learning literature, which motivates the current work. Finally, in addition these reproducibility and interpretability concerns, identifiability is a key component in many applications of latent variable models including causal representation learning [59], independent component analysis [1 1]. and topic modeling [3, 5]. See Ran and Hu [55] for additional discussion and examples.
Auxiliary information and iVAE It is well-known that assuming independence of the latent factors-i.e. -ie. Z; 1. Zy-is insufficient for identifiability [28]. Recent work, starting with iVAE, shows identifiability by additionally assuming that a k k-dimensional auxiliary variable is observed such that p(= u) is conditionally factorial, i.c. Zi IL Zj U. This extra information serves to break symmetries in the latent space and is crucial to existing proofs of identifiability.
To make the connection with this work clear, observe that assumption (PI) is equivalent to assuming
that there is additional hidden state U [1, [1.....J}} J} such that P(Z = z U = =) (²(2) and
P(U = j) = Ay- More generally, U = (UI,..... Ux) may be multivariate. In this way, a direct parallel between work and previous work is evident, with several crucial caveats:
We do not assume that U is observed partially-or known in any way:
We allow for the Z to be arbtrarily dependent even after conditioning on U, and this dependence need not be known:
5

---

We do not even require the number of states J to be known, and we do not require any bounds on J (e.g. iVAE requires J 2 m 1).
In the case where U is multivariate (i.e is == dUm > 1), we do not require the number of latent dimensions k. the state spaces, or their dependencies to be known.
The original iVAE paper only proves identifiability of f up to a nonlinear transformation (see Lemma G.2 in Appendix G for details). By contrast, will show identifiability of up to an affine transformation, without knowing U.
In order to break the symmetry without knowing anything about U or its dependencies, we develop fundamentally new insights into nonparametric identifiability of latent variable models.
3
Main results
For any positive integer d, let [d] = [l.....dd dj. By (PI), we can write the model (I) as follows. dim(U;) and k == dim(U): we allow U to be
Let U (U,..... U2) [di] x... [da] where multivariate (k > 1) and dependent-i.c., we do not assume that the U; are marginally independent. It follows trivially from (PI) that P(U1 "1 "1...... UR = ux) E {AI..... },} and J II d, where we recall that J is the unknown number of mixture components in P(Z). Denote the marginal distribution of U, which depends on Xj. by Px. The variables (U, Z) are unobserved and encode the underlying latent structure:
U = u ~ PAUU 11U=u) =u) u)
2 U = u| ~ N(ua: En)
L
(3)
[X 2 = =] ~ f(z) tE, E E~ ~~100,0-)
Here, PA is the distribution on U described above. Our goal is to identify the latent distribution P(U, Z) and/or the nonlinear decoder f from the marginal distribution P(X) induced by (3). We will additionally assume throughout that m I n:; see Remark 3.6 for a discussion of the overcomplete case with
n.
Our main results (Theorems 3.9-3.10) provide a hierarchy of progressively stronger conditions under which P(U, Z). f. or both, can be identified in progressively stronger ways. The idea is to illustrate explicitly what conditions are sufficient to identify the latent structure up to affine equivalence (the weakest notion of identifiability
consider), equivalence up to permutation, scaling, and translation, and permutation equivalence (the strongest notion of identifiabiliyy we consider, and the strongest possible for any latent variable model).
We defer the statement of the main results to Section 3.3, after the main conditions have been described. As preview to the main results, we first present the following corollary:
Corollary 3.1. Suppose k = dim(U) = 1, (U, 2) are unobserved, and X is observed. (a) If f is an invertible ReLU network, then both PÚÙ Z) and f are identifiable up an affine transformation. (b) If f is only weakly injective (cf. (F2)), then PUU, Z)) is still identifiable up to an affine transformation.
For comparison, Corollary 3.1 already strengthens existing results, since U is not required to be known and we are able to identify f. In fact, the latter answers an open question raised by Wang et al. [69]. What's more, this is just the weakest result implied by our main results: Under stronger assumptions on the latent structure, the affine equivalence presented above can be strengthened further.
Taken together, the results this section have the following concrete implication for practitioners: For stably training variational autoencoders, there is now compelling justification to work with GMM prior and deep Rel.U/Leekk-RRel. networks. As we saw above, this is commonly done in practice already.
3.1 Possible assumptions on
To distinguish cases where / and is not identifiable, we require the following technical definition. Recall that for sets A, B, (AA) (x f(x) A} and f(B) (f(x) B}.

---

Definition 3.2. Let m S (see Remark 3.6) and f R. + R
(F2) We say that f is weakly injective (i) there exists zo E R and 8 S.t. (--) =1 for every I E B(xo, 6) n /JRR ((R"), and (ii) (r E R" [ oo) C [ mh) has measure zero with respect to the Lebesgue measure on f(R²)
(F3) We say that f is observably injective if {æ E R" : (f-+(( > 1] S f ((RTT) has measure zero with respect to the Lebesgue measure on S(RR") In other words, is injective for almost every x in its image [(RR (i.e. almost every "observable" x).
(F4) We say that f is injective if ((x)) = 1 for every x E f(2").
Remark 3.3. For piecewise affine functions assumption (F2) is weaker than assumption (F3), which in turn is weaker than (F4). Therefore, for piecewise affine functions we have the chain of implications:
(F4)
(F3)
(F2).
In the sequel, we mostly focus on (F2) and (F4) for simplicity; although we prove results for (F3) in Appendix D.I. See also Remarks 3.5, 3.11.
Example 3.4. In general, deep ReLU network may be either injective or observably injective, or 0). For example, although + ReLU(x) not injective, is observably injective, where RcLU(x)) max x) is the usual rectified linear unit. To see this, note that image of RcLU the set IR= = {y y 0}, and ReLU has the unique preimage for every y E R
neither (e.g.
(y
0}. Clearly, (R2 RR> {O} has measure zero inside Rz.
At the same time, x + and + |a|| are not even weakly injective.
Remark 3.5. In Appendix H. we show that ReLU networks or Leaky ReLU networks are generically observably injective (and hence also weakly injective) under simple assumptions on their architecture.
Remark 3.6. We restrict attention to the case m I n, which is a standard assumption, as it is common to think of latent space be a low-dimensional representation of the observed space. In the overcomplete case, i.e. when m > n, we believe that identifiability is unlikely unless stronger assumptions are made, or weaker notions of identifiability are considered. To see this, consider the projection fzz y) = which is trivially affine. Then we can arbitrarily transform the y y-coordinate without changing P, i.e. (J' g)p P = fgP, where g(a, y) = (ar, h(y)) for any h. an example of identifiability in the overcomplete regime under stronger assumptions, when the auxiliary variable known, [36] show that the feature maps f and g in conditional energy-based models (for which p(x u) x ex)(f(x)))
{()))) can be identified up to an affine transformation.
3.2 Possible assumptions on Z
Our weakest result requires no additional assumptions on Z beyond (PI); see Corollary 3.1. Under stronger assumptions, more can be concluded. As with the previous section, the assumptions presented here are not necessary, but may be imposed in order to extract stronger results.
The first condition is mild condition that allows us to strengthen affine identifiability:
(P2) Z, 11 Z, U for all * j and there exist pair of states U2 and U u2 such that all ((Ew,) )z/ (Euz) )ss t E [m]) are distinct. (Note that this implies J 2 2).
The second condition is more technical, and is only necessary k > 1 and we wish to identify P(U) in addition to P(Z). ). In fact, not only will we recover P(U), but also the (unknown) number of hidden variables (i.e. k) and their state spaces (i.e. d,). Note that P(U) is not needed to sample from
(I), as long as we have P(Z). Before introducing this condition, we need preliminary definition.
Definition 3.7. Let U-; denote (U, i * i). We define no(U,) = [m] \ ft: Ze IL U; U-;) and ne(2,) = {t: Z, no(U.)). For a subset Z' C Z, ne(Z') = UZ,E2' no Zz).
The neighborhood ne(U:) collects the variables Z, that depend on U; directly.
(P3) The following conditions hold:
(a) For all Z' C Z and + uz P(Z ne(Z") w1) 4 P(Z'| ne(Z") = uz))
(b) If P(U", X) = P(U, Z, X). then dim(U") < dim(U'); and
7

---

(c) For any Us * U, the set ne(U;) is not subset of ne(U,).
Condition (P3) a "maximality" condition that is adapted from Kivva et al. [38]: We are interested in identifying the most complex latent structure with the most number of hidden variables. This is in fact necessary since we can always merge two (or more) hidden variables into a single hidden variable without changing the joint distribution. Moreover, if two distinct hidden variables U; + U3 have the same neighborhood (or one is a subset of another), then it is known that P(U) cannot be identified [17, 38, 54]. Evidently, if we seek learn P(U) in addition to P(Z), then this must be avoided. Finally, as the proof will indicate, this condition is slightly stronger than what is needed (see Remark F.3 for details).
Remark 3.8. Condition (P3) should be contrasted with the stronger "anchor words" assumption that has appeared in prior work [5, 51]: In fact. the existence of anchor word for each Uj automatically implies that ne(U;) is not a subset of ne(Uy) for # j. Thus, anchor words are sufficient but not necessary condition for identifiability, whereas Condition (P3) is indeed necessary as described above.
More details and discussion these assumptions be found in Appendix F.
3.3 Main identifiability results
When dU) there additional structure in U to learn, and so the setting simplifies considerably. We begin with this special case before considering the case of general multivariate U.
Theorem 3.9. Assume dim(U) Under (PI)-(FI), we have the following:
(a) (F2) - P(U,Z) 2) identifiable from P(X) up an affine transformation of Z.
(b) (F2)+(P2) = P(U,Z) Z) is identifiable from P(X) up to permutation, scaling, and/or translation of Z.
(c) In either (a) or (b), if additionally (F4) holds and f is continuous, then f is also identifiable from P(X) up to an affine transformation.
The next result generalizes Theorem 3.9 to arbitrary (possibly multivariate) discrete U. This is an especially challenging case: Unlike previous work such iVVEE that assumes U (and hence its structure) is known, we do not assume anything about U known. Thus, everything about U must be reconstructed based on P(X) alone, hence the need for (P3) to identify P(L') below.
Theorem 3.10. Under (PI)-(FI), we have the following:
(a) (F2) = P(Z) is identifiable from P(X) up to an affine transformation.
(b) (P2)+(P2) translation.
P(Z) is identifiable from P(X) up to permutation, scaling, and/or
(c) (F2)+ tion U, and P(Z) is identifiable up to permutation, scaling, and/or translation.
(k,, (k.,d....., dg, P(U)) are identifiable from P(X) up permuta-
(d) In any of (a), (b), (c), if additionally (F4) holds and f continuous, then is also identifiable from P(X) ) up to an affine transformation.
Without (P3), Kivva et al. [38] have shown that it is not possible to recover the high- high-dimensional latent state U, however, we can still identify the continuous latent state Z, which is enough to generate random samples from the model (1). In order to have fine-grained control over the individual variables in U. however, it is necessary to assume (P3).
Remark 3. 1. If (F4) is relaxed to (F3) f may not be identifiable up to an affine transformation, but
"essentially" identifiable in the following sense. Let S = (x : [[=)((x)) > 1}. On every to an affine transformation (which may connected component of R. (S) f is identifiable
depend on the connected component). Note, for f defined by a ReL.U NN points of are atoms of P(X).
8

---

NLL
NLL
NLLL
18
1.3
1
-2
Figure 1: Selected examples of the negative log-likelihood for different runs. In each figure, one parameter from a model (e.g. 3 is a weight in the neural network defining f) is selected, and the value of the negative log-likelihood is visualized as function of this parameter. Vertical lines indicate the ground truth and (global) minimizer, which always coincide. Three particularly interesting, nonconvex examples are shown here. See Appendix J.3.1 for details.
Remark 3.12. the assumption (F2) that f weakly injective is removed, then the claim of Theorem 3.9 is not true anymore. Consider (ax) = f(x [x]] and
3 N(-2,00) N(-1,07) 1 N(3, N(3, and
(4)
P' 1--ING N(- N(-2, N(1, 2(,,0)))
N(3,0%) ((3,,0
It is easy to verify that P cannot be transformed into P' by affine transformation, but ftP and g: are equally distributed.
Remark 3.13. In Theorems (a) and 3. 3.1)(a), the identifiability up affine transformation is the best possible if no additional assumptions on Z are made (i.e. beyond (PI)). Indeed, for an arbitrary invertible affine map h IR" IR.", h ( has GMM distribution, I h-1 affine map, and (U, Z, f) and (U, ((),, o h h-1) in model (3) generate the same distribution.
1 is invertible piecewise
Experiments
There has been extensive work already verify empirically that the model (1) under (P1)-(F1) is identifiable. For example, [70] observe that deep generative models with clustered latent spaces empirically identifiable, and compared this directly models that rely side information, and [18] show that meaningful latent variables can be learned consistently in fully unsupervised manner even when U has high-dimensional structure. Moreover, [18] indicate that high-dimensional structure is important for improved performance. Beyond these, it is well-known that VAEs with mixture priors such as VaDE [32] achieve competitive performance on many benchmark tasks; see [16, 18, 33, 41, 41, 44, 71] for additional experiments and verification. Building upon the established success of these methods, we augment these experiments as follows: We use simple examples to verify that the likelihood indeed has unique minimizer at the ground truth parameters; 2) We train VaDE on (misspecified) simulated toy models; and 3) We measure stability (up to affine transformations) of the learnt latent spaces on real data. To measure this, we report the Mean Correlation Coefficient [36, Appendix A.2] metric, which is standard, and an L alignment metric (denoted by dist 12). Definitions of these metrics and additional details on the experiments can be found in Appendix
Maximum likelihood We simulated models satisfying (P1)-(F1) by randomly choosing weights and biases for a single-layer ReI.U network and randomly generating a GMM with J 2 or components. These models are simple enough that exact computation of the MLE along the likelihood surface feasible via numerical integration (Figure 1). In all our simulations (50 total), the ground truth was the unique minimizer of the negative log-likelihood, examples also illustrate a small-scale test misspecification in the theoretical model: We include cases where J is misspecified and ff fails to satisfy (F4), but the MLE succeeds anyway.
predicted by the theory. These
Simulated data In our experiments on synthetic datasets we consider, to obtain an experimental evidence of identifiability of model (3) we fit VaDE to observed data 5 times (see Figure 2). Let

---

Observed X
Latent GMM #1
Latent GMM #2
Latent GMM #3
Latent GMM #4
Latent GMM #5
P
Figure 2: Recovered latent spaces for 5 runs of VaDE on pinwheel dataset with 3 clusters
2(1) 2(2) ..... (6) be the learned latent spaces. For every pair Z z we evaluate the MCC and dist L2 loss. For instance, for the pinwheel dataset with three clusters as in Figure 2, the average dist Alt Le..,,e(Pt (pl. P2) across 20 pairs o Z is 0.113 with standard deviation 0.065. The average weak MCC is 0.87 and the average strong MCC is 1.0. This shows strong evidence of recovery of the latent space up to affine transformations.
Real data We measure stability of the learnt latent space by training MFCVAE [18] on MNIST 10 times with different initializations and then comparing the latent representations learnt. It becomes computationally infeasible to compute dist 12 therefore report only MCC. The strong MCCs are computed to be 0.7 (ReLU), 0.69 (LeakyRel.U) and the weak MCCs are computed to be 0.91 (ReL.U) 0.94 (LeakyRelLU), These observations validate the observations first made in [70]. who ran extensive experiments on VaDE and iVAE on several large datasets including MNIST, SVHN and CIFAR10. These strong correlations confirm theory and are of particular importance to practitioners for whom stability of learning is of the essence.
5 Conclusion
We have proved a general series of results describing a hierarchy of identifiability for deep generative models that are currently used in practice. Our experiments confirm both on exact and approximate simulations that identifiability indeed holds in practice. An obvious direction for future work is to study finite-sample identifiability problems such as sample complexity and robustness (i.e. how many samples are needed to ensure that the global minimizer of the likelihood reliably close to the ground truth?). Theoretical questions aside, developing better understanding of the ELBO and its effect on optimization is an important practical question. For example, an important limitation of the current set of results is that they apply only to the likelihood, which is known to be nonconvex and intractable to optimize (see Figure for concrete examples). It is an important open question to use these insights to develop better algorithms and optimization techniques that work on finite-samples with misspecified models (i.e. real data).
More generally, although our assumptions map onto architectures and priors that are widely used in practice, is important to emphasize the relevant distinction between models and estimators. That is, the architectures used in practice represent the estimators used, and may not reflect realistic assumptions on the model itself (which typically misspecified). For example, the piecewise affine assumption may not accurately reflect valid assumptions about real- world problems. Given the lack of purely unsupervised, nonparametric identifiability results in the literature, we view our results as an important technical step towards understanding practical identifiability for deep generative models. Thus, an important future direction is to replace our assumptions with more appropriate modeling assumptions that relevant for practical applications.
6
Acknowledgements
We thank anonymous reviewers for useful comments and suggestions. G.R. was partially supported by NSF grants CCF-1816377 and CCF-200892. B.A. was supported by NSF IIS- 195639,,3,, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business. P.R. was supported by ONR via N00014181288 1812861, and NSF via IIS- 1909816. 115-595553,, 11902
10

---

References
[1] S. Achard and C. Jutten. Identifiability of post-nonlinear mixtures. IEEE Signal Processing Letters, (2(5):223-422)) 2005.
[2] E. Allman, C. Matias, and J. A. Rhodes. Identifiability of parameters in latent structure models with many observed variables. Annals of Statistics, pages -039- 2009.
[3] A. Anandkumar, D. J. Hsu, M. Janzamin, and S. M. Kakade. When are overcomplete topic models identifiable? uniqueness of tensor tucker decompositions with structured sparsity. Advances in neural information processing systems, 26, 2013.
[4] B. Aragam, C. Dan, E. P. Xing, and P. Ravikumar. Identifiability of nonparametric mixture models and bayes optimal clustering. Ann. 4tariss,,4844)22777 48(4))2277-23022 2020. ISSN 0090-5364. doi: 10.1214/9-AOSS AOS1800S188 187. arXiv:1802.04399
[5] S. Arora, R. Ge, and A. Moitra. Learning topic models beyond svd. In 2012 IEEE 53rd amnual symposium on foundations of computer science, pages 1-10. IEEE, 2012.
[6] S. Arora, R. Ge, Y. Halpern, D. Mimno, A. Moitra, D. Sontag, Y. Wu, and M. Zhu. practical algorithm for topic modeling with provable guarantees. In International Conference on Machine Learning, pages 280-288. PMLR, 2013.
[7] Y. Bansal, P. Nakkiran, and B. Barak. Revisiting model stitching to compare neural representa- tions. Advances in Neural Information Processing Systems, 34, 2021.
[8] O. Barndorff-Nielsen Identifiability of mixtures of exponential families. Journal of Mathemati- cal Analysis and Applications, (2(1):11-112) 1965.
[9] L. Bordes, S. Mottelet, and P. Vandekerkhove. Semiparametric estimation of two-component mixture model. Annals of Statisties, 34(3):1204-111 1204-123222 2006.
[10] J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. arXiv preprint arXiv:2203. /6437, 2022.
[11] P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
[12] A. Csiszárik, Kórösi-Szabó, Á. Matszangosz, Papp. and Varga. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34, 2021.
[13] B. Dai, Z. Wang, and D. Wipf. The usual suspects? reassessing blame for vae posterior collapse. In International Conference Machine Learning, pages 311-23222 PMLR, 2020.
[14] A. D' D'Amour, K. Heller, Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint 1xXx:22011.03399 2020.
[15] G. Darmois. Analyse des liaisons de probabilité. Proc. lnt. Stat. Conferences 1947, page 231, 1951.
[16] N. Dilokthanakul, A. Mediano, M. Garnelo, M. Lee, H. Salimheni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv: (tt.:///1.2000 1.0664,, 2016.
[17] J. Evans. Graphs for margins of bayesian networks. Scandinavian Journal of Statistics, 43 5)x6256448. 2016.
[18] Falck, Zhang, M. Willetts, G. Nicholson, C. Yau, and C. Holmes. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
[19] É. Gassiat, A. Cleynen, and Robin. Inference finite state space non parametric hidden markov models and applications. Statistics and Computing, 26(1):61-71,216
11

---

[20] L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mech- anism analysis, new concept? Advances Neural Information Processing Systems, 34, 2021.
[21] P. Hall and X.-H. Zhou. Nonparametric estimation of component distributions in multivariate mixture. Annals of Statistics, pages 201-224, 2003.
[22] H. Hälvä and Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939- 948. PMLR, 2020.
[23] H. Hälvä, S. L. Corff, I.. Lehéricy, So, Y. Zhu, E. Gassiat, and A. Hyvarinen. Disen- tangling identifiable features from noisy data with structured nonlinear ica. arXiv preprint arXiv:2106.. 09620,, 2021.
[24] He, D. Spokoyny, G. Neubig, and T. Berg- KergKKikkpatrick Lagging inference networks and posterior collapse in variational autoencoders. In International Conference Learning Repre- sentations, 2018.
[25] D. R. Hunter, Wang, and T. Hettmansperger. Inference for mixtures of symmetric distribu- tions. Annals of Statistics, pages 224-251, 2007.
[26] A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29, 2016.
[27] A. Hyvarinen and H. Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[28] A. Hyvärinen and P. Pajumen. Nonlinear independent component analysis: Existence and uniqueness results. Neural nelworks, 12(3):429-443 1999.
[29] A. Hyvarinen, Sasaki, and Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[30] Ishikawa, T. Teshima, Tojo, K. Oono, M. Ikeda, and Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204.07415. 2022.
[31] T. Iwata, D. Duvenaud, and Z. Ghahramani. Warped mixtures for nonparametric cluster shapes. In Proceedings of the Twenty-Nitth Conference on Uncertainty in Artificial Intelligence, pages 311-320, 2013.
[32] Z. Jiang. Y. Zheng. H. Tan, B. Tang, and H. Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv: 1.05148, 2016.
[33] M. Johnson, D. K. Duvenaud, A. Wiltschko, P. Adams, and S. R. Datta. Composing graph- ical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29, 2016.
[34] C. Jutten, J. Karhunen, et al. Advances in nonlinear blind source separation. Proc. of the 4th Int. Symp. Independent Component Analysis and Blind Signal Separation (ICA2003), pages 246-256, 2003.
[35] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[36] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Ice-beem: Identifiable conditional energy-based deep models. NeurIPS2020, 2020.
[37] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[38] B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[39] D. A. Klindt, L. Schott, Y. Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethge, and D. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2020.
[40] J. Klys, J. Snell, and R. Zemel. Learning latent subspaces in variational autoencoders. Advances in Neural Information Processing Systems, 31, 2018.
[41] D. B. Lee, D. Min, S. Lee, and S. Hwang. Meta-gmvae: Mixture of gaussian vae for unsupervised meta-learning. In International Conference on Learning Representations, 2020.
[42] K. Lene and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
2015.
[43] S. Li, B. Hooi, and G. H. Lee. Identifying through flows for recovering latent representations. arXiv preprint aivv 1909. 12555, 2019.
[44] X. Li, Chen, L. K. Poon, and N. L. Zhang. Learning latent superstructures in variational autoencoders for deep multidimensional clustering, arXiv preprint arXiv: 2018.
[45] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14-44.4. PMLR, 2019.
[46] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly- supervised disentanglement without compromises. International Conference on Machine Learning, pages (348-6359, PMLR, 2020.
[47] Y. Lu and J. Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems, 33:3094-3105 2020.
[48] G. Luise, M. Pontil, and C. Ciliberto. Generalization properties of optimal transport gans with latent distribution learning. arXiv preprint 2rXi 14641, 2020.
[49] A. Markham and M. Grosse-Wentrup. Measurement dependence inducing latent causal models. In Conference on Uncertainty in Artificial Intelligence, pages 590-599. PMLR, 2020.
[50] G. Mita, M. Filippone, and P. Michiardi. An identifiable double vae for disentangled represen- tations. M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7769-7779. PMLR, 18-24 Jul 2021.
[51] G. E. Moran, D. Sridhar, Y. Wang, and D. M. Blei. Identifiable variational autoencoders via sparse decoding. arXiv preprint arXiv:21 10. 10804, 2021.
[52] E. Nalisnick, L. Hertel, and P. Smyth. Approximate inference for deep latent gaussian mixtures.
In NIPS Workshop on Bayesian Deep Learning, volume 2, page 131, 2016.
[53) H. D. Nguyen and G. McLachlan. On approximations via convolution-defined mixture models. Communications in Statistics-Theory and Methods, +8(16):3945-3955 2019.
[54] J. Pearl and T. S. Verma. A statistical semantics for causation. Statistics and Computing, 2(2): 91-95, 1992.
[55] Z.-Y. Ran and B.-G. Hu. Parameter identifiability in statistical machine learning: a review. Neural Computation, 29(5):111111000 2017.
[56] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In E. P. Xing and T. Jebara, editors, Proceedings of the st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-1286, Bejing, China, 22-24 Jun 2014. PMLR.
[57] A. Ritchie, R. A. Vandermeulen, and C. Scott. Consistent estimation of identifiable nonparamet- ric mixture models from grouped observations. arXiv preprint arXiv:2006.0744 2020.
13

---

[58] G. Roeder, L. Metz, and D. Kingma. On linear identifiability of learned representations. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9030-9039 PMLR, 28-24 Jul 2021.
[59] B. Schölkopf. F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021.
[60] L. Schott, J. von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Lo- catello, and W. Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2107.08221, 2021.
[61] Sorrenson, C. Rother, and U. Köthe, Disentanglement by nonlinear ica with general incompressible-foww networks (GIN). In International Conference on Learning Represen- tations, 2019.
[62] Stock and R. Gribonval. An embedding of relu networks and an analysis of their identifiability. arXiv preprint arXiv:2107.003370, 2021.
[63] H. Teicher. Identifiability of finite mixtures. The annals of Mathematical statistics, pages 1265-1269, 1963.
[64] H. Teicher. Identifiability of mixtures of product measures. The Annals of Mathematical Statistics, 38(4): 130001300-1322 1302, 1967.
[65] T. Teshima, I. Ishikawa, Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems, 33:3362-3373, 2020.
[66] Tomezak and M. Welling. Vae with vampprior. International Conference on Artificial Intelligence and Statistics, pages 1214-1223. PMLR, 2018.
[67] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[68] R. A. Vandermeulen, D. Scott, et al. An operator theoretic approach to nonparametric mixture models. Annals of Statistics, 47(5):2704-2733, 2019.
[69] Y. Wang, D. Blei, and J. P. Cunningham. Posterior collapse and latent variable non- non-identifiabilityy Advances in Neural Information Processing Systems, 34, 2021.
[70] M. Willetts and B. Paige. I don't need u: Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05253 2021.
[71] M. Willetts, S. Roberts, and C. Holmes. Disentangling to cluster: Gaussian mixture variational ladder autoencoders. arXiv preprint arXiv: / /501, 2019.
[72] Y. Yacoby, W. Pan, and F. Doshi- Velez. Failure modes of variational autoencoders and their effects on downstream tasks. In ICML Workshop Uncertainty and Robustness in Deep Learning (UDL), 2020.
[73] X. Yang, Y. Wang, J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ica using volume- preserving transformations. In International Conference on Learning Representations, 2021.
[74] K. Zhang and L. Chan. Minimal nonlinear distortion principle for nonlinear independent component analysis. Journal of Machine Learning Research, 2455-22245 2008
[75] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12990. PMLR, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5 and Broader Impacts statement in the appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See (PI)-(P3) (F1)-(F4)
(b) Did you include complete proofs of all theoretical results? materials.
[Yes] See Supplementary
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplement.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [N/A] All assets are public and open-source.
(c) Did you include any assets either in the supplemental material
URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "kh6xsWAV4CN", "reviewer": "Reviewer_w4oC", "summary": "This paper explores the problem of DGMs' identifiability. The main contribution is theoretical as it proves a sequence of theorems about the identifiability of DGMs under different assumptions on decoder functions and latent distributions. It also has several experiments trying to verify the theorems on synthetic data and MNIST.", "questions": "1. The theorems in sec. 3 show that under certain conditions, all global minima of a DGM are the same up to an equivalence relation. But the current gradient based training of DGMs cannot guarantee finding a global minimum. Is there a way to analyze the identifiability of learned parameters of a DGM?", "limitations": "The authors have discussed limitations.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have some experience on deep generative models, but I'm not quite familiar with the problem of identifiability. As a result, my review might not be very targeted and discussions are very welcome from the authors and other reviews.\n\nPros:\n1. Identifiability of DGMs is a very interesting question. It's an important path towards more reliable machine learning systems. It will also make analyzing DGMs easier.\n2. This paper discusses several different assumptions for Z and f in detail, and proves how strong identifiability we can achieve under each assumption. The conclusion doesn't rely on knowing any auxiliary variable and provides a more comprehensive characterization of the problem.\n3. The paper is well written. It starts from simple cases and gradually delves into details, which makes it a pleasure to read it.\n\nCons:\n1. The theorem lacks a stability analysis. All theorems in this paper require the data distribution to exactly match. It's not clear what's the situation like if we allow a small mismatch between the real and learned data distributions. Taking Eqn. 4 in the paper as an example, we can find f and f', which maps P and P' to f#(P) and f'#(P'), which can be arbitrarily close to each other, but cannot exactly match because of the piecewise affine assumption.\n2. The experiments fail to convince me that the theory can be applied to more complicated problems. The synthetic data is simply a transformation of three-component MoG. The real data is MNIST, which has obvious clustering property. Both datasets are too simple.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S5PAngYwKD3", "reviewer": "Reviewer_dpBS", "summary": "This paper proves a series of identifiability results for commonly used modern generative models composed of Gaussian mixture priors and piecewise affine (ReLU) decoder networks. Up to affine or simpler transformations on the latent space for the decoder function, the latent space and/or the decoder function can be specified given a learned distribution. Unlike previous studies on more generic network types, this framework does not require additional auxiliary information. The assumptions made for the prior and decoder are more restrictive but correspond to common practice. As far as I understand, the main intuition for the proofs come from identifiability of Gaussian mixture models (in reduced form) and nice but non-obvious properties of piecewise-affine functions. For example, each affine segment transforms a GMM into another GMM, which is identifiable. ", "questions": "## Detailed suggestions. \n1. Line 176, I think here the $f'$ and $P'$ inherit from the previous point, so perhaps should clarify that. Only $f$ is shared with the definition in line 172. \n2. Line 181, the paper's focus is mostly on the functions rather than parameters, so should perhaps change \"parameters\" to \"functions\". \n3. Line 183. \"Failure model\" may be understood as a (likelihood) performance failure, interpretability failure or identifiability failure. I would try to specify a little. \n2. Definition 3.7. does $ne(U_i)$ refer to the set of indices or variables? The definition suggests it's an index set, but later uses assume they are variables. \n2. Line 331, the $x$ inside $f^{-1}$ should be inside braces. \n3. Line 812, should $h$ be a better function symbol to use as an example?\n3. Definition C.8, there is a condition that the preimage is finite, but then in 859 the subspaces covering the non-affine supports can be infinite. I think the authors probably mean \"countable\" in Definition C.8, so that the preimage are made of atoms rather than continuous regions. \n2. Throughout the development of the proofs, $x$ can appear as the input or output of the decoder function $f$, which can be quite confusing and takes time for the reader to follow. \n2. equation (11), I think the notation $p|_{\\cdot}$ deserves a specific definition. \n2. Various symbols on page 20 can be much better illustrated by an example of $f\\in\\mathbb{R}\\to\\mathbb{R}$ that is piecewise affine, where $x_0$ is a value on the vertical axis, then y's on the horizontal axis, B is a region around one $y_i$, etc. The concept here is actually quite simple, though notationally heavy, for a reader like me. ", "limitations": "I don't see any concerning limitations of the analyses. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. To my knowledge, this paper is the first to set out rigorously the conditions needed for identifiability in deep generative models without auxiliary labels. Although the assumptions made are stronger than in previous work, they correspond to standard practice and thus are of great significance for the field. \n1. The authors present a series of analyses from weaker to stronger assumptions of the model, demonstrating a substantial contribution to the field. \n2. Writing is another strength of the paper. The main text gave a comprehensive overview of previous work and discussed the subtle details of the assumptions and concepts. Some crucial examples are given to clarify the definitions. These are presented in a well-designed layout that I find things very clear and easy to refer back and forth. \n2. The appendix has a lot of background information and analysis compared to very relevant literature, including the iVAE. \n2. Due to limited time availability, I checked the proof up to page 23 of the appendix. They seem correct to me and make intuitive sense, but I may be wrong for very technical details. \n\n##  Weakness:\n1. The main text, among a very clear introduction of the background and problem setup, is a repetitive sale of the achievement compared to the existing literature rather than a substantive and intuitive description of the theory. I'd encourage the authors to try to reduce the significance statement of the work, however impactful it may be. For example, the fact that \"the theory does not assume any additional information\" appears so many times in the main text that some readers may find it annoying. \n2. The main numerical results should be listed in a small table or figure, some results currently in the appendix should also appear in the main text. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OYAnbdaIj8P", "reviewer": "Reviewer_SEs8", "summary": "This paper introduces new theoretical results for the identifiability of Variational Autoencoders with Gaussian mixture priors under several progressively stronger conditions on the latent distribution and the decoder (such as the decoder being a piecewise-affine function). The work extends and generalizes previous results on the identifiability of deep generative models, such as the iVAE. The key contribution is identifiability results that do not depend on auxiliary variables being observed. For example, the authors show that that under weak assumptions P(Z) is identifiable from P(X) up to an affine transformation and that additional the decoder parameters are also identifiable if the function family is injective.\n\nThe authors also include experiments on toy data and MNIST to verify the identifiability results empirically.", "questions": "I'm not sure I fully understand the definition of weak injectivity here (F2). What does $B(x_0, \\delta)$ denote? Can you give a more intuitive explanation?\n\nWhat is mean by \"in the sequel\"? Is that referring to the subsequent section?", "limitations": "The authors address some limitations and future work in the conclusion and discuss related negative results in the main text. Broader impacts are not discussed, though this is a largely theoretical work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary strength of this work is that it shows that modeling choices already commonly made in practice for Variational Autoencoders, such as ReLU networks and Gaussian mixture priors, maintain identifiability (up to affine transforms) for the latent codes and possibly the decoder parameters. This strengthens previous work, and makes a useful contribution to the theoretical analysis of deep generative models. It also helps to justify modeling practices which are commonly used, even in the more common case where auxiliary information is not available. In my opinion, the main results presented in theorems 3.9 and 3.10 are a sufficiently novel and useful contribution to the field. (I have not yet had a chance to fully verify the correctness of the proofs provided for these results, the supplement provides extensive details on the theoretical results).\n\nThe paper also generalizes to more complex cases with multiple discrete latent groups and more general exponential family mixtures.\n\nThe paper is also well-written, generally providing the necessary background, the key contributions, the assumptions and results in a clear, well-organized manner, with a few exceptions, see below.\n\nThe experiments section could use improvement, though understandably space for this section is limited. It is not quite clear to me what figure 1 results are showing. The likelihood as a function of decoder parameters? Latent codes? The right and middle plots have the same x-axis label. Figure 2 is somewhat more clear.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept