Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proves a series of identifiability results for commonly used modern generative models composed of Gaussian mixture priors and piecewise affine (ReLU) decoder networks. Up to affine or simpler transformations on the latent space for the decoder function, the latent space and/or the decoder function can be specified given a learned distribution. Unlike previous studies on more generic network types, this framework does not require additional auxiliary information. The assumptions made for the prior and decoder are more restrictive but correspond to common practice. As far as I understand, the main intuition for the proofs come from identifiability of Gaussian mixture models (in reduced form) and nice but non-obvious properties of piecewise-affine functions. For example, each affine segment transforms a GMM into another GMM, which is identifiable. 
strengths_and_weaknesses: ## Strengths:
1. To my knowledge, this paper is the first to set out rigorously the conditions needed for identifiability in deep generative models without auxiliary labels. Although the assumptions made are stronger than in previous work, they correspond to standard practice and thus are of great significance for the field. 
1. The authors present a series of analyses from weaker to stronger assumptions of the model, demonstrating a substantial contribution to the field. 
2. Writing is another strength of the paper. The main text gave a comprehensive overview of previous work and discussed the subtle details of the assumptions and concepts. Some crucial examples are given to clarify the definitions. These are presented in a well-designed layout that I find things very clear and easy to refer back and forth. 
2. The appendix has a lot of background information and analysis compared to very relevant literature, including the iVAE. 
2. Due to limited time availability, I checked the proof up to page 23 of the appendix. They seem correct to me and make intuitive sense, but I may be wrong for very technical details. 

##  Weakness:
1. The main text, among a very clear introduction of the background and problem setup, is a repetitive sale of the achievement compared to the existing literature rather than a substantive and intuitive description of the theory. I'd encourage the authors to try to reduce the significance statement of the work, however impactful it may be. For example, the fact that "the theory does not assume any additional information" appears so many times in the main text that some readers may find it annoying. 
2. The main numerical results should be listed in a small table or figure, some results currently in the appendix should also appear in the main text. 
questions: ## Detailed suggestions. 
1. Line 176, I think here the $f'$ and $P'$ inherit from the previous point, so perhaps should clarify that. Only $f$ is shared with the definition in line 172. 
2. Line 181, the paper's focus is mostly on the functions rather than parameters, so should perhaps change "parameters" to "functions". 
3. Line 183. "Failure model" may be understood as a (likelihood) performance failure, interpretability failure or identifiability failure. I would try to specify a little. 
2. Definition 3.7. does $ne(U_i)$ refer to the set of indices or variables? The definition suggests it's an index set, but later uses assume they are variables. 
2. Line 331, the $x$ inside $f^{-1}$ should be inside braces. 
3. Line 812, should $h$ be a better function symbol to use as an example?
3. Definition C.8, there is a condition that the preimage is finite, but then in 859 the subspaces covering the non-affine supports can be infinite. I think the authors probably mean "countable" in Definition C.8, so that the preimage are made of atoms rather than continuous regions. 
2. Throughout the development of the proofs, $x$ can appear as the input or output of the decoder function $f$, which can be quite confusing and takes time for the reader to follow. 
2. equation (11), I think the notation $p|_{\cdot}$ deserves a specific definition. 
2. Various symbols on page 20 can be much better illustrated by an example of $f\in\mathbb{R}\to\mathbb{R}$ that is piecewise affine, where $x_0$ is a value on the vertical axis, then y's on the horizontal axis, B is a region around one $y_i$, etc. The concept here is actually quite simple, though notationally heavy, for a reader like me. 
limitations: I don't see any concerning limitations of the analyses.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces new theoretical results for the identifiability of Variational Autoencoders with Gaussian mixture priors under several progressively stronger conditions on the latent distribution and the decoder (such as the decoder being a piecewise-affine function). The work extends and generalizes previous results on the identifiability of deep generative models, such as the iVAE. The key contribution is identifiability results that do not depend on auxiliary variables being observed. For example, the authors show that that under weak assumptions P(Z) is identifiable from P(X) up to an affine transformation and that additional the decoder parameters are also identifiable if the function family is injective.

The authors also include experiments on toy data and MNIST to verify the identifiability results empirically.
strengths_and_weaknesses: The primary strength of this work is that it shows that modeling choices already commonly made in practice for Variational Autoencoders, such as ReLU networks and Gaussian mixture priors, maintain identifiability (up to affine transforms) for the latent codes and possibly the decoder parameters. This strengthens previous work, and makes a useful contribution to the theoretical analysis of deep generative models. It also helps to justify modeling practices which are commonly used, even in the more common case where auxiliary information is not available. In my opinion, the main results presented in theorems 3.9 and 3.10 are a sufficiently novel and useful contribution to the field. (I have not yet had a chance to fully verify the correctness of the proofs provided for these results, the supplement provides extensive details on the theoretical results).

The paper also generalizes to more complex cases with multiple discrete latent groups and more general exponential family mixtures.

The paper is also well-written, generally providing the necessary background, the key contributions, the assumptions and results in a clear, well-organized manner, with a few exceptions, see below.

The experiments section could use improvement, though understandably space for this section is limited. It is not quite clear to me what figure 1 results are showing. The likelihood as a function of decoder parameters? Latent codes? The right and middle plots have the same x-axis label. Figure 2 is somewhat more clear.



questions: I'm not sure I fully understand the definition of weak injectivity here (F2). What does $B(x_0, \delta)$ denote? Can you give a more intuitive explanation?

What is mean by "in the sequel"? Is that referring to the subsequent section?
limitations: The authors address some limitations and future work in the conclusion and discuss related negative results in the main text. Broader impacts are not discussed, though this is a largely theoretical work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores the problem of DGMs' identifiability. The main contribution is theoretical as it proves a sequence of theorems about the identifiability of DGMs under different assumptions on decoder functions and latent distributions. It also has several experiments trying to verify the theorems on synthetic data and MNIST.
strengths_and_weaknesses: I have some experience on deep generative models, but I'm not quite familiar with the problem of identifiability. As a result, my review might not be very targeted and discussions are very welcome from the authors and other reviews.

Pros:
1. Identifiability of DGMs is a very interesting question. It's an important path towards more reliable machine learning systems. It will also make analyzing DGMs easier.
2. This paper discusses several different assumptions for Z and f in detail, and proves how strong identifiability we can achieve under each assumption. The conclusion doesn't rely on knowing any auxiliary variable and provides a more comprehensive characterization of the problem.
3. The paper is well written. It starts from simple cases and gradually delves into details, which makes it a pleasure to read it.

Cons:
1. The theorem lacks a stability analysis. All theorems in this paper require the data distribution to exactly match. It's not clear what's the situation like if we allow a small mismatch between the real and learned data distributions. Taking Eqn. 4 in the paper as an example, we can find f and f', which maps P and P' to f#(P) and f'#(P'), which can be arbitrarily close to each other, but cannot exactly match because of the piecewise affine assumption.
2. The experiments fail to convince me that the theory can be applied to more complicated problems. The synthetic data is simply a transformation of three-component MoG. The real data is MNIST, which has obvious clustering property. Both datasets are too simple.
questions: 1. The theorems in sec. 3 show that under certain conditions, all global minima of a DGM are the same up to an equivalence relation. But the current gradient based training of DGMs cannot guarantee finding a global minimum. Is there a way to analyze the identifiability of learned parameters of a DGM?
limitations: The authors have discussed limitations.
review_title: null

Meta-review: metareview: All three reviewers gave solid recommendations for acceptance. Authors provided a fairly detailed response to each review; each reviewer confirmed they would maintain their positive scores. Clear accept.
confidence: Certain
Decision: Accept