Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors investigate the reasoning behind the posterior collapse in VAEs and attribute it to non-identifiability. They propose an identifiable VAE that avoids the posterior collapse from happening.
main_review: General comments:
The paper is well-structured, and the motivation is well-defined. There is a need to understand the tendency of posterior collapse to ensure that latent variable models can scale to more significant problems. Therefore, it is highly relevant to the research community.

I like the motivation of using Brenier maps.

The caveat of this research is 1) that it has not been properly compared to previous research on the topic, e.g., Chen et al. 2017, and 2) that it has not been evaluated on state-of-the-art hierarchical VAE architectures, e.g., Maaloe 2019, Vahdat 2020, and Child 2021. Finally, I'm skeptical of the reasoning of Definition 1 in the paper (see comments below), which significantly influences the rest of the analysis.

With a revision considering points 1) and 2) I would change my decision.

More in-depth comments:
You state that a posterior collapse occurs when the inferred posterior, p(z|x; best_theta), equals the uninformed prior, p(z). You also state that "when posterior collapse occurs, it prevents the latent variable from providing a meaningful summary of the dataset". Is this, in fact, true? Isn't there cases for which a neural network model can fit a complex input space into a simple non-informative prior distribution, e.g., N(0,I)? You should support your argument.

In Eq. 3 you should introduce the variables, e.g., z'.

I find the argument on non-identifiability to be very similar to the bits-back coding and information preference proposed by Chen et al. 2017, in which they introduce the Variational Lossy Autoencoders. I think that the comparison should be stated much more clearly.

From the LVAE in Sonderby et al, 2016, and other research papers, they show that posterior collapse especially occur in hierarchical latent variable models. I think that the proposed method should be tested on these cases? Furthermore, what about the recent state-of-the-art results on very deep VAEs by Maaloe 2019, Vahdat 2020, and Child 2021?

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a parametrization of the VAE decoder which ensures that it is injective in the latent space, for a fixed observation. They show that such an injective decoder implies "latent identifiability", which in turn is equivalent to he VAE avoiding posterior collapse.
main_review: The work provided by the authors is significant. The paper proves an equivalence between latent variable unidentifiability and posterior collapse. To prevent the latter, it is enough to use injective decoders in a VAE, which would result in latent variable identifiability. 
Their important main theoretical result clearly shows that posterior collapse is a property of the model and/or data rather than the optimization of inference scheme. 
The authors propose to use Brenier maps, along with an injective linear map to match dimensions, to define the decoder in their VAE. 
Finally, this is accompanied by an empirical study that validates the capabilities of their model, and their superior performance in avoiding posterior collapse when compared to baseline methods.

However, the same can't said about the clarity and quality of writing, mainly due to bad choices when it comes to notations:

1- The examples in section 2.2 are not clear. This mostly ties to bad notations: from equation on line 166, $\theta$ seems to only parameterize f, but the examples (when referring to true parameters in line 176 for e.g.) use a $\theta$ that includes both $\mu$ and $\Sigma$.

2- The authors use bold variables $\mathbf{x}$ to refer to a dataset, and non-bold variables $x_i$ to refer to the $i$-th element of the dataset, but then use a bold $\mathbf{z}_d$ in line 318 to refer to some variable that is not even defined, and I'm still not sure what it means. I don't understand why standard notations (bold letters for vectors, non-bold letters for scalars) where not used, which will tremendously improve the clarity of the paper.

3- What does the notation $x \sim EF(u; v)$ mean? I can understand that $v$ is the vector of parameters for the exponential family, but what is $u$ here?  The use of this notation is also inconsistent, c.f. eq (7) and eq (1) for e.g.

4- what is the nature of $h_l$ in equation (6)? Is it a function from $R^d$ to $R$, or a pointwise function from $R^d$ to $R^d$? surely the $z_l$s in this equations are vectors (again, use of bold letters would have helped here..) since they are multiplied by matrices $W_l$ and $A_l$, yet they are equal to $h_{l-1}(...)$, which implies that it can't be a function from $R^d$ to $R$. But then, the authors say that $h_l$ is non-decreasing, which surely implies that it is a function between 2 ordered sets ($R^d$ is not an ordered set). All of this confusion can be solved by specifying that $h_l$ is a pointwise function for e.g., or by using better notations.

5- What are $\mathcal{Z}$ and $\mathcal{X}$? These are not defined anywhere in the paper.

6- inconsistent use of $w$ and $z$ between equation (9) and unnumbered equation after line 166.

7- The proposed model is called identifiable VAE, yet this terminology was used in one of the references they cite [23].

8- in paragraph on line 149, they argue that existing approaches can help mitigate posterior collapse. If we take $\beta$-VAE for e.g., this model seeks disentangled representation by putting a high weight, via the hyperparameter $\beta$, on the KL term of the ELBO. This pushes the posterior to match the factorized prior, thus achieving "disentanglement". This leads to posterior collapse, which is opposite to what is claimed in this section, unless I misunderstood something.

While I think that the contribution of the authors is worthwhile, the current presentation is not up to the standards of NeurIPS. But I'm happy to increase my score if the authors provide solutions to my concerns above
limitations_and_societal_impact: The discussion at the end of the paper is short, and serves more as a conclusion and a recap of the contribution than the limits of their method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an explanation for the phenomenon of posterior collapse in variational autoencoders and proves that an exact notion of posterior collapse happens if and only if the latent variables are non-identifiable. Then, it proposes a new variational autoencoder model based on Brenier maps which ensures that the latent variables are identifiable. The new model can simply be “plugged into” existing variational autoencoder models. Experiments demonstrate that the new model mitigates posterior collapse.
main_review: On the theoretical side, the main contribution seems to be formalizing the definitions for posterior collapse and latent variable non-identifiability, and having the idea to study posterior collapse from this perspective. Otherwise, Theorem 1 in itself is, as the paper mentions, straightforward.

The limitation is, obviously, that the definitions and the theorem assume exactness. Approximate posterior collapse can be just as harmful, and practically indistinguishable from exact posterior collapse. The paper acknowledges some of this in lines 155-161. Nevertheless, I feel that this limitation should be emphasized earlier and more clearly, e.g., in the paragraph 37-41 or somewhere else in the introduction. When people talk about posterior collapse, my guess is that they mean approximate posterior collapse too (at least with “small epsilon”).

The Brenier maps idea is interesting. As before, it only ensures that no exact posterior collapse takes place, but the experiments seem to do a good job of showing that in practice the technique prevents approximate posterior collapse too.

On the empirical side:

In figure 1, (c) and (d), why is GMVAE trained for fewer epochs that ID-GMVAE? My guess is that it converges by that point, but I feel that something should be said explicitly, otherwise the figure looks incomplete. Also, in figure 1, (d), the legend hides a part of the data; I’m sure there’s some way to make it fit, or to move the legend somewhere else.

The results in Table 1 look solid, and they confirm the main claims of the paper.

Typo: 223-224: “is is”.

----

UPDATE: I acknowledge reading the authors' response as well as the other reviews. I maintain my original score.
limitations_and_societal_impact: I mentioned in the main review that it should be emphasized more that the theoretical guarantees only apply to exact posterior collapse. Otherwise, the paper seems fine regarding limitations.
review_title: null

Meta-review: The authors focus on a particular notion of posterior collapse in VAEs, namely $p(z|x) = p(z), \forall x$, as well as a notion of injectivity of the decoder---namely that $p(\cdot|z) \neq p(\cdot|z')$. They show that in the *exact* sense, these two notions are equivalent. Moreover, they provide an architecture that ensures injectivity by using Brenier maps (i.e. derivatives of convex functions) and provide some empirical data that this architecture is better at avoiding posterior collapse.  

The theory in the paper is quite simple and straightforward. The empirical results are fairly small-scale, but suggest this architecture may have promise to avoid posterior collapse. 

Decision: Accept (Poster)