Posterior Collapse and Latent Variable Non-identifiability
Authors: Yixin Wang
David Blei
John Patrick Cunningham
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep generative model
posterior collapse
latent variable identifiability

Abstract: Variational autoencoders model high-dimensional data by positing
low-dimensional latent variables that are mapped through a flexible
distribution parametrized by a neural network. Unfortunately,
variational autoencoders often suffer from posterior collapse: the
posterior of the latent variables is equal to its prior, rendering the
variational autoencoder useless as a means to produce meaningful
representations. Existing approaches to posterior collapse often
attribute it to the use of neural networks or optimization issues due
to variational approximation. In this paper, we consider posterior
collapse as a problem of latent variable non-identifiability. We prove
that the posterior collapses if and only if the latent variables are
non-identifiable in the generative model. This fact implies that
posterior collapse is not a phenomenon specific to the use of flexible
distributions or approximate inference. Rather, it can occur in
classical probabilistic models even with exact inference, which we
also demonstrate. Based on these results, we propose a class of
latent-identifiable variational autoencoders, deep generative models
which enforce identifiability without sacrificing flexibility. This
model class resolves the problem of latent variable
non-identifiability by leveraging bijective Brenier maps and
parameterizing them with input convex neural networks, without special
variational inference objectives or optimization tricks. Across
synthetic and real datasets, latent-identifiable variational
autoencoders outperform existing methods in mitigating posterior
collapse and providing meaningful representations of the data.

Posterior Collapse and Latent Variable Non-identifiability
Yixin Wang University of Michigan yixinwe umich. edu
David M. Blei
John P. Cunningham Columbia University jpc2 1 19c2181@columbia. edu
Columbia University
davidd.blee@coluuii...co
Abstract
Variational autoencoders model high- ligh-dimensional data by positing low low-dimensional latent variables that are mapped through a flexible distribution parametrized by a neural network. Unfortunately, variational autoencoders often suffer from posterior collapse: the posterior of the latent variables equal to its prior, rendering the variational autoencoder useless as means to produce meaningful representations. Existing approaches to posterior collapse often attribute to the use of neural networks or optimization issues due to variational approximation. In this paper, we consider posterior collapse as a problem latent variable non-identifiabilit.. We prove that the posterior collapses if and only if the latent variables non- identifiable in the generative model. This fact implies that posterior collapse is not phenomenon specifie to the of flexible distributions approximate inference. Rather, can occur in classical probabilistic models even with exact inference, which also demonstrate. Based on these results, propose class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. This model class resolves the problem of latent variable non-identifiability by leveraging bijective Brenier maps and parameterizing them with input convex neural networks, without special variational inference objectives or opúmization tricks. Across synthetic and real datasets, latent-identifiable variational autoeneoders outperform existing methods mitigating posterior collapse and providing meaningful representations of the data.
Introduction
Variational autoencoders (VAE) are powerful generative models for high-dimensional data [28, 46]. Their key idea combine the inference principles of probabilistic modeling with the flexibility of neural networks. In VAE, each datapoint is independently generated by a cow-dimensional latent variable drawn from prior, then mapped a flexible distribution parametrized by neural network.
Unfortunately. VAE often suffer from posterior collapse, an important and widely studied phc- nomenon where the posterior of the latent variables is equal prior [6, 8, 38, 62]. This phenomenon also known as latent variable collapse, KI. vanishing, and over-pruning. Posterior collapse ren- ders the VAE useless produce meaningful representations, in much as its per-datapoint latent variables all have the exact same posterior.
Posterior collapse is commonly observed in the VAE whose generative model is highly flexible, leading to the common speculation that posterior collapse occurs because VAE involve flexible neural networks in the generative model [11], because uses variational inference [59]. Based these hypotheses, many of the proposed strategies for mitigating posterior collapse thus focus on modifying the variational inference objective (c.g. [44]), designing special optimization schemes for variational inference in VAB (e.g. [18, 25, 32D, or limiting the capacity of the generative model (e.g. [6, 16, 60).)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we consider posterior collapse as problem of latent variable non non-identifiability. We prove that posterior collapse occurs if and only if the latent variable non-identifiable in the generative model, which loosely means the likelihood function does not depend on the latent variable [40, 42, 56]. Below, we formally establish this equivalence by appealing to recent results in Bayesian non-identifability [40, 42, 43, 49, 58).
More broadly, the relationship between posterior collapse and latent variable non-identifiability implies that posterior collapse is not a phenomenon specific to the use of neural networks or variational inference. Rather, it can also occur in classical probabilistic models fitted with exact inference methods, such as Gaussian mixture models and probabilistic principal component analysis (PPCA). This relationship also leads to a new perspective on existing methods for avoiding posterior collapse, such as the delta- VAE [44] or the B-VAE [19]. These methods heuristically adjust the approximate inference procedure embedded in the optimization of the model parameters. Though originally motivated by the goal of patching the variational objective, the results here suggest that these adjustments are useful because they help avoid parameters at which the latent variable non- identifiable and, consequently, avoid posterior collapse.
The relationship between posterior collapse and non-identifiability points a direct solution the problem: we must make the latent variable identifiable, To this end, we propose latent-identifiable VAE, a class of VAE that is as flexible as classical VAE while also being identifiable. Latent- identifiable VAE resolves the latent variable non- con-ieentifiability by leveraging Brenier maps [36. 39] and parameterizing them with input-convex neural networks [2, 35]. Inference on identifiable VAE uses the standard variational inference objective, without special modifications or optimization tricks. Across synthetic and real datasets, we show that identifiable VAE mitigates posterior collapse without sacrificing fidelity to the data.
Related work. Existing approaches to avoiding posterior collapse often modify the variational
inference objective, design new initialization or optimization schemes for VAE, or add neural network links between each data point and their latent variables [1, 3, 6, 8, 12, 15, 16, 17, 18, 21, 25, 27, 32, 34, 38. 44, 50, 51, 52, 55, 61, 62, 63]. Several recent papers also attempt to provide explanations for posterior collapse. Chen et al. [8] explains how the inexact variational approximation can lead to inefficiency of coding in VAE, which could lead to posterior collapse due to a form of information preference. Dai et al. [11] argues that posterior collapse can be partially attributed to the local optima in training VAE with deep neural networks. Lucas et al. [33] shows that posterior collapse is not specific to the variational inference training objective; absent a variational approximation, the log marginal likelihood of PPCA has bad local optima that can lead to posterior collapse. Yacoby et al. [59] discusses how variational approximation can select an undesirable generative model when the generative model parameters are non-identifiable. In contrast to these works, we consider posterior collapse solely as a problem of latent variable non-identiffability, and not of optimization, variational approximations, or neural networks per se. We use this result to propose the identifiable VAE as way to directly avoid posterior collapse.
Outside VAF, latent variable identifiability in probabilistic models has long been studied in the statisties literature [40, 42, 42, 43, 49, 56, 58]. More recently, Betancourt [5] studies the effect of latent variable identifiability on Bayesian computation for Gaussian mixtures. Khemakhem et al. [23, 24] propose to resolve the non-identifiabiiit in deep generative models by appealing to auxiliary data. Kumar& & Poole [29] study how the variational family can help resolve the non- identifiability of VAE. These works address the identifiability issue for a different goal: they develop identifiability conditions for different subsets of VAE, aiming for recovering true causal factors of the data and improving disentanglement or out-of-distribution generalization. Related to these papers, demonstrate posterior collapse
additional way that the concept of identifiability, though classical, can be instrumental in modern probabilistic modeling. Considering identifiability leads to new solutions to posterior collapse.
Contributions. We prove that posterior collapse occurs if and only if the latent variable in the generative model is non-identifiable. We then propose iitett-ieettfiable VAE, class of VAE that are as flexible as classical VAE but have latent variables that are provably identifiable. Across synthetic and real datasets, we demonstrate that latent-identifiable VAE mitigates posterior collapse without modifying VAE objectives or applying special optimization tricks
2

---

2 Posterior collapse and latent variable non-identifaabiiity
Consider a dataset x ,, xn); each datapoint is m-dimensional. Positing n latent variables z = (2),..,,22)) a variational autoencoder (VAE) assumes that each datapoint x; is generated by K-dimensional latent variable 2;:
(I) where x; follows exponential family distribution with parameters fo(2;): fe parameterizes the conditional likelihood. In a deep generative model fo is a parameterized neural network. Classical probabilistic models like Gaussian mixture model [45] and probabilistic PCA [10, 47, 48, 54] are also special cases of Eq. 1.
2 ~ p(z;),
x; 11, - = |2;;
(f((2)),
To fit the model, VAE optimizes the parameters 0 by maximizing variational approximation of the log marginal likelihood. After finding an optimal 0, we can form a representation of the data using the approximate posterior dz(2 xx) with variational parameters & or its expectation E lx]
Note that here we abstract away computational considerations and consider the ideal case where the exact. This choice is sensible: if the exact posterior suffers from poste- rior collapse then so will the approximate posterior (a variational approximation cannot "uncollapse" a collapsed posterior). That said we also note that there exist in practice situations where variational inference alone can lead to posterior collapse. A notable example is when the variational approx- imating family is overly restrictive: it is then possible to have non-collapsing exact posteriors but collapsing approximate posteriors.
variational approximation
2.1 Posterior collapse <> Latent variable non-identifiability
We first define posterior collapse and latent variable non-identifiabiiltyy then proving their connection. Definition 1 (Posterior collapse [6, 8, 38, 62)). Given probability model x a parameter value 0 0, and a dataset x (x)). the posterior of the latent variables z collapses if
p(z|=;0)= = p(z).
The posterior collapse phenomenon can occur in a variety of probabilistic models and with different latent variables. When the probability model is a VAE, only has local latent variables z = (2), and Eq. 2 is equivalent the common definition of posterior collapse p(z; xx; ; 0) p(2;)
for all i [12, 17,33, 33, 44]. Posterior collapse has also been observed in Gaussian mixture models [5]; the posterior of the latent mixture weights resembles their prior when the number of mixture components in the model is larger than that of the data generating process. Regardless of the model, when posterior collapse occurs, it prevents the latent variable from providing meaningful summary of the dataset.
Definition 2 (Latent variable non-identifiability [42, 56]). Given a likelihood function p(x æ 0), parameter value 0 Ô, and a dataset x ,,,...,,,,.>
N,). the latent variable z non-identifiable if
p ; p(x θ)
where I denotes the domain of z. and z' æ' refer two arbitrary values the latent variable z can take. As a consequence, for any prior p(z) on z, we have the conditional likelihood equal to the marginal p(x z 2; p(x ()p(z) p(x; 0) Z.
Definition says latent variable z is non-identifiable when the likelihood of the dataset x does not depend z. It is also known as practical non-identifiability [42, 56] and is closely related to the definition of z being conditionally non -in-ienttffible (or conditionally uninformative) given & [40, 42, 43, 49, 58]. To enforce latent variable identifiability, it is sufficient to ensure that the likelihood p(x 2, 0) is an injective (a.k.a. one-to-one) function for all 0. this condition holds then
p(x| ;'; * p(xl = =2;0).
Note that latent variable non non- non-identifiability only requires Eq. 3 be true for given dataset and parameter value Thus a latent variable may be identifiable in a model given one dataset but not another, and at one but not another. See examples in Appendix A.
Latent variable identifiability (Definition 2) [42, 56] differs from model identifiability [41], a related notion that has also been cited as a contributing factor posterior collapse [59]. Latent variable
3

---

identifiability is a weaker requirement: it only requires the latent variable be identifiable at particular parameter value 0 0, while model identifiability requires both a and 0 be identifiable.
We now establish the equivalence hetween posterior collapse and latent variable non-identifiability. Theorem (Latent variable con-identifiability <> Posterior collapse). Consider a probability model p(x,2; 2; 0), a dataset x, and parameter value 0= 0. The local latent variables non-identifiable at if and only if the posterior of the latent variable z collapses, p(z xx) p(z).
Proof. To prove that non-identifiability implies posterior collapse, note that, by Bayes rule,
p(z (x; xx x p(z)p(x1 x; 2; 0) p(z)p(x;;)) 0x p(z),
(5)
where the middle equality is due to the definition of latent variable non non-identifiability. It implies p(z Ix; = p(z) as both are densities. To prove that posterior collapse implies latent variable non- identifiability, we again invoke Bayes rule. Posterior collapse implies that p(z) = p(z Ix; 0) p(z), p(x (); which further implies that p(x) z; 0) is constant in z. If p(æ|2; 0) nontrivially depends 2, then p(z) must be different from p(z) p(xx 0) function of z.
The proof of Theorem is straightforward. but Theorem has an important implication. shows that the problem of posterior collapse mainly arises from the model and the data, rather than from inference or optimization. If the maximum likelihood parameters Ó of the VAE renders the latent variable non-identifiable, then we will observe posterior collapse. Theorem 1 also clarifies why posteriors may change from non-collapsed to collapsed (and back) while fitting a VAE, When fitting VAE, Some parameter iterates may lead to posterior collapse: others may not.
Theorem points to why existing approaches can help mitigate posterior collapse. Consider the β- VAE [19], the VAE lagging encoder [18], and the semi-amortized VAE [25]. Though motivated by other perspectives, these methods modify the optimization objectives or algorithms of VAE to avoid parameter values 0 at which the latent variable is non non-identifiable The resulting posterior may not collapse, though the optimal parameters for these algorithms no longer approximates the maximum likelihood estimate.
Theorem 1 can also help us understand posterior collapse observed in practice, which manifests as the phenomenon that the posterior is approximately (as opposed to exactly) equal to the prior, p( Ix; Ô) = p(z). In several empirical studies of VAE (e.g. [12. 18, 25]), we observe that the Kullback-Leibler (KL) divergence between the prior and posterior is close to zero but not exactly zero, a property that stems from the likelihood p(x 22) being nearly constant in the latents z. In these cases, Theorem I provides the intuition that the latent variable is nearly non-identifiable. p(x|=))= = p(x) (()])) Vž, E' and so Eq. holds approximately.
2.2 Examples of latent variable non-identifiability and posterior collapse
We illustrate Theorem 1 with three examples. Here we discuss the example of Gaussian mixture VAE (GMVAE). See Appendix A for probabilistic principal component analysis (PPCA) and Gaussian mixture model (GMM).
The GMVAE [13, 51] is the following model:
p(2)) Categorical(1/K), p(w; (2:; N (12, f.)= Im),
where N4 's are d-dimensional, Σn are d x d-dimensional, and the parameters are 0 0= (,,, ) Suppose the function f fully flexible; thus f(w;) can capture any distribution of the data. The latent variable of interest is the categorical Zz,, If its posterior collapses, then p(z; 1: (x) = 1/K for all le I.
Consider fitting a GMVAE model with K = to a dataset of 5,000 samples. This dataset is drawn from GMVAE also with K 2 well-separated clusters; there is model misspecification. A GMVAE is typically fit by optimizing the maximum log marginal likelihood 0 argmax, log p(x)() Note there may be multiple values of 0 that achieve the global optimum of this function.
We focus on two likelihood maximizers. One provides latent variable identifiability and the posterior of 2; does not collapse. The other does not provide identifiablity; the posterior collapses.

---

1. The first likelihood- maximizing parameter 01 is the truth; the distribution of the K fitted clusters correspond the K fata-generating clusters. Given this parameter, the latent variable 2; is identifiable because the K data-generating clusters are different; different cluster memberships zi must result in different likelihoods p(x; 2 O2). The posterior of 2; does not collapse.
2. In the second same distribution, each of which is equal to the marginal distribution of the data. Specifically, (g) (0,Ia) for all h, and each fitted cluster is mixture of the K original data generating clusters, i.c., the marginal. At this parameter value, the model is still able to fully capture the mixture distribution of the data. However, all the K mixture components the latent variable 2; is non-identifiable; different cluster membership 2; do not result in different likelihoods p(x; 2 02), and hence the posterior of 2; collapses. Figure la illustrates fit of this (non-identifiable) GMVAE to the pinwheel data [22]. In Section 3, we construct an latent- identifiable VAE (LIDVAB) that avoids this collapse.
parameter 0z. however, all K fitted clusters share the
the same, and thus
Latent variable identifiability is a function of the both the model and the true data-generating distribution. Consider fitting the same GMVAE with K = 2 but to a different dataset of 5,000 samples, this one drawn from a GM with only one cluster. (There is model misspecification) One maximizing parameter value 0g is where both of the fitted clusters correspond to the true data generating cluster. While this parameter value resembles that of the first maximizer 0o above- both correspond to the true data generating cluster- this dataset leads variable identifiability. The two fitted clusters are the same and so different cluster memberships do not result in different likelihoods of p(x; 12 0g). The latent variable 2; is not identifiable and its posterior collapses.
different situation for latent
Takeaways. The GMVAE example in this section (and the PPCA and GMM examples in Appendix A) illustrate different ways that a latent variable can be non-identifiable in a model and suffer from posterior collapse. They show that even the true posterior wwithout variational inference can collapse in non-identifiable models. They also illustrate that whether a latent variable is identifiable can depend on both the model and the data. Posterior collapse is an intrinsic problem of the model and the data, rather than specific to the use of neural networks or variational inference.
The equivalence between posterior collapse and latent variable non-identifiability in Theorem also implies that, to mitigate posterior collapse, we should try to resolve latent variable non-identifiability. In the next section, we develop such class of latent-identifiable VAE.
3 Latent-identiffable VAE via Brenier maps
We now construct latent-identifiable VAB, a class of VAE whose latent variables are guaranteed to be identifiable, and thus the posteriors cannot collapse.
3.1 The latent-identifiable VAE
To construct the latent-identifiable VAE, we rely on key observation that, to guarantee latent variable identifiability, it is sufficient to make the likelihood function P(x; |2;; 0) injective for all values of O. If the likelihood is injective, then, for any 6, each value of will lead to different distribution Pxx; ()) In particular, this fact will be true for any optimized Ó and so the latent z must be identifiable, regardless of the data. By Theorem 1, its posterior cannot collapse.
Constructing dieettiideniiiiale VAE thus amounts constructing an injective likelihood function for VAE. The construction is based on a few building blocks of linear and nonlinear injective functions, then composed into an injective likelihood p(x; |2; ; 0) mapping from 2d to 2mm where .Z and & indicate the set of values 2; and x; take. For example, if x; is an m-dimensional binary vector, then 2 = (0, 1ym; if z; is K- K-dimensional real-valued vector, then . = =dd_
The building blocks of LIDVAE: E: Injective functions. For linear mappings from Rd to R (d2 2 d1), we consider matrix multiplication by a d xdz-dimensional matrix B. For a d -imeensiona variable 2, left multiplication by a matrix B is injective when / has full column rank [53]. For example, matrix with all ones in the diagonal and all other entries being zero has full column rank.
For nonlinear injective functions, we focus on Brenier maps [4, 37J. A d d-dimensional Brenier map is is the gradient of a convex function from 2d to R. That is, a Brenier map satisfies VT for some
5

---

convex function T:R - R. Brenier maps are also known as monotone transport map. They are guaranteed to be bijective [4, 37] because their derivative is the Hessian of a convex T, which must be positive semidefinite and has a nonnegative determinant [4].
To build VAE with Brenier maps, require a neural network parametrization of the Brenier map.
As Brenier maps are gradients of convex functions, we begin with the neural network parametrizaton
of convex fuunction, namely the input convex neural network (ICNN) [2. 35j. This parameterization of convex functions will enable Brenier maps to be paramterized as the gradient of ICNN.
An L-layer ICNN is neural network mapping from Rd to IP. Given an input u E R2, its Ith layer is
zo = u,
=+ + A Ab),,
(I=00..,,L-11
(6)
where the last layer z must be a scalar, (Wy) are non- non-negative weight matrices with Wo = 0. The functions (/1 : R R) are convex and non-decreasing entry-wise activation functions for layer I; they are applied clement-wise to the vector (Wzz)+ + Ayu + by). A common choice of ho : R
is the
square of leaky RELU, ho(x) = xx,)) with 0.2; the remaining h1 's are set to be leaky RELU, (x)x)= max(a x, x,)). This neural network is called "input convex" because it is guaranteed to be convex function.
Input convex neural networks can approximate any convex function on compact domain in sup norm (Theorem 1 of Chen et al. (9].) Given the neural network parameterization of convex functions, we can parametrize the Brenier map g-(-) as its gradient with respect to the input go(u) = / This neural network parameterization of Brenier map is universal approxiamtor of all Brenier maps compact domain, because input convex neural networks are universal approximators of convex functions [9].
The latent-identifaable VAE (LIDVAE). We construct injective likelihoods for LIDVAE by composing two bijective Brenier maps with an injective matrix multiplication. As the composition of injective and bijective mappings must be injective, the resulting composition must be injective. Suppose 1 : R RK and ø2 RD Rd are two Brenier maps, and f is K D D-dimensional matrix (D 2 K) with all the main diagonal entries being one and all other entries being zero. The matrix B has full column rank, so multiplication by 6 injective. Thus the composition g2 g1,(()) must be an injective function from a low-dimensional space RK to a high-dimensional space RP.
Definition CLateett-ieentfiable VAE (LIDVAE) via Brenier maps), An LIDVAE via Brenier maps generates D-dimensional datapoint x;,E E [1,...,n]] by:
2 ~ p(z)),
x - EF(x; 1g2 g1,((2)))),
where EF stands for exponential family distributions; z; is a -diimnnsional latent variable, discrete or continuous. The parameters of the model are 0 = (g1,9,82,),, where g1,8 : R - R and
82 : ReD - RDP are two continuous Brenier maps. The matrix is K D-dimensional matrix (D 2 K) with all the main diagonal entries being one and all other entries being zero.
Contrasting LIDVAE (Eq. 7) with the classical VAE (Eq, I), the LIDVAE replaces the function fo: ==_ with the injective mapping (2 g),) composed by bijective Brenier maps g 1,,00 and zero-one matrix B with full column rank. As the likelihood functions of exponential family are injective, the likelihood function p(æ; z;; 0) (f g g1, of LIDVAE must be injective. Therefore, replacing an arbitrary function fo 7 2 with the injective mapping "2 gg plays crucial role in enforcing identifiability for latent variable and avoiding posterior collapse in LIDVAE. As the latent 2; must be identifiable in LIDVAE, its posterior does not collapse.
Despite its injective likelihood, LIDVAR are as flexible as VAE; the use of Brenier maps and ICNN does not limit the capacity of the generative model. Loosely, LIDVAE can model any distributions in R because Brenier maps can map any given non-atomic distribution in Rd to any other one in Rd [37]. Moreover, the ICNN parametrization is a universal approximator of Brenier maps [2]. We summarize the key properties of LIDVAE in the following proposition.
Proposition 2. The latent variable z; is identifiable LIDVAE, i.e. for all i E (1, we have
plx; 12, =3) p(x, l2; = 0)
V
(8)
Moreover, for any VAE-generated data distribution, there exists an LIDVAE that can generate the same distribution. (The proof is in Appendix B.)

---

Hoyer o-gees
loye gouss
ligg
2
E-layer SPWWI
loyer sntvid
100
200 300
200
E22
(a) Non-ID GMVAF
(h) IDGMVAR
(c) Accuracy
(d) Log-likelihood
Figure 1: (a)-(b): The posterior of the classical GMVAE [13, 26, 51] collapses when fit to the pinwheel dataset; the latents predict the same value for all datapoints. The posteriors of latent- identifiable Gaussian mixture VAE (LIDGMVAE), however, do not collapse and provide meaningful representations. (c)-(d The latent-identifiable GMVAT produces posteriors that are substantially more informative than GMVAE when fit fashion MNIST. It also achieves higher test log likelihood.
3.2 Inference in LIDVAE
Performing inference in LIDVAE is identical the classical VAE, as the two VAE differ only in their parameter constraints. To fit an LIDVAF, we use the classical amortized inference algorithm of VAE; we maximize the evidence lower bound (ELBO) of the log marginal likelihood [28].
In general, LIDVAE are a drop-in replacement for VAE. Both have the same capacity (Proposition 2) and share the same inference algorithm, but LIDVAE identifiable and does not suffer from posterior collapse. The price we pay for LIDVAE is computational: the generative model (i.e. decoder) parametrized using the gradient of neural network; its optimization thus requires calculating gradients of the gradient of neural network, which increases the computational complexity of VAE inference and can sometimes challenge optimization. While fitting classical VAE using stochastic gradient descent has O(k-p) computational complexity, where & is the number of iterations and p is the number of parameters, fitting fatent-identifiable VAE may require Ole p²) computational complexity.
3.3 Extensions of LIDVAE
The construction of LIDVAE reveals a general strategy to make the latent variables of generative models identifiable: replacing nonlinear mappings with injective nonlinear mappings. We can employ this strategy to make the latent variables of many other VAE variants identifiable. Below we give two examples, mixture VAE and sequential VAE.
The mixture VAE, with GMVAE as special case, models the data with an exponential family mixture and mapped through a flexible neural network to generate the data. We develop its latent- identifiable counterpart using Brenier maps.
Example 1 (Latent-identifable mixture VAE (LIDMVAE)). An LIDMVAE generates a D- dimensional datapoint x;, (1,...,n) by
Categorical(1/K)) wilz; ~ EF(w; 1 (J 21), xilw; EF(x; (2 g1,(w;))),
(9)
where W; is a K-dimensional one-hot vector that indicates the cluster assignment. The parameters of the model are (g,,) where the functions g1 .R. - R and g2 BR RD are two continuous Brenier maps. The matrices Bi and ß2 are a K x M-dimensional matrix (M z K) and M x D-dimensional matrix (D M) respectively, both having all the main diagonal entries being one and all other entries being zero.
The LIDMVAE differs from the classical mixture VAE in p(x;22;), where we replace its neural network mapping with its injective counterpart, i.c. composition of two Brenier maps and matrix multiplication g
( f g-)())). As a special case, setting both exponential families in Example I as Gaussian gives us LIDGMVAE, which we will use to model images in Section 4.
Next derive the identifiable counterpart of sequential VAE, which models the data with an autoregressive model conditional on the latents.
Example 2 (Catent-identtffable sequential VAE (LIDSVAE)). An LIDSVAE generates a D- dimensional datapoint x;, i (1,...,n)) by
Zi p(z)),
xi xilzi, EF g10((2,,
7

---

Aishion-MNIS,
Omniglot
AU
KL
MI
LL
AU
KL
MI
LL
VAB [28]
0.1
0.2
0.9
-258.8
0.02
0.0
0.1
-862.1
SA-VAE [25]
0.2
0.3
1.3
-252.2
0.1
-853.4
Lagging VAE [18]
0.4
0.6
1.6
-248.5
0.5
1.0
3.6
-849.4
B- BAV [19] (f=0.2)
0.6
1.2
2.4
-245.3
0.7
1.4
5.9
-842.6
LIDGMVAE (this work)
1.0
1.6
2.6
-242.3
1.0
1.7
7.5
-820.3
Synthetic
Yahoo
Yelp
AU
KL
MI
LL
AU
KL
MI
LL
AU
KL
MI
LL
VAE [28]
0.0
0.0
0.0
-46.5
0.0
0.0
0.0
-519.7
0.0
0.0
0.0
-635.9
SA VAE [25]
0.4
0.1
0.1
400.2
0.2
1.0
0.2
-520.2
0.1
1,9
0,2
-631.5
Lagging VAE [18]
0.5
0.1
0.1
-40.0
0.3
1.6
0,4
-518.6
0.2
3.6
0.1
-631.0
B-VAE (B=0.2)
1.0
0.1
0.1
-39,9
0.5
4.7
0.9
-524.4
0.3
10.0
0.1
-637.3
LIDSVAE
1.0
0.6
-40.3
0.8
7.2
-519.5
0.7
9.1
-634.2
Table 1: Across image and text datasets, LIDVAE outperforms existing VAE variants in preventing posterior collapse while achieving similar goodness-of-lit the data.
where Xej (x1,..., represents the history of x before the ith dimension. The function fo Kei - R maps the history X into an H-dimensional vector. Finally, (zi,fo(x<<)] an (K + +D)x vector that represents row-stack of the vectors (2j)g=1 and (fe(x<;)H<<1
Similar with mixture VAE, the LIDSVAE also differs from sequential VAE only in its use of (2 g-)(-) function p(x; xi,x) We will use LIDSVAE model text Section 4.
4 Empirical studies
We study LIDVAE on images and text datasets, finding that LIDVAE do not suffer from posterior collapse increase the capacity of the generative model, while achieving similar fits to the data. We further study PPCA, showing how likelihood functions nearly constant in latent variables lead to collapsing posterior even with Markov chain Monte Carlo (MCMC).
4.1 LIDVAE on images and text
We consider three metries for evaluating posterior collapse: (1) KL divergence between the posterior and the prior, x))l(())) (2) Percentange of active units (AU):AU
d=1 ((Covptx))(Ettta [=,]) 2 c), where zạ = (21d...., is the dth dimension of the latent variable for all the data points. In calculating AU, we follow Burda et al. [7] to calculate the posterior mean, (E [x,]),,,,,,, xx, ] for all data points, and calculate the sample variance of IE [2id | across i's from this vector. The threshold is chosen be 0.01 [7]: the theoretical maximum of AA
U is one; (3) Approximate Mutual information (MI) between x; and 2; [xx 2) = Ex [ [log(q(2 [x)]]]
Ex [E (z)]{ We also evaluate the model using the importance weighted estimate of log- likelihood on held-out test set [7]. For mixture VAE, also evaluate the predictive accuracy of the categorical latents against ground truth labels to quantify their informativeness.
Competing methods. We compare LIDVAE with the classical VAE [28]. the BAV (p=0.2) [19], the semi-amortized VAE [25], and the lagging VAE [18]. Throughout the empirical studies, we use flexible variational approximating families (ReaINVPs [14] for image and LSTMss [20] for text).
Results: Images. We first study LIDGMVAE on four subsampled image datasets drawn from
pinwheel [22], MNIST [31], Fashion MNIST [57], and Omniglot [30]. Figures la and 1b illustrate fit of the GMVAE and the LIDGMVAL
the pinwheel data [22]. The posterior of the GMVAE latents collapse, attributing all datapoints to the same latent cluster. In contrast, LIDGMVAE produces categorical latents faithful to the clustering structure. Figure examines LIDGMVAE as we increase the flexibility of the generative model. Figure le shows that the categorical latents of the LIDGMVAE are substantially more predictive the true labels than their classical counterparts. Moreover, its performance does not degrade the generative model becomes more flexible. Figure ld

---

12
12
12
52
-2
2
-2
-2
posteror
Densty
Denstoy
(a) 0.2
(b) 0.5
(c) 110
(d) 1.5
Figure 2: As the noise level increases in PPCA, the latent variable becomes closer to non-identifiable because the likelihood and more susceptible to posterior collapse. Its likelihood surface becomes flatter and its posterior becomes closer to the prior. Top panel: Likelihood surface of PPCA function of the two latents 21. When increase, the likelihood surface becomes flatter and the latent variables 21.22 are closer to non-identifiable. Bottom panel: Posterior of 2 under different values. When ar increase, the posterior becomes closer to the prior.
shows that the LIDGMVAE consistently achieve higher test log-likelihood. Table 1 compares different variants of VAE in a 9-layer generative model. Across four datasets, LIDGM VAE mitigates posterior collapse. It achieves higher AU, KL and MI than other variants of VAE. It also achieves higher test log-likelihood.
Results: Text. We apply LIDSVAE to three subsampled text datasets drawn from synthetic text dataset, the Yahoo dataset, and the Yelp dataset [60]. The synthetic dataset generated from a classical two-layer sequential VAE with five-dimensional latent. Table compares the LIDSVA VAE with the sequential VAE. Across the three text datasets, the LIDSVAE outperforms other variants of VAE in mitigating posterior collapse, generally achieving higher AU, KI,, and MI.
4.2 Latent variable non-identifiabiliy and posterior collapse in PPCA
Here show that the PPCA posterior becomes close to the prior when the latent variable becomes close to be non-identifiable. We perform inference using Hamiltonian Monte Carlo (HMC), avoiding the effect of variational approximation posterior collapse.
Consider a PPCA with two latent dimensions, p(z;) (2;; 0,12), p(a; (2; =); o. Is), where the value of o² is known, zz 's are the latent variables of interest, and w is the only parameter of interest. When the noise 6² is set large value, the latent variable 2; may become nearly non-identifiable. The reason is that the likelihood function p(x; (2;) becomes slower-varying as o increases. For example, Figure shows that the likelihood surface becomes flatter d² increases. Accordingly, the posterior becomes closer to the prior as o² increases. When o = 1.5, the posterior collapses. This non-identifiability argument provides an explanation to the closely related phenomenon described in Section 6.2 of [33].
5 Discussion
In this work, we show that the posterior collapse phenomenon is problem of latent variable non- identifiability. It is not specific to the use of neural networks or particular inference algorithms in VAE. Rather,
an intrinsic issue of the model and the dataset. To this end, we propose class of LIDVAE via Brenier maps to resolve latent variable non-identifiability and mitigate posterior collapse. Across empirical studies, we find that LIDVAE outperforms existing methods mitigating posterior collapse.

---

The latent variables of LIDVAE are guaranteed to be identifiable. However, it does not guarantee that the latent variables and the parameters of LIDVAE are jointly identifiable. In other words, the LIDVAE model may not be identifiable even though its latents are identifiable. This difference between latent variable identifiability and model identifiability may appear minor. But the tractability of resolving latent variable identifiability plays key role in making non-identifiability fruitful one perspective of posterior collapse. To enforce latent variable identifiability, it is sufficient to ensure that the likelihood p(x Ix,0) 2) is injective function of In contrast, resolving model identifiability for the general class of VAE remains long standing open problem, with some recent progress relying auxiliary variables [23, 24]. The tractability of resolving latent variable identifiability is key catalyst a principled solution to mitigating posterior collapse.
There are a few limitations of this work. One is that the theoretical argument focuses on the collapse of the exact posterior. The rationale that, if the exact posterior collapses, then its variational approximation must also collapse because variational approximation of posteriors cannot "uncollapse" a posterior. That said, variational approximation may "collapse" a posterior, i.e. the exact posterior does not collapse but the variational approximate posterior collapses. The theoretical argument and algorithmic approaches developed in this work does not apply to this setting, which remains an interesting venue of future work.
second limitation is that the batent-identifiabl VAE developed in this work bear higher computa- tional cost than classical VAE. While the latent-identifaable VAE ensures the identifiability of its latent variables and mitigates posterior collapse. it does come with a price in computation because its generative model (i.e. decoder) is parametrized using gradients of a neural network. Fitting the latent-identifaable VAE thus requires calculating gradients of gradients of neural network, leading to much higher computational complexity than fitting the classifical VAE. Developing computation- ally efficient variants of the latent-identifiable VAE is another interesting direction for future work.
Acknowledgments. We thank Taiga Abe and Gemma Moran for helpful discussions, and anonymous reviewers for constructive feedback that improved the manuscript. David Blei is supported by ONR N00014- 17-11411- ONR Amazon, and the Simons Foundation. John Cunningham supported by the Simons Foundation, McKnight Foundation, Zuckerman Institute, Grossman Center, and Gatsby Charitable Trust.
NSF CCF-1740833 DARPA SD2 FA8750-18- 18-C--133,
10

---

References
[1] Alemi, A. A.. Poole. B., et al. (2017). Fixing broken ELBO. arXiv preprini arXiv:1711.00464.
[2] Amos, B., Xu, L., & Kolter, J. Z. (2017). Input convex neural networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70 (pp. 146-155), JMLR. org.
[3] Asperti, A. (2019). Variational autoencoders and the variable collapse phenomenon. Sensors & Transducers, 234(6), 1-8.
[4] Ball, K. (2004). An elementary introduction to monotone transportation. In Geometric aspects of functional analysis (pp. 41-52). Springer.
[5] Betancourt, M. (2017). Identifying Bayesian mixture models. https: haassss//mmsstttee org/users/ Aocumentations case- studies/ wtwprriyyyyssmiiueeennnoom
Accessed: 2021-05-04.
[6] Bowman, S., Vilnis, L.., et al. (2016). Generating sentences from continuous space. In Proceedings The 20th SIGNLL Conference on Computational Natural Language Learning (pp. 10-21).
[7] Burda, Y., Grosse, R., & Salakhutdinov, R. (2015). Importance weighted autoencoders. arXiv preprint arXiv:/509.005199
[8] Chen, X., Kingma, D. P., et al. (2016). Variational lossy autoencoder. arXiv preprint arXiv:1611.0273
[9] Chen, Y., Shi, Y., & Zhang, B. (2018), Optimal control via neural networks: A convex approach. arXiv preprint arXiv: 1805. 18835.
[10] Collins, M., Dasgupta, S., Schapire, R. (2001), generalization of principal components analysis to the exponential family. In Nips, volume 13 (pp.23).
[11] Dai, B., Wang, 7., Wipf, D. (2019). The usual suspects? Reassessing blame for VAE posterior collapse. arxiv preprint orvi 19t.. 10702.
[12] Dieng, A. B., Kim, Y., Rush, A. M., & Blei, D. M. (2018), Avoiding latent variable collapse with generative skip models, arXiv preprint 1oXXiv/8807.444
[13] Dilokthanakul, N., Mediano, P. A., et al. (2016). Deep unsupervised clustering with Gaussian mixture variational autoencoders. arXiv preprint
[14] Dinh, I.,, Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using real NVP. arXiv preprint arXiv:/605.0880
[15] Fu, H., Li, C., et al. (2019). Cyclical annealing schedule: A simple approach to mitigating KL vanishing. arXiv preprint arXiv:1903 1903. 10145.
[16] Gulrajani, I., Kumar, K., al. (2016), Pixelvae: latent variable model natural images. arXiv preprint
arXiv:1611.0501.
[17] Havrylov, Titov, (2020). Preventing posterior collapse with Levenshtein variational autoencoder. arXiv preprint arXiv:2004.1477
[18] He, J., Spokoyny. D., Neubig, G.. Berg- KinK-aikpa T. (2019). Lagging inference networks and posterior collapse variational autoencoders. arXiv preprint vrXiv:1901.05::
[19] Higgins, I., Matthey, L., al. (2016). B-VAE: Learning basic visual concepts with constrained variational framework.
[20] Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735 1780.
[21] Hoffman, M. D. & Johnson, M. J. (2016). ELBO surgery: Yet another way to carve up the variational evidence lower bound.
[22] Johnson, M. J., Duvenaud, D. K., Wiltschko, A., Adams, R. P., Datta, S. R. (2016). Composing graphical models with neural networks for structured representations and fast inference. In Advances in neural information processing systems (pp.
[23] Khemakhem, I., Kingma, P., Hyvärinen, A. (2019). Variational autoencoders and nonlinear ICA: unifying framework. arXiv preprint 19070:19077044800
[24] Khemakhem, I., Monti, R. P., Kingma, D. & Hyvarinen, (2020). ICF-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA.
[25] Kim, Y., Wiseman, S., Miller, A.. Sontag, D., & Rush, A. (2018). Semi-amortized variational autoencoders.
In International Conference on Machine Learning (pp. 2678-2687)
[26] Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Advances in neural information processing systems (pp. 3581-3589).
[27] Kingma, D. P., Salimans, T., et al. (2016). Improved variational inference with inverse autoregressive flow.
In Advances in neural information processing systems (pp. 074.4743-440 4751).
11

---

[28] Kingma, D. P.& & Welling, M. (2014). Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), volume 1.
[29] Kumar, A. & Poole, B. (2020). On implicit regularization in B-vaes. In International Conference on Machine Learning (pp. 5480-5400000 5490).: PMLR.
[30] Lake, B. M., Salakhutdinov, R., Tenenbaum, J. B. (2015). Human-level concept learning through probabilistic program induction, Science, 350(6266), 1332-1338.
[3I] LeCun, Y., Cortes, C., & Burges, C. (2010). MNIST handwritten digit database. ATT Labs [Online]. 2.
Available: http://www..euu...//eeee///iiiea
[32] Li, B., He, J., Neubig. G., Berg- Kirrgarrka. T., & Yang. Y. (2019). A surprisingly effective fix for deep latent variable modeling of text. arXiv preprint 1xXiv:199,000663
[33] Lucas, J., Tucker, G., Grosse, R. B., & Norouzi, M. (2019). Don't blame the ELBO! A lincar VAE perspective on posterior collapse. In Advances in Neural Information Processing Systems (pp. (403-9413).
[34] Maalo L., Fraccaro, M., Liévin, V., & Winther, O. (2019). BIVA: A very deep hierarchy of latent variables for generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox,
R. Garnett (Eds.), ) Advances in Neural Information Processing Systems, volume 32: Curran Associates, Inc.
[35] Makkuva, A. V., Taghvaei, A., Oh. S., & Lee. J. D. (2019). Optimal transport mapping via input convex neural networks. arXiv preprint arXiv:1908. 10962.
[36] McCann, R. J. et al. (1995). Existence and uniqueness of monotone measure- ceaasur-prresevvng maps. Duke Mathematical Journal, 80(2), 309-324.
[37] McCann, R. & Guillen, N. (2011). Five lectures optimal transportation: geometry, regularity and applications. Analysis and geometry of metric measure spaces: Lecture notes of the séminaire de Mathématiques Supérieure (SMS) Montréal, (pp. 145-180).
[38] Oord, A. d., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning, arXiv preprint 1rXiv:1111.00937
[39] Peyré, G., Cuturi, M., al. (2019). Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6), 355-607.
[40] Poirier, D. (1998). Revising beliefs in nonidentificd models. Econometric Theory, 14(4), 483-509.
[41] Rao, B. & Prakasa, R. (1992). Identifubillity in Stochastic Models: Characterization of Probability Distributions. Probability and mathematical statistics. Academic Press.
[42] Raue, A., Kreutz, C., et al. (2009). Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics, 25(15), 1923-1929.
[43] Rauc, A., Kreutz, C., Theis, J., & Timmer, (2013). Joining forces of Bayesian and frequentist methodology: a study for inference in the presence of non-identifiability. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1984), 201 10544.
[44] Razavi, A., Oord, A. v. d., Poole, B., Vinyals, O. (2019). Preventing posterior collapse with VAFs.. arXiv preprint arXiv: 1901.03416.
[45] Reynolds, D. A. (2009). Gaussian mixture models. Encyclopedia of biometrics, 741, 659 259-663.
[46] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4000
[47] Roweis. S. (1998). Em algorithms for pea and spea. Advances in neural information processing systems, (pp. 626-632).
[48] Rowcis, S. & Ghahramani, Z. (1999). A unifying review of lincar gaussian models. Neural computation, 11(2). 305-345.
[49] San Martin, E. & González, J. (2010). Bayesian identifiability: Contributions to an inconclusive dehate. Chilean Journal of Statistics, 1(2), 69-91.
[50] Seybold, B., Fertig, E., Alemi, A., & Fischer. I. (2019). Dueling decoders: Regularizing variational autoencoder latent spaces. arXiv preprint arXiv: arXXvv/905.000 190.00788
[51] Shu, R. (2016). Gaussian mixture VAE: Lessons in variational inference, generative models, and deep nets. [52] Senderby, C. K., Raiko, T., Maaløe, I.., Sonderby, K., Winther, O. (2016). How train deep variational autoencoders and probabilistic ladder networks. In 33rd International Conference on Machine Learning (ICML 2016).
[53] Strang, G., Strang, G., Strang, G., & Strang, G. (1993). Introduction linear algebra, volume 3. Wellesley- Cambridge Press Wellesley, MA.
[54] Tipping, M. E. & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodologyj, 61(3). 611-622.
12

---

[55] Tomezak, J. M. & Welling, M. (2017). VAE with VampPrior. arXiv preprini arXiv:
[56] Wieland, F.G., Hauber, A. L., Rosenblatt, M., Tönsing, C., & Timmer, J. (2021). On structural and practical identifiability. Current Opinion in Systems Biology.
[57] Xiao, H., Rasul, K., & Vollgraf. R. (2017). Fashion-MNIST. novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747
[58] Xie, Y. & Carlin, B. P. (2006). Measures of Bayesian learning and identifiability in hierarchical models. Journal of Statistical Planning and Inference, 136(10), 3458 -477.
[59] Yacoby, Y.. Pan, W., & Doshi- Velez, (2020), Characterizing and avoiding problematic global optima of variational autoencoders.
[60] Yang, Z., Hu. Salakhutdinov, R., Kirkpairka, (2017). Improved variational autoencoders for text modeling using dilated convolutions. Proceedings of the 34th International Conference on Machine Learning- Volume 70 (pp. 3889-3.
[61] Young, S., Kannan, A.. Dauphin, Y., & Fei-Fei, L. (2017). Tackling over-pruning in variational autocncoders. arXiv preprint arviv: 1eoe:010666666 03643.
[62] Zhao, T., Lee, K., & Eskenazi, M. (2018). Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volame 1: Long Papers) (pp. 1098-1107).
[63] Zhao, Y., Yu, P., Mahapatra, S., Su, Q. & Chen, C. (2020). Discretized bottleneck in VAE: Posterior- collapse-free
learning. arXiv preprint urXiv:2004.10603
13


Reviews:
[{"review_id": "qkKqJRc6uWC", "reviewer": "Reviewer_dUTF", "summary": "The paper proposes an explanation for the phenomenon of posterior collapse in variational autoencoders and proves that an exact notion of posterior collapse happens if and only if the latent variables are non-identifiable. Then, it proposes a new variational autoencoder model based on Brenier maps which ensures that the latent variables are identifiable. The new model can simply be “plugged into” existing variational autoencoder models. Experiments demonstrate that the new model mitigates posterior collapse.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the theoretical side, the main contribution seems to be formalizing the definitions for posterior collapse and latent variable non-identifiability, and having the idea to study posterior collapse from this perspective. Otherwise, Theorem 1 in itself is, as the paper mentions, straightforward.\n\nThe limitation is, obviously, that the definitions and the theorem assume exactness. Approximate posterior collapse can be just as harmful, and practically indistinguishable from exact posterior collapse. The paper acknowledges some of this in lines 155-161. Nevertheless, I feel that this limitation should be emphasized earlier and more clearly, e.g., in the paragraph 37-41 or somewhere else in the introduction. When people talk about posterior collapse, my guess is that they mean approximate posterior collapse too (at least with “small epsilon”).\n\nThe Brenier maps idea is interesting. As before, it only ensures that no exact posterior collapse takes place, but the experiments seem to do a good job of showing that in practice the technique prevents approximate posterior collapse too.\n\nOn the empirical side:\n\nIn figure 1, (c) and (d), why is GMVAE trained for fewer epochs that ID-GMVAE? My guess is that it converges by that point, but I feel that something should be said explicitly, otherwise the figure looks incomplete. Also, in figure 1, (d), the legend hides a part of the data; I’m sure there’s some way to make it fit, or to move the legend somewhere else.\n\nThe results in Table 1 look solid, and they confirm the main claims of the paper.\n\nTypo: 223-224: “is is”.\n\n----\n\nUPDATE: I acknowledge reading the authors' response as well as the other reviews. I maintain my original score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I mentioned in the main review that it should be emphasized more that the theoretical guarantees only apply to exact posterior collapse. Otherwise, the paper seems fine regarding limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gSa9rKH9YUw", "reviewer": "Reviewer_Fkyr", "summary": "The authors investigate the reasoning behind the posterior collapse in VAEs and attribute it to non-identifiability. They propose an identifiable VAE that avoids the posterior collapse from happening.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General comments:\nThe paper is well-structured, and the motivation is well-defined. There is a need to understand the tendency of posterior collapse to ensure that latent variable models can scale to more significant problems. Therefore, it is highly relevant to the research community.\n\nI like the motivation of using Brenier maps.\n\nThe caveat of this research is 1) that it has not been properly compared to previous research on the topic, e.g., Chen et al. 2017, and 2) that it has not been evaluated on state-of-the-art hierarchical VAE architectures, e.g., Maaloe 2019, Vahdat 2020, and Child 2021. Finally, I'm skeptical of the reasoning of Definition 1 in the paper (see comments below), which significantly influences the rest of the analysis.\n\nWith a revision considering points 1) and 2) I would change my decision.\n\nMore in-depth comments:\nYou state that a posterior collapse occurs when the inferred posterior, p(z|x; best_theta), equals the uninformed prior, p(z). You also state that \"when posterior collapse occurs, it prevents the latent variable from providing a meaningful summary of the dataset\". Is this, in fact, true? Isn't there cases for which a neural network model can fit a complex input space into a simple non-informative prior distribution, e.g., N(0,I)? You should support your argument.\n\nIn Eq. 3 you should introduce the variables, e.g., z'.\n\nI find the argument on non-identifiability to be very similar to the bits-back coding and information preference proposed by Chen et al. 2017, in which they introduce the Variational Lossy Autoencoders. I think that the comparison should be stated much more clearly.\n\nFrom the LVAE in Sonderby et al, 2016, and other research papers, they show that posterior collapse especially occur in hierarchical latent variable models. I think that the proposed method should be tested on these cases? Furthermore, what about the recent state-of-the-art results on very deep VAEs by Maaloe 2019, Vahdat 2020, and Child 2021?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "ZjeuFkHmh1", "reviewer": "Reviewer_qzPP", "summary": "The authors propose a parametrization of the VAE decoder which ensures that it is injective in the latent space, for a fixed observation. They show that such an injective decoder implies \"latent identifiability\", which in turn is equivalent to he VAE avoiding posterior collapse.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work provided by the authors is significant. The paper proves an equivalence between latent variable unidentifiability and posterior collapse. To prevent the latter, it is enough to use injective decoders in a VAE, which would result in latent variable identifiability. \nTheir important main theoretical result clearly shows that posterior collapse is a property of the model and/or data rather than the optimization of inference scheme. \nThe authors propose to use Brenier maps, along with an injective linear map to match dimensions, to define the decoder in their VAE. \nFinally, this is accompanied by an empirical study that validates the capabilities of their model, and their superior performance in avoiding posterior collapse when compared to baseline methods.\n\nHowever, the same can't said about the clarity and quality of writing, mainly due to bad choices when it comes to notations:\n\n1- The examples in section 2.2 are not clear. This mostly ties to bad notations: from equation on line 166, $\\theta$ seems to only parameterize f, but the examples (when referring to true parameters in line 176 for e.g.) use a $\\theta$ that includes both $\\mu$ and $\\Sigma$.\n\n2- The authors use bold variables $\\mathbf{x}$ to refer to a dataset, and non-bold variables $x_i$ to refer to the $i$-th element of the dataset, but then use a bold $\\mathbf{z}_d$ in line 318 to refer to some variable that is not even defined, and I'm still not sure what it means. I don't understand why standard notations (bold letters for vectors, non-bold letters for scalars) where not used, which will tremendously improve the clarity of the paper.\n\n3- What does the notation $x \\sim EF(u; v)$ mean? I can understand that $v$ is the vector of parameters for the exponential family, but what is $u$ here?  The use of this notation is also inconsistent, c.f. eq (7) and eq (1) for e.g.\n\n4- what is the nature of $h_l$ in equation (6)? Is it a function from $R^d$ to $R$, or a pointwise function from $R^d$ to $R^d$? surely the $z_l$s in this equations are vectors (again, use of bold letters would have helped here..) since they are multiplied by matrices $W_l$ and $A_l$, yet they are equal to $h_{l-1}(...)$, which implies that it can't be a function from $R^d$ to $R$. But then, the authors say that $h_l$ is non-decreasing, which surely implies that it is a function between 2 ordered sets ($R^d$ is not an ordered set). All of this confusion can be solved by specifying that $h_l$ is a pointwise function for e.g., or by using better notations.\n\n5- What are $\\mathcal{Z}$ and $\\mathcal{X}$? These are not defined anywhere in the paper.\n\n6- inconsistent use of $w$ and $z$ between equation (9) and unnumbered equation after line 166.\n\n7- The proposed model is called identifiable VAE, yet this terminology was used in one of the references they cite [23].\n\n8- in paragraph on line 149, they argue that existing approaches can help mitigate posterior collapse. If we take $\\beta$-VAE for e.g., this model seeks disentangled representation by putting a high weight, via the hyperparameter $\\beta$, on the KL term of the ELBO. This pushes the posterior to match the factorized prior, thus achieving \"disentanglement\". This leads to posterior collapse, which is opposite to what is claimed in this section, unless I misunderstood something.\n\nWhile I think that the contribution of the authors is worthwhile, the current presentation is not up to the standards of NeurIPS. But I'm happy to increase my score if the authors provide solutions to my concerns above", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion at the end of the paper is short, and serves more as a conclusion and a recap of the contribution than the limits of their method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors focus on a particular notion of posterior collapse in VAEs, namely $p(z|x) = p(z), \forall x$, as well as a notion of injectivity of the decoder---namely that $p(\cdot|z) \neq p(\cdot|z')$. They show that in the *exact* sense, these two notions are equivalent. Moreover, they provide an architecture that ensures injectivity by using Brenier maps (i.e. derivatives of convex functions) and provide some empirical data that this architecture is better at avoiding posterior collapse.  

The theory in the paper is quite simple and straightforward. The empirical results are fairly small-scale, but suggest this architecture may have promise to avoid posterior collapse. 

Decision: Accept (Poster)