Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders
Authors: Hien Dang
Tho Tran Huu
Tan Minh Nguyen
Nhat Ho
Venue: ICLR 2024 poster (2024)
Keywords: variational autoencoders
posterior collapse
Primary area: generative models
TL;DR: We prove and study posterior collapse occurence for conditional and hierarchical variational autoencoders

Abstract: The posterior collapse phenomenon in variational autoencoder (VAE), where the variational posterior distribution closely matches the prior distribution, can hinder the quality of the learned latent variables. As a consequence of posterior collapse, the latent variables extracted by the encoder in VAE preserve less information from the input data and thus fail to produce meaningful representations as input to the reconstruction process in the decoder. While this phenomenon has been an actively addressed topic related to VAE performance, the theory for posterior collapse remains underdeveloped, especially beyond the standard VAE. In this work, we advance the theoretical understanding of posterior collapse to two important and prevalent yet less studied classes of VAE: conditional VAE and hierarchical VAE. Specifically, via a non-trivial theoretical analysis of linear conditional VAE and hierarchical VAE with two levels of latent, we prove that the cause of posterior collapses in these models includes the correlation between the input and output of the conditional VAE and the effect of learnable encoder variance in the hierarchical VAE. We empirically validate our theoretical findings for linear conditional and hierarchical VAE and demonstrate that these results are also predictive for non-linear cases with extensive experiments.

Published as a conference paper at ICLR 2024
BEYOND VANILLA VARIATIONAL AUTOENCODERS: DETECTING POSTERIOR COLLAPSE IN CONDITIONAL AND HIERARCHICAL VARIATIONAL AUTOENCODERS
Hien Dang
Tho Tran
FPT Software Al Center danghoanghien1122 1ee com
FPT Software Al Center
Photrannuu99 9mali9999mmmm1.mmmmm com
Tan Nguyen
Nhat Ho
Department of Mathematics
Department of Statistics and Data Sciences University of Texas at Austin
National University of Singapore tanmn@nus .ww.enuussuuuuuse.......ee edu.sg
minhnhat@utexea edu
ABSTRACT
The posterior collapse phenomenon variational autoencoder (VAE), where the variational posterior distribution closely matches the prior distribution, hinder the quality of the learned latent variables. As a consequence of posterior collapse, the latent variables extracted by the encoder in VAE preserve less information from the input data and thus fail to produce meaningful representations as input the reconstruction process in the decoder. While this phenomenon has been an actively addressed topic related to VAE performance, the theory for posterior col- lapse remains underdeveloped, especially beyond the standard VAE. In this work, we advance the theoretical understanding of posterior collapse to two important and prevalent yet less studied classes of VAE: conditional VAE and hierarchical VAE. Specifically, via a non-trivial theoretical analysis of linear conditional VAE and hierarchical VAE with two levels of latent, we prove that the cause posterior collapses in these models includes the correlation between the input and output of the conditional VAE and the effect of learnable encoder variance in the hierarchi- cal VAE. We empirically validate theoretical findings for linear conditional and hierarchical VAE and demonstrate that these results are also predictive for non-linear cases with extensive experiments.
INTRODUCTION
Variational autoencoder (VAE) (Kingma & Welling, 2013) has achieved successes across unsuper- vised tasks that aim find good wiw-demnsiioal representations of high-dimensional data, ranging from image generation (Child, 2021; ; Vahdat & Kautz, 2020) and text analysis (Bowman et al., 2015; Miao al., 2016; Guu et al., 2017) clustering (Jiang al., 2016) and dimensionality duction (Akkari et al., 2022). The success of VAE relies on integrating variational inference with flexible neural networks to generate new observations from an intrinsic low-dimensional latent struc- ture (Blei et al., 2017). However, it has been observed that when training to maximize the evidence lower bound (ELBO) of the data's log- log-liielhhod,, the variational posterior of the latent variables in VAE converges to their prior. This phenomenon is known as the posterior collapse. When posterior collapse occurs, the data does not contribute to learned posterior distribution of the latent vari- ables, thus limiting the ability of VAE to capture intrinsic representation from the observed data. It widely claimed the literature that causes of posterior collapse are due to: i) the Kull- back Lack-eeibler (KL) divergence regularization factor in ELBO that pushes the variational distribution towards the prior, and ii) the powerful decoder that assigns high probability to the training samples when posterior collapse occurs. A plethora of methods have been proposed mitigate the effect of the KL-regularization term in ELBO training process by modifying the training objective functions (Bowman al., 2015; Huang et al., 2018; Sønderby al., 2016; Higgins al., 2016;
"Co-last authors. Please correspond danghoanghien 123 123@gmail.com

---

Published as a conference paper at ICLR 2024
Razavi et al., 2019) by redesigning the network architecture of the decoder to limit its representa- tion capacity (Gulrajani et al., 2017; Yang et al., 2017; Semeniuta et al., 2017: Van Den Oord et al,, 2017: Dieng et al., 2019; Zhao al., 2020). However, the theoretical understanding of posterior collapse has still remained limited due to the complex loss landscape of VAE.
Given that the highly non-convex nature of deep nonlinear networks imposes a sig- nificant barrier to the theoretical understanding of posterior collapse, linear VAEs are good candi- date model for providing important theoretical insight into this phenomenon and have been recently studied in (Wang& & Ziyin, 2022; Lucas et al., 2019). Nevertheless, these works only focus on the simplest settings of VAE, which are the linear standard VAE with one latent variable (see Figure ((a) for the illustration). Hence, the theoretical analysis of other important VAEs architectures has remained elusive.
Contribution:
In this paper, we advance the theory of posterior collapse to two important and prevalently used classes of VAE: Conditional VAE (CVAE) (Sohn et al., 2015) and Markovian Hierarchical VAE (MHVAE) (Luo, 2022). CVAE is widely used in practice for structured prediction tasks (Sohn et al., 2015; Walker et al., 2016). By conditioning on both latent variables and the input condition in the generating process, CVAE overcomes the limitation of VAE that the generating process cannot be controlled. On the other hand, MHVAE is an extension of VAE to incorporate higher levels of latent structures and is more relevant to practical VAE architecture that use multiple layers of latent variable to gain greater expressivity (Child, 2021; Vahdat & Kautz, 2020; Maaløe et al., 2019). Moreover, studying MHVAE potentially sheds light on the understanding of diffusion model (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020), since diffusion model can be interpreted as a simplified version of deep MHVAE (Luo, 2022). Following common training practice, we consider linear CVAE and MHVAE with adjustable hyperparameter IS's before each KL-regularization term to balance latent channel capacity and independence constraints with reconstruction accuracy as in the 3- VAE (Higgins et al., 2016). Our contributions are four-fold:
1. We first revisit linear standard VAE and verify the importance of learnability of the encoder variance to posterior collapse existence. For unlearnable encoder variance, we prove that posterior collapse might not happen even when the encoder is low-rank (see Section 3).
2. We characterize the global solutions of linear CVAE training problem with precise condi- tions for the posterior collapse occurrence. We find that the correlation of the training input and training output is one of the factors that decides the collapse level (see Section 4.1).
3. We characterize the global solutions of linear two-latent MHVAE training problem and point out precise conditions for the posterior collapse occurrence. We study the model having separate fos and find their effects the posterior collapse of the latent variables (see Section 4.2).
4. We empirically show that the insights deduced from our theoretical analysis are also pre- dietive for non-linear cases with extensive experiments (see Section 5).
Notation: We will use the following notations frequently for subsequent analysis and theorems.
For imput data x E Ro,, we denote A := Ex (ax ). the second moment matrix. The eigenvalue decomposition of A is A P 4P A with &b do and P R Doxoo where do S Do is the number of positive eigenvalues of A. For CVAE, in addition to the eigenvalue decomposition of condition æ as above, we define similar notation for output y. B = E,, dd2x and P De where d2 D2 is the number of positive eigenvalues of B. Moreover, consider the whitening transformation ã 4- 11/2p] E Idd and v 1 By E RR,, It is clear that Ez Idg- (ūg = Id, and v, can be written
PB v with v e
= P 1/2 and y P a 1/2g. The KL divergence of two probability distributions P and Q is denoted as DRL. (PlI)).
BACKGROUND ON VARIATIONAL AUTOENCODERS
Variational Autoencoder: VAE represents a class of generative models assuming each data point x is generated from an unseen latent variable. Specifically, VAE assumes that there exists a latent vari- able 2 E Rd,, which can be sampled from a prior distribution p(2) (usually a normal distribution), and the data can be sampled through conditional distribution p(a) (2) that modeled as decoder. Because the marginal log-like likeliiodood log p(x) is intractable, VAE uses amortized variational infer- ence (Blei al., 2017) approximate the posterior p(2(1:) via variational distribution (=2(x)). The
2

---

Published as a conference paper at ICLR 2024
(x(aa)))
qulair,a) at)
(o(2i(a)
(x(22le))
puly
x
po(a|z)
y
pa(ala))
pa(silas)
(a) Standard VAE
(b) Conditional VAE
(c) Markovian Hicrarchical VAE
Figure 1: Graphical illustration of standard VAE, CVAE, and MHVAE with two latents.
variation inference q(2[2) ) is modeled as an encoder that maps data a into the latent 2 in latent space. This allows tractable approximate inference using the evidence lower bound (ELBO):
ELBOVAR == (lostt))loopp(())
DRu (x(())() I (og)(
Standard training involves maximizing the ELBO, which includes a reconstruction term [tts):/llogp(()))2 and KL--ivergence regularization term DxL (g(=)=)\l(<)))
Conditional Variational Autoencoder: One of the limitations of VAE is that we cannot control its data generation process. Assume that we do not want to generate some random new digits, but some certain digits based on our need, or assume that we are doing the image inpainting problem: given an existing image where a user has removed an unwanted object, the goal is to fill in the hole with plausible- lookkng pixels. Thus, CVAE is developed to address these limitations (Sohn et al., 2015; Walker et al., 2016). CVAE is an extension of VAE that include input condition æ, output variable y. and latent variable 2. Given training example (x, y). CVAE maps both and y into the latent space the encoder and use both the latent variable 2 and input x in the generating process. Hence, the variational lower bound can be rewritten follows:
ELBOcvae (=lea)))) [log (y =)] DxL (xa(ca((ls,,))lp((ee))) log poyl())
(2)
where p( (=((x)) still a standard Gaussian because the model assumes sampled independently of at test time (Doersch, 2016).
Hierarchical Variational Autoencoder: Hicrarchical VAE (HVAE) is a generalization of VAE by introducing multiple latent layers with hierarchy to gain greater expressivity for both distributions (<(2))) and ()(x)2) (Child, 2021). ) In this work, we focus on special type of HVAE named the Markovian HVAE (Luo, 2022). In this model, the generative process is Markov chain, where decoding each latent z, only conditions on previous latent 21+1- The ELBO in this case becomes
log p (ax 21:T) 2o (ZU.T x)
( Po (æ 21) I Po (#c-1 21)
Eg (x (m)
(x((zur(m) log
ao
(21
x)) II 96 (2e 2t-1)
3 REVISITING LINEAR VAE WITH ONE LATENT AND MORE
We first revisit the simplest model, i.e., linear standard VAE with latent variable, which has also been studied in (Lucas et al., 2019; Dai et al., 2020: Wang & Ziyin, 2022), with two settings: learn- able and unlearnable (i.e., predefined and not updated during the training of the model) diagonal encoder variance. For both settings, the encoder a linear mapping W E Rdd XDo that maps input data Po latent space 2 E Rd Applying the reparameterization trick (Kingma Welling, 2013), the latent z is produced by further adding noise term & (O, E), i.c., = Wr + & with Σ E Rdh xd; is the encoder variance. Thus, we have the recognition model qo (2(x) = (Wwx, E). linear map that parameterizes the distribution p=(1]2) U Doxd, and Tjaec is unlearnable. The prior p(z) is (O0, v with known Teenc Note that we do not include bias in the linear encoder and decoder, since the effect of bias term is equivalent centering both input and output data to zero mean (Wang & Ziyin, 2022). Therefore, adding the bias term does not affect the main results. After dropping the multiplier 1/2 and other constants, the negative ELBO becomes
The decoder
N(U2,11ae with
1 _E. [|| xll2 + Aracee(U")) U+ Be (U
LVAE
3c 1<]|]= log E/
(3)
2 Taee
3

---

Published as a conference paper at ICLR 2024
Mase//eee Bene. Our contributions are: i) we characterize the global solutions of linear VAE training problem for unlearnable Σ with arbitrary elements on the diagonal, which is more general than the Proposition 2 in (Wang & Ziyin, 2022) where only the unlearnable isotropic Σ o²1 is considered, and ii) we prove that for the case of unlearnable Σ, even when the encoder matrix is low-rank, posterior collapse may not happen. While it has been known that for learnable Σ, low-rank encoder matrix certainly leads to posterior collapse (Lucas et al., 2019; Dai et al., 2020). Thus, learnable latent variance is among the causes of posterior collapse, opposite to the results in Section 4.5 in Wang & Ziyin (2022) that "a learnable latent variance is not the cause of posterior collapse". We will explain this point further after Theorem 1. Recalling the notations defined in Section 1, the following theorem characterizes the global minima (U", W)) when minimizing CVar for unlearnable Σ case. In partieular, we derive the global minima's s SVD with closed-fone form singular values, via the SVD and singular values of the matrix Z == B.g (a:37) and other hyperparameters.
where e M
Theorem 1 (Unlearnable E). Let Z = (xT ) = RRSS is the SVD of Z with singular val- i= in non-increasing order and define V := WP WP441/2 With unlearnable Σ = diag (of................. . (2)) ). the optimal solution of (U", W-) of CvAe as follows:
ues (0.) do
U²=R2T RRTT V = W' -
(4)
where T E Rdixxd xds is an orthonormal matrix that sort the diagonal of E in non-decreasing order, L.e. Σ = -EETT = T diag(o"?....,, yT with of² .... S o²2 Ω E RDxdd and A E Rdixdo are rectangular diagonal matrices with the following elements, Vie [di]:
w/- = - Vmax 0,
V Biidee
00 BBr,
Tddec
Idde 'Jonc
) x3 = mmx (0, nenco
(0 (0:-v3o) Bo,
Ndec Tlene
If do < d1, we denote 0₂- = 0 for do < <is d1.
The proof of Theorem 1 in Appendix D. We note that our result allows for arbitrary predefined values of { thus is more general than the Proposition 2 in (Wang & Ziyin, 2022) where 's are all equal to a constant. Under broader settings. there are two notable points from Theorem that have not been captured in the previous result of (Wang & Ziyin, 2022): i) at optimality, the singular matrix T sorts the set (or) in non-decreasing order, and ii) singular values w; and X; are calculated via the i-th smallest value of of the set { not necessarily the i-th element d;-
The role of learnability of the encoder variance to posterior collapse: From Theorem 1, the ranks of both the encoder and decoder depend on the sign of 0, /Bo' The model becomes low-rank when the sign of this term is negative for some i. However, low-rank V is not sufficient for the occurrence of posterior collapse, it also depends on the sorting matrix T that sorts {a: { in non-decreasing order. For the isotropic Σ case where all a; are all equal, T can be any orthogonal matrix since { are in non-decreasing order already. Therefore, although A and AST has zero rows due to the low-rank of V V TAST might have zero rows and Wx V might have no zero component. For the k-th dimension of the latent = Wz + E to collapse, i.e., p( (xr) (O, ). we need the k-th dimension of the mean vector Wa of the posterior q(=[x) to equal 0 for any x. Thus, posterior collapse might not happen. This is opposite to the claim that posterior collapse occurs in this unlearnable isotropic Σ setting from Theorem 1 in (Wang & Ziyin, 2022). If all values d are distinct, T must be permutation matrix and hence, V = TAST has d1 rank(V") zero rows, corresponding with dimensions of latent z that follow (N))
For the setting of learnable and diagonal Σ, the low-rank structure of U and V surely leads to posterior collapse. Specifically, at optimality, we have E' Bmace TU* + Bc21)-1, and thus, T is diagonal. As a result, U* can be decomposed as U RΩ with orthonormal matrix R E Do and Ω is a rectangular diagonal matrix containing its singular values. Hence, U* has d zero columns with r == rank(U"). Dai et al. (2017) claimed that there exists an inherent mechanism to prune these superfluous columns to exactly zero Looking at the loss LAV at Eqn. (3), we see that these di zero columns of U* will make the corresponding di -r dimensions of the vector Wr to not appear in the reconstruction term UWz - 1 and they only appear in the V2 VII These dimensions of Wa subsequently becomes zeroes at optimality. Therefore, these dj - r dimensions of the latent 2 = Wz + & collapse to its prior N (0, ). The detailed analysis for learnable E case is provided in the Appendix D.2.
regularization term |
4

---

Published as a conference paper at ICLR 2024
4 BEYOND STANDARD VAE: POSTERIOR COLLAPSE IN LINEAR CONDITIONAL AND HIERARCHICAL VAE
4.1 CONDITIONAL VAE
In this section, we consider linear CVAE with input condition x E Po, the latent & E R and output y R 22_ The latent is produced by adding noise term { ~~ (O, Σ) to the output of the linear encoder networks that maps both x and y into latent space, i.e., = WiT + W2y + &. where Σ Rddxdd is the encoder variance, W1 E Rd xDo, and W2 E Rd. Hence, (o(=|0, y) = NNW + W2y, )) The decoder parameterizes the distribution p=(y)z, = N2 + U2 M D), where U₁ E DDxx, , U2 E RDxxDo, and predefined Videc- We set the prior ((z) (O, M) with pre-defined Teenc- An illustration of the described architecture is given in Figure 1(b). We note that the linear standard VAE studied in (Wang & Ziyin, 2022: Lucas al., 2019) does not capture this setting. Indeed, let us consider the task of generating new pictures. The generating distribution p(y(2) considered in (Wang & Ziyin, 2022; Lucas et al., 2019), where 2 (0, I), does not condition on the input x on its generating process.
Previous works that studied linear VAE usually assume Σ is data-independent or only linearly de- pendent to the data (Lucas al., 2019; Wang Ziyin, 2022). We find that this constraint can be removed in the analysis of linear standard VAE, CVAE, and MHVAE. Particularly, when each data has its own learnable Σg, the training problem is equivalent to using single Σ for all data (see Ap- pendix C for details). Therefore, for brevity, will use the same variance matrix E for all samples. Under this formulation, the negative ELBO loss function in Equ. (2) can be written as:
1 EM, [ W + U2) + (U, W2 1)yl
LCVAE(W, W2. U1, 22 Σ)
Valec
+ +trace U, EU + Bc (1[ + W2yll + trace Σ))] 3d1 3 log Σ
(5)
where c := Titec/ /eeee/Tene Tene and 8 > Comparing to the loss LVAE of standard VAE, minimizing LevAE is a more complicated problem due to the fact that the architecture of CVAE requires two additional mappings, including the map from the output to latent 2 the encoder and the map from condition to output y in the decoder. Recall the notations defined in Section 1, we find the global minima of Levar and derive the closed-form singular values of the decoder map U, in the following theorem. We focus on the rank and the singular values of U1 because they important factors that influence the level of posterior collapse (i.c., how many latent dimensions collapse), which we will explain further after the theorem.
Theorem 2 (Learnable E). Let = Mide// Z = En.g(ūTT) Redoxd and define E P/2 Z ZZZ))//2p3 increasing order. The optimal solution of (U;, U2 Wi, We. E²) of LeVAE is as follows:
PeQ be the SVD of E with singular values {0,] in non-
U = PQR T Σ- = Braee (UT (UTTU +3e
where R Rddi is an orthonormal matrix. Ω is the rectangular singular matrix of Ui with diagonal elements {wi ) li=1 and variance Σ" diag(o?, o?...., .2,) with:
1 w/ - max(0, (0,x 02 Brine): of =
v Bene /iee| Vo, nenc.
if 0; 2 Indee Vie [di]
(6)
if0: < Br
Tooe
where {o is permutation of J i.e. diaglo o'a
R diag(o).....o.)))))) Ra, If
d2 < d1. we denote f; = ffr d2 i d1. The other matrices obey:
U&P U 11/2 = P/22
W P W1/P ZTZ = (U; + -U WWPpy
Z ZT)
WiP
The proof of Theorem 2 is given in Appendix E. First, we notice that the rank of Uj depends on the sign of 8; - 3y When there are some i's that the sign is negative, the map Uf from a to y in the generating process becomes low-rank. Since the encoder variance Σ is learnable, the posterior collapse surely happens in this setting. Specifically, assuming r is the rank of U and < d1, i.e. U is low-rank, then U. has dj zero columns at optimality. This makes the
5

---

Published as a conference paper at ICLR 2024
corresponding d1 dimensions of W1 and Way no longer appear in the reconstruction term IKU,, + -2)2 + (U1W2- - Tyl of the loss Levab defined in Equ. (5). These dimensions of WII and W2y only influence the term | + Wayll coming from the KL-regularization, and thus, this term forces these d1 r dimensions of W + Way to be O. Hence, the distribution of these d1 dimensions of latent variable W i Way È collapses exactly to the prior min which means that posterior collapse has occurred. Second, the singular values O's of E decide the rank of Ui and therefore determine the level of the posterior collapse of the model. If the data (z, y) has zero mean, for example, we can add bias term to have this effect, then E PE PBV' /2((1 Cov(ğ, 3) Cov(ū. =) ) positive and negative) berween r and y leads to the larger level of posterior collapse. In an extreme case, where = y, we have that E = O and U₁ = 0. This is reasonable since = J can then be directly generated from the map U22, while the Kl-reegulaizzatinn term converges to due to the complete posterior collapse. Otherwise, if is independent of y. the singular values of E are maximized, and posterior collapse will be minimized in this case, Lastly, Theorem 2 implies that sufficiently small B Midee will mitigate posterior collapse. This is aligned with the observation in (Lucas et al.. 2019; Dai et al., 2020; Wang & Ziyin, 2022) for the linear VAE model. We also note that the i-th element d; does not necessarily correspond with the i-th largest singular 0,, but it depends on the right singular matrix R of U to define the ordering.
Thus, given the same y. the larger correlation (both
4.2 MARKOVIAN HIERARCHICAL VAE
We extend our results to linear MHVAE with two levels of latent. Specifically, we study the case where the encoder variance matrix of the first latent variable En is unlearnable and isotropic, while the encoder variance of the second latent variable E₂ is either: learnable, or, ii) unlearnable isotropic. In these settings, the encoder process includes the mapping from input æ E DD to first latent 2 E Idd via the distribution (o(21 = -N(W i2, E1, W, E Red. xDo, and the mapping from 21 to second latent 22 E Rdd via qo (22|=1) = N 221, Σ2), W2 E Rddxxd, Similarly, the decoder process parameterizes the distribution (221/2) [U222, Mac I), U2 E Rdixd and p(x122)) 2U I) U1 E Doxx,, The prior distribution is given by p(21) NO I) A graphical illustration is provided in Fig. ((c). Our goal is to minimize the following negative ELBO (the detailed derivations are at Appendix F.2):
LHVAE = - EE, (2l (log - ( (a(2) =))p((()))) - (e
En Noss
| æ²I2 trace U; -, U2W:W1z Wizl| + Bi trace U2)
+ Bi trace(((2
+' (race(W221)) w.) + trace Σz))
- Bi log Σi| - B2 log Ez].
Although the above encoder consists of two consecutive linear maps with additive noises, the ELBO training problem must have extra KL-regularizer term between the two latents, i.e., DKI.
This term, named "consistency term" in (Luo, 2022), complicates the training problem much more, can be seen via the differences between LIVAR and LVAE in the standard VAE setting in Eqn. (3). We note that this model shares many similarities with diffusion models where the encoding process of diffusion models also consists of consecutive linear maps with injected noise, and their training process requires to minimize the consistency terms at each timestep. We characterize the global minima of LIVAE for learnable Σg in the following theorem. Similar as above theorems, Theorem 3 derives the SVD forms and the closed-form singular values of the encoder and decoder maps to analyze the level of posterior collapse via the hyperparameters.
Theorem (Unlearnable isotropic E,, Learnable Στ). Let c Naec/ Neno Z E,, (xăT)
ROST the SVD of Z with singular values (0 in non non-increasing order, and unlearnable Σ₁ = of with 01 > O. Assuming do 2 d1 d2. the optimal solution of (Ui, U2, Wi, W Σÿ) of LIVAE is
do
Vi WiP 1/2 PAR U3 = P
W U, (U:U;T + +c21 ", zvi (ViVi +
and Σj = 81 8, Tece U U + 1 where P, Q are square orthonormal matrices. A and Ω are rectangular diagonal matrices with the following elements, for i E [di]:

---

Published as a conference paper at ICLR 2024
a) 1) Buyz (a): B2 nace): x Vaaele 02 Be w/ =
of62
Be Vg.
Ba Vew
Bi Yeer
b) 1f <
Payie max n and of 2 f Wdec' 21 = 0, wt = =o} - 12/ /2/B1)//een
1) <
Benin ma
82 31 naee) and of < 31 Vaec x = max (0,
82
ay (0: vBia1)) w/ = O. v3.
The detailed proof of Theorem is in Appendix F.I. The proof uses zero gradient condition of critical points to derive U₁ as a function of V1 and W₂ as function of U2 to reduce the number of variables. Then, the main novelty of the proof is that we prove V1 VI and U2 are simultaneously diagonalizable, and thus, we are able to convert the zero gradient condition into relations of their singular values X's and w's. Thanks to these relations between A's and w's, the loss function now can be converted to function of singular values. The other cases of the input and latent dimensions, e.g., do < dj, are considered with details in Appendix F.I.
Theorem 3 identifies precise conditions for the occurrence of posterior collapse and the low-rank structure of the model at the optimum. There are several interesting remarks can be drawn from the results of the above theorem. First, regarding the posterior collapse occurrence, since Σ₂ is learnable and diagonal, if there are some i that W; = O. i.c., U; is low-rank, the second latent variable will exhibit posterior collapse with the number of non-collapse dimensions of 22 equal the number of non-zero wi's s. However, the first latent variable might not suffer from posterior collapse even when Vi is low-rank due the unlearnable isotropic Σ,, with the same reason that discuss in the standard VAE case in Section 3. Second, all hyperparameters, including Tidee, $1, 83, but except TJenc. are decisive for the rank of the encoders/decoders and the level of posterior collapse. In S2nie particular, the singular value wi > 0 when either 0; 2 /Bien or of 2 B2 nāce Therefore, having sufficiently small /3 and 7ldec or a sufficiently large B1 can mitigate posterior collapse for the second latent. Given W; > 0, A > 0 if 0² Bymaec > O. Hence, sufficiently small 32 and Tice will also increase the rank of the mapping from the input data x to the first latent 21. In summary, decreasing the value of f32 and 7/dec or increasing the value of 31 can avoid posterior collapse, with the former preferred since it also avoids the low-rank structure for the first latent variable. We also characterize the global minima of two-latent MHVAE with both latent variances that are unlearnable and isotropic in Appendix F.2. In this setting, posterior collapse might not happen in either of the two latent variables.
3,
Remark 1. If the second latent variable z2 suffers complete collapse, the generating distribution becomes p0(21[22) (0, I) since all columns of U2 are now zero columns. Therefore, the MHVAE model now becomes similar standard VAE with the prior NO, I) We conjecture this observation also applies to MHVAE with more layers of latent structures: when a complete posterior collapse happens latent variable, its higher-level latent variables become inactive.
4.3 MITIGATE POSTERIOR COLLAPSE
The analysis in Sections 4.1 and 4.2 identifies the causes of posterior collapse in conditional VAE and hierarchical VAE and implies some potential ways to fix it. Although the methods listed in Table are the implications drawn from results with linear setting, we empirically prove their ef- fectiveness in non-linear regime in the extensive experiments below in Section 5 and in Appendix A.
EXPERIMENTS
In this section, we demonstrate that the insights from the linear regime can shed light on the be- haviors of the nonlinear CVAE and MHVAE counterparts. Due to the space limitation, we mainly present experiments on non-linear networks in the main paper. Experiments to verify our theorems for the linear case and additional empirical results for nonlinear VAE, CVAE and HVAE along with hyperparameter details can be found in Appendix
5.1 LEARNABILITY OF ENCODER VARIANCE AND POSTERIOR COLLAPSE
In this experiment, we demonstrate that the learnability of the encoder variance Σ is important to the occurrence of posterior collapse. We separately train two linear VAE models on MNIST dataset. The first model has data-independent and learnable Σ, while the second model has fixed and unlearnable
7

---

Published as a conference paper at ICLR 2024
Conditional VAT
Markovian Hierarchical VAT
sufficiently small [3 Titec can mitigate poste- rior collapse.
A sufficiently small 83 "a can both mitigate
collapse of the second latent and increase the rank of encoder/decoder of the first latent.
Unlearnable encoder variance can prevent col- lapse. This is also true for MНVAA.
Surprisingly, using larger 31 value can mitigate the collapse for the second latent.
Since high correlation between the input condi-
Create separate maps between the input and each latent in case of complete collapsed latent causes higher-level latents losc information of the input.
tion and the output leads to strong collapse, decor-
relation techniques can help mitigate the collapse.
Table 1: Insights to mitigate posterior collapse drawn from our analysis
0.25
x: leanable
Uw
Σ
Use 0.70
Oux
(a) Linear VAE
(b) ReLU CVAE
3.55
0.5
9 mM
0.5
4 A= 10
0.75
3 2.
28
(c) ReLU MHVAE
Figure 2: Graphs of (c, collappe with varied hyperparameters (out of 5) latent dimensions collapse immediately at e = 8 x 10-5 while collapse does not happen with unlearnable and (c) Larger value of B2 Tice triggers more latent dimensions to collapse, whereas larger value of 31 mitigates posterior collapse.
0.05). (a) For learnable E,
I. (b) Larger value of or Veeec makes more latent dimensions to collapse,
E I. The latent dimension is 5, and we intentionally choose 3 = 4, Videe 1 to have 3 (out of 5) singular values 0 equal To measure the degree of posterior collapse,
the (e, -coollapse definition in (Lucas et al., 2019). Specifically, a latent dimension i of latent = is (e, -coollppd if c] 2 1 5. It is clear from Fig. 2(a) that with learnable Σ, 3 out of dimensions of the latent 2 collapse immediately at small e, while the unlearnable variance does not.
F. [KL [x)xx=))lp((z))
5.2 CVAE EXPERIMENTS
We perform the task of reconstructing the MNIST digits from partial observation as described in (Sohn et al., 2015). We divide each digit image in the MNIST dataset into four quadrants: the bottom left quadrant is used as the input x and the other three quadrants are used as the output
Varying B, Mdee experiment. We train a ReLU CVAE that uses two-layer networks with ReLU activation as its encoder and decoder with different values of 3 and M/ttc- Fig. 2(b) demonstrates that decreasing 3 and Mitec can mitigate posterior collapse, as suggested in Section 4.3.
Collapse levels on MNIST dataset. Theorem 2 implies that training linear CVAE on dataset with a lower set 0, 's i.e., the singular values of matrix defined Theorem 2, more prone to posterior collapse. To verify this insight in nonlinear settings, separately train multiple CVAE models, including linear CVAE, ReLU CVAE, and CNN CVAE, on three disjoint subsets of the MNIST dataset. Each subset contains all examples of each digit from the list 7,9). To com- pare the values of 0 between datasets, we take the sum the top-16 largest 0, 's and get the list (6.41, 13.42, 18.64) for the digit (1,9, 7), respectively. The results presented in Fig. 3(a) empiri- cally show that the values of f, 's are negatively correlated with the degree of collapse.
5.3 MHVAE EXPERIMENTS
Varying B1, 32, nae experiment. We train a ReLU MHVAE that uses two-layer networks with ReLU activation the encoder and decoder, with multiple values of 8,, 32 and Y/dec- Fig. 2(c)
8

---

Published as a conference paper at ICLR 2024
6
5
7
8
0.1
7
7
7
8
2
9
3 3
0
7 1
8
0
4
9
3
9
3
7
2
0
3
0123456789
3
3 7
8
0
2
3
5
8
8 0
8
6
6
4 7 3 3
5
6
6
9
4
6
6
2
7
1
7
3
4
0
1
O
9
3
0
5
8
8
9
01734567
9
1
5 1 9
3
1
3
3
4
0
7
2
6 2
6
5
6
6
9 1 3
9
5
6 9
9
9
1
2
/
2
1
7
7
3
0
012345672
B1
3
3
3
3
3
0.5
1.0
32
0.1
Oigt
Opk
Digi7
(a) Linear, ReLU and CNN CVAE
(b) ReLU MHVAE
Figure 3: (a) Graphs of (c, collapse for several CVAEs trained separately on each of three digit 11,9,7 7} subsets of MNIST (6 = 0.05). Dataset with smaller 0,'s 's (1 + 9 - in increasing order) has more collapsed dimensions, and (b) Samples reconstructed by nonlinear MHVAE. Smaller B/ alleviates collapse and produces better samples, while smaller 31 has the reverse effect.
demonstrates that decreasing 3g and Mea reduce the degree of posterior collapse, while decreasing 3 has the opposite effect, Theorem suggests.
Samples reconstructed from ReLU MHVAF with varied 31 and 32. We train the ReLU MHVAE with E; 0 and parameterized learnable Σz(x) ('Tanh(MIP(z)))
MNIST dataset. Fig. 3(b) aligns to the insight discussed in Section 4.3 that decreasing the value of 32 help mitigate collapse, and thus, produce better samples, while decreasing 31 leads to blurry images. The full experiment with $1, 32 E {0.1, 1.0, 2.0, 6.0} can be found in Fig. 4 in Appendix A.
6 RELATED WORKS
To avoid posterior collapse, existing approaches modify the training objective to diminish the effect of KL-regularization term in the ELBO training. such as annealing a weight on KL term during training (Bowman et al., 2015; Huang et al., 2018; Senderby et al., 2016; Higgins et al., 2016) or constraining the posterior to have a minimum KI-distance with the prior (Razavi et al., 2019). Another line of work avoids this phenomenon by limiting the capacity of the decoder (Gulrajani et al., 2017; Yang et al., 2017: Semeniuta et al., 2017) or changing its architecture (Van Den Oord al., 2017; Dieng et al., 2019; Zhao al., 2020). On the theoretical side, there have been efforts to detect posterior collapse under some restricted settings. (Dai et al., 2017; Lucas et al., 2019; Rolinek al., 2019) study the relationship of VAE and probabilistic PCA. Specifically, (Lucas et al., 2019) showed that linear VAE can recover the true posterior of probabilistic PCA. (Dai et al., 2020) argues that posterior collapse is direct consequence of bad local minima. The work that is more relatable to our work is (Wang& & Ziyin, 2022), where they find the global minima of linear standard VAE and find the conditions when posterior collapse occurs. Nevertheless, the theoretical understanding of posterior collapse in important VAE models such CVAE and HVAE remains limited. Due to space limitation, we defer the full related work discussion until Appendix B.
7 CONCLUDING REMARKS
Despite their prevalence in practical use as generative models, the theoretical understanding of CVAE and HVAE has remained limited. This work theoretically identifies causes and precise con- ditions for posterior collapse occurrence in linear CVAE and MHVAE from loss landscape perspec- tives. Some of our interesting insights beyond the results in linear standard VAE include: i) the strong correlation between the input conditions and the output of CVAE is indicative of strong pos- terior collapse, ii) posterior collapse may not happen if the encoder variance unlearnable, even when the encoder network is low-rank. The experiments show that these insights are also predictive of nonlinear networks. One limitation of our work is the case of both encoder variances are learn- able in two-latent MHVAE is not considered due to technical challenges and left future works. Another limitation is that our theory does not consider the training dynamics that lead to the global minima and how they contribute to the collapse problem.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research/project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: ALSG2-TC-2023-012-SGII NH acknowledges support from the NSF IFML 2019844 and the NSF Al Institute for Foundations Machine Learning.
REFERENCES
Nissrine Akkari, Fabien Casenave, Elie Hachem, and David Ryckelynck. A bayesian nonlinear reduced order modeling using variational autoencoders. Fluids, 7(10), 2022. ISSN 2311-5521. doi: 10.3390/fluid771003344 URL https: wwww.www......o///// www...ddp.cooom 12311-552111 1ttmmm2311-5521//110000
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizzaton,, 2018.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statis- ticians. Journal of the American Statistical Association, 112(518)859-877 apr 2017. doi: 10110800166445..211.11555 2017 1285773.
URL https: http::///oo..oog/000.00000020000
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from continuous space. In Conference on Computational Natural Language Learning, 2015.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509, 00519, 2015.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. International Conference on Learning Representations, 2021. URL https:
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint 1 1488 2017.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? reassessing blame for vae posterior collapse. International conference on machine learning, pp. 2313-2322. PMLR, 2020.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397-2405. PMLR, 2019.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05000 2016.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixeIVAE: A latent variable model for natural images. In International Conference Learning Representations, 2017. URLhttps: htt.ttps:///pnnevieew.... http//pppneeewe.eeoeee wttp:eewweeeetruremeeee 1d=BJKKvt55q.
Shuxuan Guo, Jose M. Alvarez, and Mathicu Salzmann. Expandnets: Linear over-parameterizztion to train compact convolutional networks, 2021.
Kelvin Guu, Tatsunori Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 099 2017. doi: 10. 10.1112//aclaa.000 a_00030.
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In 5th International Confer- ence Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL hwwph:tpp///cccccc TyxBORtxx.
https:://pperrwwiee //tp///ppere.....eoo/eeee net forum?id
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statisties, 50(2):949-986 2022.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2066999 URL https:://////
0. 1tt.//.....11111111 10/1/09//P.../ 201 6.90.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representa- tions, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 2020.
Chin-Wa Wei Huang, Shawn Tan, Alexandre Lacoste, and Aaron C Courville. Improving explorability in variational inference with annealed variational objectives. Advances in neural information processing systems, 31, 2018.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks, 2023.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. International Joint Con- ference Artificial Intelligence, 2016.
Kenji Kawaguchi. Deep learning without poor local minima. Advances neural information pro- cessing systems, 29, 2016.
Diederik P Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Yuri Kinoshita, Kenta Oono, Kenji Fukumizu, Yuichi Yoshida, and Shin-ichi Maeda. Control- ling posterior collapse by an inverse lipschitz constraint on the decoder network. arXiv preprint 2rXiv:2304.12270. 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Anna Kuzina and Jakub M Tomezak. Analyzing the posterior collapse in hierarchical variational autoencoders. arXiv preprint UrXiv:2302.09976 2023.
Thomas Laurent and James H. von Brecht. Deep linear networks with arbitrary loss: All local minima are global. International Conference on Machine Learning, 2018.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't 't blame the elbo! linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32, 2019.
Calvin Luo. arXiv:2208.11970, 2022.
Understanding diffusion models: A unified perspective.
arXiv preprint
Lars Maalóe, Marco Fraccaro, and Ole Winther. Semi-supervised generation with cluster-aware generative models. ArXiv, abs/1704.00637, 2017.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther, Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Interna- tional conference on machine learning, pp. 1727-1736. PMLR, 2016.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 2021.

---

Published as a conference paper at ICLR 2024
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In International Conference on Learning Representations, 2019. URL https: // /poppneeeiiww.c..mmme net. http://eeeooeeaaaaaaoaaoaaaaoaoo
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406-12415, 2019.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions the nonlinear dy- namics of learning in deep linear neural networks. CoRR, abs/1312.6120,, 2013.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. A hybrid convolutional variational toencoder for text generation. In Conference on Empirical Methods in Natural Language Pro-
cessing, 2017.
Jascha Sohl- bohl-Dicksttei, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation us- ing deep conditional generative models. In C. Cortes, N. Lawrence, D. Lee. M. Sugiyama, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 28, Cur- Associates, Inc., 2015. URL https:::: htt.ttpss//pprcceednnssssss nwwrins. www.rps..ceappp...eeeeee wwtppceepppppeeeeeeeeeeeeeeeeeee firefs/le////s file http://///.....22266666662222220
Casper Kaae Sønderby, Tapani Raiko, Lars Maaloe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances neural information processing systems, 29, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:1 13:19667-19679, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In Computer Vision-ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pp. 835-851. Springer, 2016.
Yixin Wang, David M. Blei, and John P. Cunningham. Posterior collapse and latent variable non- identifiability, 2023.
Zihao Wang and Liu Ziyin. Posterior collapse of a linear latent variable model. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Pro- cessing Systems, 2022. URL https: //penrevieeee net. forum?id=zAc22 6__aHb.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. Improved varia- tional autoencoders for text modeling using dilated convolutions. In International conference on machine learning, pp. 3881-3890. PMLR, 2017.
Yang Zhao, Ping Yu, Suchismit Mahapatra, Qinliang Su, and Changyou Chen. Improve variational autoencoder for text generationwith discrete latent bottleneck. arXiv preprint arXiv:2004, 10603, 2020.
12

---

Published as a conference paper at ICLR 2024
Appendix for "Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders"
Table of Contents
Additional Experiments and Network Training Details
14
A.I
Details of network training and hyperparameters in Section 5 in main paper
14
A.1.1
Learnability of encoder variance and posterior collapse
14
A.1.2
CVAB experiments
14
A.1.3
MHVAE experiments
14
A.2
Additional experiments
15
A.2.1
Additional experiment for VAE
15
A.2.2
Additional experiments for CVAE
16
A.2.3
Additional experiments for MHVAE
18
B Related works
20
Sample-wise encoder variance
22
C.1 Conditional VAE
22
C.2 Markovian Hierarchical VAE
22
Proofs for Standard VAE
23
D.1 Unlearnable diagonal encoder variance Σ
23
D.2 Learnable encoder variance Σ
24
E Proofs for Conditional VAE
26
Proofs for Markovian Hierarchical VAE with 2 latents
29
F.1 Learnable Σ₂ and unlearnable isotropic Σ
30
F22 Unlearnable isotropic encoder variances Σ, Σg
34
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTS AND NETWORK TRAINING DETAILS
We define Leee as the reconstruction loss for both CVAE and MHVAE. For CVAE, 1kL. is the KL- divergence DxL (gs((a (2 (eteell,,))ppppe )))=(()))) Similarly for MHVAE, IKL, and L,L,, are the KL--divergence terms (gs((s(z) (x)]ppa (21 22)) and DKL (g.(22|z1) (22]=)))(pp (22)), respectively. To measure the discrep- ancy between the empirical singular values and the theoretical singular values of the encoder and decoder networks, we define the metric DMA (fu)): {vi) = D Σen | U/ to be the mean abso- lute difference between two sets of non-increasing singular values {u,) foz ) (if the number of nonzero singular values is different between two sets, we extend the shorter set with O's match the length of the other set).
A.1 DRTAILS OF NETWORK TRAINING AND HYPERPARAMETERS IN SECTION IN MAIN PAPER
In this subsection, we provide the remaining training details and hyperparameters for experiments shown in main paper. Unless otherwise stated, all the experiments in Section 5 are trained for 100 epochs with ELBO loss using Adam optimizer, learning rate x 10-3, and batch size of 128.
A. LEARNABILITY OF ENCODER VARTANCE AND POSTERIOR COLLAPSE
Effeet of learnable and unlearnable Σ on posterior collapse (Fig. 2(a)): In this experiment, we demonstrate that the learnability of the encoder variance Σ is important to the occurrence of posterior collapse. We separately train 2 Linear VAE models on MNIST dataset. The first model has shared and learnable E, while the second model has fixed Σ I. We set do 784, dj 5, 1/ene Tiee = 1, hidden dim = 256, optimized with Adam optimizer for 100 epochs, learning rate set to 1 x 10-4 for both models. It is clear from Fig. 2(a) that with learnable Σ, 3 out of 5 dimensions of the latent = collapse with the KI. divergence smaller than 8 x 10-5, while in the same setting, the unlearnable encoder variance does not result in posterior collapse.
A.1.2 CVAE EXPERIMENTS
Varying B, Neee experiment (Fig. 2(b)): In this experiment, train ReLU CVAE with data- independent and learnable Σ on the task of reconstructing the original MNIST digit from the bot- tom left quadrant. ReLU CVAE model is obtained by replacing all the linear layers in both the encoder and the decoder in the linear CVAE by two-layer MLP with ReLU activation. We set do 196, d1 16, d2 588, Tea = 1.0 and hidden dimension h of the MLPs set to 16, learning rate set to 1 x 10-4, (0.1.0.5, 1.0,2.03. 2.03. then we run the other experiment with fixed B = 1.0 and vary Y/dec from the set
4 We first run the experiment with fixed Take = 1.0 and / chosen from the set
(0.25, 0.5, 1.0, 2.0}.
Collapse level of MNIST digit datasets (Fig. 3(a)): We separately train CVAs models, namely linear CVAE, ReLU CVAE and CNN CVAE on 3 subsets of the MNIST dataset, each subset contains all examples of each digit (1,7,9) 7,9. The linear CVAE and ReLU CVAE have Teen = Milec 0.5, 8 = 1.0 and other parameters the same as the experiment of Fig. 2(b). For CNN CVAE model, replace all hidden layers in ReLU CVAE models by convolutional layers (with ReLU activation) and other settings stay the same. For the encoder of CNN CVAE, we convolutional layers with kernel size 3 x 3 x 32, 3 x 3 x 16, and stride = 2. For its decoder, we use transposed convolutional layers with kernel sizes 3 x 3 32, 3 x 3 16, 3 x 3 x and stride = 2.
A.1.3 MHVAE EXPERIMENTS
Varying 3, 32, naee experiment with Relu MHVAE (Fig. 2(c)): In this experiment, we 2-layer MIP with ReLU and Tanh activation functions to replace all the linear layers in both encoder and decoder of Linear MHVAE. We train the model on MNIST dataset with Eg parameterized by 2-layer MLP with latent 2 as the input. We set /enc = 0.5, Σ₁ 0.521, the latent dimensions and the hidden dimension are di d2 16 and h 256, respectively. We run 3 sub-experiments as follow: i) we fixed 83g = 1, 7aee = 0.5 and then vary 3₁ from the set f0.1. 0.5, 1.0, 2.0}. ii) we fixed
31 = 1, Riee 0.5 and then vary 82 from the set {0.1, 0.5, 1.0, 2.0}. and iii) we fixed 31 = B2
and then vary fidee from the set (0.25, 0.5. 1.0, 2.0).
Samples reconstructed from ReLU MHVAF with varied 3, and 82 (Fig. 4): We vary 31, /3 from the set (0.1, 2.0, 4.0), mene and nee is set to 0.5. Other hyperparameters and the architecture of ReLU MHVAE model is identical to the experiment Fig. 2(c).
14

---

Published as a conference paper at ICLR 2024
122 2
0.1 7 1
4
6 2 07
4
5
5
8 2
8
0 7 9 2
7
4
9
9
3
0 J
2
7 0
2
20200000
9
2 7 3 6
9
3
2
0
0
3
3 4
9
#
0
9 0 G
3
9
0 0
9 10
8
9
0 2
0 0
2
7
2
0 8
313
3
2
S
0
3
8
0 0 2 & 5 4 / 9 J G / 3 3 9
2
9
4
66748 5
6
6 2 9 7 3
4
2/ 7
2
7
2
0130 =
O
3
2
31331
8 3
S
2
66995
6 G 3 4 5 2 I 7 S 3 \ 3 6 1 3 1 3 6
F 9 1 3
9
&
9
& 7 D 1
8 / 5 7
7
3
9 8 Ø
$1
27178 9
7
4
7 3 0
4
g
0130 $
O
O S
0 5 3 1676
G
5
51375
B
5
4
6699 7 5
6
6
4 5 8 3 3 6
6
2
2
2/17 4
2
7 F
2
3
&
0730 5
O
O 5 1
O
0 5 f
6
O I &
O
3 3 4
0 5
1131
3
5
3
I 6 1 7
5
66998
4 5
6
9 7 7
2 1
7 4 3
1130 0
7
O
3 3
Figure 4: Samples reconstructed by nonlinear МНVAE with different (2) combinations. Smaller 32 alleviates collapse and produces better samples, while smaller B1 has the reverse ef- fect.
Log-likelihood
KL
AU
Learnable Σ
-744.99
9.54
69%
Unlearnable Σ
- -743.63
9.80
100%
Table 2: Test log-likelihood and posterior collapse degree of Rel.U VAE on MNIST with learnable and unlearnable encoder variance.
A.2 ADDITIONAL EXPERIMENTS
In this part, we empirically verify our theoretical results for linear VAE, CVAE and MHVAE by conducting experiments on both synthetic data and MNIST dataset. Furthermore, we continue to show that the insights drawn from our analysis are also true for non-linear settings.
A.2.1 ADDITIONAL EXPERIMENT FOR VAE
Log-likelihood, KL and AU of VAE with learnable and unlearnable Σ on MNIST (Table 2): In this experiment, we use 2-layer MLP with ReLU activation to replace all linear layers in Linear VAE. We set = 0.5 and other settings for this experiment to be identical to the experiment in Fig. 2(a). We evaluate the model performance estimate of log-likelihood on a separate test set. To evaluate posterior collapse, we use two metries: 1) the KI divergence between the posterior and the prior distribution, DxL. (g(()))) the active units (AU) percentage (Wang et al., 2023; Burda et al., 2015) with = 0.01. Higher AU percentage means that more latent dimensions are utilized by the model. In this experiment, we set Do = 784, dj = 16, Teeee = Tidee 1. In Table 2, the unlearnable encoder variance with 100% AU (compared 69% AU of the learnable case) indicates that unlearnable encoder variance can help to mitigate posterior collapse. This experiment provides further evidences that unlearnable encoder variance can help alleviate posterior collapse.
generative tasks using the importance weighted
and 2)
ResNet-18 VAE with learnable and unlearnable Σ on CIFARIO (Fig. 5): In this experiment, we train ResNet-18 VAE model on CIFAR10 dataset (Krizhevsky al., 2009). The model uti- lizes ResNet- 18 architecture (He et al., 2016) to transform the input image into the latent vec- tor 21 in the encoder. In the decoder, transposed convolution layers are employed with kernel sizes of 3 x 3 x 32, 3 x 3 x 8,3 3 x 3, and stride of 2. To maintain the appropriate di- mensions for the ResNet-15 architecture and transposed convolution layers, two 2-layer MLPs utilized for intermediate transformations. For this experiment,
with Relu activation
set Dn = 3072, dj = 128, dhidtten = 512, Tenc = natec = 1.0, 3 = 2, Eunteumable I, and train the model for 100 epochs with Adam optimizer and learning rate of x 10-3 with batch size 128.Fig.
15

---

Published as a conference paper at ICLR 2024
120
100
80
E learnable
Collapsed
60
Σ
40
20
5
6
Figure 5: Graph of (e, 6)-collase of ResNet-18 VAE model with learnable E and unlearnable
Σ = I, 7nne 1 (6 0.01). Learable Σ suffers posterior collapse when most of the latent dimensions collapse to the prior at small e, while unlearnable Σ does not.
70
60
15
40
10
20
5
10
0
1
2
3
4
83
82
Alg
Sylit
(a) Linear CVAE
(b) Linear MHVAE
Figure 6: Linear CVAE and MHVAE losses on MNIST dataset with / and 32 vary, respectively. Our theory correctly predicts complete posterior collapse at / = 3.33 for CVAE, and at B3 6.17 for MHVAF
illustrates that the unlearnable encoder variance model exhibits fewer collapsed latent dimensions compared to the learnable encoder variance model with same threshold. These results justify that using an unlearnable encoder variance can alleviate posterior collapse, as pointed out in the paper.
A.2.2 ADDITIONAL EXPERIMENTS FOR CVAE
Linear CVAE (Fig. 6(a)): In this experiment, we train linear CVAE model verify the the- oretical results by checking the sign of 0 3 for posterior collapse described in Theo- rem 2. The top-1, 2, 4, 16. 32, 64 leading singular vales 0,'s of MNIST dataset are (3.33, 2.09, 1.59, 0.84 0.44, 0.19, 6.2 10 10-2). In this experiment,
set do 196, d1 64, d2 588, 7em = 7eee = 1, learning rate set to x 10-4 Thus, to determine the value of 33 that cause mode collapse, we simply set = 0. Fig. 6(a) demonstrate that the convergence of Blkt. agrees precisely with the threshold obtained from Theorem 2.
Verification of Theorem (Fig. 7, 8): To verify Theorem 2, measure the difference between the empirical singular values and theoretical singular values w and variances o² in two experiments for linear CVAE: synthetic experiment and MNIST experiment.
In the synthetic experiment, we optimize the matrix optimization problem derived in the proof of Theorem 2, which is equivalent to the minimizing negative ELBO problem. We randomly initialize each index of r, y by sampling from (0, 0.1²) and optimize the matrix optimization objective and Teac Mae
We use Adam optimizer for 200 iterations with learning rate 0.1. Fig. 7 corroborates Theorem 2 by demonstrating that DMA (Mux). ) {w])
with do = di d2
and (M (a(3) converges to O, which indicates the learned singular values (wi; and learned variances {of} converges to the theoretical singular values {wf) } and variances {o
In the MNIST experiment, we train linear CVAE with ELBO loss on the task of reconstructing three remaining quadrants from the bottom left quadrant of MNIST dataset. Then, we compare the set of singular values {w; } and variances {o?) with the theoretical solutions {w]}. {o,²) described in Theorem 2. In this experiment, do 196. d1 64, d2 588, and we set Meee ndee 3 1. The linear CVAE network is trained for 100 epochs with Adam optimizer, learning rate 1 x 10 and batch size 128. From Fig. 8, it is evident that both DMA ((wi), , (w)) }) and (MA((a,), (a(})
16

---

Published as a conference paper at ICLR 2024
LOVAE
DMA (kLA((u)) (wf])
Dma(tost,1103)
10"
10
2.5
10-3
10"
2.0
10-2
1.5
10"
1.0
0.5
10
0.0
10"
50
100
150
200
0
50
100
150
200
30
100
130
200
Iteration
lteration
Iteration
Figure 7: Evolution of DMA metrics across training iterations for linear CVAE synthetic dataset.
LCVAE
PMA (fwc) {w())
DMal(0t (d(1)
180
0.8
0.6
160
0.7
0.5
0.6
0.4
120
100
0.3
0.2
60
40
25
50
100
25
100
75
100
Epoch
Epoch
Epoch
Figure Evolution of DA metrics across training epochs for linear CVAE trained on MNIST dataset.
Log-like log-likelihood
KI.
AU
0.25
-177.14
18.85
56%
0.5
117.772
15.42
56%
B
1.0
-173.93
10.22
44%
2.0
-174.32
6.37
37%
3.0
-176.43
3.57
25%
0.25
142.56
17.58
50%
0.5
-173.93
10.22
44%
Tieec
0.75
-392.68
5.64
38%
1.0
-553.32
2.10
13%
2.0
-951.16
0.00
0%
Table 3: Test ligeeiidl and posterior collapse degree of ReLU CVAE on MNIST. As Table stated, smaller B and Taaec mitigate collapse and have more active units.
converge low value (less than 0.08 the end of training), which indicates that the values {wi), }o,} approaches the theoretical solution.
Log-likelihood, KL and AU of CVAE with varied 3, Ree (Table 3): All settings in this experiment are identical to the experiment in Fig. 2(b). We measure the log-likelihood, KL divergence of the model and AU of the model in Table 3. is clear that decreasing 8, nase alleviate posterior collapse. We observe that varying Vatec greatly affects the log-likelihood of the model, while changing 3 has mixed effects on this metric.
Correlation of y and posterior collapse (Table 4): We train ReLU CVAE model on synthetic dataset (x, 3). which generated by sampling 1000 samples y 128 ~ A/ I) and then sampling with different correlation level with y as depicted in Table 4. In this experiment, dy = 128, nen = 1.0, Maae = 0.5 and train the models with Adam optimizer for 1000 epochs with batch size 16. The results in Table 4 justify that higher correlation between and leads stronger collapse degree (higher AU percentage and KL divergence). as pointed out in our paper and especially, this insight also applies for non-linear conditional VAE.
set dg di
17

---

Published as a conference paper at ICLR 2024
y
N
RL
AU
y (correlation I)
0.01
0.1%
X = 2Y v
u (correlation 0.51)
66.20
54.1%
T + VI5
(correlation 0.251)
77.10
63.0%
x y + v0
u (correlation 0.1251)
79.64
64.7%
N(O.I)
80.37
67.2%
Table 4: Correlation and posterior collapse degree of ReLU CVAE on synthetic data. Higher corre- lation between the input condition x and output y leads to a stronger collapse.
32
2
8
16
9
6
6
9
6
0
8
5
4
9 5
6
9
5 8
0 5
9
0
8
9 6 9 S 9
6
0
5
2
2
9
9
2 2
5 7
2
5 6 2
7
9
8
7
9
9
A
# 7
6
0
9
7 2
6
2
7
9
9
6
6 999 5
6
5 S 5 9 S O 9 6 3 8
6
3 5 O 9 0 8 8 1 5 5 O 8 2
8
6 5 5 2 5 6 0 6 3 8 6
4 4 4 5 7 A 5 O 4
9 9 F
9 1509 9 0
9
9
8 4
9
2 8 7
3
7
8 9
7
8
9
7 16666
2
7
7
6
15958 S
O 9 5 5 9 5 O 9 0
5 9 6 6 6 5 7 6
6 5 9 3 O 3 8 G
9
6 6
8 O 0
9 7
3
9 1559 O
9
9
18855
7 & S
2
7 4
1686
19933 S
5
5
6 5 9 S SS9
9 S G
S 6
9
9
B88 6
& 8
4 3 8
6
G L 4
6 9 6
6 3 5 4 0 8
6 4 5
7 6 0 6 S 9
6
159 S
6 5 9 5
S SSS
9
S O 5
9 9 6
9 6 3 8
5 0
0
2 8 5
6
2
6
6 6 7
6
5
7
G
4
2
8
6
9
Figure 9: Samples generated by CNN Hierarchical VAE (with ReLU activation) with different (3i, (2) combinations. Smaller B2 alleviates collapse and produces better samples, while smaller 31 has the reverse effect.
A.2.3 ADDITIONAL. EXPERIMENTS FOR MHVAE
Samples reconstructed from CNN MHVAE with varied 81 and B3 (Fig. 9): Similar to the exper- iment studying the quality of samples reconstructed ReLU MHVAE with different combinations of B1 and 83 in Fig. 4, we train the CNN MHVAE model with Σ; I and parameterized (2(x) depends on ø1. this experiment, we replace all hidden layers in ReLU MHVAE models by con- volutional layers (with ReLU activation). The encoder of CNN MHVAE uses convolutional layers with kernel size 3 x 64, 3 x x 32 and stride = 2. The decoder of CNN MHVAE consists of transposed convolutional layers with kernel size 3 x 64,3 3 x 32, x 3 1, and stride 2. We set VJenc = 7idec = 0.5, o = 0.5. Fig. illustrates that decreasing the value of 33 help mitigate collapse and produce better samples, while decreasing 31 causes the images to be blurry, which is similar to the result in the case of of ReLU MHVAE.
Varying Bi, B2, 1/dec experiment with ResNet-18 MHVAE CIFARIO (Fig. 10): In this ex- periment, we train R8 MHVAE on CIFARIO dataset for 100 epochs with Adam opti- mizer, learning rate of 1 x 10-3 and batch size of 128. Within the model, ResNet-18 architec- ture is utilized to map the input image to the latent vector 21 in the encoder, the transforma- tion of 21 to y in the decoder parameterized by transposed convolutional layers with kernel size 3 xx3 x 32, 3 3 x 3 3x3x and stride 2. Similar to 18 VAE, two 2-layer MLPs with Relu activation are utilized for intermediate transformations. Furthermore, the mappings from 21 to 22 and 22 to 21 are implemented using 2-layer MLPs with Relu activation and hidden dimension 1024. We set dg = 3072, dj = d2 = 64, Nen = σ1 = 0.1. Similar to the varying 31, B2, 7aee experiment in Section 5.3, we alternatively run 3 sub- bobbepprriments as follow: i) we fixed B2 1, 7ake 0.1 and then vary 8 from the set {0.1, 0.5, 1.0, 2.0}. ii) we fixed B1 = 1, Nate = 0.1 and then vary 32 from the set 10.1, 0.5, 1.0, 2.0). and iii) we fixed B1 B2 and then vary Dae from the set
18

---

Published as a conference paper at ICLR 2024
60
81 0.1
60
8/ m 0.1
60
Wiee 0.25
50
50
50
Collapsed
30
9 0.5
9/ = 0.5
40
30
Rbe
10
20
2.0
20
10
= 2.0
1.0
20
10
Mix
Nir 1.0
0
4
90
0
2
3
6
9:
0.0
0.2
0.4
0.6
0.8 1,0
Figure 10: Graph of -coollsed for ResNet-18 MHVAE model trained CIFARIO dataset with varying hyperparameters 31, A2 and Vjaec. (ō = 0.05). As Table suggests, smaller (32 and fidec mitigate collapse and have more active units, while 31 has the reverse effect.
LHAAE
DMA((Ath (A73)
DMA((w)) (w7))
23
22
21
20
10:
10-2
19
10.
10
17
10-1
10"
16
10
100
150
Iteration
Iteration
lteration
Figure 11: Evolution of DA metrics across training iterations for linear MHVAE with unlearnable isotropic Σ and learnable Σg on synthetic dataset.
LHVAE
PMA([Ad (1)))
DMA((u)), (w/})
57.5
1.6
55.0
0.30
1.4
$2.5
0.25
50.0
0.20
1.0
47.5
0.8
0.6
42.5
0.4
0.05
0.2
40.0
25
50
75
100
0
25
50
75
100
0.0
50
75
100
Epoch
Epoch
Epoch
Figure 12: Evolution of DMA metrics across training epochs for linear MHVAE with unlearnable isotropic Σj and learnable Eg trained on MNIST dataset.
(0.25, 0.5, 1.0, 2.0). The experiments depicted in Fig. 10 clearly support Theorem by demonstrat- ing that decreasing (92 and T]dte reduces the degree of posterior collapse, while decreasing 31 has the opposite effect.
Linear MHVAE (Fig, 6(b)): In this experiment, we train the two-latent linear MHVAE model with unlearnable Σ; and learnable Σg on the MNIST dataset with latent dimensions dj d2 64. The experiment aims to check the threshold that cause wf 's to be 0 described in Theorem 3. We keep 31 = and then gradually increase 32 to check whether posterior collapse happens the threshold predicted. Fig. 6(b) shows that the convergence of KL loss for to zero agrees with the threshold obtained from Theorem 3. which, the top-1, 4, 8, 16, 32, 64 leading singular 0,'s used for computing 32 thresholds are [6.17, 201 1.80, 1.24, 0.89, 0.58, 0.34].
Verification of Theorem (Fig. 11, 12): To verify Theorem for linear MHVAE with learnable Σg. we further perform two experiments: synthetic experiment and MNIST experiment.
In the synthetic experiment for linear MHVAE with learnable Σ,, we initialize each index x, by sampling from.N (0, 0.1²). We optimize the matrix optimization problem in the proof of Theorem 3, which equivalent to the minimizing negative ELBO problem. We choose dy = d1 = d2 = 5, Σj I and Maee 7i 31 1, B3 2. We use Adam optimizer for 200 iterations with learning rate 0.1. The convergence of DMA { {x;3) and DMA(fW;) (w, }) to 0 depicted in Fig. empirically corroborate Theorem 3.
19

---

Published as a conference paper at ICLR 2024
CEAAE
DMA(tWs (w())
10a
10"
10a
15.6
10
15.4
10-1
10
15.2
10
15.0
50
100
150
200
10a4
0
50
100
150
200
10
50
100
150
200
lteration
lteration
lteration
Figure 13: Evolution of DA metries across training iterations for linear MHVAE with unlearnable isotropic E1 Eg on synthetic dataset.
LIAAE
DMA((A) (Ma(())){AA)))
DMA((w)), (w)))
0.5
0.12
0.4
0.08
0.3
55
0.06
0.2
50
0.04
45
0.02
M W M-MM
0.0
0
25
50
75
100
25
50
75
103
25
50
75
100
Epoch
Epoch
Epoch
Figure 14: Evolution of DMA metrics across training epochs for linear MHVAE with unlearnable isotropic E, Eg trained MNIST dataset.
In the MNIST experiment with learnable Σg. train two-latent linear MHVAE, learnable and data-independent Dg by minimizing the negative ELBO. In this experiment, we set do 784, dj = d2 10, nene Tidec 0.5, En 0.5ºI. The ELBO loss is optimized with Adam optimizer with learning rate 1 x 10 and batch size 128 for 100 epochs. Fig. 12 demonstrates that both DMA ( (A;3) and DMA DMA(( (fwi), {w;}) converges to low value, which empirically verifies Theo- rem 3.
Verification Theorem (Fig. 13, 14): Similar as above, but in the setting unlearnable isotropic Σ,, D2 studied in Theorem 5, we also verify it by performing two similar experiments synthetic dataset and MNIST dataset.
In the synthetic experiment, except pre-defined Σ; I. the other settings of this experiment is identical to the synthetic experiment for MHVAE with learnable Σn above. The clear convergence of (MA( (fAr) {);)) and DM (fw,), {w,)) to demonstrated in Fig. 13 empirically verified Theo- rem
In the MNIST experiment, linear MHVAE is trained with pre-defined and unlearnable E1 Eg 0.521. Other hyperparameters and training settings is identical to the MNIST experiment for linear MHVAE with learnable Eg above. Fig. 14 also corroborate the convergence of the sets of singular values {Ar), {w;} to the theoretical values.
ligelioel,oood, KL and AU of HVAE with varied B, Naee (Table 5) All settings in this experi- ment are identical to the experiment Fig. 2(c). Table 5 demonstrates that increasing 31 alleviate posterior collapse and increasing 82 and 7/dec have the opposite effect. We also notice that changing Vice greatly affects the log-likelihool of the model, while varying 31, 82 has mixed effects on this metric.
B RELATED WORKS
Posterior collapse: To avoid posterior collapse, existing approaches modify the training objec- tive to diminish the effect of KL-regu KL-regularization term in the ELBO training. This includes heuristic approaches such as annealing a weight on KL term during training (Bowman et al., 2015; Huang al., 2018 8; Sønderby al., 2016; Higgins et al., 2016), finding tighter bounds for the marginal log-
20

---

Published as a conference paper at ICLR 2024
Log-likelihood
KL
AU
0.25
-229.88
1.09
34%
0.5
-225.89
4.59
89%
B
1.0
-225.82
8.77
100%
2.0
-226.64
13.00
100%
3.0
-227.49
16.19
100%
0.25
-228.66
18.07
100%
0.5
-226.72
13.04
100%
82
1.0
-225.82
8.77
100%
2.0
-226.05
4.24
82%
3.0
227.41
1.98
40%
0.25
211.79
18.07
100%
0.5
-225.82
8.77
100%
dlee
0.75
-522.61
3.45
68%
1.0
-740.24
0.49
18%
2.0
-1278.18
0.00
0%
Table 5: Test log- log-likelhhood and posterior collapse degree of ReLU two-latent MHVAE trained on MNIST dataset. As Table suggests, smaller 32 and Tidec mitigate collapse and have more active units, while Bi has the reverse effect.
likelihood (Burda et al., 2015) constraining the posterior family to have minimum KL-distance with the prior (Razavi et al., 2019). Another line of work avoids this phenomenon by limiting capacity of the decoder (Gulrajani et al.. 2017: Yang et al., 2017; Semeniuta al., 2017) changing its architecture ( (Van Den Oord et al., 2017: Dieng et al., 2019; Zhao et al., 2020). (Kinoshita et al., 2023) proposes potential way to control posterior collapse by using inverse Lipchitz network in the decoder. Using hierarchical VAE is also demonstrated to alleviate posterior collapse with good performances (Child, 2021; Sohn et al., 2015; Maaløe al., 2017; Vahdat & Kautz, 2020; Maaløe et al., 2019). However, (Kuzina & Tomczak, 2023) empirically observes that this issue is still present in current state-of-the-art hierarchical VAE models. On the theoretical side, there have been efforts to characterize posterior collapse under some restricted settings. (Dai et al., 2017; Lucas et al., 2019; Rolinek al., 2019) study the relationship of VAE and probabilistic PCA. Specifically, (Lucas al., 2019) showed that linear VAE can recover the true posterior of probabilistic PCA. They also prove that ELBO does not introduce additional bad local minima with posterior collapse linear VAE model. (Dai et al., 2020) argues that posterior collapse is direct consequence of bad local minima of the loss surface and prove that a small nonlinear perturbations from the linear VAE can produce such minima. The work that is more relatable to our work is (Wang & Ziyin, 2022), where they find the global minima linear standard VAE and find the conditions when posterior collapse occurs. Nevertheless, the theoretical understanding of posterior collapse in important VAE models such as CVAE and HVAE remains limited.
Linear network: Analyzing deep linear networks is important step in studying deep nonlin- ear networks. The theoretical analysis of deep nonlinear networks is very challenging and, in fact, there has been no rigorous theory for deep nonlinear networks yet the best of our knowledge. Thus, deep linear networks have been studied to provide insights into the behavior of deep nonlinear networks. For example, using only linear regression, (Hastie al., 2022) can recover several phe- nomena observed in lcrge-scale deep nonlinear networks, including the double descent phenomenon (Nakkiran al., 2021). (Saxe al., 2013; Kawaguchi, 2016; Laurent & von Brecht, 2018 8; Hardt & Ma, 2017) empirically show that the optimization of deep linear models exhibits similar properties those of the optimization of deep nonlinear models. pointed out in Saxe et (2013), despite the linearity of their input-output map, deep linear networks have nonlinear gradient descent dy- namics on weights that change with the addition of each new hidden layer. This nonlinear learning phenomenon is proven to be similar those seen in deep nonlinear networks.
In practice, deep linear networks can help improve the training and performance of deep nonlinear networks Huh et al. (2023); Guo al. (2021); Arora et al. (2018). Specifically. Huh al. (2023) empirically proves that linear overparameterization in nonlinear networks improves generalization on classification tasks (see Section 4 in Huh al. (2023)). In particular, Huh et al. (2023) expands
21

---

Published as a conference paper at ICLR 2024
each linear layer into succession of multiple linear layers and does not include any non- non-linearities in between. Guo et al. (2021) applies a similar strategy for compact networks, and their experiments show that training such expanded networks yields better results than training the original compact networks. Arora et al. (2018) shows that linear overparameterivation, i.e., the use of a deep linear network in place of classic linear model, induces gradient descent particular precondition- ing scheme that can accelerate optimization. The preconditioning scheme that deep linear layers introduce can be interpreted using momentum and adaptive learning rate.
C SAMPLE-WISH ENCODER VARIANCE
C.I CONDITIONAL VAE
In this section, we extend the minimize problem in Equ. (5) to data-dependent encoder vari- ance (xx). Indeed, assume the training samples are {(2,, ) and q(z(2,,)) y) ~ ANWia +
W2y1 2,) Vi [M), have:
- -ELBOCV AE 1, W2. 1, U2 1 Ex)
--Ex.y (x(a))))))
[pe (y/ I 2)] 3Dx (2 v)ll(((()))
3 None
Σ Bggls
|1 + U24; will? - د E. 3 log [
||
Wdiec
Σ
1
[[U:W; + U2)x; + (UIW2 I)yill 2 traco (U,
N
Vicco
+ (I + W2yil traco(
[dz 3 log |Σ:|
Taking the derivative w.r.t each E. we have:
ØELBO 1
ØΣ (U, + -3,
2
3 Σ; = Bni UU + 1
(8)
We have E; Σ for all optimal, and thus, the above minimizing negative ELBO problem is equivalent to the training problem in Eqn. (5) that use the same Σ for all data.
C.2 MARKOVIAN HIERARCHICAL. VAE
Similarly, we consider the negative ELBO function for MHVAE two latents with data-dependent encoder variance Σ. Indeed, assume training samples {x:)/ and dropping the multiplier 1/2 and some constants in the negative ELBO, we have:
EELBOHVAE WT, Wz, U2 U2 (E, (221) = Ex Begeettt
- B
(2g)=1) (zgl) (DeL (go (2) - (Dx
1
1 Σ
|| HUU,WaT - 2112 (race(U,E,, UH) + ( W
N
Tace
+ 3r trace
U2E2 + 31 trace (U, 1) (U, - ])
+ 2 ( 2 + trace W.E,, W + trace (E2,)) - B1 log E1,5 - B2 log Ezgs1
where the details of the above derivation from the proof in Appendix F.1.
22

---

Published as a conference paper at ICLR 2024
Taking the derivative w.nt each Σ1,; and Σ,,,, Vi E [M], we have at critical points of - -ELBO
@ELBO = U - 1 (U2W; 1) + W -3 =0 -N
Vicc
=E1, Bose (U 33 1) (U:W; 3 W2)
ØELBO
(3 U2 + 2 - 32 =0
2 Idee
B2
(U2 U2 22 + 2 32 31
I)-
Idec B1
Thus, we have at optimal, Σ1, and E, are all equal for all input data. Hence, we can consider equivalent problem of minimizing the negative ELBO with same encoder variances Σn and Σ₂ for all training samples. Similar conclusions for linear standard VAE can be obtained by letting U2 = W2 Σ2,1 -...
E2,N =...=2..
the above arguments.
D PROOFS FOR STANDARD VAE
In this section, we prove Theorem in Section D.1. We also derive the similar results for learnable
E case Section D.2.
Recall that A = (xxx) PAOPA I $ 1/2pp A and Z == Ex (xxT E IR Doxdo, Also, let
V1 = W,P. P 1/2 E Rd
We minimize the negative ELBO loss function (after dropping multiplier 1/2 and some constants);
CV AE E,
log :)] +
1
|IUw -y1l trace trace 86 (| (IWxz + trace(E)) log Σ
Ex
Vake
||UV - 2 + trace (U + 3c. V |3 + 3e² trace Σ) - -1 -Blog E| (10)
Nosc
D.1 UNLEARNABLE DIAGONAL ENCODER VARIANCE Σ
Proof of Theorem I. Since the Σ = diag(of, ....,,, ,0,)) Se trace E):
is fixed, we can drop the term
LVAF
|UV 213 + (UaceeUUUS) + 36² |
(11)
Take
At critical points of Lv AE:
OL
1
(UT (UV +)
Bc2 0.
2 8V
Nace
2
(UVV - -z-v +UE) = 0.
(12)
2
Take
From 2V
0, have:
- -
(13)
and:
36
(14)
Denoting (0 and (wm)mi U. respectively. Let and be the singular matrices Z and U with non-increasing diagonal, respectively. We also denote E' diag(o?.....
din(do,d1) with non-increasing order be the singular values of Z and
oz2) ...
rearrangements of Σ such that of <
23

---

Published as a conference paper at ICLR 2024
S... Thus, Σ =ETT with some orthonormal matrix T I It is clear that when all diagonal entries are distinct, T is a permutation matrix with only =1's and 0's. When there some equal entries in E. T may includes some orthonormal blocks on the diagonal when these equal entries are near to each other.
Plugging Eqn. (13) and Eqn. (14) into the loss function in Eqn. (11), have:
nideo Lv AE
/ trace
ZV UT) trace U (U)
| F
trace ZZ
UU U
3c²1) - + trace U UTT'T"
2
trace
362) -2 + trace(22T>)))
do
d1
wff.
d1
Σ
07
Σ
w
Wi
Be
1=1
1=1
i=1
da
BC
0
wfo '2 3c
Σ
(15)
w
i=d
where we use two trace inequalities:
trace (ZZTUZUTTUT U I6 U $ trace(99 (2 + 2 -c2)))))
(16)
trace U UΣ trace(S? n
(17)
The first inequality is from Von Neumann inequality with equality holds if and only if Zz and UU (B "U" are simultaneous ordering diagonalizable by some orthonormal matrix R. The second inequality is Ruhe's trace inequality, with equality holds if and only if there exists an orthonormal matrix that E = T T and UUTTT is diagonal matrix with decreasing entries.
By optimizing each w; in Eqn. (15), have that:
w = max (0, VBc of (0, - BBoo))
(18)
In order to let the inequalities above to become equality, we have that both R U and T TUUTT diagonal matrix with decreasing entries. Thus, U R.T From (13), by letting
Z = ROS with orthornomal matrix Id xáo we have the singular values (in decreasing order)
of V as:
V = T(Ω 5 1 '22011200 122,
X; max (0, 1 (0, - /Bcooi)
(19)
/Bc
L
the case when V has d1 zero matrix T that decides whether posterior collapse happen or not. Specifically, if all values { are distinct, T is permutation matrix and hence, U T is diagonal matrix. Thus, using the same arguments as in the proof D.2, we have partial collapse will happen (although the variance of the posterior can be different from )e
Remark 2.
zero singular values with r == rank(V) ), it will depend on
On the other hand, if is chosen to be isotropic, T can be any orthonormal matrix and thus the number of zero rows of V can vary from 0 (no posterior collapse) to d1 (partial posterior collapse). It is clear that when 0g cat V i, W and we observe a full posterior collapse.
D.2 LEARNABLE ENCODER VARIANCE Σ
For learnable encoder variance Σ in linear standard VAE, we have the following results.
Theorem 4 (Learnable E). Let Z Eg (xxT) ROS is the SVD of Z with singular values (0, do 1 con-innceasing order and define V == WP WP 1112, the optimal solution (U*, W E*)
of LVVE is given by:
U' UR= V TAS E' = =3n2 ARR.uUTT (UU U
24

---

Published as a conference paper at ICLR 2024
where T Rdixds is an orthonormal matrices. The diagonal elements of Ωt and A as follows, vi E [d1]:
max -Bndec)
x
Nenn
02 V mmx 09 - -Brlee)-
nenc
If du < d1. we denote 0, = 0 for do <is d1.
Now we prove Theorem
Proof of Theorem 4. Recall the loss function in Eqn. (10);
LV
|IV - 2 + trace(U
36 |V 1/ + 36 trace(E))
- 3 log [Σ]. (20)
Vide
We have critical points of LV
ac
- 3 - =0 = Bree (U Bc'
OΣ
lddec
(21)
Plug Σ = B1/aac
(UT "0
B--21) into Lv AE and dropping constant terms, we have:
CV
[|UV 21β + 32 + log U U + B6
(22)
2 Tdee
At critical points of AE:
1 0c
1
(UT (UV - -2) +3+ =0,
2 0V
1 Le
1
(UV ZJVT + +UU U B 0.
(23)
2 80
Ndee
From aC JV
0, we have:
V (U 3621) ( z,
-
(24)
and:
3e'v V V Z).
(25)
Denoting [0 and {wi minn(dgdd1) with decreasing order be the singular values of and U, respectively. Let O and Ω be the singular matrices of Z and U, respectively. Plug Eqn. (24) and (25) to L', have:
1
trace ZV U Bog U U + 3621
Viec
1
[||
trace ZZ U(U U 36 U ) 3 log U U 3c21
Viec
|||
trace(00 22 Ω + T T )} + log U T U + 3c²1
Náce
do
di
1
03
3 log(w> + Bc²
Vlace
W + B
do
d
30
di
=
0 +
w + 96
Σ
Blog(w? + 3,2))
(26)
di
1=1
25

---

Published as a conference paper at ICLR 2024
where we use Von Neumann inequality for ZZ and ( UUT - The equality condition holds if these two symmetric matrices are simultaneous ordering diagonalizable (i.e., there exists an orthonormal matrix diagonalize both matrices such that the eigenvalues order of both matrices are in decreasing order).
Consider the function:
2202 /Tae log(w> Bc²), w h(w)
(27)
w² + =3c²
This function is minimized at w w- 0. Applying this result for each wi in Eqn. (26), we have:
(62
Br if H 2 noce Otherwise, if 0² < nace
w
max 82 Br Vi [di]
(28)
Meoc
Denote {A+]==1 min the singular values V and A as corresponding singular matrix. At the minimizer of LVAB. U and Z shares a set of left singular vectors due to equality of Von Neumann trace inequality. Thus, using these shared singular vectors, from Eqn. (24), the singular values V are:
X
Bene
max 0f Br
(29)
0g
From Eqn. (21), now we consider E that are diagonal in the set of global parameters of the loss function. We have:
v Brecctooc
if 0, 2 v BBase
of
(30)
if 0, < v /BTaS
Teenc:
where {a is permutation of {o,) Since E (U U 3,2r) at optimal, if choose Σ to be diagonal, have U diagonal the global optimum. Thus, U can be decomposed as U = R.Y' with orthonormal matrix R E Do and S2' is a diagonal matrix with diagonal entries a permutation of diagonal entries of ft. Hence, U will have di -r zero columns with r := rank(U) (U). From the loss function in Eqn. (10), we see that the corresponding rows of V with the zero columns of U, will have no effect on the term UV Z|J. Thus, the only term in the loss function that relates to these rows of V is |V[le. To minimize V [[,, these rows of V will converge zero rows. Therefore, from Wz z Vä, we see that these zero rows will correspond with latent dimensions that collapse to the prior distribution of that dimension (partial posterior collapse). O
E PROOFS FOR CONDITIONAL VAE
In this section, we prove Theorem 2 in main paper.
We consider the conditional VAE described in Section 4.1. For any input data (x, y), have:
Encoder: q(2|2,, y) NWW + W2y, Σ), W, E RR D
W₂ RdxiDs
Decoder: p(y[ 2) (( + U22, n I), U, R Daxdi U2 E D2x
Prior: p(z[z) = (0, I),
note that write Wi + W2y + with & ~ N Σ) for given (z,
To train CVAE, we minimize the following loss function (Sohn et al., 2015; Doersch, 2016; Walker et al., 2016):
LCVAE -E
(e a. [Po =)] + 3D (
[p{]))))
-Ex,vis
|
U2x - y|1 , 1g - log Σ
+ nenc
||
Macc
1
Er,y (1, W1 U2)T + (U, rgyl 2 + trace traco(U, (U,
naco
+ 3d2 (I) Wix+ Wzyll 2 + trace Σ - 3 Blg [Σ|
where c == nace nan-- Note that we have dropped the multiplier 1/2 and constants in the above derivation.
26

---

Published as a conference paper at ICLR 2024
Proof of Theorem 2. For brevity in the subsequent analysis, we further denote V1 W,P
Rdd Xdo 22 W2PpW1l2 E IRdixds, T2 U2P E D2xdo and D = PBWp122 E
D2xd2, we have:
BU W + U2) (U, W2 1)yll)) = U W + U2) 0 (U, W2 I)P 13 /11/2
= =| V1 + T2ll + |U V2 D + 2rrace (U, , V1 + 2222()
B(| W2yll*) = B(IV ã + V2ūll I |1 + | |/2 trace V 2
Therefore, the negative ELBO becomes
LOVAE U1, 11, 22 T2, Σ)
UIV T2 p
11V2 D12 + 2 2 +
Vatec
(U. V2 1)l1?
(c)))=1) + 3 ( V2 E tracee | (2V.))
-Blog [
Next, we have at critical points of LCVAE:
OC
- -
Vilec
+ 3agee
(U, U U1+ 3
(31)
Plugging Σ = Made (U] U1 + 1c 1 in the loss function LCVAE and dropping some constants,
have:
LOVAE
U + 12 + U, V2 - D + trace (U, + V2 D)
2
Vile
+ Be (1/V, 2
|V21 + 2racee
(V.2v 2trace(V
+ Bog U, 3
+
(32)
We have, at critical points of CCV
nhe dL
+ + (U.V2 +
D)ZT =0
OT,
na dc'
=U + T2) + V2 - D)Z + 3eV1 + V2Z =0.
2 OV1
2 Oc
UT - D) U (U + T +B 36 V2 0.
2 OV2
ndoe OC
(U,V, DVV, + +U, V + ViZV.) DZ VV
0U,
+ + T.ZV, + Bmaee U U -
(33)
ac
From
0, we have:
ST,
T2 - D
(34)
ac'
From PV.
0 and Eqn. (34), we have:
V1 V.Z 0.
(35)
27

---

Published as a conference paper at ICLR 2024
ac 2 0 and Eqn. (35), we have:
From
U - D) - D 3c 2 -=0
(U - -
+ (U, - -2" -c -- W2D(1-''s
W.D(I- Z) U1 + +
U, D(I z)
W.E = UU - E,
(36)
where we define : -I(I-Z'))D for brevity. Using above substitutions Eqn. (34), Eqn. (35) and Eqn. (36), we have:
U + Tall + U 2 - DI F + trace(((U, VI + T2) - -D)"
[I(U, 22 D)Z T |1 + |U V2 - DI 21race((U, - D)Z Z(U, ((U,V=-D) -D)
Lrace( (U, V2 DD (U V2 D) Z))
trace( W. 1)" (U, W2 - I)D(I Z Z
(U,
= trace W, U. W2E) - 2 trace U, W2E) -trace
V | V F trace V 2V, VV 1/2 V2 2 trace V2Z V2ZIzV,)
= trace V2 Z 2)V = trace W2EW,
trace W, W U.U,U, W2 2 trace (u, W2E) + 36 trace W2EW,)
= trace((U, + (,W,
trace U, W2E) U, + - 2 trace (U, (U U, + 3e - -U
trace(U, UTa (U, U,
tra U, T) -U E).
Denote { and {0 be the singular values of U, and E in non-increasing order, respec- tively. If d2 d1, we denote 0, with d2 d1. We now have, after dropping constant trace(E)) (E):
COVAE = trace UU; + Bc21) - E) + 3 log U, + 1
Nake
X203
Σ
3 log(X? + 30²)
3e
Vieec
i=1
i=1
3 4
202
d
3 log( 3e²),
Mlee
322
where we used Von Neumann trace inequality for (U U, + 3c ---1))-'x and E. We consider the function below:
c²2²
g(t) 2
+ log(t)) t> 3c...
(37)
dic
It is easy to see that g(t) is minimized at 1"
d002 if 0 > -ygge Otherwise, if 0 < 31 Vil
t" = Bcc. If 0 = O. clearly log(X² + Bc?) is minimized at À = 0. Applying this result for each A we have:
x;
/ax 03 -B V [di],
(38)
Tenc
note that the RHS also be applied when 0, 0 for i [di].
28

---

Published as a conference paper at ICLR 2024
Consider the global parameters that Σ is diagonal. The entries of E can be calculated from Eqn. (31):
v Meecn 0i:
0; 2 VBrdue
d,
(39)
neac
< VBraaee
where {o,} is permutation of {oi}.
11
Remark 3. Posterior collapse also exists in CVAE. When at the global parameters such that E is diagonal, we have U Un is diagonal. Thus, U1 can be decomposed as U, RΩ with orthonor- mal matrix R E IR Do and Q is a diagonal matrix with diagonal entries are permutation of diagonal entries of St. Hence, U1 will have d1 -r zero columns with = rank(U,).
If there is column (says, i-th column) of U1 is zero column, then in the loss function in Equ. (5), the i-th row of Wiz 1z + W2y will only appear in the term W global minimum, Ez,,y W 1 + W2yll?? will be pushed Thus, for each pair of input v), we have w! x. + w y (posterior collapse).
12 + 22 Thus, at the
T
F PROOFS FOR MARKOVIAN HIERARCHICAL VAE WITH 2 LATENTS
In this section, we prove Theorem 3 in Section F.I. We also analyze the case that both encoder variances En and Eg are unlearnable isotropic in Section F.2. We have:
Encoder: (<=1<>) ~ WNW 1., E1), W E gdd xdo Ei 1gdixd.
((<2|=1) ~ >N(W 221. E2), W₂ E Bdaxdr
En E Radad
Decoder: p(Z1|22) 2V(U2=2, Noce I), U2 E Rdixdd
(40)
ndee 1), U1 ER Prior: p(22) (O, ((0,
p(y|z1) ~N(UU =1, nde
Let A == (xxxx) PA4PT i = 4 1/1p a and Z == E, (xx) E Doxd, Also, let
V, = W,P. P.P4 11/2 E RdIxD, thus V VT W,AWT E, W LE WT).
We minimize the negative ELBO loss function for MHVAE with layers of latent:
LIVAA E,, (ea - B1 E (xslls) (DKr, (aaa
-32E, E DRI (2)))
= Lx [log p( =1) + 31 log Dog((=1 (22) + B2 logg(22) - B1 log (bg((=)(>)) - B2 log ((22/=1)]
31
32
Br,ei.Ca
U1 - 2 +
U222
21
||2 + + BETTETT
?
Mdee
ndce
Venc
B1 log Eil
3 Σx 1n E2 B2 log Σz|
Eg U =11² + trace U, U,E,U + 311 U/
Wizl trace U,E.U,)
Tãae
+ 81 trace( (U, (
I) (U,W; W, T) + +3 W W |- trace tr,. + trace Ez))
-Bid1 B2d2 31 log Σ1 82 log Σz
[|]U, V1 - 2 + trace U. U, (1 1)V, + 31 trace U,J U2Σ2)
+ 81 trace( (U, U,2 T - 1}Σ,) + 23 W.V. + + trace W,Σ,) + e Bg trace Eg)
-iid B2d2 - 33, log Σ1 33 log Σ2|
29

---

Published as a conference paper at ICLR 2024
F. LEARNABLE Σg AND UNLEARNABLE ISOTROPIC Σ1
Proof of Theorem 3. With unlearnable isotropic Σ₁ ofI, the loss function LAVAR becomes (af- ter dropping some constants):
LEVAE =
1
2 + & W2 - 1)V + trace U1 (2 + 31 trace (race(U, U2Σ2)
Tocc
B1 trace( (U2W; (Taaee(U22))) W2 (U,(W,, 1)E,) (26 W2V1 B2c trace (W W2Zi)
32c² trace Σg)] B2 log Ez2.
Taking the derivative of LHVAE w.n. Σz:
1 0C B1
32
U U2
0
1)
2 2
Ticc
B1
=I
32
Idee
U
U
U2
t 32
1
(41)
31
Plugging this into the loss function and dropping constants yields:
LHVAE= =
1 [| 2 + 3:11(1,, W2 - 1)Vill + trace(U, U,E,)
Tace
+ Irace (U2W; (OTT))(TTT (U2W2 1)Σ,) 320 W2V
112 v 320² trace W W)
+ 32 log U2+ +
31 ll.
(42)
At critical points of CHVAE
Maae Oc'
UU,V - 2) +),A( - (U,W, - 1)V, + 32d W, V, =0
2 0V.
(43)
Maae OC
(U,V, Z +UE =0
(44)
2 0U,
nac 3c
V W + B2n U2 U, + e² 8 1 2
(UW,
(1 I W =0,
2 OU2
31
(45)
Tioe ec
U, V)v. &UT 1) + 3 W.V.VI + +
aW2
(46)
From Eqn. (46), we have:
U. (U2W, 1) + WV) En)= 0
31 U, (U2W2-1)
W2
(since V, V Σ, is positive definite)
320
W2 (U, U2 + c² 32
U..
31
31
W2 (U2U, + I)-
(47)
31
U2UJ + L2 -- B
82
W
(48)
B
From Eqn. (44), we have:
U, - (V.V ΣT-
(49)
30

---

Published as a conference paper at ICLR 2024
From Eqn. (43), with the use of Eqn. (48) and (47), have:
&2
+ 2 244
2211 Vv
U, - U, z+
(U2U.
31
31
+C,3,UU2U 2 (U2UU + + 2 1)
1
31
32 1) V;
+ + 2, +
-
(50)
3
= -U, + 2 2 1)
VV.V U, zv. U.U.V = UT
- U.V.VV -U
3
(51)
where the last equality is from Eqn. (44).
We have the assumption that E =o From Eqn. (51), its LHS is symmetric because the RHS is symmetric, hence two symmetric matrices V1 V and (U2U2 + c² I) commute. Hence, they orthogonally simultaneous diagonalizable. As consequence, there exists orthonormal matrix P Rd xd, such that P V, V P and P (UU2U d -
(P U2U c2 2 I
are diagonal matrices (thus, P UU is also diagonal). Note that we choose P such that the eigenvalues of V, VT are in decreasing order. by altering the columns of P. Therefore, we can write the SVD of V, and U2 as V = PAQ T and U2 = PΩNT with orthonormal matrices R2 xd, Q E RRdoxda, ,, N E R2d2xd2 and singular matrices A RRd xdo, Z xdd, We min(di,dg) and {0, (0. respectively. denote the singular values of 11, U2 Z are {= miin(dodd1) {wr]
Next, by multiplying U. to the left of Eqn. (45), we have:
Bendee U.TUUU U2
3 1
+o
= = (U,U, + + 22 + +ofD(U.UT
+ 2 1)
U2.
(52)
31
31
Plugging the SVD forms of 1, and U2 in Eqn. (52) yields:
82 'NT = NNNN + I2 +of
+ +22 -
No
NΩ
Ω
Vide
31
B1
B1
W
2
=
( + of)
Vi [min(d1,d)) d2))
(53)
w + 2 82
(w] + c B,
-4 = 0 or w + 2 B2 2
+
+
Vie imin(d1, d2)].
(54)
B1 Tace
Using above equations, the loss function can be simplified into:
|| 1 trace(U, (U, (U,TT ZVT) + 6,33 trace (UUUU 2 2
C HVVE 2
- + E))]
B
32 U.U,2 U2 + 2 B2 l|-
(55)
31
31

---

Published as a conference paper at ICLR 2024
Computing the components in Liv AE+ have:
c² 22
log(wp + c² 2202)
log U U2
Σ ==1
81
31
traco(U]
ZV])
trace U ZJ
trace V (ViV) + -
X20
X
i=
trace (UU + + 2 -- VV. +of1) B1
d1
X + 3 + 2
where we used Von Neumann inequality for VV if and only if these two matrices are simultaneous ordering diagonalizable by some orthonormal matrix R.
and Z Z with equality holds
We assume d1 = d2 S do for now. Therefore, we have:
da
d1
X2
d1
X
o
di log(w? + c2
2200)
Voee LE 2
8
P2
+ 32
w +
2
82
31
1=1
1=1
31
1=1
do
010?
X o
B2
07
+
+ 3 log(w? ? +
(56)
X2 +
w +
2
82
B1
1=1
01
g(diwes)
Consider the function:
0,0² g(A.w) X2 + oi
X2 x
+ B2 log(w> + c2222
+ + B2
w²
32
B
31
We consider cases:
If w 0, have g(,,0)
0?0² X²+of
322 +ff + 323 log(c2 22. It is easy to
that g(A,0) 0) is minimized at:
σ1 (0 v1111)
x
V max
0.
(57)
VBn
Iff + 20 è² /2/1220
Nace (X2 + of)
13-0i nān
then we have:
oi02
g x2 + of
+ B2 log(X² + of) B2nic B2n log(rien))
(58)
Since we require both == x2 + of 2 of and w2 (X2 + of
B. nax)) / (t
la
B ( 1 //20 2 consider three cases:
If of 2 n g is minimized at the minima to Pari
o70²
if to 2 of. otherwise, g is
minimized at of. Thus:
x / B2Muec
σ1
max 02 32maac)
Bg * Tuke
0f0²
32 ②
max
(59)
n
3
Nsse
B1
We can easily check that this solution is optimal after comparing with the case w above.
32

---

Published as a conference paper at ICLR 2024
of 31 Vaeee and if the minima g at t0 82 t = 3i Viee thus w = 0 and we know from the case w = 0 above that:
oi0² Bantic
82 M 9 is minimized at 31
01 VB1
(0- (0- VBio.)
max
0,
w O.
(60)
If of 31 Naees and if the minima of g at to
82
0i0² B2
82
neee 9 is minimized Bi
0f and thus:
to X2 + oî Pang
x
01
v V02
32Maace
v B2muue
of02
B2
c².
(61)
32 Vdee
Bi
We can easily check this solution is optimal in this case after comparing with the case above.
We call the optimal singulars above the "standard" case. For other relations between da, di and d2, we consider below cases:
If do < d1 < d2: For index i S do. the optimal values follow standard case. For do < i S
d1, clearly A = (recall V1 E R xd)), then wi = mmxx(0, of 82 m
For d1,
is clear that X; = wi
If du < d2 - d1: For index i S do, the optimal values follow standard case. For do < of ani For > d2, it is clear that
s d2, clearly X; O. then W; V max((,
w;= 0.
If d1 & min(do, d2): For index i < d1, the optimal values follow standard case. For di x; wi (recall V1 Rdix and U2 Rddxxd)).
d2 s do < d1: For index i s dg. the optimal values follow standard case. For d2 < <ig
do, w; = 0 and A = / max (0, Vps
(0 BBoi)) For i > do. A; = wi
If d2 < d1 - do: For index i < d2, the optimal values follow standard case. For de < <i
d1, wi = 0 and A = max (0, vs.
a (0 vBi01)) For > d1, A = wi =
Remark 4. Eg is diagonal, we can easily calculate the optimal En via the equation Ej Wa (U, U, + c Also, in this case, U2 will have orthogonal columns and can be writ- ten U,_ TΩ' with orthonormal matrix T and Ω is diagonal matrix. Therefore, W₂ U.J. UU _ + ----1=- 2 (S'n2T +cTI) will have zero rows (posterior collapse in second latent variable) if ffrnk((..) = rank(U2) < dz.
For the first latent variable, posterior collapse may not exist. Indeed, we have V1 has the form Vi PAR Thus, P will determine the number of zero rows of V,. The number of zero rows of Vi will vary from 0 to dj rank(Vi) since P can be chosen arbitrarily.
33

---

Published as a conference paper at ICLR 2024
F.2 UNLEARNABLE ISOTROPIC ENCODER VARIANCES Σ1. Σg
In this section, with the setting as in Eqn. (40), we derive the results for MHVAE two latents with both encoder variances are unlearnable and isotropic Σ ofI. Σ₂ I.L. We have the following results.
Theorem 5. Assume Σ1 I. Σg I for some 01 22 Assuming do dl d2. the optimal solution of (Ui, U. Vi, W2) of CHVAE is given by:
V = PAR U PPQ' T, W - (U202 +c21)=1 " ZV; (ViVi" + +
where Z = ROST is the SVD of Z and orthonormal matrix P E Rdixd,, The diagonal elements of and S2 are as follows, with E [di]:
1f 2 max V 1203201102
83 eo v
of
Vo-- 0
@3885000003.
/3132000
VB2001
330t2
w/
B103
03-
V
3107
82 c002
1f0; max V v
B1 10
and viro 2 BB2002
/ 01
x = 0,
B2 00
82 2
w
31
$1
If0; < max V V 13.3001102
8, 20²
} and v < B20022
v9i
0. 01 (0 v B101)
A max
/B
w/ 0.
Now we prove Theorem 5.
Proof of Theorem 5. CHYAE =
The loss function is this setting will be:
[| Z 1 + (U, - 1)Vill2, + trace(U, U,E1) + 31 trace U.] U,Σ2)
Taae
B1 tr, [U,,, - -)),) 82c² W,
V || 3 2 W,E,)].
We have at critical points of LIVVE
Vane dc
+ 0V, U.U.V. - 2) + - & W,V, =0
2
(62)
T ac
(U,V, Z U,E, 0
(63)
2 0U,
2 0C
(U - I)V, WT + -U - W,E,W, = =0,
(64)
2 0U2
2 0C
IJV, VJ + +.. W - +2 + +8 W.E, = 0.
2 aW2
(65)
34

---

Published as a conference paper at ICLR 2024
From Eqn. (65), we have:
(BIU2 -D + +Ei =0
31
W2
2 (U,W2 1)
(since V + + PD)
c²
*2 (U,) U2 +e 2 1) U.
31
*W₂ = (U2U + 2 2
1)-
(66)
31
U2W - - 32
(U2U + 2 2 1)-
(67)
B
31
From Eqn. (63), we have:
U = I -
(68)
From Eqn. (62) with the use of Eqn. (67) and (66), we have:
U U + c+ 83
U
V1
31
31
+ +U, + 2 22
UU2 + + 32
I) 1 =0
B1
32
+U
- = U
(69)
3
= (U,U, + 2 L
VV. = UJ - U.U.V. = -Z =
-
B1
(70)
where the last equality is from Eqn. (63).
Now, go back to the loss function LHV AE. we have:
V Z ( W2 - I)V 11 + & 1I
-trace( ( U. - -2VV + ||]| + 3 11 W2 - 1
+ + (U2UJ + +V
=trace(U V,, -2V UT)+ +
+ 3 trace (VI UU,UU + c² -
1U2 + B2 IU, + +2 222 'v))
31
B1
31
ttrace(U, U. 2U, ZVT) | | + e B2 trace (v. (U2U, + 2 82
1 V1)
31
- --traceUU, 1, trace U zv - | + 2B trace (V. UU + 0 3
I v1))
where we use Eqn. (68) in the last equation.
We also have:
81 tracee((,,)))) (U,W, I)E, -) 1)
) + 32 trace WW
c432
31 (U2U, + +2
3
1)
- 1 (U:UJ + 2 2 1
3
3
(U2UJ + + B2
(72)
+ cB trace Bua (U,U, + +2 l)
Σ
1
B
3
= =d trace (U,U, 2 3
-
31
35

---

Published as a conference paper at ICLR 2024
Thus, using Eqn. (71) and (72) to the loss function LHV AE yields:
trace ZV,) + 32 trace (U202 + 0² 22 Z F
r)
LEVAE= =
Vdee
31
B1 trace( U2E2U2) + B2 trace E EU(UU +
Naee I) )]
Venc
1 - [IZ]| trace
(U, (race(U
+ 31 trace (2
Tike
e² B2 trace (UUJU, + e² 82 -1 (V.V + E,))],
(73)
B1
We have the assumption that Σ of From Eqn. (70), its LHS is symmetric because the RHS is symmetric, hence two symmetric matrices V1 VT and (U2U2 + 3. I) commute. Hence, they are orthogonally simultaneous diagonalizable. As consequence, there exists an orthonormal matrix P [dd xd1 such that Vi P and PT (U + c² I -p -P
(P U2U P 22 -
diagonal matrices (thus, PTT PUU is also diagonal). Note that we choose P such that the eigenvalues of V1 VIVi are in decreasing order, by altering the columns of P. Therefore, we can write the SVD of V, and U2 V1 PAQ and U2 PΩN with orthonormal matrices P Rdi Xd1
Q E Rdo xdo N E Iedexda and singular matrices A E IRd1 ddo, Q E IR Specifically, min(do.dd))
min(do.da) respectively. the singular values of V1, U2, Z are {M: (A))mi {w))== ) mnin(d2,d5) da)) and {0: (0.)min(do
Next, from Eqn. (64), we have:
of U U2
------ UU (U,. - DI - N.W
c2 U (U2U,) UU2U, + 2 82
-V.VV + (UUUU +
(74)
31
31
B1
where we use Eqn. (67) and Eqn. (66) in the last equality. Using the SVD forms of V, and U2 in Eqn. (74) yields:
Na SNNT
2
B2
NΩ.
+ 22
I (
+
2 2 TONT
-
B1
31
B1
B2 wf ( + tof
Vi (mi,(d1,d2))
B1 (w} + 2 22))
B2 ² 32 2 + = ow +
D 0 or Computing the components in the loss function Eqn. (73), we have:
Vi E [min(di, d2)].
(75)
3
trace UU, = trace U22
= trace (NΩ T ΩN Σg) = of trace(n. .) = o
Σ
w/
1=1
trace V V.U.Z. Z)
trace V (ViV, + -
da
X
X +
Irace (UUUU + c2 2 - (V, V7
4
X
+o
+ +of
= Σ
31
w
82 Mi
where we denote {0, ) are the singular values Z and we use Von Neumann inequality for ZT Z and Vī (V + - V,. The equality condition holds if these two symmetric matrices are simultaneous ordering diagonalizable.
36

---

Published as a conference paper at ICLR 2024
We assume that d1 d2 S do-
From the loss function in Eqn. (73) and above calculation, we have:
do
do
X203
de
X3 + of
AE Ndec LVV
+
$102
32
oo
w + c
2
do
1p
X-03
d1
di
X of +
+ +110
2
c
82
+o
w
2 31
1=1
=1
1=1
1=1
do
o90O
C2 + oi)
0
+
310300
(76)
X of
w/
+ + 8/
where we denote c == 1/dec/ Mene- We consider two cases derived from Eqn. (75):
If w 0:
0702 0(A.0) 22 + of + +
(77)
We can easily see that if 2 15 we have is minimized at x = |00 (0 v
=v vBi
with 9" =2/310,0 = 91- Otherwise, X" 0 and g" = 0 + Biot 92.
If 2 + 2
82
= 02 3 + o.. Let + (1 2 σ1), we have that w²
31
e
12
t-
2
2 0, hence t 2 co2
We have:
D1
3,
B1
o70²
w(),w)
+ 2c02 v 3152
X2 + of - 320203
X²+ of
0,0²
12 + 2002 v 31 320 20 = -i(t, 0), t2 max(o1,,o 002)
(78)
Taking the derivative of h w.r.t yields:
0h
20702
+ 2000 v 3,,3,
(79)
01
13
Oh
0
Öt
0702
*1 =
= lo-
(80)
00.531132
We can see that to is the minimum of h(t, if to 2 max will prove that the minimum of fu(t,, 0) achieved max(o1, co21 three following cases about to:
(2). B1
Otherwise,
We consider
B1
2 V C0 /00102//113 d'02
If 2 max(o1, co2
0 8 di vBi
The minimum of g is achieved t" to with corresponding function value 93 3( V o010021//135) - 20302. We compare as with gi and ge from the case w = 0:
91 - 22=2 + 3, 3/BB2280f0300 3
-VB + v 1i - B1320 20 0. (Cauchy-Schwarl inequality)
g2 9o = 0: + 310 + 3 313200000300
20. (Cauchy-Schwarz inequality).
37

---

Published as a conference paper at ICLR 2024
Thus, 9" 9s and:
o
==
V01 V
031822207032
13132002
VB2001
13clo1
w
3103
VB2
1
B1o7
(81)
If o1 2
V
la co2 > to + 0;
do
S
VV313200102
Thus, vBlai 2
vD, 01
v B2002.
From the condition t 2 max(o1, v B
co2). we obtain
01 to. For to.
we have:
@h(t)
@h(to)
Oh Öt
(since
is an increasing function w.r.t ()
Öt
Ot
So the minimum of g is achieved at t" = with corresponding function value:
94 = 8² + 220 / 31 - 3202
(82)
We only need to compare with 92 since 0: < VV313,0010 S v(Viia1
V We have:
92 94 3 32c²02
200 / B132 2 0.
(Cauchy-Schwarce inequality)
Thus, 9" 94 and:
== 0,
82 00
82
w'
c²
31 02
81
If 01 > to 2
+
c'
S 0, < v VV133201102 Thus, vBlai 2
vDL
VB2002-
Similar to the previous case, the minimum of is achieved at t" 01 with corre- sponding function value:
g" = 94 82 2200102 3132 3202
(83)
and minimizers:
=,
32 00
82 c2
w-
V B1 02
81
If V co2 > max to v A co22 and thus, w 0. We already know that
when w" = 0, A = 0 when 0 < V and X' V
a.0 Jn
of when 0 2 VF101-
In conclusion:
If max V VB13200102.
B é vih 01 /:
of
X*
V0 1
BB.32002
VB2co
B30o0
V02
3/
B10
V
38

---

Published as a conference paper at ICLR 2024
8. c00
If < max(V VB18200102
and v 2 VB2002:
v
X
B2
82 62 11
w
VV
31 02
If < max (V
1B13200112
c and vBioi < 32002:
vo,
X = =0
If < mmx V 1310200102
&. c0 and v S 0 and v < 32002:
v
w" 0,
01 (0 vBio)).
X
VB1
Remark 5. The singular values of W2 and U₁ can be calculated via Eqn. (66) and Eqn. (68). From Eqn. (66), we have that:
W₂ = U. (UJU) 21 + +c21)-1PT.
(84)
Since can be arbitrary orthonormal matrix, the number of zero rows of Wa can vary from (no posterior collapse) to d2 rank(W.2). Similar argument can be made for V1. Indeed, we have b has the form V1 = PAR T. Thus, P will determine the number of zero rows of V1. The number of zero rows of V1 will vary from 0 dj rank (V) since P can be chosen arbitrarily.
39


Reviews:
[{"review_id": "ebz6aH8GKM", "reviewer": "Reviewer_vZoP", "summary": "This work provides a theoretical analysis of posterior collapse in Variational Autoencoder (VAE) models. First the authors extend previous results for the linear VAE case (Variational Autoencoders with only a single linear layer as encoder and decoder) and provide a novel condition for posterior collapse. Then these results are extended and the authors provide conditions for posterior collapse for two further VAE models: the Conditional VAE (CVAE), and the Markovian Hierarchical VAE (MHVAE).", "questions": "In Fig. 3 (b) the case for beta_1=1.0 and beta_2=1.0 has been left out. Just for completeness it might be interesting to visualize the result for this \"standard\" ELBO (beta_1=beta_2=1.0).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work provides important theoretical insights into the conditions for posterior collapse in Variational Autoencoders. Although these conditions are only presented for the linear case, and for MHVAE the case that only one of the two encoder variances is learnable, this work provides an important step forward for the understanding of the inner workings of VAEs and provides the basis and theoretical tool set for future research. The empirical evaluation confirms that the theoretical insights gained for the single layer encoder and decoder case extend, at least empirically, to the nonlinear (multi layer) case.", "weaknesses": "The conditions provided in the paper are only for the linear (single layer encoder and decoder) case, which of course limits the practical applicability of the framework. Yet, I believe that the work provides an important step forward in the understanding of VAEs and do not see a big drawback in this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQr2fOpy21", "reviewer": "Reviewer_Bn1w", "summary": "This work extends previous theoretical work on the linear variational autoencoder (VAE), showing the conditions under which linear conditional VAEs (CVAEs) and Markovian hierarchical VAEs (MHVAEs) will exhibit posterior collapse. Empirical experiments are conducted that supports theoretical results in both linear and nonlinear cases.", "questions": "- Can you clarify on the novelty claim for theorem 1?\n- On the hierarchical VAE results, you mentioned that it is advisible to create separate maps between the input data and each level of hierarchy, which seems closely related to the ideas in (Sønderby et al., 2016; Zhao et al., 2017; Vahdat & Kautz, 2020). Maybe it will be helpful to discuss these models since the theoretical results here seem to provide motivation for them?\n\nReferences:\n\nCasper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.\n\nZhao, S., Song, J., & Ermon, S. (2017). Learning hierarchical features from generative models. arXiv preprint arXiv:1702.08396.\n\nArash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667–19679, 2020.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **The results are potentially novel and can have interesting implications.** This work extends the previous results on more complex VAE models which are widely used in practice. This might give practical directions for people deploying these variant of VAEs that previous work did not cover.\n- **Experiments clearly support the main theoretical results.** Both linear and nonlinear models are trained for different parameters and subsets of data. The results very clearly support the theoretical predictions and shows evidence on how the linear results can be predictive of the general nonlinear case.", "weaknesses": "- **The paper seems to claim a result that has already been proven.** Theorem 1 seems to be exactly identical to proposition 2 in (Wang & Ziying, 2022). Maybe the functional form of the result is identical but the authors have more general assumptions on $\\sigma_i$ (I doubt this after having scanned the proof in Wang & Ziying). In either case, the statement that this theorem is one of the novel contributions is false or at best misleading.\n- **It is unclear how much novelty there is in extending the proof to CVAEs and MHVAEs.** Since I did not follow the proof for theorem 2 and 3 closely, and I am also not familiar with the literature, I cannot be certain whether extending the existing results on linear VAEs to CVAEs and MHVAEs is methodologically novel. In particular, the expression for $\\omega^*$ in theorem 2 seems quite similar to that of theorem 1 (which I do not consider novel, see previous point), and it is unclear to me if the derivation follows almost identical strategies. I think it might be helpful if the authors can discuss why extending the proof to the CVAE and MHVAE cases are interesting and nontrivial. This might just be a minor concern as one can argue that the implications of the results might be the more important and interesting aspect of this work.\n- **Graphics are not very easy to read.** It is difficult to see how the results shown in the plots support the theoretical results. This is only a minor critique and can be easily improved by using a different coloring scheme or just rearranging the labels in a vertical list and/or remind the reader what the theoretical predictions are (larger $\\beta1$ and smaller $\\beta2$ is good, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kSqEbwn70y", "reviewer": "Reviewer_VR5S", "summary": "This paper presents a theoretical and empirical analysis of the problem of posterior collapse in two types of VAE, specifically conditional and hierarchical. Under a linear VAE setting they prove certain specific conditions as contributors to posterior collapse. They then conduct various experiments on MNIST, focusing in their empirical work on non-linear VAEs, and demonstrate that their theoretical results line up with the experimental findings of what settings lead to collapse.", "questions": "You claim that Wang & Ziyin 2022 arrived at the opposite conclusion as you regarding the role of learnable latent variance. Can you say more about why you would have found different results and what that means?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper puts forward theoretical analysis of posterior collapse in previously unstudied models that are more complex than vanilla VAEs\n- The paper performs experiments that corroborate the theoretical claims, albeit in a more realistic nonlinear setting", "weaknesses": "- Much of the empirical results are relegated to the appendix with the paper itself being somewhat light on experiments. This isn’t necessarily a bad thing but it would be nice to see more thorough experiments in the main paper, e.g. vanilla VAE, linear CVAE/MHVAE, learnable vs. unlearnable variance, just for comparison.\n- The theoretical results are generally restricted to highly specific scenarios, such as linear and two latent for MHVAE, and the setting of both variances being learnable in MHVAE is not considered. It would be nice to see more theoretical motivation for why these results might generalize to more realistic training settings, but instead that is largely directed towards prior work.\n- Even in the empirical setting the majority of models studied are restricted to fairly shallow networks. The results are entirely on a simple dataset, MNIST, and therefore it’s hard to get a sense for how the experiments would behave in a more complex setting with deeper networks, modern architectures, and less standardized data.\n- One of the most noticeable absences is that this paper doesn’t consider the effects of SGD and optimization strategies. The setting where global solutions are directly obtainable is not really representative of most real world VAEs which would involve deep nonlinear networks that are most likely to only reach a local optimum. While the experiments themselves do use Adam to train the models, there’s not much analysis of how the optimization strategy interacts with other phenomena.\n- Many of these weaknesses could be levied against prior work in this space. For what it’s worth this paper does go beyond those works in that they do rigorous theoretical analysis of more complex model types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZWICdRqGb", "reviewer": "Reviewer_fUP2", "summary": "The paper presents the first theoretical analyses of posterior collapse in the conditional VAE and markovian hierarchical VAE. Both models contrast with standard VAEs, which have theoretical results related to posterior collapse, in that they have more complex latent structure. Mathematical analyses derive conditions under which we should see posterior collapse for both classes of models. Qualitative observations are extracted from the theorems. Experimental results on MNIST test and validate the qualitative observations.", "questions": "My main questions are related to the limitations. I would like to hear how the authors would plan to revise to clarify and sharpen exposition.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The VAE is a widely used, classic generative modeling framework. Theoretical results providing a better understanding of performance, and lack thereof, are interesting and valuable. \n- The theoretical results and qualitative observations provide satisfying and interesting insights into the behavior of the model. \n- The empirical experiments provide a nice, if modest, evaluation of the predictions.", "weaknesses": "- The primary weakness was in exposition. The paper tends to discuss intuitions after theorems, which makes the theorems a bit challenging on first read. \n- Moreover, at several point the authors assume a lot of the reader. Specifically, consider the case of unlearnable Sigma, and comparisons between theorem 1 and the previous work (the result from which is not stated in the text). \nTaken together the paper was more difficult than need be to understand. \n\nMinor comments: \n\"Interestingly, we find that the correlation of the training input and training output is one of the factors that decides the collapse level\" Why is this interesting? \n- \"We study model with\" typo\n- \"beside the eigenvalue\" typo\n- \" i) we characterize the global solutions of linear VAE training problem for unlearnable Σ case, which generalizes the result in (Wang & Ziyin, 2022) where only the unlearnable isotropic Σ is considered, and ii) we prove that for the case of unlearnable Σ, even when the encoder matrix is low-rank, posterior collapse may not happen. Thus, learnable latent variance is among the causes of posterior collapse, opposite to the results in Wang & Ziyin (2022) that it is not the cause of posterior collapse.\" This warrants some explanation. Why is the conclusion opposite? \n- What does it mean for Sigma to be unlearnable? This isn't properly explained. \n- What is the meaning of Theorem 1? It isn't yet clear to me what we are proving or why? \n- \"We note that our results generalize Theorem 1 in (Wang & Ziyin, 2022) where σi’s are all equal to a constant.\" It would really be helpful to the reader to present the previous result and explain clearly why. \n- Ok the explanation after the theorem is helpful. It would be preferable to help readers see what is coming ahead of time, too. \n- I am still unsure about what an unlearnable Sigma is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aM2cV6swrY", "reviewer": "Reviewer_1CHE", "summary": "In this paper, the authors analyze the issue of posterior mode collapse for conditional and hierarchical linear VAEs. They demonstrate conditions required to observe posterior collapse, which provides meaningful insight into the design and training of more complex VAEs.", "questions": "Please clarify what new designing insights are obtained from your results that couldn't be guessed from other works in the literature on analytical understanding methods for VAEs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Technical derivations seem correct, and the paper methodology is solid.\n- Theoretical insights into the understanding and design of VAEs.", "weaknesses": "The reviewer appreciates little contribution in the paper. Yes, the authors generalize the results obtained for the linear VAE by Wang & Ziying (2022) and Dai et al. 2020 to linear CVAEs and hierarchical linear VAES, and this contribution is acknowledged. However, I wonder if generalizing the analysis to these new models resulted in different conclusions or insights about avoiding posterior collapse. In other words, is anything in Table 1 significantly different from what we learned from Wang and Ziying? At the end, the conclusion is that both $\\beta$ and $\\eta_{dec}$ should be small ...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)