Posterior Collapse of a Linear Latent Variable Model
Authors: Zihao Wang
Liu Ziyin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian deep learning
loss landscape
posterior collapse
linear model
VAE

Abstract: This work identifies the existence and cause of a type of posterior collapse that frequently occurs in the Bayesian deep learning practice. For a general linear latent variable model that includes linear variational autoencoders as a special case, we precisely identify the nature of posterior collapse to be the competition between the likelihood and the regularization of the mean due to the prior. Our result also suggests that posterior collapse may be a general problem of learning for deeper architectures and deepens our understanding of Bayesian deep learning.

Posterior Collapse of a Linear Latent Variable Model
Zihao Wang" Department of CSE HKUST
Liu Ziyin' Department of Physics The University of Tokyo
Abstract
This work identifies the existence and cause of a type of posterior collapse that frequently occurs in the Bayesian deep learning practice. For general linear la- tent variable model that includes linear variational autoencoders as a special case, we precisely identify the nature of posterior collapse be the competition be- tween the likelihood and the regularization of the mean due to the prior. Our result suggests that posterior collapse may be related neural collapse and di- mensional collapse and could be a subclass of a general problem of learning for deeper architectures.
1 Introduction
Bayesian approaches to deep learning have attracted much attention because they allow for a more principled treatment of inference and uncertainty estimation (Mackay, 1992; Neal, 2012: Wang and Yeung, 2020; Jiang and Ahn, 2020; Zhao et al., 2021; Liu, 2021). One long-standing and unresolved problem for the Bayesian deep learning practice is the problem of posterior collapse, where the posterior distribution of the learned latent variables partially or completely collapses with the prior (Bowman et al., 2015; Huang et al., 2018: Lucas et al., 2019; Razavi et al., 2019; Kingma et al., 2016; Wang et al., 2021). Up to now, the study of the nature of the cause of the posterior collapse problem has been limited. There are two main challenges that prevent our understanding of the problem: (1) posterior collapses mainly occur in deep learning, and the landscape of deep neural networks hard to understand in general; (2) the use of approximate loss functions such as the evidence lower bound (ELBO) complicates the problem.
Consider problem where wants to model the data distribution p(2r, vy) through latent variable z. The evidence lower bound (ELBO) loss function reads
Ey.pl [g.p[-BB(cso)) log(p(>l=))] + F. (P
(1)
free
where q is the approximate distribution, we rely on to approximate the true distribution p. This loss is more general than the standard ELBO for variational autoencoders (VAE) (Kingma and Welling, 2013). Meanwhile, it can be seen as the simplest type of loss for a conditional VAE (Sohn et al., 2015), where aims to model conditional distribution pzy|z). The distribution p(2) is the prior distribution of the latent variable x and is often a low-copplexity distribution such as a zero-mean unit-variance Gaussian. This loss function thus has a clean interpretation the sum of predic- tion accuracy term (the first term Eree) that encourages better prediction accuracy and a complexity term (the second term (KL) that encourages simpler solution. Learning under this loss function proceeds by balancing the prediction error and the model simplicity. Moreover, learning under this loss function has also been used one of the primary theoretical models in neuroscience (Friston, 2009), and its understanding may also help advance theoretical neuroscience. This work provides an in-depth study of the posterior collapse problem of Eq. (I), when the decoder ((y|z) and encoder (2(2z) are each parametrized by linear model.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).
Equaal contribution.

---

Specifically, our contributions include:
we find the global minima of a general linear latent variable model that includes the línear
VAE as a special case under the Objective (1);
we find the precise condition when posterior collapse occurs, where the global minimum is
the origin:
we pinpoint the cause of the posterior collapse to be the excessively strong regularization effect on the mean of the latent variables due to the prior.
To the best of our knowledge, our work is the first pinpoint the cause of the posterior collapse problem. This work is organized as follows. The next section discusses the previous literature. Section 3 describes the theoretical problem setting. Section 4 presents
main technical results and analyzes them in detail. Section presents numerical examples. The last section concludes this work and points to the remaining open problems. The Appendix B investigates the effect of the bias term, Appendix C details the effect of a data-dependent encoder variance, and Appendix D treats the case of a learnable decoder variance.
2 Related Works
Approximate Bayesian Deep Learning. Bayesian deep learning in general and VAE training, in particular, rely heavily approximate methods such the ELBO objective because the exact probabilities are intractable. The connection of approximate Bayesian learning and probabilistic PCA (pPCA) has been extensively studied (Nakajima and Sugiyama, 2010; Nakajima et al., 2013, 2015; Lucas et al., 2019).
Causes of Posterior Collapse. Earlier touches the problem tend attribute the cause of posterior collapse to the use of approximate methods, namely, to the use of the ELBO (Bowman et al., 2015; Huang al., 2018; Razavi et al., 2019). Another line of work attributes the cause to the high capacity of modern neural networks (Alemi et al., 2018; 8; Ziyin et al., 2022c). However, Lucas et al. (2019) showed that for simplified linear model, the ELBO is not the cause of posterior collapse because the posterior collapse exists even in the exact posterior. collapse is not due the high capacity of the models because linear models have limited capacity. Lucas et al. (2019) then suggested that making the decoder variance learnable can fix the collapse problem and that an unlearnable decoder variance is the cause of the posterior collapse. However, our results show that this is not the case: for both learnable and unlcarnable decoder variance, there exist situations where a collapse happens or does not happen, which implies that the learnability of the decoder variance does not have a causal relation with posterior collapse, nor is it sufficient to fix the problem (Section D). terms of the problem setting, ours is also more general than Lucas et al. (2019) because our result (1) applies to general latent variable models (one example being the conditional VAE (Sohn et al., 2015)) and (2) considers the case of 3- 3-VAE with general 3 when the decoder variance is learned. An important implication of our work is that posterior collapses can be ubiquitous problem for deep-learning-ban -baeed latent-variable models (not just unique to autoencoding models) and that they share common cause. Meanwhile, (Lücke et al., 2020) shows that posterior collapse can happen due to the tradeoff between the decoding performance and the decoding entropy. (Shekhovtsov et al., 2022) demonstrated the relationship between model consistency and posterior collapse and suggested that a proper choice of data processing or architecture may alleviate collapse.
also implies that the posterior
Linear Networks. Deep linear nets have been extensively used to understand the landscape of non- linear networks. For example, linear regressors are shown to be relevant for understanding the gen- eralization behavior of modern overparametrized networks (Hastie et al., 2019). Saxe et al. (2013) used a two-layer linear network understand the dynamics of learning nonlinear networks. The linear nets are the same as linear regression model in terms of expressivity. However, the loss landscape highly complicated due to depth. (Kawaguchi, 2016; Hardt and Ma, 2016; Laurent and Brecht, 2018; Ziyin et al., 2022a). Our work essentially studies the loss landscape of linear networks. While each encoder and decoder we use consists of a single lincar layer, they effectively constitute two-layer linear network when trained together.
3 Problem Setting
We consider a general linear latent variable model with input space E R Do, latent space e RR and target space y Rdd, general, y = f(x) is an arbitrary function of x. When the target is identical to the input æ. it reduces to the standard VAE. The VAE formalism assumes that there

---

is an intermediate "latent variable" that captures the data generation process, In the main text, the encoder and decoder are linear transformations without bias terms, and the learnable bias is treated in Appendix B, which shows that the effect of the bias terms is equívalent to centering both the input and target to be zero-mean (x - - E[ y/ - y F[y)). Incorporating the bias terms thus does not affect the main results. Specifically, the encoder is defined as = = WT x: where e ~~ (0, 2) is the noise distribution introduced by the reparameterization trick where the variance matrix Σ diagg(?).... ...,, o,) is assumed to be diagonal and independent from x. The decoder parametrizes the distribution p(y]=) 2U, 7), where the variance Taae 7 is to be isotropic and input-independent. In alignment with the standard practice, we also assume the prior distribution of latent variable p(=) = .V (0, veecc)) is an isotropic normal distribution, and the encoding variances matrix M is learned from the data distribution while na is not Icarnable. Lastly, we weigh the KL term by a coefficient /8, which is a common practice in VAE training (Higgins et al., 2016). Hence, the objective of such linear model reads,'
TVAE (U, W,
(2)
= E, (B log(p(ul2)) +
)]
(3)
= En,e
|U( x - v112 + Ndoc
[ x
+ Σ
di
8
o
log
o
(4)
2M
nene
2
2
1=1
Eg GulUWT x- v12 TT(EEUT) + nace nene
3
0
0
=
27
Tr (WT AW)
1-1
2
nene
1
log
Nenc
2
Emenn
Esar
where A == [xg[xx]]]] the second moment of the input data. Note that crucial feature of the KL term is that it decomposes into two terms, one that regularizes the variance of = (Evar) and another that regularizes the mean of (Emean). We will see that it is precisely the Emean term that causes the posterior collapse. Eq. (5) has ignored the partition function of the decoder because we treat deee as a constant. We study the case of learnable Tidec in Section D. In comparison to the previous works (Lucas et al., 2019) that have treated the case of a learnable ndec. our result is more general because our result also considers the effect of and allows for the case d, 2 d2. is also worth commenting on the difference between this setting and that of the pPCA setting (Nakajima et al., 2015): (1) the effect of / is included in the VAE loss, (2) the prior of VAE is over the latent variable, whereas pPCA has it over the model parameters, and (3) the model can be overparametrized (d) 2 do).
denote the input variable, latent variable, and tar- get variable, respectively. Ez denotes the expectation over the training set. A == Ex [xx""" is the second moment matrix of the input x. A is thus positive semidefinite by definition. The eigenvalue decomposition of A is A = P where & diag(é), Od,) ) is the diagonal matrix for all do Do positive eigenvalues and PA = [Pi,... Pdo are matrices by concatenating do eigenvectors p; Ro, W and U are learnable linear transformation matrices for the linear encoding and decod- ing processes. Σ is the learnable diagonal latent variance matrix for encoder with diagonal entries og MJone is the standard deviation of the prior distribution p(=). T/doe is the standard deviation of decoded samples. A frequently used quantity is a whitened and rotated æ: ā == a 1 PT Note that this transformation can be inverted:
Notation.
To summarize, we use y. and
PA P f. We see that F T. Furthermore, we define Z : Ex [y Ex[ [x[yx" Px $] IRdxxdo Let Z - FE be the singular value decomposition of Z, where F E Rd and G E dooxdo are two orthogonal matrices. Σg E Rd is rectan- gular diagonal matrix with d" min(d, d2) singular values of Z in the non-increasing order, i.e.,
C 2 <o 2 ... 2 Sa- 20 0.
We note that E. is the expectation over the training set. Also, we use the subscript "VAE" because the model can be seen as a conditional VAE, even though it may be more proper to call it a "general latent variable model."
3

---

Main Results
This section discusses the main results, whose proofs are presented in Appendix E. While is often a learnable parameter, we first assume that the KL term is sufficiently strong such that σ = od fJenc is close to the prior value. We then compare with the case when it is learnable, and this comparison reveals that an optimizable og is not essential to the posterior collapse problem.
4.1 General Result
In this section, we prove results that will be useful for understanding the nature of the VAE training objective and will be useful for us to find the global minimum. We first show that the VAE objective is equivalent to matrix factorization problem with special type of regularization.
Proposition Let ā == x- al Exly J. and
(U", (V,, mig L(U,V) arg min -UVT- TY(USUT) + 3dde V (Ú,V)
(6)
(U.V)
Nene
Given fixed E, the minimizer of LVAE(U, W, is (U", ("", where W is any solution of ④b PAW= V
Proof sketch. The term hpar is irrelevant to finding the optimal U and V when E is fixed. Thus, the relevant objective can be obtained with the change of variables ã x PT
The condition & PTW V* shows that when the data is low-rank, each solution (U*, V*) corre- sponds to manifold of solutions in the original parameter space. The effective loss L(U, V) can be compared with the regularized singular value decomposition problem (Zheng al., 2018). We see that the first term is the standard matrix factorization objective, while the second and third are unique regularization effects due to the VAE structure and the ELBO objective. addition, the term E in the second is the strength of the regularization for the norm of U, and a crucial difference with standard regularized matrix factorization that also a learnable matrix.
The next proposition finds, for any fixed Σ, the global minima (U* V*) " of Eq. (6). In particular, the learning characterized by the learning of the singular values of U and V.
Proposition 2. The optimal solution (U* V*) of ming.v L(U.V) is given by
U = =AP V*=GOP,
where F e Rd2xda and G E doxod are orthogonal matrices derived by the SVD of Z, P is an arbitrary orthogonal matrix in Rdixd, and A E Iddaxd and θ Rdaxd, are rectangular diagonal matrices with the diagonal elements
V /maec
O:Monc
A
max
0,
C:
0,
max 0,
4:
oefeee
Meee
BBnddc
Menc
(8)
For convention, we let S5 when i d' min( (o. dz).
Proof sketch. The optimal V* a function of U under the zero gradient condition. Thus, the objective reduces to single-variate with respect to U. The optimal U* is constructed by its SVD U= QAP, where the optimal Q" and can be determined given the SVD of Z, and left as free orthogonal matrix. V* is determined once U is obtained. D
The readers are recommended to examine the form of the solutions closely. There are a few inter- esting features of the global minimum. One note that the sign of the term Ç, //aaa//ae//////// Dadee/lenee is crucial, and can encourage the parameters U and to be low-rank. Recall that d; is the eigenvalue value of ZZT E[y [[ya one can roughly identify ?? as the the strength of the alignment between the input æ and the target y. To see this, consider a simplified scenario where the target y > is linear function of the input, where is the overall strength of the signal and ||M]] a normalized orientation matrix, then
ZZT 2 AM.
(9)

---

of L2 which is positive semidefinite matrix. We see that there are two distinctive sources of contribution the magnitude of the eigenvalues of ZZ". Its eigenvalues are large if either the overall strength is large or if the orientation matrix M aligns well with the covariance of the input feature A. Additionally, in the case of VAE, yM T, and ZZT is nothing but the covariance of input features, and ç are the eigenvalues of A.
4.2 Linear VAE without Learnable Σ
We first consider the case where d; is a constant that is completely determined by the prior: oz Menc- This allows find a simplified form for the global minimum. The proof follows by plugging
02 =
Aenc into Proposition 2.
Theorem 1. Let 0; Tooe for all i. Then, the global minimum has
max (0, BBrdde (G- ---Baaas)) 0, =
n BBdee
(c: / V
max 0,
(10)
Tiec
There are three interesting observations of the global minimum. First of all, it depends crucially on the sign of - VBndee for all i. When the sign is negative for some i, the learned model becomes low-rank. Namely, some of the dimensions collapse with the prior. When the signs are all negative, we have a complete posterior collapse: both U and V are identically zero, so the latent variables have distribution identical to the prior. A complete posterior collapse happens and only if max; Çi /BBlaor $ O. A partial posterior collapse happens if there exists such that 2, /Bndee O. These two conditions give the precise conditions of posterior collapse in this scenario. This implics that having sufficiently small / will always prevent posterior collapse. The second observation is that the effect of / is identical to that of TJdec because v3 and Tjdec always appear together, and so one alternative way to fix posterior collapses is to use sufficiently small Mdeec- From Bayesian perspective, the latter method of tuning Mdee is better because Tadcc comes directly from the (assumed) likelihood p(z p(z]/ttee). In contrast, the / parameter is only an implementation technique that has obscure meaning the Bayesian framework. Therefore, using small Mee can be fix to the problem that is justified by the Bayesian principle. The third observation that the condition for posterior collapse completely independent of the parameter TJene- which is the desired variance according to the prior p(z). This means that under a Gaussian assumption, the prior does not affect the posterior collapse all.
Lastly, one also notices a potential problem. The eigenvalue of the second layer increases with /Brdec. while the first layer decreases with / and so having too-small 3 or nade causes the model to have very large norm, which can cause a significant problem for both empirical optimization and generalization. This problem is well-known in the studies about the regularization deep learning: suppose we apply weight decay to two different layers of a ReLU net, and decrease the weight decay strength of one layer zero, then the norm of this layer will tend to infinity, and the norm of the other layer will tend to zero (Mehta et al., 2018). However, in the next section, will see that this problem miraculously solved for VAE when o; is learnable.
4.3 Linear VAE with Learnable Σ
Now, we consider the more general case of a learnable Σ. In practice, Σ is often dependent on the input æ. We make the simplification that Σ is just inta-dnnd optimizable diagonal matrix, which is the common assumption in the related works (Lucas et al., 2019). In Section C, we consider the case when is data-dependent and show that our result remains unchanged. The following Corollary gives the optimal training objective a function Σ and is direct consequence of proposition Corollary 1.
di min L(U.V)
- (
Σ
?
(11)
Taagngae Menn
U.V
1=1
Jene
4,>
imd,+ +1
where the indicator 1 1>0= = when the corresponding inequality condition > true, and f>o O otherwise.
5

---

The constant term Eidd +1 G in Equation (11) only appears when the latent dimension di is less than d min(do, d2). This is the common situation for VAE applications. It indicates that the model learns the large eigenvalues and ignores the small eigenvalues. This means that to find the optimal of Eq. (5), one only has to find the global minimum of a reduced objective:
0
log oi nenc
min (VAE(U, W, U.W
min L(U,V)+
+
2 1=1
(12)
U.V 2Mdec
2
d
of M
2M 2=1
2
nenc
SiS
V Bdee
Beue
Bidec
Tene
log
+ constant.
:l)((r)
(13)
The optimal of thus be obtained by minimizing each independently: of arg mig 4(0). Proposition 3. The optimal of of(((r) is
of Braes
[eenc
if Bracc < ??: if Brracce 2 ??
(14)
Jeue
This proposition gives an explicit expression for of. On the one hand, we see that there is a threshold value for B. If 3 is sufficiently large, a; will identical to the prior value T/enes in agreement with our assumption in the previous section. On the other hand, the learned variance a, is a function of B |3 is below a threshold. We will see that this threshold is the necessary and sufficient condition for posterior collapse happen in a learnable E setting. Thus, the learned variance being identical to the prior variance is also a signature posterior collapse. The following theorem gives the precise form of the global minimum.
Theorem 2. The global minimum of LVAE (U, W, W,E) is given by
T==FAP,
(15)
W is the solution of
Φb P.W. GOP,
(16)
where and are derived by the SVD of Z, P is an arbitrary orthogonal matrix in Rdixd, and A diag(A Ad,) and θ diag ( ..., 0a,) are diagonal matrices such that
1
A
max (0, C² - :neae)
(17)
nenc
max (0, B31800)
when G > 0;
0
(18)
=
otherwise.
The optimal V diag(o;?
Jo such that
VBnane
Brdeo G7 Bm Gi-
of =
TJenc G
(19)
nenc
for s min(do, d2). For i min dz). 0 =0. O.
Proof. The optimal solution U* W* n are obtained by combining proposition 1. 2, and 3.
Comparing with the solution in section 4.2, one notices two things: (a) the conditions for complete or partial posterior collapse remain unchanged, which implies that a learnable latent variance neither qualitatively nor quantitatively relevant for the posterior collapse problem even though the functional form of the eigenvalues changed; (b) the magnitude of each of the two layers no longer scales with / Brjacc and using small 3 77 will not directly cause the model to diverge in the

---

norm, which suggests using that making Σ learnable can have the unexpected practical advantage of stabilizing the training.
Additionally, one also notices that Madee has the effect of keeping the learned model low-rank by removing all the eigenvalues of the learned model below it. This be directly compared with the effect of using latent dimension smaller than the input dimension: d1 < do- In the latter case, difference between the two types
the smallest d1 do singular values are also pruned. There of low-rankness: using a large 3rgec both removes all the singular values below it and shrinks the remaining ones while using small latent dimension only removes the smaller singular values with- out affecting the rest. This is similar to the difference between soft thresholding estimation and hard thresholding estimation in statistics (Wasserman, 2013). This suggests that partial posterior col- lapses are not necessarily undesirable because, during a partial posterior collapse. the latent variable models automatically perform a degree of sparse learning, which is theoretically understood to help denoising the signal and lead to better generalization (Markovsky, 2012). That being said, complete posterior collapse should always be avoided.
4.4 Learnable Tidec
Our result can also be extended the case when 'ade is learnable, which has been suggested by Lucas et al. (2019) as a remedy for the posterior collapse. To do this, we need to include the partition function of the decoder, proportional to log vaee> that has been ignored in Eq. (5). We present the detailed analysis in Section D. Our analysis shows that even if Titeo is learnable, posterior collapse can happen for some datasets. In addition to the fact that is also possible for collapses not to happen when Teec is not learned, we conclude that Mee does not have a causal relation with posterior collapse. Our analysis also suggests a way to fix posterior collapse for VAE: make ndee learnable and set 3 dz/d". Note that the condition / dg/d" is tight in the sense that if it does not hold, then there cxists a data distribution such that complete collapse can happen. This condition also highlights that it is important to introduce the 33 coefficient for VAEs because, for VAE, dz/d" 1, and this condition translates to 3 < 1; namely, vanilla VAE cannot avoid complete collapse.
Our result also implies that 33 has a highly nonlinear effect on learning depending on the architecture. For example, when the model is underparameteriized (d, < dd),, using a small / does not cause any problem, whereas for an overparametrized model, small 3 causes the decoder variance to converge towards
4.5 Implications
Our main results have implications for both the problem of posterior collapse and the practice of latent variable models in general.
The cause of posterior collapse. One important implication is the identification of the cause of the posterior collapse problem and the potential ways to fix it. Our results suggest that
learnable (data-dependent or not) latent variance is not the cause of posterior collapse; changing the variance of the prior cannot fix or influence the posterior collapse problem;
comparing with the results in Lucas et al. (2019), 7/dec being learnable or not is causally
related to the posterior collapse problem;
the values of Maee and [ are crucial for the posterior collapse;
choosing appropriate /3 is still needed: a sufficiently small /3 can avoid posterior collapse.
Note that the effect of small /3 (large "/eee) weakens the prior (reconstruction) term, and so the cause of the posterior collapse must be the competition between the prior term, which regularizes the com- plexity of the model, and the likelihood term, which encourages accurate reoogniton/reconstuutiin Our results suggest that one can ignore the effect of the Eear term in studying the mechanism of posterior collapse. Ignoring the Luar Eq. 5, one sees that the posterior collapse is caused by the competition between the likelihood and lmean, which is precisely the regularization effect on the mean of 2.
There is an interesting alternative perspective on the nature of the posterior from the viewpoint of the loss landscape geometry. The following theorem states that the origin (where all parameters are zero) is either a saddle or the global minimum for this problem. Since we have shown that o; does not affect the collapse, we simply let a; = Nenn as in Section 4.2.
7

---

20
1.0
15
Are
0.6
d:
Bkv.
VAE
0.4
0.0
Figure I: Training loss L and o; versus on synthetic regression dataset. 0; is measured by averag- ing over the training set. The vertical dashed lines show where the theory predicts partial collapse. Complete posterior collapse happens at roughly 3 = 14.
0.8
20
free
0.6
flxx.
10
LyAE
0.2
0.0
2
B
Figure 2: Training loss and o; versus 3 for MNIST dataset. The vertical dashed lines show where the theory predicts partial collapse. The posterior collapse happens for the MNIST dataset at around ==5.
Theorem 3. The Hessian of Eq. 5 at 0 is positive semidefinite if and only if is the global minimum.
The surprising aspect is that for the latent variable model, there is intermediate case where the origin is local minimum but not global. Therefore, the origin is, in fact, a very special point in the landscape of latent variable model, in the sense that key global property of the landscape (namely, the global minimum) is determined by the local geometry of the model at the origin. Noting that our model can be seen as a direct generalization of the Bayesian linear regression a deeper architecture, it also becomes reasonable to suspect that the posterior collapse problem is a unique problem of deep learning because the standard Bayesian linear regression does not suffer posterior collapse because the origin can never be local maximum of the posterior (Bishop and Nasrabadi, 2006). Dai et al. (2020) also finds the origin to be very special point in a general deep nonlinear VAE structure and that it can be local minimum under various settings. However, the implication of our work is broader. The origin not only a special point for the autoencoding model families but actually be special point for a very broad of model classes (namely, the model class of general latent variable models). The problem of posterior collapse is thus not limited to autoencoders but can also be relevant to common regression and classification tasks.
Connection other types of collapses. Our result suggests that there are some interesting connec- tions between the posterior collapse phenomenon and the neural collapse phenomenon in supervised learning (Papyan et al., 2020) and dimensional collapse phenomenon (SSL) (Jing et al., 2021). Ziyin et al. (2022a) and Ziyin and Ueda (2022) shows that the neural collapse phenomenon for a two-layer model can be understood through the change of the stability at the origin, which is determined by the competition between the signal strength (B[xry) of the data distribution and the regularization strength of weight decay. For SSL, Ziyin et al. (2022b) also shows that the stability of the origin is important and that it is decided by the competition between the level data variation and the data augmentation strength. Our result suggests that the posterior collapse problem can also be understood through the stability at the origin. This might imply that there could be some universal cause of all these collapses that have been discovered independently in different subfields of deep learning, and one important future direction would be study these phenomena from a unified perspective.
supervisdd learning
Insights for latent variable model practices. While we have primarily focused discussing the phenomena of posterior collapse, our results also shed light on latent variable models (including VAE) in practice when there is complete posterior collapse. Specifically, our results suggest that

---

3
7
0
6
9
9
1
6
7
7
7
2
5
3
0
3
7
0
0 9 0
6
3
3
2
7
1
6
9
1
9
9
9
6
7
7
O
1
8
7
8
9
6
6
3
6
9
8
0
9
1
8
7
8
8
8
0
0
/
0
7
3
5
4
5
9 5
8
7
/
3
0
7
6
7 0
(a)
1, remaining modes= 5
(b) 8
2.75, remaining modes: 4
(c) remaining modes:
1
9
0
7
0 /
7
0
8
?
0
1
3
3
1
7
0
9
0
9
9
0
9
9
9
8
8
0
1
7
3
0
9
0
8
7
8
8
9
8
9
9
9
0
9
7
9
9
9
/
9
(d) B 3.5, remaining modes: 2
(c) 8 4, remaining modes: 1
(f) 3 6, remaining modes: 0
Figure 3: MNIST generation under different B. The generated images lose diversity and variation as 9 increases. The number of modes left is estimated by the theoretical prediction of thresholds of each singular value.
latent variable models perform sparse learning through soft thresholding or hard threshold- ing or both;
thus. partial posterior collapse may actually be desirable;
making the latent variance learnable can help stabilize training and avoid divergence of model parameters;
when Yaeec not learned, the effect of increasing identical to the effect of decreasing
Mdec;
when 7ide is learned, one needs to pay special care to choose a suitable B.
5 Numerical Examples
This section empirically examines our theoretical claims for linear models and demonstrates that our key theoretical insights generalize well to nonlinear models and natural data.
Setring. We illustrate results both synthetic data and natural data. For synthetic data, sample input data æ from multivariate normal distribution (0, and target data y = Mæ is ob- tained by linear transformation. Specifically, data, we also experiment with the standard MNIST data. Following common practices, we choose Vddor: 'Jene 1. For non-linear VAE models, we consider two-layer fully connected neural net- works for the encoder and decoder with both ReLU and Tanh activation functions and with hidden dimension dz. For synthetic dataset d, 8, and dn = 2048 for real-world data. In contrast to our assumption that the variances Σ of encoded z are independent from the imput we parameter- ize the variance of each encoded by linear transformation or a two-layer neural network, i.c., This data-dependent modeling is closer to the common practice, and the comparison can justify the correctness of our theory. The model is optimized by Adam with a learning rate of 10-3, The results are reported after the convergence. For MNIST, the learning rate 10
choose do d2 5. As example of natural
(x) MitearrMMP](aa
Results. Linear models are found to agree precisely with the theoretical results, so we only present the results in the appendix. We focus exploring the nonlinear models in the main text. We first consider a simple regression task with MI.P encoder and decoders with the ReL.U activation (Fig- ure 1). Here, we see that the theoretical prediction of loss function LVAe agrees well with empirical observation. Moreover, the threshold of complete posterior collapse is also perfectly predicted. For completeness, we also present the case when (1) the activation is Tanh in Appendix A. We note that the results are similar. The observation is similar to the standard MNIST dataset with a nonlinear encoder and decoder. See Figure 2. For illustration, we also present the generated MNIST images by non-linear 3- V-V trained with different choices of 3 in Figure 3. The latent dimension is five as described before. When there are non-collapsed modes, the generated images are both sharp and contain meaningful variations. As the number of remaining non-collapsed modes reduces to zero, we see that the generated images become increasingly blurred, and the variation between the data also diminishes. When the model completely collapses, the model outputs a constant, as the theory

---

suggests. Moreover, we note that the values of B are chosen according to the theoretical thresh- olds for each mode to collapse, i.e, the top-5 & are [5.12, 34,4, 2.57]. We see that the theoretical thresholds provide good predictive power for the behavior of mode collapse qualitatively.
6 Outlook
In this work, we have tackled the problem of posterior collapse from a loss landscape point of view. Our work also contributes to the fundamental theory of deep learning. The linear VAE architecture can seen as a deep linear model with two layers, whose loss landscape highly nontrivial. In this perspective, our results advance those results in Ziyin et al. (2022a), where the dimension of the output space is limited to ld. The limitation of our work is obvious: our theory only deals with the landscape, and it unclear how the dynamics of gradient-based methods could contribute the collapse problem. In fact, there is strong evidence that stochastic gradient descent can bias the model towards low-rank or sparse solutions (Arora et al., 2019; Ziyin al., 2021). and, in the context of posterior collapse, these are precisely the collapsed solutions. One important future direction is thus study the role of dynamics in influencing posterior collapse.
References
Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., and Murphy, K. (2018). Fixing a broken clbo. International Conference on Machine Learning, pages 159-168. PMLR.
Arora, S., Cohen, N., Hu, W., and Luo, Y. (2019). Implicit regularization deep matrix factoriza- tion. Advances in Neural Information Processing Systems, 32.
Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning, volume 4. Springer.
Bowman. S. R., Vilnis, L.. Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, (2015), Generat- ing sentences from continuous space. arXiv preprint arXiv:1511.06349
Dai, B., Wang, Z., and Wipf, D. (2020). The usual suspects? reassessing blame for vae posterior collapse. International Conference on Machine Learning, pages 2313-2322. PMLR.
Friston, K. (2009). The free-energy principle: rough guide the brain? Trends in cognitive sciences, 13(7):293-301.
Hardt, M. and Ma, T. (2016). Identity matters in deep learning. arXiv preprint arXiv:1611.0422.
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, J. (2019). Surprises high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv: 1903.08566
Higgins, I., Matthey, I.,, Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016). beta-vae: Learning basic visual concepts with constrained variational framework.
Huang, C.-W., Tan, S., Lacoste, A., and Courville, (2018). Improving explorability in varia- tional inference with annealed variational objectives. Advances in Neural Informaion Processing Systems, 31.
Jiang, J. and Ahn, (2020). Generative neurosymbolic machines. Advances in Neural Information Processing Systems, 13:12222-22582.
Jing, I.., Vincent, P., LeCun, Y., and Tian, Y. (2021). Understanding dimensional collapse in con- trastive self-supervised learning. arXiv preprint /
Kawaguchi, K. (2016). Deep learning without poor local minima. Advances in Neural Information Processing Systems, 29:586-594 594.
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, L, and Welling, M. (2016). Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29.
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. 141
arXiv preprint
10

---

Laurent. T. and Brecht, J. (2018). Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pages 2902-2907. PMLR.
Liu, K.-H. (2021). Relational learning with variational bayes. In International Conference on Learning Representations.
Lucas, J., Tucker, G., Grosse, R. B., and Norouzi, M. (2019). Don't blame the elbo! a linear vac perspective on posterior collapse. Advances in Newral Information Processing Systems, 32.
Lücke, J., Forster, D., and Dai, Z. (2020). The evidence lower bound of variational autoencoders of three entropies. arXiv preprint arXiv:2010. 14860.
converges
Mackay, D. J. C. (1992). Bayesian methods for adaptive models. PhD thesis, California Institute of Technology.
Markovsky, I. (2012). Low rank approximation: algorithms, implementation, applications, volume 906. Springer.
Mehta, D., Chen, T., Tang. T., and Hauenstein, D. (2018). The loss surface of deep linear networks viewed through the algebraic geometry lens. arXiv preprint arXiv:1810.07716.
Nakajima, S. and Sugiyama, M. (2010). Implicit regularization in variational bayesian matrix fac- torization. In ICML.
Nakajima, S., Sugiyama, M., Babacan, S. D., and Tomioka, R. (2013). Global analytic solution of fully-observed variational bayesian matrix factorization. The Journal of Machine Learning Research, 14(1):1-37.
Nakajima, S., Tomioka, R., Sugiyama, M., and Babacan, S. D. (2015). Condition for perfect dimen- sionality recovery by variational bayesian pea. Mach. Learn. Res., 16:3757-381.
Neal, R. M. (2012). Bayesian learning for neural networks, volume 118. Springer Science & Business Media.
Papyan, V.. Han, X., and Donoho, D. L. (2020). Prevalence of neural collapse during the ter- minal phase of deep learning training. Proceedings of the National Academy of Sciences, 117740):4452-24663.
Razavi, A., Oord, A. v. d., Poole. B., and Vinyals, O. (2019). Preventing posterior collapse with dalta-vaes. arXiv preprint trXivv/901.000
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprini arXiv: 1312.6120
Shekhovtsov, A., Schlesinger, D., and Flach, B. (2022). VAE approximation error: ELBO and exponential families. In International Conference on Learning Representations.
Sohn, K., Lee, H., and Yan, X. (2015). Learning structured output representation using deep condi- tional generative models. Advances neural information processing systems, 28.
Von Neumann, J. (1962). Some matrix inequalities and metrization of matrix space, tomask univer- sity review (1937) 286-300. reprinted in ah taub (ed.), john neumann collected works (vol. iv).
Wang, H. and Yeung, D.-Y. (2020). A survey on bayesian deep leamning ACM Computing Surveys (CSUR), 53(5):1-37.
Wang, Y., Blei, D., and Cunningham, P. (2021). Posterior collapse and latent variable non- identifiability. Advances in Neural Information Processing Systems, 34.
Wasserman, L. (2013). All of statistics: a concise course in statistical inference. Springer Science & Business Media.
Zhao, M., Hoti, K., Wang. H., Raghu, A., and Katabi, D. (2021). Assessment of medication self- administration using artificial intelligence. Nature medicine, 2744)7777-735

---

Zheng, S., Ding, C., and Nie, F. (2018). Regularized singular value decomposition and application to recommender system. arXiv preprint arXiv:1804.05000
Ziyin, I., Li, B., and Meng, X. (2022a). Exact solutions of deep linear network.
Ziyin, L., Li, B., Simon, J. B., and Ueda, M. (2021). Sgd can converge to local maxima. In International Conference Learning Representations.
Ziyin, L., Lubana, E. S., Ueda, M., and Tanaka, H. (2022b). What shapes the loss landscape of self-supervised learning? arXiv preprint arX1v:2210.06638.
Ziyin, L. and Ueda, M. (2022). Exact phase transitions in deep learning. arXiv:2205.1255 12510..
arXiv preprint
Ziyin, L., Zhang, H., Meng, X., Lu, Y., Xing, E., and Ueda, M. (2022c). Stochastic neural networks with infinite width are deterministic.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] The experi- ments are only for demonstration and are straightforward to reproduce following the theory.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] The fluctuations are visually negligible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are done on single 3080Ti GPU.
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "iQ4mTUewKVI", "reviewer": "Reviewer_R7JS", "summary": "This paper analyzes the globally optimal solutions to linear latent variable models. The authors show that these solutions exhibit low-rank structure that is equivalent to so-called posterior collapse.\n\nThe paper introduces a \"(generalized) variational autoencoder\" objective where the input and target spaces need not match. They show (Proposition 1) that solutions of the VAE objective with a fixed approximate posterior variance are aligned with solutions of a related matrix factorization problem (up to an [assumed] full-rank matrix multiplication). The optimal solution of this matrix factorization problem is computed and the (possibly) low-rank structure is characterized.\n\nThe results are then generalized (Section 4.3) to the setting where the variance of the approximate posterior is learned (and is assumed to be data-independent, which is optimal in the linear VAE setting [Lucas et al. 2019]). The authors interpret their results in terms of the posterior collapse phenomena that occurs in deep VAEs and empirically investigate their findings on synthetic data and the MNIST dataset.\n\n**Post-rebuttal update:** The authors have made significant changes to their paper following the reviews. With these, they address the vast majority of the issues that I presented below. They also pointed out that my claims on $\\beta$ being unnecessary in the linear VAE setting are not technically correct. Following this, I have updated my score to recommend that the paper be accepted. I hope that the authors will continue to improve the presentation of the work.", "questions": "I feel that the following is too strong a statement: \"Up to now, there has not been any precise identification of neither the nature nor the cause of the posterior collapse problem.\" See discussion above.\n\n$A$ is positive semi-definite but it is then assumed that $A^{1/2}$ is invertible. This is equivalent to assuming that all eigenvalues of the data covariance matrix are strictly positive, and it is stated in footnote that it is straightforward to extend to the low-rank case. But this is not evident to me from the proofs. For such a claim, ideally, the proofs in the supplementary material would apply to the low-rank case. You could keep the assumption in the main paper if it heavily simplifies notation etc., but I do not quite see how.\n\nWhy does notation appear at the end of the problem setting section? There is some duplication here, as much of it has already been defined above.\n\nThere is some confusing notation in the statement of Proposition 1. It swaps between $U$ and $W$ with $U^*$ doubly defined as the optimum of both objectives implicitly. Notation is also swapped in the proof. Further, the proof of Proposition 1 claims that these minimization problems are identical. It is have shown that,\n$L_{\\mathrm{VAE}}(U,W) = L(U, A^{1/2}W)$, which is not quite the claim (particularly in the case where $A$ is not full-rank, then each minimizer of the former may correspond to a manifold of solutions for the latter). The proposition statement compares *the* minimizers, but note that both objectives are non-convex and a unique minimizer has not yet been established. Perhaps a more accurate statement is that any stationary point has these dual forms (under a full-rank $A$).\n\n#### Proof of proposition 2\n\nThe proof begins by stating conditions on the global minimum; but this is a non-convex optimization problem and so this condition guarantees only a stationary point. As written, this is not incorrect but is a little confusing. The conditions are established for a stationary point and then solved for the global minimum by finding the stationary parameters with the smallest loss.\n\nI encourage the authors to state the SVD of Z within the proof again (otherwise the definition of F can be a source of temporary confusion).\n\nPlease state the trace inequality that you are using exactly (trace of matrix product upper bounded by sum of product of eigenvalues). It is also worth including the derivation of equation 45 --- the proof is difficult to follow without this. Similarly for equation 46 (how did you verify that only the larger solution is related to the minimum?).\n\n#### Proof of proposition 3\n\nIt is written that the gradient is an increasing function and so the minimum value is determined when the gradient is zero. You also need to add the condition that $\\ell'(\\sigma) < 0$ for some $\\sigma$ which is then given in closed-form.\n\n\nMinor comments:\n\nL23: in what sense is this more general? Due to `x` and `y`? Or due to the stochastic reparameterization trick used in VAEs? There is also no marginalization over `y` and it is unclear what `x` and `y` represent here. (This is corrected later, in Section 3).\n\nL34: \"in the deep learning context\" -> suggests you are doing this. But it is a stretch to claim this when studying linear models only.\n\nL44-45: a strong claim given referenced work. May need further justification.\n\nL245: your posterior collapse condition also depends on the decoder variance. You also write above that the decoder variance can also be controlled to alleviate posterior collapse. And finally, you do not analyze learning the decoder variance but if you included the partition function in the objective I expect that the same analysis of Lucas et al. could be repeated in this setting to the same general effect.\n\nL284: Similar to L245, this is only true when excluding the partition function from the objective (and restricting what we mean by \"the effect\").\n\nL324: \"to [what] extent the result carries\"\n\nL326-327: \"Moreover, can we design a Bayesian-principled method for avoiding posterior collapse that does not involve hand-tuning β or ηdec?\" This is exactly one of the contributions of Lucas et al., in the case of the linear VAE.", "limitations": "Limitations are discussed adequately within the paper. However, it is my feeling that the framing of the paper relative to the related work is inaccurate and the discussion of limitations should be updated therein.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe theoretical results are well-connected and the proofs are (mostly) well-written and easy to follow. This is also true of the main paper itself (bar a few minor notational issues).\n\nThe authors analyze (a variant of) the beta-VAE objective which is widely used in practice.\n\n## Weaknesses\n\nThe objective presented in Equations (2)-(5) is incorrect and does not include the log partition function from the decoder observation model. There is an argument to be made that this matches practical implementations or perhaps that this term can be ignored when the decoder variance is not learned. But as I will discuss further, this is a problematic assumption that has been carefully studied for this setting in prior work.\n\nA limitation of the analysis, relative to Lucas et al., is that only the form of the global optimum is analyzed under this objective and none of the other stationary points are characterized. Moreover, beta plays the role of controlling the rank of the global optimum but in the linear VAE with learned decoder variance there is no need for a beta parameter as all low-rank stationary points become saddle points. I suspect that all other stationary points here are also saddles, but many of the higher rank solutions have larger loss until beta is reduced. Further to this point, the authors write that \"one alternative way to fix posterior collapse[s] that have not been suggested in the field is to use a sufficiently small [decoder variance]\". But this is exactly the suggestion of Lucas et al. (!) where the decoder variance is treated both as a learned or fixed parameter.\n\nIt is also not surprising that the choice of the prior variance scale has no effect on posterior collapse. In the models considered (and in deep VAEs), this variance can be trivially accounted for by scaling the encoder and decoder weights up and down respectively. This can be understood easily via appendix B of Lucas et al., where such a transformation is shown under which the likelihood term is invariant --- this (full-rank) transformation can be chosen to adjust for any choice of prior variance.\n\nAnd finally, the point could be made that beta VAEs are a common method used in practice and certainly they are worthy of study in their own right. Though, note that Lucas et al. discuss beta VAEs in Chapter 5 and draw a clear connection between the role of beta and of the observation noise in the general ELBO objective with Gaussian likelihood. In particular, that when treating both as fixed hyperparameters the exact same gradient update can be computed in either case.\n\nThe remaining contribution of this work that is not supported by prior work is that the authors consider more general models than VAEs (with different output dimensions). However, I do not consider this contribution to be significant enough to overcome the negative points discussed elsewhere in my review.\n\nTo my knowledge, the authors have provided novel theoretical results. They generalize past results in the sense that they account for a KL-annealing term and a different output dimensionality. But they are weaker in the ways discussed above. Considered in balance, I feel that these contributions are not significant enough to warrant acceptance.\n\nFinally, the empirical results may not be a primary contribution of this work but they are markedly weaker than similar publications at previous venues. The authors train linear VAEs on MNIST and a synthetic dataset. And two-layer fully-connected encoder/decoder networks are also trained on MNIST. [Note that the variance is not global as in the theoretical analysis but is instead data-dependent, but I agree with the authors that this is a reasonable choice.] For comparison, Lucas et al. 2019 train fully-connected VAEs on MNIST, and convolutioanl VAEs on CelebA. Their empirical evaluation included different objectives, varying encoder architectures, KL annealing schedules, fixed/learned decoder variance, and more. Dai et al. 2020 train on MNIST, Fashion-MNIST, SVHN, CIFAR10, CIFAR100, and CelebA. They used fully-connected, convolutional, and residual architectures and evaluated different objectives and model capacity. Shekhovtsov et al. 2022 (see below) have a similarly thorough empirical evaluation. In summary, while I see no significant issue with the empirical evaluation in this paper I also am unable to recommend it as a strength of the work relative to other papers investigating the same issue.\n\n### Missing discussion of related work\n\nThe connection between linear VAEs and matrix factorization are already well-known [1,2,3]. In these references, non-amortized variational inference is used but many of the findings overlap with that of this paper. A suitable discussion and comparison must be included.\n\nSome other work investigating posterior collapse has also been missed in discussion [4,5].\n\n[1] S. Nakajima and M. Sugiyama. Implicit regularization in variational bayesian matrix factorization. ICML, 2010.\n\n[2] S. Nakajima, M. Sugiyama, S. D. Babacan, and R. Tomioka. Global analytic solution of fully observed variational bayesian matrix factorization. Journal of Machine Learning Research, 2013.\n\n[3] S. Nakajima, R. Tomioka, M. Sugiyama, and S. D. Babacan. Condition for perfect dimensionality recovery by variational bayesian PCA. Journal of Machine Learning Research, 2015. \n\n[4] A. Shekhovtsov, D. Schlesinger, and B. Flach. VAE approximation error: ELBO and exponential families. ICLR, 2022.\n\n[5] J.Lücke, D. Forster, and Z. Dai. The evidence lower bound of variational autoencoder converges to a sum of three entropies. ICLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gPMHtWAwjWB", "reviewer": "Reviewer_pzUk", "summary": "The paper provides a theoretical analysis of (a simplified) linear VAE in order to improve our understanding of the posterior collapse often observed with similar nonlinear Bayesian models. The main finding is that these models naturally perform a thresholding of singular values, which is at the root of the issue. This can then be linked to the prior on the mean. ", "questions": "* Line 217 (a): it is argued that a learnable variance does not change the posterior collapse. This is under the assumption that the learnable variance is data independent. Do the comments still hold true if the variance is data dependent (as is the common case) ?\n* Line 262-275: It is observed that an optimum may exist where the parameters are equal to zero. Is it correctly understood that this is due to the data being zero mean, i.e. that the mode is for the model to predict the data mean?", "limitations": "The most fundamental limitation of the paper is that it studies a linear model in the hope that this may tell us something about nonlinear models. Here we can only hope. The paper is quite clear on this limitation.\n\nA second limitation is that all noise variances are assumed to be data independent, which is quite far from common practice (esp. for the encoder) and is at odds with some of the driving ideas of VAEs. I think the paper could be more clear about this limitation.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The paper is fairly easy to read, and the authors do a nice job of summarizing their findings along the way to keep the reader engaged.\n* The main findings are interesting, and inspirational.\n* The empirical results on nonlinear models, while not the focus of the paper, provides evidence in support of the findings from linear models may generalize.\n\n### Weakness\n* The paper treats both encoder and decoder as data independent, which is quite far from the usual VAE setting. I understand that using linear models to predict uncertainties would be rather useless, but I still think it would be good if the paper acknowledged the limitation.\n* I really miss a greater discussion of the link to (P)PCA in the paper. If I understand correctly, it is these links that are examined in the prior work by Lucas et al., but given that a linear VAE is basically a variant of PCA, I found it rather confusing that these links are not discussed in the paper.\n* In line 179, it is stated that one solution to the posterior collapse problem is to pick the variance of the likelihood to be small. First, this doesn't really work for non-Gaussian likelihoods (which are the most common), and second it seems like a poor solution to effectively have a delta-likelihood in a probabilistic model (then there is little point in being probabilistic in the first place).\n\n### Minor comments\n* Line 105 says that A is full rank, while in 111 it is semidefinite. That's confusing.\n* In line 136, it is stated that Sigma is learnable, yet earlier it was not learnable. Again, confusing.\n* In lines 299-300 it would be good to use the proper notation $10^{-3}$ rather than 1e-3.\n* Late in the paper it is stated that the introduction of beta (as in the beta-VAE) makes little sense from a probabilistic perspective. I think this should be stated when beta is initially introduced.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PKUhjol0X03", "reviewer": "Reviewer_gw2z", "summary": "The work identifies the cause and the precise condition of posterior collapse of a linear latent variable model, and proposed methods to alleviate the problem by making the latent variance learnable.", "questions": "Some minor problems with the writing and derivations, for which the reviewer hopes the author(s) could clarify or fix:\n\nTypos:\n1. Line 117, $\\Psi$ should be $\\Sigma_Z$.\n2. Line 140 and Eq (7), $U=F\\Lambda P$, but in Line 466 of appendix, $U=Q\\Lambda P$. Also F has been used by SVD of Z in Line 116 and in proofs. Please fix U's definition here to make things consistent. \n3. Line 151 and 152, $\\sigma_i$ should be $\\zeta_i$, as introduced in Line 118.\n4. Line 200, it should be $\\mathrm{min}(d_0, d_2)$ instead of $\\mathrm{min}(d_1, d_2)$\n\nIn Appendix:\n\n 5. Line 444, the mean should be $U z + b_d$.\n 7. Eq 24, should be $\\frac{\\partial L_{VAE}}{\\partial b_d}$.\n 6. Eq 36, should be $VU^TU - Z^TU + \\cdots, sign is wrong for the second term.\n 7. Line 469, $\\Sigma$ should be $\\Sigma_Z$. Line 470, $\\sigma_i$ should be $\\zeta_i$ as of the singular value $\\Sigma_Z$.   \n 8. Eq 52, $\\zeta^2_i$ should be $\\zeta_i$.\n\nDerivations:\n\n 9. For proof of Corollary 1, Appendix Eq (49) is correct. But going from App. Eq(49) to Eq(11) is unclear to the reviewer.\n 10. The proof of Proposition 4, in appendix. To reach Eq (23), it already used $b^*_{d}=E_x[y]$ The correct proof should be put together two equations for $\\frac{\\partial L_{VAE}}{\\partial b_d}=0$ and $\\frac{\\partial L_{VAE}}{\\partial b_e}=0$, two equations together to reach the $b^*_{d,e}$ solutions.\n 11. For Eq (44), the author(s) should add the condition, when the equality holds for von Neumann's Trace Inequalities. (btw, it's \"von Neumann, \"v\" without capitalisation) \n\nSome of questions to help the reviewer understand better:\n\n1. To the understanding of the reviewer, he believes that there is a physical interpretation of the term $\\frac{\\sqrt{\\beta}\\sigma_i\\eta_{dec}}{\\eta_{enc}}$. Inspired by the author(s) to explain \"$\\zeta_i$ as the strength of the alignment between the input $x$ and the target $y$\" (Line 152), the term could be explained as the relative variance of the alignment between the model prediction. Then the thresholding in Eq(8) is to encourage U and V to be low-ranking by discarding the modes that the model variance scale $\\frac{\\sqrt{\\beta}\\sigma_i\\eta_{dec}}{\\eta_{enc}}$ is higher than the data signal $\\zeta_i$. Is the understanding correct?   \n\n2. In Line 169, \"we have a complete posterior collapse: both U and V are identically zero\". It seems suffice to have posterior collapse with V (henceforth W as in line 97) being zero. Is there any significance for U being identically zero? \n\n3. In line 176, \"one alternative way to fix posterior collapses ... is to use a sufficiently small $\\eta_{dec}$\". Does it mean a more deterministic decoder is less likely to have posterior collapse? If so, does it generalise to nonlinear models? ", "limitations": "The author(s) discussed some limitations and future work plans in 6. Outlook.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " * Originality: The theoretical analysis and the direct pinpointing of the cause of posterior collapse problem for linear latent variable models are novel and insightful.\n * Quality: The derivation and theoretical analysis are sound. \n * Clarity: The paper was written in a clear and concise way, with detailed derivation to prove the propositions/corollary/theorems in the appendix. Some minor fixes are required, for which the reviewer listed in the Questions section.\n * Significance: Better understanding of the posterior collapse is important not only for theoretical understanding of deep learning, but also for designing better principles to build models and training strategies to improve training stabilities. This work is important on this front.\nOverall, the paper is theoretically sound, well written and carefully explained. The reviewer enjoys the reading. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept