The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
Authors: Bin Dai
Ziyu Wang
David Wipf
Venue:  (2020)
Keywords: variational autoencoder
posterior collapse

Abstract: In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions.  Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice.  However, contrary to conventional wisdom that largely assigns blame for this phenomena on the undue influence of KL-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks.  In particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances.  Regardless, the underlying message here is not meant to undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.

Under review as a conference paper at ICLR 2020
THE USUAL SUSPECTS? REASSESSING BLAME FOR VAE POSTERIOR COLLAPSE
Anonymous authors Paper under double-blind review
ABSTRACT
In narrow asymptotic settings Gaussian VAE models of continuous data have been shown possess global optima aligned with ground-truth distributions. Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice. However, contrary conventional wisdom that largely assigns blame for this phenomena on the undue influence of Kl.-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks. particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances. Regardless, the underlying message here is not meant undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.
INTRODUCTION
The variational autoencoder (VAE) (Kingma & Welling 2014 Rezende et al. 2014) represents a
powerful generative model of data points that are assumed to possess some complex yet unknown latent structure. This assumption is instantiated via the marginalized distribution
Po (z) (=))))))(=)dd
which forms the basis of prevailing VAE models. Here E R" is collection of unobservable latent factors of variation that, when drawn from the prior p(z), are colloquially said to generate an observed data point æ Rdd through the conditional distribution po(x]z). The latter is controlled by parameters 8 that can, at least conceptually speaking, be optimized by maximum likelihood over (=( given available training examples.
In particular, assuming n training points X [2x(1), ,.... (.). maximum likelihood estimation is tantamount to minimizing the negative log-like likelihood expression Σ. log [pe (zz()) Proceed- ing further, because the marginalization over æ in (T) is often intractable, the VÁE instead minimizes a convenient variational upper bound given by L(0 &
(0( [iogPo Pa
( [z)
+ KL (z) (p(x))
log po (2)(1)
with equality iff { (x)) = po (=(2220)) for all i. The additional parameters & govern the shape of the variational distribution (2 x)) that is designed to approximate the true but often intractable
latent posterior (=)x).)
The VAE energy from composed of two terms, data-fitting loss that borrows the basic struc- ture of an autoencoder (AE), and KL-diiergence--aaee baased regularization factor. The former incen- tivizes assigning high probability to latent codes z that facilitate accurate reconstructions of each (+)) In fact, if qs (x) is Dirac delta function, this term is exactly equivalent to a deterministic AE with data reconstruction loss defined by log Po (x z)). Overall, it is because of this association that qo (2(az) is generally referred to the encoder distribution, while pe (ær z) denotes the decoder

---

Under review as a conference paper at ICLR 2020
distribution. Additionally, the KL regularizer KIL (g)(()/)))l(((() pushes the encoder distribution towards the prior without violating the variational bound.
For continuous data, which will be our primary focus herein, it typical to assume that
p(z) - - Pa (a|z) Ma ( and qo (z|x) (= Ez),
where > 0 is a scalar variance parameter, while the Gaussian moments μr, = Hs (2: 0). M4 1: (x; Φ), and Σ; = diag a= (æ: o)]² are computed via feedforward neural network layers. The encoder network parameterized by Φ takes æ as an input and outputs μ4 and Σ, Similarly the decoder network parameterized by converts latent code z into pzz. Given these assumptions, the generic VAE objective from (2)
can be refined to
[(0, 0) = :/ +5
E do (2)20())
[111 - Mr (2;
(4)
+ d doloog + o= (z(1) :0) log diag [o.
ao (z))) [))] (x))
excluding an inconsequential factor of 1/2. This expression can be optimized over using SGD and a simple reparameterization strategy (Kingma & Welling, 2014 Rezende et al., 2014) to produce parameter estimates {0", ). Among other things, new samples approximating the training data [2 0, I) and new ~ Pa- (x)==>00).
can then be generated via the ancestral process new N
Although it has been argued that global minima of may correspond with the optimal recovery of ground truth distributions in certain asymptotic settings (Dai & Wipf 2019). it is well known that in practice, VAE models are at risk of converging to degenerate solutions where, for example, it may be that 9s (z|z) p(z). This phenomena, commonly referred to as VAE posterior collapse (He al. 2019 Razavi et al. 2019), has been acknowledged and analyzed from variety of dif- ferent perspectives as we detail in Section 2 That being said, we would argue that there remains lingering ambiguity regarding the different types and respective causes of posterior collapse. Con- sequently, Section provides a useful taxonomy that will serve to contextualize our main technical contributions. These include the following:
Building upon existing analysis of affine VAE decoder models, in Section 4 arbitrarily small nonlinear activations can introduce suboptimal local minima exhibiting posterior collapse.
we prove that
We demonstrate in Section 5 that if the encoder/decoder networks are incapable of sufficiently reducing the VAE reconstruction errors, even in a deterministic setting with no KL-divergence regularizer, there will exist an implicit lower bound on the optimal value of y. Moreover, we prove that if this 7 is sufficiently large, the VAE will behave like an aggressive thresholding operator, enforcing exact posterior collapse, i.c., de (z (x) = p(z).
Based on these observations, depth/capacity is increased, even for deterministic AE models with no regularization, reconstruc- tion errors become worse. This bounds the effective VAE trade-off parameter 7 such that posterior collapse is essentially inevitable. Collectively then, we provide convincing evidence that poste- rior collapse is, at least certain settings, the fault of deep AE local minima, and need not be exclusively consequence of usual suspects such as the KL-divergence term.
present experiments in Section 6 esttabiishnng that network
We conclude in Section 7 with practical take-home messages, and motivate the search for improved AE architectures and training regimes that might be leveraged by analogous VAE models.
RECENT WORK AND THE USUAL SUSPECTS FOR INSTIGATING COLLAPSE
Posterior collapse under various guises is one of the most frequently addressed topics related to VAE performance. Depending on the context, arguably the most common and seemingly trans- parent suspect for causing collapse is the KL regularization factor that is obviously minimized by (x(x)))) p(z). This perception has inspired various countermeasures. including heuristic anneal- ing of the KL penalty or KL warm-start (Bowman et al. 2016), tighter bounds the likeliilldod (Burda et al.
2015 Huang et al.) 2018 Sønderby et al.]
2015 Rezende & Mohamed 2015), more
2

---

Under review as a conference paper at ICLR 2020
complex priors (Buuer& & Mnih 2018 Tomezak & Welling 2018),
modified decoder architectures
(Cai et al. 2017 Dieng et al., 2018; Yeung et al. (00 or efforts to explicitly disallow the prior from ever equaling the variational distribution (Razavi al. 2019). Thus far though, most published results do not indicate success generating high-resolution images, and in the majority of cases, eval- uations are limited to small images and/or relatively shallow networks. This suggests that there may be more nuance involved in pinpointing the causes and potential remedies of posterior collapse. One notable exception though is the BIVA model from (Maaløe et al., (201)), which employs bidirec- tional hierarchy of latent variables, in part to combat posterior collapse. While improvements in NLL scores have been demonstrated with BIVA using relatively deep encoder/decoders, this model is significantly more complex and difficult to analyze.
On the analysis side, there have been various efforts to explicitly characterize posterior collapse in restricted settings. For example, [uuaas et al. (2019) demonstrate that if is fixed to sufficiently large value, then a VAE energy function with an affine decoder mean will have minima that over- prune latent dimensions. A related lincarized approximation to the VAE objective is analyzed in (Rolinek et al. 2019); however, collapsed latent dimensions are excluded and remains somewhat unclear how the surrogate objective relates to the original. Posterior collapse has also been associ- ated with data-dependent decoder covariance networks (2; 0) + ?I (Mattei & Frellsen 2018), which allows for degenerate solutions assigning infinite density single data point and diffuse, collapsed density everywhere else. Finally, from the perspective training dynamics, (He et al. 2019) argue that a lagging inference network can also lead to posterior collapse.
3 TAXONOMY OF POSTERIOR COLLAPSE
Although there is now vast literature the various potential causes of posterior collapse, there remains ambiguity as to exactly what this phenomena is referring to. this regard, we believe that it is critical differentiate five subtle yet quite distinct scenarios that could reasonably fall under the generic rubric of posterior collapse:
(i) Latent dimensions of that are not needed for providing good reconstructions the training data are set to the prior, meaning (z) p(zz) = ~N 1) at any superfluous dimension Along other dimensions o will be near zero and μı= will provide usable predictive signal leading to accurate reconstructions of the training data. This case can actually be viewed as desirable form of selective posterior collapse that, argued in (Dai & Wipf 2019). is necessary (albeit not sufficient) condition for generating good samples.
(ii)
The decoder variance y is not learned but fixed to a large value" such that the KL term from 2 is overly dominant, forcing most or all dimensions of to follow the prior N(,0 1). In this scenario, the actual global optimum of the VAE energy (conditioned on being fixed) will lead to deleterious posterior collapse and the model reconstructions of the training data will be poor. In fact, even the original marginal log- ligliikl can potentially default to trivial/useless solution if y is fixed too large, assigning a small marginal likelihood to the training data, provably so in the affine case (Lucas et al. 2019).
(iii) As mentioned previously, if the Gaussian decoder covariance is learned as a separate network structure (instead of simply (z; 0) = >/), there can exist degenerate solutions that assign infinite density to single data point and a diffuse, isotropic Gaussian elsewhere (Mattei & Frellsen 2018). This implies that (4) can be unbounded from below at what amounts to a posterior collapsed solution and bad reconstructions almost everywhere.
(iv) When powerful non- -on-Gaussian decoders are used, and in particular those that can parameter- ize complex distributions regardless of the value of x (c.g., PixcICNN-basce (Van den Oord ct al. 2016)), it is possible for the VAE to assign high probabbbbyyilityy to the training data even ( p(z) (Alemi et al. 2017 Bowman al. 2015 Chen al. 2016). This category of posterior collapse is quite distinct from categories (ii) and (ii) above in that, although the reconstructions are similarly poor, the associated NLL scores can still be good.
(v) The previous four categories of posterior collapse can all be directly associated with emergent properties of the VAE global minimum under various modeling conditions. In contrast, a fifth type of collapse exists that the explicit progeny of bad VAE local minima. More
Orr equivalently, KL scaling parameter such as used by the 3-VAE Higgins al. 2017 is large.
3

---

Under review as a conference paper at ICLR 2020
specifically, will argue shortly, when deeper encoder/decoder networks used, the risk of converging to bad, overregularized solutions increases.
The remainder of this paper will primarily focus on category (v), with brief mention of the other types for comparison purposes where appropriate. Our rationale for this selection bias is that, un- like the others, category (i) collapse is actually advantageous and hence need not be mitigated. In contrast, while category (ii) is undesirable, it be can be avoided by learning 7. As for category (iii), this represents an unavoidable consequence of models with flexible decoder covariances capable of detecting outliers (Dai et al. 2019). In fact, even simpler inlier/outlier decomposition models such as robust PCA are inevitably risk for this phenomena (Candès et al., 201T). Regardless, when (x; 8) II this problem goes away. And finally, we do not address category (iv) in depth sim- ply because is unrelated the canonical Gaussian VAE models of continuous data that we have chosen to examine herein. Regardless, it is still worthwhile to explicitly differentiate these five types and bare them in mind when considering attempts to both explain and improve VAE models.
INSIGHTS FROM SIMPLIFIED CASES
Because different categories of posterior collapse can be impacted by different global/local minima structures, a useful starting point is restricted setting whereby we can comprehensively characterize all such minima. For this purpose, we first consider VAE model with the decoder network set to an affine function. As is often assumed in practice, we choose Σ, 7I, where y > 0 is a scalar parameter within the parameter set In contrast, for the mean function we choose μ = W , b for some weight matrix Wa and bias vector 6 The encoder can be arbitrarily complex (although the optimal structure can be shown to be affine as well).
Given these simplifications, and assuming the training data has r > K. nonzero singular values, it has been demonstrated that at any global optima, the columns of W = will correspond with the first principal components of X provided that we simultancously learn
set it to the optimal value (which is available in closed form) (Dai et al. 2019 Lucas et al. 2019 Tipping & Bishop 1999). Additionally, it has also be shown that no spurious, suboptimal local minima will exist. Note also that if r < x the same basic conclusions still apply: however, W. & will only have r nonzero columns, each corresponding with different principal component of the data. The unused latent dimensions will satisfy 4 (=|æ) = NO0, I). which represents the canonical form of the benign category (i) posterior collapse. Collectively, these results imply that if we converge to any local minima of the VAE energy, we will obtain the best possible linear approximation to the data using a minimal number of latent dimensions, and malignant posterior collapse is not an issue, i.e., categories (ii)(vv will not arise.
Even so, if instead of learning 7, we choose a fixed value that is larger than any of the significant singular values of X X then category (ii) posterior collapse can be inadvertently introduced. More specifically, let F, denote the number of such singular values that are smaller than some fixed value. Then along
F.y latent dimensions (z(zx) = (O, I), and the corresponding columns of W. will be set to zero at the global optima (conditioned on this fixed y), regardless of whether or not these dimensions are necessary for accurately reconstructing the data. And has been argued that the risk of this type of posterior collapse at conditionally-optime global minimum will likely be inherited by deeper models as well (Lucas et al. 2019), although learning 7 can ameliorate this problem.
Of course when we move to more complex architectures, the risk of bad local minima or other suboptimal stationary points becomes a new potential concern, and it is not clear that the affine case described above contributes to reliable, predictive intuitions. To illustrate this point, we will now demonstrate that the introduction of an arbitrarily small nonlinearity can nonetheless produce pernicious local minimum that exhibits category (v) posterior collapse. For this purpose, we assume the decoder mean function
(5) The function Ta is nothing more than a soft- coft-threshold operator as is commonly used in neural net- work architectures designed to reflect unfolded iterative algorithms for representation learning (Gre- ggor & LeCun| 2010; SSprechmann et al.) 2013). In the present context though, we choose this non- linearity largely because it allows (5) to reflect arbitrarily small perturbations away from strictly
M = T. (Wan) 222 + by, with To (u) A sign(u) (lul - -a) a
4

---

Under review as a conference paper at ICLR 2020
affine model, and indeed if a 0 the exact affine model is recovered. Collectively, these specifica-
(8) tions lead to the parameterization 0 = (Wz: z: bz: 7) and Φ = {u a Ji=1 and energy (excluding irrelevant scale factors and constants) given by
| 10 - no (W=z) -bo0 2
()),6) =
(6)
+dlog y + ||000 log
diag o (2)
2
+
(t)
where ftz and o² denote arbitrary encoder moments for data point (this is consistent with the assumption of an arbitrarily complex encoder used in previous analysis of affine decoder models).
Now define 7 nd Σ, ||x!() - =112. with æ A E, We then have the following result:
Proposition 4.1 For any a > O, there will always exist data sets X such that O6 has a global minimumn that perfectly reconstructs the training data, but also a bad local minimum characterized by
(=(=)=) = (o(2)))=N(((0,] and po pa( (a 71).
Hence the moment we allow for nonlinear (or precisely, non-affine) decoders there exist a poor local minimum, across all parameters including a learnable 7, that exhibits category (v) posterior collapse
2 In other words, no predictive information about passes through the latent space, and woeeless/nonninfommattie distribution (x emerges that is incapable of assigning high probability the data (except obviously in the trivial degenerate case where all the data points are equal to the empirical mean ã). We will next investigate the degree to which such concerns can influence behavior in arbitrarily deep architectures.
EXTRAPOLATING TO PRACTICAL DEEP ARCHITECTURES
Previously we have demonstrated the possibility of local minima aligned with category (v) posterior collapse the moment we allow for decoders that deviate ever so slightly from an affine model. But nuanced counterexamples designed for proving technical results notwithstanding. it reasonable to examine what realistic factors are largely responsible for leading optimization trajectories towards such potential bad local solutions. For example, is it merely the strength of the KL regularization term, and if so, why can we not use KL warm-start to navigate around such points? In this section we will elucidate a deceptively simple, alternative risk factor that will be corroborated empirically in Section
From the outset, we should mention that with deep encoder/decoder architectures commonly used in practice, stationary point can more-or-less always exist at solutions exhibiting posterior collapse. As a representative and ubiquitous example, please see Appendix [A.4] But of course without further details, this type of stationary point could conceivably manifest as saddle point (stable or unstable), a local maximum, or a local minimum. For the strictly affine decoder model mentioned in Section there will only be harmless unstable saddle point at any collapsed solution (the Hessian has negative eigenvalues). In contrast, for the special nonlinear case elucidated via Proposition 4. we can instead have bad local minima. We will now argue that as the depth of common feedforward architectures increases, the risk of converging to category (v)-like solutions with most or all latent dimensions stuck at bad stationary points can also increase.
Somewhat orthogonal to existing explanations of posterior collapse, our basis for this argument is not directly related to the VAE KL-divergence term. Instead, we consider a deceptively simple yet potentially influential alternative: Unregularized, deterministic AE models can have bad local solutions with high reconstruction errors when sufficiently deep. This in turn can directly translate category (v) posterior collapse when training corresponding VAE model with matching deep architecture. Moreover, to the extent that this is true, KL warm-start or related countermeasures
TThis result mirrors related efforts examining linear DNNs, where has been previously demonstrated that under certain conditions, all local minima are globally optimal (Kawaguchi (016). while small nonlinearities induce bad local optima (Yun et al. 2019). However, the loss surface of these models is completely different from VAF, and hence we view Proposition 4. as complementary result.
5

---

Under review as a conference paper at ICLR 2020
will likely be ineffective in avoiding such suboptimal minima. We will next examine these claims in greater depth followed by a discussion of practical implications.
5.1 FROM DEEPER ARCHITECTURES TO INEVITABLE POSTERIOR COLLAPSE
Consider the deterministic AE model formed by composing the encoder mean M = My (::0) and decoder mean p4 puted via á = 2
(::) networks from VAE model, i.c., reconstructions 2 are com-
xx: ;): We then train this AE to minimize the squared-error loss E= x a (0)
producing parameters {faes Due Analogously, the corresponding VAE trained to minimize arrives parameter set denoted {Oroc: Dvae). In this scenario, it will typically follow that
Mx M= (z.)); Quc) bac) 0.e
11
n
[|ax10 M4 (z; Deac) [13]
Adune (2)
nd
nd
(8) meaning that the deterministic AE reconstruction error will generally be smaller than the stochastic VAE version. Note that if o + 0, the VAE defaults to the same deterministic encoder as the AE and hence will have identical representational capacity; however, the KL regularization prevents this from happening, and any a² > O can only make the reconstructions worse Likewise, the KL penalty factor | l2 can further restrict the effective capacity and increase the reconstruction error of the training data. Beyond these intuitive arguments, we have never empirically found case where (8) does not hold (see Section 6 for examples).
We next define the set
S₂
0,0
(1)
SE
for any 0. Now suppose that the chosen encoder/decoder architecture is such that with high probability, achievable optimization trajectories (c.g.. via SGD or related) lead to parameters {lue. ) & Sg, i..., Prob (f0ze, ) E S&)) se O. It then follows that the optimal VAE noise vari- ance denoted y", when conditioned on practically-achievahh values for other network parameters, will satisfy
E Nowan (2)00()) [|I=(0 11, (z; Ovac) 113
2 E.
(10)
2=1
The equality in (LOp can be confirmed by simply differentiating the VAE cost w.r.t. Y and equating to zero, while the inequality comes from (8) and the fact that (faes dae] $ S..
From inspection of the VAE energy from 44. it is readily apparent that larger values of will discount the fata-fitting term and therefore place greater emphasis on the KL divergence. Since the latter is minimized when the latent posterior equals the prior, we might expect that whenever e and therefore 7" increased per (TO), we are at a greater risk of nearing collapsed solutions. But the nature of this approach is not at all transparent, and yet this subtlety has important implications for understanding the VAE loss surface in regions at risk of posterior collapse.
For example, one plausible hypothesis is that only as 7" - x do we risk full category (v) collapse. If this were the case, we might have less cause for alarm since the reconstruction error and by association " will typically be bounded from above at any local minimizer. However, will now demonstrate that even finite values can exactly collapse the posterior. In formally showing this, is helpful introduce slightly narrower but nonetheless representative class of VAE models.
E ( [||==(0) m₂ (z; 0) 112). i.e., the VAE data term evaluated at single data point without the 1/y scale factor. We then define well- behaved VAE as model with energy function [4) designed such that Vn. (2,,0,,, x()) and J..J (M,, a 2, 0 xz ()) are Lipschitz continuous gradients for all i. Furthermore, specify a degenerate decoder as any (z; 8) with set to 0 value such that Vo, (M. 0, a (1))
Specifically, let ff 2,,
Except potentially in certain contrived adversarial conditions that do represent practical regimes.

---

Under review as a conference paper at ICLR 2020
for some constant >> 0 that can be arbitrarily small. This ensures that is an increasing fune- tion of a, a quite natural stipulation given that increasing the encoder variance will generally only serve to corrupt the reconstruction, unless of course the decoder is completely blocking the signal from the encoder. In the latter degenerate situation, it would follow that Vuzf (1z, 2. 0, x x(1)=
Vo.f (#z, s, 8 xx O, which more-or-less tantamount to category (v) posterior collapse. Based on these definitions, we can now present the following:
Proposition 5.1 For any well-behaved VAE with arbitrary, non-degenerate decoder p. 0 f). there will always exist a of < 0o such that the trivial solution 1h 0 + 0) and (=(2))) p(z) will have lower cost.
Around any evaluation point, the sufficient condition we applied to demonstrate posterior collapse (see proof details) also be achieved with some >" 7 if we allow for partial collapse, i.e., qo- (z)(2z) [x) = p(zy) along some but not all latent dimensions j E {1....,kk] K}. Overall, the analysis loosely suggests that the number of dimensions vulnerable to exact collapse will increase monoton- ically with 7.
Proposition 5. also provides evidence that the VAE behaves like strict thresholding operator, com- pletely shutting off latent dimensions using a finite value for y. This is analogous to the distinction between using the L versus Eg norm for solving regularized regression problems of the standard form minz | - Aul 2 + 7 muu), where A is a design matrix and is a penalty function. When the & norm, some or all elements of can be pruned to exactly zero with sufficiently large but finite Zhao & Yu (2006). In contrast, when the &g norm is applied, the coefficients will be shrunk to smaller values but never pushed all the way to zero unless 7 + xc.
5.2 PRACTICAL IMPLICATIONS
In aggregate then, if the AE base model displays unavoidably high reconstruction errors, this implic- itly constrains the corresponding VAE model to have a large optimal value, which can potentially lead to undesirable posterior collapse per Proposition 5. In Section 6 we will demonstrate empiri- cally that training unregularized AE models can become increasingly difficult and prone to bad local minima (or at least bad stable stationary points) as the depth increases; and this difficulty can persist measuress such as skip connections. Therefore, from this vantage point would argue that it is the AE base architecture that is effectively the guilty party when it comes to category (v) posterior collapse.
even with
The perspective described above also helps to explain why heuristics like KL warm-start are not always useful for improving VAE performance. With the standard Gaussian model (4) considered herein, KL warm-start amounts to adopting a pre-defined schedule for incrementally increasing y starting from small initial value, the motivation being that small y will steer optimization trajectories away from overregularized solutions and posterior collapse.
However, regardless of how arbitrarily small y may be fixed at any point during this process, the VAE reconstructions are not likely to be better than the analogous deterministic AE (which is roughly equivalent to forcing y = 0 within the present context). This implies that there can exist an implicir > as computed by (T) that can be significantly larger such that, even if KL warm-stan is used, the optimization trajectory may well lead to a collapsed posterior stationary point that has this 7* as the optimal value in terms of minimizing the VAE cost with other parameters fixed. Note that if full posterior collapse does occur, the gradient from the KL term will equal zero and hence to be at stationary point it must be that the data term gradient is also zero. In such situations, varying manually will not impact the gradient balance anyway.
6 EMPIRICAL ASSESSMENTS
In this section we empirically demonstrate the existence of bad AE local minima with high recon- struction errors at increasing depth, as well as the association between these bad minima and immi- nent VAE posterior collapse. For this purpose, with 1, 2, 4, 6, 8 and 10 hidden layers on the Fashion-MNISs dataset (Xiao et al. 2017). Each hidden layer is 51 512-dimensional and followed by ReLU activations (see Appendix A. for further
first train fully connected AE and VAE models
7

---

Under review as a conference paper at ICLR 2020
AE
RuaBleck Spotial Scale
VAE VAE KL ennouling
VAE
Awwmge Momaltad MiC
Ractlldd Spattial Scale
-4" VAE
KL annouling
BlceBleck// Sputiial Scote
Mean
Depth
Layer/ Spatial Scale
Layera/esBlock
Figure 1: Reconstruction errors for various encoder/decoder models of varying complexity. Left: Fully connected networks with different depths trained on Fashion-MNIST. Middle: Convolution networks with increasing depth/# of spatial scales trained on Cifar100, 100. Right: Averaged AE results from residual networks with varying number of residual blocks and block depth trained on SVHN, Cifar10, Cifar and CelebA. In plots, once the encoder/decoder complexity is sufficiently high, the reconstruction errors begin to increase.
details). The reconstruction emror is shown in (left. As the depth of the network increases, the reconstruction error of the AE model first decreases because of the increased capacity. However, when the network becomes local minima (or at least stable stationary point/plateau) that is unrelated to KL-divergence regular- ization. The reconstruction error of a VAE model is always worse than that of the corresponding AE model as expected. Moreover, while KL warm-start/annealing can help to improve the VAE reconstructions to some extent, performance still worse than the AE as expected.
deep, the enror starts to increase, indicating convergence to bad
We next train AE and VAE models using a more complex convolutional network on Cifar data (Krizhevsky & Hinton, 2009). At each spatial scale, we use 1 to convolution layers fol- lowed by ReLU activations. We also apply 2 x 2 max pooling to downsample the feature maps to smaller spatial scale in the encoder and use transposed convolution layer to upscale the feature map in the decoder. The reconstruction errors are shown in Figure mmdddle). Again, the trend is similar to the fully-connected network results. See Appendix A. for an additional ImageNet example.
It has been argued in the past that skip connections can increase the mutual information between ob- servations (*) and the inferred latent variables (Dieng et al. 2018), reducing the risk of posterior collapse. And it is well-known that ResNet architectures based on skip connections can improve performance on numerous recognition tasks (He et al. 2016). To this end, we train number of AE models using ResNet-inspired encoder/decoder architectures on multiple datasets including Cifar 10, C0far100, SVHN and CelebA. Similar to the convolution network structure from above, we use 1, 2, and residual blocks within each spatial scale. Inside each block, we apply to convolution layers. For aggregate comparison purposes, normalize the reconstruction error obtained on each dataset by dividing it with the corresponding error produced by the most shallow network structure (1 residual block with 2 convolution layers). We then average the normalized reconstruction errors over all four datasets. The average normalized errors are shown in [ that adding more convolution layers inside each residual block can increase the reconstruction error when the network is too deep. Moreover, adding more residual blocks can also lead to higher recon- struction errors. And empirical results obtained using different datasets and networks architectures, beyond the conditions of Figure [ also show a general trend of increased reconstruction error once the effective depth is sufficiently deep.
where we observe
We emphasize that in all these models, as the network complexity/depth increases, the simpler mod- els are always contained within the capacity of the larger ones. Therefore, because the reconstruction error on the training data is becoming worse, it must be the case that the AE is becoming stuck at bad local minima or plateaus. Again since the AE reconstruction error serves as a probable lower bound for that of the VAE model, deeper VAE model will likely suffer the same problem, only exacerbated by the KL-divergence term in the form of posterior collapse. This implies that there will be more a values moving closer to 1 as the VAE model becomes deeper; similarly μ. values will push towards O. The corresponding dimensions will encode no information and become completely useless.
8

---

Under review as a conference paper at ICLR 2020
100
2.5 188
55.32
2.5
220
Count
Count
0.5
0
0.2
0.4
6.6
58
12
0.2
0.6
0.6
0.8
04
06
0.
Figure 2: Histogram of o² values VAE encoder/decoder network depth is varied. There are 2, 4 and 5 convolution layers in each spatial scale from left to right. As depth increases, the reconstruc- tion error grows and more o values are near 1, indicative of impending posterior collapse.
To help corroborate this association between bad AE local minima and VAE posterior collapse, we plot histograms of VAE o values as network depth varied in Figure 2 The models are trained on CelebA and the number of convolution layers in each spatial scale is 2, 4 and from left to right. As the depth increases, the reconstruction error becomes larger and there are more o, near
7 DISCUSSION
In this work we have emphasized the previously-undexpppriaaaee role of bad local minima in trap- ping VAE models posterior collapsed solutions. Unlike affine decoder models whereby all local minima are provably global, Proposition 4.I stipulates that even infinitesimal nonlinear perturba- tions can introduce suboptimal local minima characterized by deleterious posterior collapse. Fur- thermore, we have demonstrated that the risk of converging to such a suboptimal minima increases with decoder depth. In particular, outline the following practically-likel pathway posterior collapse:
Deeper AE architectures are essential for modeling high-fidelity images similar, and yet counter-intuitively. increasing AE depth can actually produce larger reconstruction errors on the training data because bad local minima (with or without skip connections). An analogous VAE model with the same architecture will likely produce even worse reconstructions because of the additional KL regularization term, which not designed to steer optimization trajectories away from poor reconstructions.
2. At any such bad local minima, the value of 7 will necessarily be large, i.e., if it is not large, we cannot be at local minimum.
3. But because of the thresholding behavior of the VAE as quantified by Proposition 5.1 be- comes larger there an increased risk exact posterior collapse along excessive latent dimen- sions. And complete collapse along all dimensions will occur for some finite sufficiently large. Furthermore, explicitly forcing to be small does not fix this problem, since in some sense the implicit y" is still large discussed in Section 5.2
While we believe that this message interesting in and of itself, there are nonetheless several practically-relevant implications. For example, complex hierarchical VAEs like BIVA notwithstand- ing, skip connections and KI. warm-start have modest ability to steer optimization trajectories to- wards good solutions; however, this underappreciated limitation will not generally manifest until networks are sufficiently deep as we have considered. Fortunately, any advances or insights gleaned from developing deeper unregularized AEs, e.g., better AE architectures, training procedures, or initializations (Li & Nguyen| 2019). could likely be adapted to reduce the risk of posterior collapse corresponding VAE models.
In closing, we should also mention that, although this work has focused on Gaussian VAE mod- els, many of the insights translate into broader non-Gaussian regimes. For example, a variety of recent VAE enhancements involve replacing the fixed Gaussian latent-space prior pl 2() with pa- rameterized non-Gaussian alternative (Bauer & Mnih 2019 Tomezak & Welling 2018). This type of modification provides greater flexibility in modeling the aggregated posterior in the latent space, which is useful for generating better samples (Makhzani et al. 2016). However, it does not immu- nize VAEs against the bad local minima introduced by deep decoders, and good reconstructions required by models using Gaussian or non- non- Gaussian priors alike. Therefore, our analysis herein still applies in much the same way.

---

Under review as a conference paper at ICLR 2020
REFERENCES
A. Alemi, B. Poole, Fischer, J. Dillon, R. Saurous, and K. Murphy. Fixing a broken ELBO. arXiv preprint vrXiv:1711.00460 2017.
M. Bauer and A. Mnih. arXiv: 1810. 1 1428, 2018.
Resampled priors for variational autoencoders.
arXiv preprint
M. Bauer and A. Mnih. Resampled priors for variational autoencoders. International Conference on Artificial Intelligence and Statistics, 2019.
S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowiz, and S. Bengio. Generating sentences from a continuous space. arXiv preprint arXiv: /5/0:3341 2015.
Y. Burda, R. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. arXiv preprint 150vv:550.0003 00519, 2015.
L. Cai, H. Gao, and S. Ji. Multi-stage variational auto-encoders for coarse-to-fine image generation. arXiv preprint AxKiv:705.0722 2017.
E. Candès, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? J. ACM, 58(2), 2011.
X. Chen, D. Kingma, T. Salimans, Duan, P. Dhariwal, J. Schulman, Sutskever, and P. Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 1xXvv:6611.02331, 2016.
B. Dai and D. Wipf. Diagnosing and enhancing VAE models. International Conference Learning Representations, 2019.
B. Dai, Wang. J. Aston, G. Hua, and D. Wipf. Hidden talents of the variational autoencoder. arXiv preprint arXiv: 11007006655444 2019.
A. Dieng, Y. Kim, A. Rush, and D. Blei. Avoiding latent variable collapse with generative skip models. arXiv preprint arXiv: 04863, 2018.
K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. International Conference Machine Learning, 2010.
J. He, D. Spokoyny. G. Neubig, and T. Berg-Kirkpatrick. Lagging inference networks and poste- rior collapse in variational autoencoders. International Conference on Learning Representations, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. 3-vae: Learning basic visual concepts with a constrained variational framework. International Conference Learning Representations, 2017.
C. Huang, S. Tan, A. Lacoste, and A. Courville. Improving explorability in variational inference with annealed variational objectives. Advances in Neural Information Processing Systems, 2018.
K. Kawaguchi. Deep learning without poor local minima. Advances in Neural Information Process- ing Systems, 2016.
D. Kingma and M. Welling, Auto-encoding variational Bayes. International Conference Learn ing Representations, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
P. Li and P.M. Nguyen. On random deep weight-tied autocncoders: Exact asymptotic analysis, phase transitions, and implications to training. International Conference on Learning Representations, 2019.
J. Lucas, G. Tucker, R. Grosse, and M. Norouzi. Understanding posterior collapse in generative latent variable models. International Conference on Learning Representations, Workshop Paper, 2019.
10

---

Under review as a conference paper at ICLR 2020
L. Maaløe, M. Fraccaro, V. Liévin, and O. Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. arXiv preprint arXiv:1902.021000 2019.
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv preprint arXiv: 2016.
P.A. Mattei and J. Frellsen. Leveraging the exact likelihood of deep latent variable models, Advances in Neural Information Processing Systems, 2018.
E. Orjebin. A recursive formula for the moments of a truncated univariate normal distribu- tion. 2014. URL httptts////sppppeeee.. smp uq. htt...ad...a.aooooaaaaaatttttaaa
projects/ http:////settttttto/o/oooraaaaaa Tooee .pt.ff
A. Razavi, Oord, Poole, and O. Vinyals. Preventing posterior collapse with 6-VAEs. Interna- lional Conference on Learning Representations, 2019.
D. Rezende and S. Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770, 2015.
D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning, 2014.
M. Rolinek, D. Zietlow, and G. Martius, Variational autoencoders pursue PCA directions (by acci- dent). 2019.
C. Sønderby, T. Raiko, Maaløe, S. Sønderby, and O. Winther. How to train deep variational autoencoders and probabilistic ladder networks. arXiv preprint arXiv:1602.02282, 2016.
P. Sprechmann, A.M. Bronstein, and G. Sapiro. Learning efficient sparse and low rank models. IEEE Trans. Pattern Analysis and Machine Intelligence, 37(9), 2015.
M. Tipping and C. Bishop. Probabilistic principal component analysis. J. Royal Statistical Society, Series B, 61()):611-62 1999.
J. Tomezak and M. Welling. VAE with VampPrior. International Conference Artificial Intelli- gence and Statistics, 2018.
A. Van den Oord, N. Kalchbrenner, L. Espeholt, O Vinyals, A. Graves, and K. Kavukcuoglu. Con- ditional image generation with PixeICNN decoders. Advances in Neural Information Processing Systems, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.
Yeung, A. Kannan, Y. Dauphin, and L. Fei-Fei. Tackling over-pruning in variational autoencoders. arXiv preprint 11t.::17000064333 2017.
C. Yun, S. Sra, and Jadbabaie. Small nonlinearities activation functions create bad local minima in neural networks. International Conference on Learning Representations, 2019.
Zhao and Yu. On model selection consistency of Lasso. Journal of Machine learning research, 7:2541-2563 2006.
A APPENDIX
A.1 NETWORK STRUCTURE, EXPERIMENTAL SETTINGS, AND ADDITIONAL IMAGENET RESULTS
Three different kinds of network structures are used in the experiments: fully connected networks, convolution networks, and residual networks. For all these structures, we set the dimension of the latent variable 64. We then describe the network details accordingly.
11

---

Under review as a conference paper at ICLR 2020
Fully Connected Netowrk: This experiment is only applied the simple Fashion- MINIST dataset, which contains 60000 28 x 28 baock-andwhiie images. These images are first flattened to a 784 dimensional vector. Both the encoder and decoder have multiple number of 512-dimensionee hidden layers, each followed by ReI.U activations.
Convolution Netowrk: The original images either 32 32 x 3 (Cifarl0, Cifar100 and SVHN) or 64 x 64 x 3 (CelebA and ImageNet). In the encoder, we use a multiple number (denoted as t) of 3 x 3 convolution layers for each spatial scale. Each convolution layer followed by ReLU activation. Then we use a x 2 max pooling to downsample the feature map to a smaller spatial scale. The number of channels is doubled when the spatial scale halved. We use 64 channels when the spatial scale is 32 x 32. When the spatial scale reaches 4 x (there should be 512 channels in this feature map), we use an average pooling to transform the feature map vector, which is then transformed into the latent variable using a fully connected layer. In the decoder, the latent variable first transformed to 4096-dimensional vector using fully connected layer and then reshaped to 2 x 2 1024. Again in each spatial scale, use 1 transpose convolution layer upscale the feature map and halve the number of channels followed by f convolution layers. Each convolution and transpose convolution layer is followed by ReLU activation layer. When the spatial scale reaches that the original image, we use a convolution layer to transofrm the feature map to 3 channels.
The network structure of the residual network is similar to that of convo- lution network described above. We simply replace the convolution layer with a residual block. Inside the residual block, use different numbers of convolution numbers. (The typical number of convolution layers inside a residual block is or 3. In our experiments, we try 2, 3, 4 and 5.)
Residual Network:
Training Details: All the experiments with different network structures and datasets trained in the same procedure. We use the Adam optimization method and the default optimizer hyper parameters in Tensorflow. The batch size is 64 and we train the model for 250K iterations. The initial learning rate is 0.0002 and it is halved every 100K iterations.
Additional Results on ImageNet: We also show the reconstruction error for convolution networks with increasing depth trained on ImageNet in Figure The trend is the same as that in Figure [|
240
AE
VAE
Mean Square Error (MSE)
200
160
120
Layer / Spatial Scale
Figure 3: Reconstruction for Convolution networks with increasing depth/# of spatial scales trained on ImageNet.
A.2 PROOF OF PROPOSITION 4.1
While the following analysis could in principle be extended to more complex datasets, for our pur- poses is sufficient consider the following simplified case for ease of exposition. Specifically, we assume that 1, d.d K, set d 2, 2, 1. and (1)) (1, x), (-1, -1).
Additionally, we will use the following basic facts about the Gaussian tail. Note that below follow from integration by parts; see Orjebin (014)).
12

---

Under review as a conference paper at ICLR 2020
Lemma A.1 Lete NN0, A>> 0; ((x), (z) be the paf and edf of the standard normal distri- bution, respectively. Then
1 ((A) S0-122
(11)
Ble1 [e = (((),,
(12)
Ele 1 1 (AA) + Ag(A)-
(13)
A.2.1
SUBOPTIMALITY OF
Under the specificed conditions, the energy from 7 has value of nd. Thus show that it is not the global minimum, it suffices to show that the following VAE, parameterized by 6, has energy + -00 6 + 0:
2 1 =-1,
W. (a + 1,a + 1), b, =0, 0,
11) o.?2 =
1,( 2(1 Tr ((a + 1)(1 + 5E)))
This follows because, given the stated parameters, we have that
2
Ø) =
Σ
(1 + 2 log EN 0.1) (a - 1)(1 + 5e)))² 2log + 0 + 1)
t=1
2
(0(1) + 2 log EN( (a + (a 1)ōe))² log 6)
1
(44 log 6 O(1)
(i) holds when 8 1 a+1 to see this, denote x := a+= (a 1)(EE). Then
Ex(c - (x))
=E. [[1 Ta (x)) 2 (=2a3] Eel(1 - (x())) (|x] [[]]
+ Bel(1 (px=-a))
E [1 (x - =] P[ a) + E& ((1 x a) ((<c-a))
(a)
(b)
In the RHS above (a) 1(a 1)8/2 using 11
13
then have
1
(b) < P(æ < a) = P $
I exp
(a - 1)8
2[(a + 1)0)²
(c) < E. ((2x + (a + 1)dE) (
(2a + (a 1)8E)²
1
-2112 ds
V2T
(4a² + [(a + 1)
1
-5212
1
1
4a + ((a 1)0)²
1
V2T
exp
21(a
1)8}²
when 5 < a+1- Thus
EN - TT (x))2
lim
=1, 1,
8++
[(a + 11
and
lim flog (e To (z)) 2 log 8} 2 log(a + 1),
8-0
or
log To (2)) 4log + O(1),
and see (i) holds.
13

---

Under review as a conference paper at ICLR 2020
A.2.2 LOCAL OPTIMALITY OF
We will now show that at 77. the Hessian of the energy has structure
W
(bx)
(o
(
(W
(62)
2
0
0
(p.d.) 0
0
0
(p.d.)
where p.d. means the corresponding submatrix is positive definite and independent of other param- eters. While the Hessian is 0 in the subspace of W.. we can show that for VAEs that only different from 7 by W.. æ, the gradient always points back to ( Thus (7) is strict local minima.
First we compute the Hessian matrix block-wise. We will identify W E R2x1 with the vector (Wz))=1= and use the shorthand notations x(() = )3 by = (bz) 200 = u1 + 02 (0)
where (00 1) (recall that 200) is scalar in this proof).
1. The second-order derivatives involving W x can be expressed
ac aw,
-2
Σ Bel(TT (1)) (u)) Ta (W) bb)]l.
(14)
i=1
and therefore all second-order derivatives involving Wj will have the form
E.[T', (W (W)2(9)FF + (W, ( Fe).
(15)
where F1, F22 are some arbitrary functions that are finite W = O, the above always evaluates to at W =.
Since T/ (0)
(0)
2. For second-order derivatives involving by we have
Oc -2 bb 7
-BB B. (z Fa (W (1)) bz)
and
0L I,
0(b,)²
2 OL
0,
(since Wa 0);
dybb,
22 ab
and
8-6 and 8 0n! ab
will also have the form of (15). thus both equal at W, = 0.
0u 0 !
3. Next consider second-order derivatives involving u or ok (t) Since the KL part of the
energy, (z(qu(2|x<)) )p(z)), only depends on μu and (6) and have p.d. Hes- sian independent of other parameters, it suffices to calculate the derivatives of the reconstruction error part, denoted as Lrecon- Since
OLrecon
-2
Eee
(1) Ne (0) b,),,
0p
(8)
7
2.j
0L
-2
Σ
Ee
(1r)) - Ta (
0o
(8)
7
2
all second-order derivatives will have the form of 115 and equal 0 at W_ - 0.
4. For 7 calculate that 0 C/Oy² 4/7"
Now, consider VAE parameters that are only different from in W Plugging b, 2, p
ookk (1)
into 14), we have
14

---

Under review as a conference paper at ICLR 2020
-2 J
0C
Es ( (Wje)s) (-ma ==1
As (T'a (W)s))) ( Ta (W < always holds, we can see that the gradient points back 7 This concludes our proof of being a strict local minima.
A.3 PROOF OF PROPOSITION
We begin by assuming arbitrarily complex encoder for convenience. This allows us remove the encoder- -poosor-pponsoeed amortized inference and instead optimize independent parameters μ and as separately for each data point. Later we will show that this capacity assumption can be dropped and the main result still holds.
We next define
m, (
(1
E
TR km and 8, - {
(d
(o!
E R (16)
which are nothing more than the concatenation of all of the decoder means and variances from each data point into the respective column vectors. is also useful to decompose the assumed non-degenerate decoder parameters via
e [eb, w]. 3b O,w,
(17)
where w [0, is a scalar such that Me (z; My (wz; vh). Note that we can always repa- rameterize an existing deep architecture to extract such a latent scaling factor which we can then hypothetically optimize separately while holding the remaining parameters vp fixed. Finally, with slight abuse of notation, we may then define the function
I (wm,, s, w88))
(18)
n
( w), 2())
ΣE E
Nr Nr (a Salpe.,ddia diag [o [|~(0) Men (wz;e) Φ
113]
This is basically just the original function ff summed over all training points, with z6 fixed at the corresponding values extracted from 0 while w serves as free scaling parameter on the decoder.
Based on the assumption of Lipschitz continuous gradients, we can always create the upper bound
(u, v) < (ū,
(19)
+ u) Vuf (u, v) + 1/2 1|u 1| + (v (v- 5) Vaf (u, vll 1/2 | 01|
where the Lipschitz constant of the gradients and we have adopted u - wm; and & wo to simplify notation. Equality occurs at the evaluation point {u, v) = {ū, ë}. However, this bound does not account for the fact that we know V. (u, v) 2 0 (i.e., I (u, v) increasing w.r.t. v) and that v 2 O. Given these assumptions, we can produce the refined upper bound
gub (u, 2 (u,
(20)
where pub (u, A
f (ū, Ū) + (u ū) Vuf (u, vulu + l/u ul12+
Ug V., (u, v) lo) (21)
and the function g IR3
R defined as
(v- 0) + (v
8)2
if v 0 and {e, 0, 8) 2 0,
(c, 0, 4
if u and
{u, 0,8) 2 0, otherwise.
(22)
21.
0
15

---

Under review as a conference paper at ICLR 2020
Given that
L arg min [v - 8) 8 + 4/ 2
(v 0) and
-6a min [10 8) 8 (v 0)
(23)
2L
the function g is basically just setting all values of (u 8) 2 || - 5|| with negative slope to
the minimum - This change is possible while retaining an upper bound because f (u, v) is non- 2T.
decreasing in v by stated assumption. Additionally, is set to infinity for all v < 0 to enforce non-negatively.
While it may be possible to proceed further using fuh, we find it useful to consider a final modifica- tion. Specifically, define the approximation
fapmr
(u, 2)
pub (ú, v)
(24)
where
fapm (u, v) A
nd Σ 5=1
0) + (u-u) ū) T
vul (u, uuuuu 1/2 1 11112122
g a
(15, Vo, (u, v) 1,,
(25)
and
-B2
82
02
if ú 4, 2 and
{u, 00,0,0} 2 0,
21.
21
gaPpr (u,
LGa
50
+ (
2 0²
if ú i and
{u, 0, 2 0,
(26)
2
2
otherwise.
While slightly cumbersome to write out, gor has a simple interpretation. By construction, we have that
min garp (v, ē, 8) = g "Pp (0, 0, 8) = min g (v, 8) (0, 6,
(27)
and
g gor (ū = (8,
(28)
At other points, go is just a simple quadratic interpolation but without any factor that is linear in v. And removal of this linear term, while retaining (27) and (27) will be useful for the analysis that follows below. Note also that although foppr (u, v) is no longer a strict bound on f (u, v), it will nonetheless still be an upper bound whenever Dy f0, 8y} for all which will ultimately be sufficient for our purposes.
We now consider optimizing the function
happr (m., 82, w) A 1 pappr (wm,, ws:) +
7/2
| log diag a (
(29)
If define L (m,, $2, w) the VAE cost from under the current parameterization, then by design follows that
heppr (m, 82, ()) L (m,, ())
(30)
and
ap (m,, 82, w)) 2 L (m.. 8,, w)
(31)
whenever wa, E {0, wā for all j. Therefore if we find such a solution {m', s'., w/} that satisfies this condition and has h. (iper(m... 8/g, w') haver (m., 82, Ø), it necessitates that L(m', 8/ w') < L(m,,, $2, w) as well. This then ensures that (m., 82, w) cannot be a local minimum.
We now examine the function happr more closely. After few algebraic manipulations and exclud- ing irrelevant constants, we have that
appr (,y, ; Sg, w)
nd
Wmzy Va, ff (2e, 8) u)luy=mmtns + (w²m², - 22mm jwme)) + u $2.)
+ m + 22 log 221
(32)
16

---

Under review as a conference paper at ICLR 2020
where Cj is the coefficient the x² term from
After rearranging terms, optimizing out m, and 82, and discarding constants, we can then obtain (with slight abuse of notation) the reduced function
26
nd
happr (w)
Vj
log(l + c,w²).
(33)
+ Bw2
where B - and yj A 1 ū t. vn (u, v) u, Note that yj must be bounded since L * and w E [0, 1I, Va, (u, v) uy=@m I L, and m are all bounded. The latter is im- plicitly bounded because the VAE KL term prevents infinite encoder mean functions. Furthermore, e, must be strictly greater than zero per the definition of a non-degenerate decoder, this guarantees that
goppr (ūsz WSS Vey f (u, v) lyssssss) gappr (o. U.S Vu, (u,, 2) D))o,=as))
(34)
which is only possible with ej > O. Proceeding further, because
V Vaah (w) = ad
By, (7 Bw222 2 +
(35)
+ cjw2
j=1
we observe that if is increased sufficiently large, the first term will always be smaller than the second since 3 and all y/, are bounded, and çj > 0 Vj. So there can never be a point whereby V wwaheppr (w) = 0 when = 7 sufficiently large. Therefore the minimum in this situation occurs on the boundary where 22 O. And finally, if w² = O, then the optimal m, and 8, is determined solely by the KL term, and hence they are set according to the prior. Moreover, the decoder has no signal from the encoder and is therefore optimized by simply setting Mr, 0; to the mean ã for all Additionally, none of this analysis requires and arbitrarily complex encoder; the exact same results hold long as the encoder can output 0 for means and for the variances.
Note also that if we proceed through the above analysis using w E R as parameterizing separate Wy scaling factor for each latent dimension j E (1,..., H²), then a smaller y value would generally force partial collapse. In other words, we could enforce nonzero gradients of haver along the indices of each latent dimension separately. This loosely criteria would then lead to qo- (zj x) Proposition
p(zz) along some but not all latent dimensions as stated in the main text below
A.4 REPRESENTATIVE STATIONARY POINT EXHIBITING POSTERIOR COLLAPSE IN DEEP VAE MODELS
Here provide example of a stationary point that exhibits posterior collapse with an arbitrary deep encoder/decoder architecture. This example is representative of many other possible cases. Assume both encoder and decoder mean functions ft and flg. as well as the diagonal encoder covariance function Σ, = diag are computed by standard deep neural networks, with layers composed of linear weights followed by element-wiss nonlinear activations (the decoder covariance satisfies Σg = =II as before). We denote the weight matrix from the first layer of the decoder mean network as W1 H=* while w wherj refers to the corresponding j-th column. Assuming p layers, we denote W and W as weights from the last layers of the encoder networks producing μt= and log o respectively, j-th rows defined as wh Me.j. and wh "z.." We then characterize the following key stationary point:
ww gradients of 4 with respect to wh.... wh.... and w
Proposition A.2 If ffw (u
0 for any j {1,2,...,, then the oZ.j. are all equal zero.
'L 0 would violate the stipulated conditions for non-degenerate decoder since it would imply that no signal from z could pass through the decoder. And of course if T. 0, would already he solution exhibiting posterior collapse.
We are assuming here that the decoder has sufficient capacity model any constant value, e.g., the output layer has hias term.
17

---

Under review as a conference paper at ICLR 2020
If the stated weights zero along dimension then obviously it must be that go(2y x)) p(2y),
i.., a collapsed dimension for better or worse. The proof is straightforward; we provide the details below for completeness.
Proof: First we remind that the variational upper bound defined in 2 We define L(x; as the loss at data point æ; i.e.
L( 0, Φ) = Eg po (a]=)] + KL (2<(2/2)]l
(36)
The total loss is the integration of C(x; 0 6) over æ. Further more, we denote Lx)( 0) and Ligen (ax; 02 6) as the KI. loss and the generation loss at respectively, i.e.
(2)(z;))
KL (9s(=
Σ KL
Σ (mi + log o 1)
(37)
2
Egen (2;; 0,0) 0)
(x) (x) [log (ar
(38)
The second equality in (37 holds because the covariance of qo (2 (x) and p(z) are both diagonal. The last encoder layer and the first decoder layer denoted as he and h If wh. wh.s 0, w then we have
0,
Mxj = w Mog. he = O, = = exp (woz.)) = ((2//>)= = AO0,
(39)
The gradient of Pzj and Oz. from Lui(;; () becomes
(a
(Lu(a;;)
Opz. Mzs =0, O,
(40)
0o.s
So the gradient of wh p..j. and w 03.. from Lut is
(Lu(x; Φ)
aLu((;3) ()
ho =0,
(41)
Ow" peu-
Opz.
1L.1(2r; ()
aLk((a; ())
he 0.
(42)
Ow
20..j :0z...
03.
Now we consider the gradient from Lgen (x: 0, Φ). We have
-8 (o)lp))))))
- 0 Oog (x(l)) Oh
(43)
Ōzg
1h
Ōzj
Since
h act
Σ w
(44)
Marf
where act(-) is the activation function, we can obtain
oh
act'
Σ w
whess
w
O.
(45)
Oz,
Plugging this back into (3) gives
log Po (2(2))) 0. dzg
(46)
According to the chain rule, we have
@C (2;
- log po
Oz,
Ex x)
0,
(47)
Ow
0z,
Ow p
18

---

Under review as a conference paper at ICLR 2020
@ yen (x;0,00 Φ)
log pe (2/zz)
dzj
Ez 1x (c)
Sw o?. 0.
(48)
ir o?.
After combining these two equations with (41 and (22
and then integrating over we have
ac Φ)
0,
(49)
Om M.
ac(0. Φ)
OwLy 0.
(50)
Then we consider the gradient with respect to w waa.i"" Since WHarj is part of 0, it only receives gradient from Egen (zx; 0, Φ). So we do need to consider the KL loss. If w whers 0, h D5=1 w k..gz is not related to z,. So po(z)z)) = po (z( where 2-j represents z without the
j-th dimension. The gradient of un Ma is
0Lgen(z;0,0) Ez)g(z())
log po (x)
Oloo po (z/2)) Öh
Ezzn(2(a) (z)
2g
(51)
ow U221
Ow Mavj
0 log po (2:1z-))
E 12zj~o((--0 (2-((()
B2y=N((011
2jj
Thd
θ log po (arl
.))
Ez-i-a(Z-1(0)
E Bes,N(0,1)
=0
ah
The integration over æ should also be 0. So we obtain
AL(0: Φ)
(52)
dw Ka
19


Reviews:
[{"review_id": "rJg6KrUpKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper is clearly written and well structured. After categorizing difference causes of posterior collapse, the authors present a theoretical analysis of one such cause extending beyond the linear case covered in existing work. The authors then extended further to the deep VAE setting and showed that issues with the VAE may be accounted for by issues in the network architecture itself which would present when training an autoencoder.\n\n\n\nOverall:\n\n1) I felt that Section 3 which introduces categorizations of posterior collapse is a valuable contribution and I expect that these difference forms of posterior collapse are currently under appreciated by the ML community. I am not certain that the categorization is entirely complete but is nonetheless an excellent step in the right direction. One source of confusion for me was the difference between sections (ii) and (v) --- in particular I believe that (ii) and (v) are not mutually exclusive.\n\nAdditionally, the authors wrote \"while category (ii) is undesirable, it can be avoided by learning $\\gamma$\". While this is certainly true in the affine decoder case it is not obvious that this is true in the non-linear case.\n\n2) Section 4 provides a brief overview of existing results in the affine case and introduces a non-linear counter-example showing that local minima may exist which encourage complete posterior collapse.\n\n3) On the proof of Proposition 4.1. In A.2.1 you prove that there exists a VAE whose ELBO grows infinitely (exceeding the local maxima of (7)). While I have been unable to spot errors in the proof, something feels odd here. In particular, the negative ELBO should not be able to exceed the entropy of the data which in this case should be finite. I've been unable to resolve this discrepancy myself and would appreciate comments from the authors (or others). The rest of the proof looks correct to me.\n\n4) I felt that section 5 was significantly weaker than the rest of the paper. This stemmed mostly from the fact that many of the arguments were far less precise and less rigorous than those preceding. I think the presentation of this section could be significantly improved by focusing around Proposition 5.1.\n\na) Section 5 depends on the decoder architecture being weak, though this is not clearly defined formally. I believe this is a sensible restriction which enables analysis beyond the setting of primary concern in Alemi et al. (and other related work).\n\nb) In the third paragraph, you write \"deep AE models can have bad local solutions with high reconstruction [...]\". I feel that this doesn't align well with the discussion in this section. In particular, I believe it would be more accurate to say that IF the autoencoder has bad local minima then the VAE is also likely to have category (v) posterior collapse.\n\nc) Equation (8) feels a little too imprecise. Perhaps this could be formalized through a bias-variance decomposition of the right hand side similar to Rolinek et al.?\n\nd) The discussion of optimization trajectories was particularly difficult to follow. It is inherently difficult to reason about the optimization trajectories of deep auto-encoding models and is potentially dangerous to do so. For example, perhaps the KL divergence term encourages a smoother loss landscape and encourages the VAE to avoid the local stationary points that the auto-encoder falls victim to.\n\ne) It is written, \"it becomes clear that the potential for category (v) posterior collapse arises when $\\epsilon$ is large\". This is not clear to me and in fact the analysis seems more indicative of collapse presented in category (ii) (though as mentioned above, I am not convinced these are entirely separate). Similarly, later in this section it is written, \"this is more-or-less tantamount to category (v) posterior collapse\". I was also unable to follow this reasoning.\n\nf) \"it is actually the AE base architecture that is effectively the guilty party when it comes to posterior collapse\". If the conclusions are to be believed, this only applies to category (v) collapse.\n\ng) Unfortunately, I did not buy the arguments surrounding KL annealing at the end of section 5. In particular, KL warm start will change the optimization trajectory of the VAE. It is possible that the VAE has a significantly worse loss landscape than the autoencoder initially and so warm-start may enable the VAE to escape this difficult initial region.\n\nMinor:\n\n- The term \"VAE energy\" used throughout is not typical within the literature and seems less explicit than the ELBO (e.g. it overlaps with energy based models).\n- Equation (4) is missing a factor of (1/2).\n- Section 3, in (ii), typo: \"assumI adding $\\gamma$ is fixed\", and \"like-likelihood\". In (v), typo: \"The previous fifth categories\"\n- Section 4, end of para 3, citep used instead of citet for Lucas et al.\n- Section 4, eqn 6 is missing a factor of 1/2 and a log(2pi) term.\n- Section 5, \"AE model formed by concatenating\" I believe this should be \"by composing\".\n- Section 5, eqn 10, the without $\\gamma$ notation is confusing and looks as though the argmin does not depend on gamma. Presumably, it would make more sense to consider $\\gamma^*$ as a function of $\\theta$ and $\\phi$.\n- Section 5 \"this is exactly analogous\". I do not think this is _exactly_ analogous and would recommend removing this word.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xJXhf6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\nThe paper theoretically investigates the role of “local optima” of the variational objective in ignoring latent variables (leading to posterior collapse) in variational autoencoders. The paper first discusses various potential causes for posterior collapse before diving deeper into a particular cause: local optima. The paper considers a class of near-affine decoders and characterise the relationship between the variance (gamma) in the likelihood and local optima. The paper then extends this discussion for deeper architecture and vanilla autoencoders and illustrate how this can arise when the reconstruction cost is high. The paper considers several experiments to illustrate this issue.\n\n2. Opinion and rationales\nI thank the authors for a good discussion paper on this important topic. However, at this stage, I’m leaning toward “weak reject”, due to the reasons below. That said, I’m willing to read the authors’ clarification and read the paper again during the rebuttal to correct my misunderstandings if there is any. The points below are all related.\n\na. I would like to understand the use of “local optima” here. I think the paper specifically investigate local optima of the likelihood noise variance, and there are potentially other local optima. Wouldn’t this be an issue with hyperparameter optimisation in general? For example, for any regression tasks, high observation noise can be used to explain the data and all other modelling components can thus be ignored, so people have to initialise this to small values or constrain it during optimisation.\n\nb. I think there is one paper that the paper should discuss: Two problems with variational expectation maximisation for time-series models by Turner and Sahani. In this paper, the paper considers optimising the variational objective wrt noise likelihood hyperparameters and illustrates the “bias” issue of the bound towards high observation noise.\n \nc. I think it would be good to think about the intuition of this as well: “unavoidably high reconstruction errors, this implicitly constrains the corresponding VAE model to have a large optimal gamma value”: isn’t this intuitive to improve the likelihood of the hyperparameter gamma given the data?\n\nd. If all above are sensible and correct, I would like to understand the difference between this class of local minima and that of (ii). Aren’t they the same?\n\ne. The experiments consider training AEs/VAEs with increasingly complex decoders/encoders and suggest there is a strong relationship between the reconstruction errors in AEs and VAEs, and this and posterior collapse. But are these related to the minima in the decoder’s/encoder’s parameter spaces and not the hyperparameter space? So that is the message that the paper is trying to convey here?\n\n3. Minor:\n\nSec 3\n(ii) assumI -> assuming\n(v) fifth -> four, forth -> fifth\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gXy3rXcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to establish an explanation for the posterior collapse by linking the phenomenon to local minima. If I am understanding correctly, the final conclusion reads along the lines of 'if the reconstruction error is high, then the posterior distribution will follow the prior distribution'. They also provide some experimental data to suggest that when the reconstruction error is high, the distribution of the latents tend to follow the prior distribution more closely. \n\nAlthough I really liked section 3 where authors establish the different ways in which `posterior collapse' can be defined, overall I am not sure if I can extract a useful insight or solution out of this paper. When the reconstruction error is large, the VAE is practically not useful.  \n\nAlso, I don't think I am on board with continuing to use the standard Gaussian prior. Several papers (such as the cited Vampprior paper) showed that one can very successfully use GMM like priors, which reduces the burden on the autoencoder. Even though I liked the exposition in the first half of the paper, I don't think I find the contributions of this paper very useful, as one can actually learn the prior and get good autoencoder reconstructions while obtaining a good match between the prior and the posterior, without having degenerate posterior distribution which is independent from the data distribution. All in all, I think using a standard gaussian prior is not a good idea, and that fact renders the explanations provided in this paper obsolete in my opinion. Is there any reason why we would want to utilize a simplistic prior such as the standard Gaussian prior? Do you have any insights with regards to whether the explanations in this paper would still hold with more expressive prior distributions? \n\nIn general, I have found section 5 hard follow. And to reiterate, the main arguments seem to be centered around autoencoders which cannot reconstruct well, as the authors also consider the deterministic autoencoder. If the autoencoder can not reconstruct well, it is not reasonable to expect a regularized autoencoder such as VAE to reconstruct, better, and therefore the VAE is already is a regime where it is not useful anyhow. I think the authors should think about the cases where the reconstruction error is low, and see if there is an issue of posterior collapse in those setups. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript investigates the posterior collapse in variational autoencoders and seeks to provide some explanations from the phenomenon. The primary contribution is to propose some previously understudied explanations for the posterior collapse that results from the optimization landscape of the log-likelihood portion of the ELBO.

The reviewers and AC agree that the problem studied is timely and interesting, and closely related to a variety of recent work investigating the landscape properties of variational autoencoders and other generative models. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the technical difficulty and importance of the results. In reviews and discussion, the reviewers noted issues with clarity of the presentation and sufficient justification of the results. There were also concerns about novelty. In the opinion of the AC, the manuscript in its current state is borderline, and should ideally be improved in terms of clarity of the discussion, and some more investigation of the insights that result from the analysis.
Decision: Reject