On the Value of Infinite Gradients in Variational Autoencoder Models
Authors: Bin Dai
Li Kevin Wenliang
David Wipf
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: variational autoencoders
sparse representations
latent variable models

Abstract: A number of recent studies of continuous variational autoencoder (VAE) models have noted, either directly or indirectly, the tendency of various parameter gradients to drift towards infinity during training.  Because such gradients could potentially contribute to numerical instabilities, and are often framed as a problematic phenomena to be avoided, it may be tempting to shift to alternative energy functions that guarantee bounded gradients.  But it remains an open question: What might the unintended consequences of such a restriction be?  To address this issue, we examine how unbounded gradients relate to the regularization of a broad class of autoencoder-based architectures, including VAE models, as applied to data lying on or near a low-dimensional manifold (e.g., natural images).  Our main finding is that, if the ultimate goal is to simultaneously avoid over-regularization (high reconstruction errors, sometimes referred to as posterior collapse) and under-regularization (excessive latent dimensions are not pruned from the model), then an autoencoder-based energy function with infinite gradients around optimal representations is provably required per a certain technical sense which we carefully detail.  Given that both over- and under-regularization can directly lead to poor generated sample quality or suboptimal feature selection, this result suggests that heuristic modifications to or constraints on the VAE energy function may at times be ill-advised, and large gradients should be accommodated to the extent possible.

On the Value of Infinite Gradients in Variational Autoencoder Models
Bin Dai Institue for Advanced Study
K. Wenliang
Gatsby Computational Neuroscience Unit University College London
Tsinghua University
baibo@plyysicc@@hoomall...coo
kevinli@gatsly ucl. ac uk
David Wipf
Shanghai Research Lab
Amazon Web Services
davidwipf@ggail.comm com com
Abstract
A number of recent studies of continuous variational autoencoder (VAE) models have noted, either directly indirectly, the tendency of various parameter gradients to drift towards infinity during training. Because such gradients could potentially contribute to numerical instabilities, and are often framed as problematic phe- nomena to be avoided, it may be tempting to shift to alternative energy functions that guarantee bounded gradients. But it remains an open question: What might the unintended consequences of such restriction be? To address this issue, we examine how unbounded gradients relate to the regularization of a broad class of autoencoder-basee architectures, including VAE models, applied to data lying on or near a oww-iimesioonl manifold (e.g., natural images). Our main finding is that, if the ultimate goal is to simultaneously avoid over-regularization (high reconstruction errors, sometimes referred to as posterior collapse) and under- regularization (excessive latent dimensions an autoencoder-bassee energy function with infinite gradients around optimal rep- resentations is provably required per certain technical sense which carefully detail. Given that both over- and under-regularization can directly lead to poor generated sample quality suboptimal feature selection, this result suggests that heuristic modifications to or constraints on the VAE energy function may at times be ill- advised, and large gradients should be accommodated the extent possible.
not pruned from the model), then
Introduction
Suppose we have access to continuous variables x that are drawn from ground-truth measure Mgt- This measure assigns probability mass Pge (dxz) to the infinitesimal dæ residing within C Rd such that we have [x Myt (dx)
This formalism allows us consider data that may lie on or near an d), capturing the notion of low-dimensional
r-dimensional manifold embedded in Rd (implying structure relative to the high-dimensional ambient space.
Because of the possibility of unknown latent manifold, it is common to approximate the corre- sponding ground-trute measure via a density model parameterized as
(x = f po
(1)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this expression A are trainable parameters and x E RR serves as low-dimensional latent repre- sentation, with fixed prior p(z) (O I) and ideally x 2 r. If some 8 were available such that [A po- (ar (a)d z [A (Mg( (dx) for any measurable A C x. then the model would adequately reflect the intrinsic underlying distribution. Of course we will generally not know in advance the value of 0*, but in principle we might consider minimizing - log Ps (a:) averaged across a set of training samples
(x(() Ji=1 drawn from Mgt, i.e., minimize +/4 - -log [po (a())) ~ J -llog [po(2)] Ugg (dar)
over 0. Unfortunately though, the marginalization required to produce pe (ax(1) is generally in- tractable for models of sufficient representational power. To circumvent this issue, the variational
autoencoder (VAE) [Kingma and Welling 2014, Rezende al., 2014] instead optimizes the tractable
variational bound 6) -
n -log po (= 12
E Ba (2)200)
[logpo po [z) + KL [0(() M[(())
1
(2)
Here (=(z))) represents a variational approximation to pa (z|xx) with additional parameters ó gov- erning the tightness of the bound. It is commonly referred to as an encoder distribution since
quantifies the mapping from æ to the latent code z. For analogous reasons, po(a)22) is labeled as the decoder distribution. When combined, the data-dependent factor - (2(0)) [log po (a/|²] can be viewed as instantiating form of stochastic autoencoder (AE) structure, which attempts to assign high probability to accurate reconstructions of each æ;; if 9s (z|zz) is Dirac delta function, then regular deterministic AE emerges with loss dictated by the decoder negative log-likelihood
- log pu ae ))) Beyond this, KL [1o (z (x)) [[x(2)] serves as a regularization factor that pushes the
encoder distribution towards the prior. The bound (2) can be minimized over {0, Φ) using SGD and simple reparameterization trick [Kingma and Welling 2014 Rezende et al., (014).
The latter requires that we assume specific functional form for the encoder distribution. In this regard, it is common to select Ia (z .x) =>(11t [
where the Gaussian moment vectors H. and a= are functions of model parameters Φ and the random variable æ. i.e., H4= = f (æ; Φ), and o= = o= (x;; Φ). Similarly, for continuous data the decoder model is conventionally parameterized as Pe (z) 2) = N(x p I), with mean defined analogously as H4, = 4 (z; 0) and scalar variance parameter 7 > O. The functions μı (#; og Φ), and n4, (z; are all instantiated using deep
neural network layers. Given this definitions, can be expressed in the more transparent form
(() =
1
E as (2(22(()) [1 (1x((0) Me (z; 0) [13 dloog
(3)
1=1
+ o= (z)(0) )12 log diag [o2 (a of .)
+ | (2(0;; b0) ||.}
Although VAE models have been successfully applied to a variety of practical problems Li and She 2017, [chott al. 2018 Walker al.) 2016). at times they exhibit potentially problematic behavior that has not been fully investigated. For example, a number of recent works have mentioned that if a trainable decoder variance parameter y is included within a VAE as in (3), then the optimal value may converge to zero resulting in infinite or unbounded gradients and potential instabilities (Dai and Wipi Mattei and Frellsen 2018 Rezende and Viola 2018 Takahashi et al.) 2018). And we emphasize that this phenomena can occur even within the confines of the stated Gaussian assumptions and inevitable regularizution effects of the KL term. While these unbounded gradients may indeed be troublesome from optimization perspective, in this work we will reframe such gradients as an integral part of successful autoencoder- based energy functions designed to model (in a sense that will be precisely quantified below) continuous data arising from low-dinensional manifold.
To accomplish this, our analysis is split into three parts. First, in Section 2 we detail how unbounded gradients contribute to an optimal, balanced form of regularization, allowing the VAE to capture oow-dimensional manifold structure via maximally parsimonious (and lossless) latent representation. Such representations turn out to be critical for tasks such as generating non-blum non-blurry samples that resemble the training data (Tolstikhin et al. 2018), or for using autoencoder-bel based models in general to robustly screen outliers [An and Cho 2015 Xu et al.) 2018). Of course it is natural to consider whether these same goals could not be achieved using an alternative energy function with strictly bounded gradients.
2

---

The second and primary component of our contribution answers this question in the negative. More concretely, our main result from Section B proves that canonical autoencoder-basee architectures will necessarily require unbounded gradients to guarantee the type of maximally parsimonious latent representation mentioned above. Thirdly, in Section we elucidate the benefits of learning 7 during training, even in situations where we know that the optimal value will be at or near zero and contribute to arbitrarily-Iarge gradients. In particular, we argue that (at the very least) learning 7 localizes troublesome unbounded gradients to narrow regions around minima of (3), while simultancously smoothing the VAE objective across optimization trajectories prior to convergence.
Overall, our contribution can be viewed as complementary to the wide body of work analyzing what commonly-refere to as posterior collapse in VAE models [He et al. 2019] Razavi et al. 2019].
The latter can be related to the situation where is too large (either implicitly [Dai et al. 2020
or explicitly [Lucas et al. 2019) and along all or most latent dimensions the posterior do x x0))
collapses to the prior N NO, leading to high reconstruction errors. In contrast, we direct our attention herein to the opposite condition whereby y is arbitrarily small and unbounded gradients invariably ensue. In this regime, the resulting latent representations obtained from bad local minimizers can potentially be under-regularized in sense that will be described in subsequent sections.
2 Optimal Low-Dimensiee
Structure via Unbounded VAE Gradients
As alluded to previously, the VAE objective will experience unbounded gradients if y 0 as has sometimes been observed (at least approximately) during training. But perhaps counter-intuitively, this phenomena nonetheless serves a critical purpose in the context of modeling data with low- dimensional manifold structure. To quantify this assertion, Section 2. will first precisely define what type of now-dimensional or sparse latent representations will be considered optimal for our present analysis; later in Section 2.2
we link this definition to practical VAE/AE applications.
2.1 Optimal Sparse (Lossless) Representations
An autoencoder- architecture (VAE or otherwise) with decoder the (:;6), con- straint E O, and arbitrary encoder μtg component produces optimal sparse representation of a training set X w.r.t. if the following two conditions simultaneously hold:
Definition
(i) The reconstruction error is zero, meaning
x (3) Mo [22 (0; :): =0
(4)
(ii) Conditioned on achieving perfect reconstructions per criteria (i) above, the number of latent dimensions such that μzz (z(()) Φ)); 0 for all is maximal across any O E ② and any encoder function μ... A j-th latent dimension so-defined provides no benefit in reducing the reconstruction error and could in principle be removed from the model.
Remark Conceptually, this definition is merely describing the most parsimonious latent repre- sentation of the training data (conditioned on the available capacity of the decoder) that nonetheless allows us to obtain perfect reconstructions. And when combined with the low-dimensional manifold assumption from Section [] it readily follows that an optimal sparse representation of X will gen- erally involve x uninformative dimensions, assuming k 2 and adequately parameterized decoder family As an illustrative example, for data lying on a low-dimensional linear subspace,
'The encoder μ, function is allowed to be unconstrained here since, unlike the decoder, it does not contribute over-fitting (in principle
infinite capacity encoder be used). Additionally. the VAE encoder from 3 will also have variance component; however, does not directly play a role Definition [ We will address the relationship between a and optimal sparsity in subsequent discussion.
*Obviously the decoder requires sufficient flexibility to capture the manifold structure; however, there finite sample regime, the decoder is allowed be arbitrarily complex, then in principle just a single nonzero latent dimension will always be sufficient to achieve zero reconstruction error regardless of the actual data structure. This form of degenerate VAE over over-fitting has
one additional nuance worth mentioning here.
been previously quantified in (Dai et al. 2018)
3

---

the corresponding optimal sparse representation obtainable via a linear decoder will be defined by the smallest subspace containing all of the data variance, i.e., the standard PCA solution.
Remark 2 Although Definition T may appear to involve overly restrictive assumptions, it nonetheless well-approximates practical situations of broad interest. For example, as has been quantified in recent study [Pope et al. 2021]. natural images do indeed have a very low intrinsic dimension relative to the high-dimensional pixel space. Hence these images can in principle be reconstructed almost exactly using oow-dimeniional representations. Moreover, many classical under-determined inverse problems have been framed in terms of obtaining perfect reconstructions of observed measurements subject to some minimal measure of parsimony (Candès and Recht 2009).
The particular lossless notion of optimality are adopting here is not meant preclude alternatives that may be tailored for different scenarios. Rather, the proposed definition is merely selected showcase class of VAE/AE usage regimes whereby infinite gradients can play an influential role. Consequently, lossy conceptions of optimal parsimony [Alemi et al. (016 Tishby al. 2000]. while useful in their own right, are laregly outside the scope of this work.
Remark 3
Remark Although the encoder is generally stochastic, prior analysis from (Dai and Wipf 2019 has revealed that the VAE global minimum is nonetheless capable of achieving something analogous to Definition]] | More concretely, for unneeded latent dimensions the posterior is pushed to the prior to optimize the KI. regularizer, i.e., 2o (zj (1) -0, 1) for all i, which amounts to uninformative noise that will be filtered by the decoder so as not to impact reconstructions. In contrast, for
informative dimensions the posterior variance satisfies d= (x();)) ;) + 0 for all i. Collectively, this allows the VAE global minima to achieve
E 90 (2)2)1)
ua [z: 0
pb p4= (2)
(5)
while relying on the fewest number of active latent dimensions, such that both criteria (i) and (ii) of Definition ean be simultaneously satisefied.
This capability requires that the VAE avoid both over- or under-regularivzaion of the latent repre- sentations. To be more precise, VAE over-regularizzaion (sometimes loosely referred to as latent posterior collapse [He et al, 2019 Razavi et al. 2019]) occurs when too many latent dimensions are uninformative (i.e., the latent posterior along these dimensions is close to the uninformative prior) such that the reconstruction error is high and criteria (i) is violated. In contrast, with under-regularized solutions criteria (i) may be satisfied, and yet in reducing the reconstruction error towards zero, an excessive number of latent dimensions are informative in violation of criteria (ii).
In avoiding both of these suboptimal scenarios, it can be shown that the VAE explicitly relies on - 0 and the attendant unbounded gradients that follow (Dai and Wipf 2019]. From an intuitive standpoint, might expect that achieving criteria (i) would require an unbounded gradient given that, if we minimize (3) over y in isolation, the optimal value satisfies
[|
Σ E 1E (2(00())
pr [2; 0
(6)
we then plug this value back into the dl log y term from 3 the result, as well as the corresponding gradients of other model parameters, is unbounded from below as the reconstruction error goes to zero (note also that in this instance, the original 1/7 data term becomes a constant, so there is no counteracting effect). Of course to actually achieve near-zero reconstruction errors, least some dimensions of a₂ must be pushed towards zero as mentioned previously, which can also lead to infinite gradients within the KL-divergence factor.
2.2 Relevance to Typical VAE Usage Regimes
Obtaining minimalist latent representations as distilled by Definition can serve a variety of practical downstream applications, such as feature extraction [Bengio et al. 2013 Ng 2011]. compression
[Ballé et al.) 2018 Donoho 2006 Minnen et al., 2018), manifold learning Silva et al.) 2006), corruption removal DDai et al.] 2018], or even the generation of realistic samples. With respect to
the latter, it has been shown in Dai and Wipi 2019) that what we have above defined as an optimal
sparse representation can be viewed as a necessary (albeit not sufficient) condition for generating

---

ambient space
inlier, train
inlier, test
2D principal subspace containing data
o outlier test
1D inlier manifold
Figure I: The importance of optimal sparse representations in screening outliers, In this example, the simple 2D principal subspace obtainable by PCA can perfectly reconstruct the inlier manifold shown in red. But this requires using two separate informative dimensions, allowing both inliers and outliers to be reconstructed with zero error within this subspace. In contrast, it is only by recovering the curved ID inlier manifold, which relies on single informative dimension, that inliers and outliers can be differentiated. Please see supplementary for practical example using real data.
samples using continuous-space Gaussian VAE that match the training distribution. In principle, deterministic AE architecture capable of producing optimal sparse representations can also be leveraged to generate realistic samples; this would simply involve first discarding the uninformative dimensions and then applying the same analysis from [Dai and Wipf. 2019]. In fact, variants of this strategy have been previously considered in Ghosh et al. 2019 Tolstikhin et al. 2018).
And as a final motivational example, any AE-based architecture capable of producing optimal sparse representations can naturally be applied to screening outliers by squeezing the latent space to the minimal number of informative dimensions needed for reconstructing inliers. In doing so, we reduce the risk that outlier points x. (out) can be accurately reconstructed by exploiting the superfluous latent flexibility. Here we are assuming that (out) ~ Pout F Hgt for some outlier distribution Hout- Figure contains an illustration of the basic rationale. The only exception to this line of reasoning would be adversarial outliers that follow the exact same low-dimensional structure as the inliers, meaning yout and Pgt both apply all of their probability mass to the same low-dimensional manifold. In this scenario, we would need to exploit differences between Pout and jgt within the manifold to reliably screen outliers, regime in which Definition is not directly applicable. That being said, differentiating Hout and Mgg once a shared low-dimensional manifold has been modeled is far easier than doing so in the original ambient space.
Additionally, in the supplementary we demonstrate that indeed, the inlier data (in this case Fashion MNIST samples) come from fow-dimensional manifold, outlier points (MNIST samples) can be reliably differentiated, provided that λ 2 r and the VAE has sufficient capacity and the learned y can converge to near zero. And because of the VAE's propensity to find optimal sparse representations where possible, even as x is raised such that x >> r, unneeded dimensions are shut off to reduce the risk of outliers masquerading as inliers (see supplementary).
2.3 Implications for 3-VAE models
The 3-VAE (Higgins al. 2017
represents a commonly-adopeee modification of the original VAE objective, whereby the KL term is rescaled by some fixed parameter 3 > O. For Gaussian VAE models (which our focus), this scale factor effectively makes no difference if a fixed decoder variance is adopted. In this situation, 33 can just be directly absorbed into y, and the dlo 7 normalization factor from (3) can be viewed as an irrelevant constant. However, if y is learned then 8 * will make non-trivial difference because of the imbalance introduced w.r.t. the now critical Gaussian normalization factor. In particular,
is too large (specifically / > d/r, where is the data dimension and is the manifold dimension), then optimal sparse representations will generally be impossible to achieve even while learning 7-
This is because, as can be inferred from the analysis in (Dai and Wipi 2019]. the VAE loss (when granted sufficient capacity) scales as (d-r) log y around optimal sparse representations as 7 becomes small. In this expression, the d log 7 factor is derived from the Gaussian normalization mentioned above, while the -r log 7 factor originates from the KI. term. However, if we scale the KI. term by B such that 8 > d/r, then (d - Br) log tends towards positive infinity as y becomes small, and the
5

---

VAE will instead sacrifice the reconstruction error such that optimal sparse representations are not possible.
3 Can we Reliably Obtain Optimal Sparse Representations without Unbounded Gradients?
As discussed in Section 2 given data originating from a now-dimensional manifold, optimal sparse representations are a necessary requirement (at least approximately) for various tasks such as gener- ating non-blurry samples aligned with the ground-truth distribution alternatively, screening for outliers. We have also discribed how the divergent gradients associated with y - O, allow VAE global minima to achieve such optimal sparse representations. But what about alternatives that circumvent such unbounded gradients altogether? For example, could we not consider a regularized AE model that, while encouraging sparse latent representations [Ng 2011]. explicitly relies on energy function terms with bounded gradients, e.g., as may be derived from a family of sparse penalty functions with bounded gradients [Chen et al. [017 Fan and Li 2001 Palmer al. [006] ? Despite this conceptual possibility, per the analysis that follows, the answer turns out to be unequivocally no within the stated context. Or more specifically, if we wish to guarantee an optimal sparse representation without additional assumptions on the decoder model and observed data, then even arbitrary AE-based objectives will necessarily require penalty terms with infinite gradients around optimal solutions.
3.1 A Generic AE-based Objective for Optimal Sparse Representations
Consider the constrained objective function
())
-
9
(0) Mx (20)
+ h 1111))) 1=1
dn
s.t. (1) = Mz (0) Vi, E θ,
where Z - [=10)] E R and z denotes the k-th row of Z. This expression can be viewed as characterizing a typical regularized AE with a generic penalty functions 9 R + R and h R+ + I the reconstruction error and the norm across training samples of each latent dimension, respectively, Additionally, the constraint 0 E θ included in 7 can, among other things, serve to prevent the trivial solution Z - O, which could occur if each x (t) is pushed to zero while the decoder n includes an unconstrained compensatory factor that grows towards infinity such that the error (+) ua (z (z()) f) 1l, can still be minimized to zero for all i. Any regularized AE must include such constraints to avoid trivial solutions, or else additional penalty terms on 0 that serve a similar purpose. Note also that if we happen to choose g = h, the provided multipliers 1/ /n,, 1/d, and 1/(dn) induce a form of proportional regularization within energy functions composed of multiple penalty factors of varying dimension designed to favor sparsity (Wipf and Wu 2012). The square-root Lasso can be viewed as a special case of this strategy that emerges when & is a square-root function [Belloni al. 2011]. However, for arbitrary selections of g and f (with any tunable trade-off parameter absorbed within), (7) reflects a broad family of AE architectures.
We can also relate 77 to various VAE instantiations. Define Ioe an indicator function satisfying Ioe (u) = 3o for u 7 0 and zero otherwise. We then have the following:
Lemma 2 Let ph_ (z; Wz + for some W E Bdxx and b E Rd, and a (æ; x) for any arbitrary 8 R=, Then the limit y + 0, the VAE loss from (3) is such that ming : (w;6) L (0, Φ) = ming (0, 6) reduces to 7 with g(-) = I-)(.) and h(.)) = Tog(?), excluding irrelevant constant factors.
Lemma 3 For any arbitrary μe (z;8) and O O if we enforce a= (x; o) + f all x and apply a log transformation to each ||2, then the VAE loss from (3) collapses to (7) with g(-) = h(-)) log(.), excluding irrelevant constant factors.
Collectively, these results point to close affiliation between 7 and the VAE loss, especially given
that 7 - 0 and o² (æx; Φ) O along many dimensions are characteristics of VAE global optima [Dai
and Wipf, 2019]. Hence it is natural to consider more general selections of g and h in the context of optimal sparse representations.

---

3.2 On the Difficulty Avoiding Unbounded Gradients
Given generic AE architecture as in (7. this section examines what possible functions g and h are such that global minimum of L. (6, @) is capable of producing an optimal sparse representation. This can be addressed as follows:
Theorem For any functions g R R and h IR+ E with bounded gradients, and any 0 Ji=t <Rdx and
dimension set {d, r) that order as d 2 BRRT >> 0, there exists data X = {x
decoder {μz, (z; ; E }} (with the capacity to reconstruct x lying within some parameterized family of k-dimensional manifolds) which satisfy the following:
(a) - rows and zero elsewhere.
- 1r [0 1 = 0 for some 0 E ④ and Z E R with | > 0 0.
(b) Minimizing C (0, 6) over 9 and any possible encoder produces either solution with
n -ا=1(=00)) Me (2))) 0 2 0 (i.e., imperfect reconstruction), or where || 0 for strictly more than r rows of Z (i.e., not maximally sparse).
1
This result effectively implies that, to guarantee every global minima corresponds with an optimal sparse reconstruction per our definition, irrespective of the decoder and observed data, the constituent penalty functions must have an unbounded gradient (at least around zero; see further intuitions below). This can be viewed as necessary, albeit not sufficient condition, for optimal sparsity, sufficiency requires additional care taking limits around zero, e.g., 7 - 0 in the case of the VAÉ. Consequently, we cannot simply replace VAE model with standard AE architecture to somehow guarantee optimal sparse representations devoid of infinite surrounding gradients (unless further assumptions on the data and decoder are introduced).
3.3 High-Level Intuition Behind Theorem 4
While the proof is predicated on a nuanced counterexample designed with a specific technical purpose in mind (see supplementary file), we can nonetheless loosely convey the basic idea through toy illustration shown in Figure 2 Here are assuming that the data points (+)) Ji=1 lie ID manifold embedded in 2D ambient space; the extension to higher dimensions is straightforward. Moreover, we stipulate that this manifold is tightly squeezed within a small non-negative e x square near zero, represented by the blue curve on the lefthand side of Figure Now consider sample point x'' [ x/2] taken from somewhere along the stated 1D manifold. We represent this point using two candidate decoder functions, both assumed to be within the capacity of p as displayed in the middle of Figure
For the simple decoder case, which is just the identity function (z, 0) = z, the values 21 = 2 and 22 = 2/2 needed for perfect reconstruction will both be small, i.e., {=1, x/2} S E by design. In contrast, the optimal decoder only requires that a single dimension of x, namely 21, be nonzero. However, the optimal value actually needed for perfect reconstruction, denoted 27, can be arbitrarily large in controlling where along the extended, labyrinthine manifold pathway x' is located (for case of presentation will assume 2 is also positive). Hence we can easily have that
z
max (z1,=6)
Because of this, ensure that =" [2] is preferred over the z' alternative, we require concave penalty function i on each encoder dimension such that any infinitesimal movement away from zero incurs arbitrarily- modest additional cost (see the green curve on the righthand side of Figure 2). From this it follows that any movement of z and z² away from zero, no matter how small, will be such that we can guarantee that the penalties on z" and z' will satisfy
cost, while increases originating from points away from zero incur only
((+) + h(0) = (xfz) ~ M((2) RJ 1(22)) < h(22) + h(22) ~ 21h(2]) - k(0),, (9)
and so z. is preferred. The righthand side Figure 2 motivates this relationship. Note also that we were to explicitly bound the slope of h around zero, then we could always select an sufficiently small such that the inequality in (9) reversed; hence an unbounded slope required to achieve the stated result.
7

---

Two candidate decoders:
h(x)) h(x)) = h((2)
simple
h(2)
sample point from 1D menrfold (bive curvej in small regian 20 ambient
optimal sparse
Mx
representation
21
2
Figure 2: 2D illustration of the intuition behind Theorem See Section
3.3 for details.
To large extent, the intuition here mirrors the basic scenario from Figure I and is emblematic of broader situations that naturally arise in practice. For example, if we run PCA on MNIST data, we find that only a 100 or so principal components are needed to achieve highly accurate reconstructions. But VAE model with only around 15 informative latent dimensions can accomplish something similar [Dai et al. 2018 by closely approximating an optimal sparse representation using a nonlinear decoder. Of course unless have an objective function with a strong preference for lower- dimensional structures, as instantiated through large gradients around optimal sparse representations, then the network may well favor or converge to a simpler, higher-dimennionall alternative (e.g.. resembling a PCA solution).
Mitigating Unbounded Gradients via 7-Dependent Smoothing
While we have argued that unbounded gradients may serve a useful purpose in obtaining optimal latent representations, they may nonetheless pose challenges from an optimization standpoint. In addressing this concern, it is worth acknowledging that energy functions involving infinite gradients and/or unbounded regions are already indispensable across wide range of structured regression and sparse estimation problems [Gorodnitsky and Rao 1997 Rao et al., 2003). This history implies that when training VAE or other related AE model, we may borrow appropriate tools designed to mitigate the risk of converging to bad local solutions or regions of instability. In this vein, one effective strategy involves partially minimizing what amounts to smoothed version of the original objective function. The degree of smoothness is then gradually reduced as the optimization trajectory moves towards optimum. Within the domain of underdetermined linear inverse problems, this procedure is frequently used to find maximally sparse representations with minimal reconstruction error [Chartrand and Yin 2008 Hu et al.) 2012 Xu al. 2013
The VAE automatically accomplishes something similar when we choose to iteratively estimate y dur- ing training rather than merely setting its value to near zero may be theoretically optimal (assuming we know that there exists sufficient network capacity to achieve negligible reconstruction errors). Initially, when the reconstruction cost is still high because encoder/decoder parameters have not converged, the learned 7 will be larger and the overall VAE energy will be relatively smooth, devoid of
many deep local minimizers. It is only later as the data fit Σ E go (2(
[|ae(() pa (z; 0)]
becomes small that y will follow suite, and by this point it is more likely that we have already approached a basin attraction capable of producing optimal sparse reconstructions. Additionally, unlike fixing Ry 0 for all training iterations, in which case gradients will be unbounded right from beginning, by learning > we will likely only encounter large gradients a narrow neighborhood around minimizing solutions. This implies that in practice, we only need accommodate such gra- dients when the reconstruction error becomes small, at which point stability countermeasures can be deployed if/when necessary, e.g., reduced step size, checks for oscillating gradient sign patterns Riedmiller and Braun 1993]. etc.
To help visualize these points, in Figure 3 we have plotted 1D slices through the objective function of a simple VAE model involving a single layer for both encoder and decoder, applied data from random low-dimensional subspace. We vary 7 E {10-3, 10-2 2 10 .1), which exposes the increasing gradients and multi-modal nature of the objective function as 7 becomes smaller. Dashed vertical lines indicate the minimal value of the respective curve for each y. Additionally, we have explicitly designed the model underpinning this visualization such that there will exist an optimal sparse representation at zero on the z-axis. Consequently, we can readily observe that becomes
8

---

00001
normalized VÁE enargy
0.5
0011
0.1
normalized VAE energy
0.5
=001
=011
100
0.5
-0.5
-1
-0.5
-1
-0.5
0.5
1D through parameter space
1D slice through parameter space
(a)
(b)
Figure 3: Plots (a) and (b) show two sets of representative 1D slices through the VAE objective function (3) as the value of is varied. Dashed vertical lines indicate the r-axis location of the minimal value of each respective slice and y setting. And for both plots (a) and (b) the 1D slices are set such that an optimal sparse representation would occur at zero on the r-axis when y - 0. It can be observed that disconnected local minima only occur when y small.
sufficiently small, the minimizing value of the VAE energy increasingly aligns with an optimal sparse representation desired. However, as y is reduced the energy is less smooth and disconnected local minima appear in both ID slices. And local minima of the VAE loss surface can at times be risk points for under-regularized representations.
To further explore the implications of this 7-ddeendennt smoothing effect, we empirically compare practical scenario whereby learning y may be better than fixing an arbitrarily small value. To this effect, we first train VAE model on CelebA data [Liu et al. 2013) and learn an appropriate small value of y denoted 7 (note that 7 need not be exactly zero since with real data and limited capacity the network will generally display some nonzero reconstruction errors). Please see the supplementary for network and training details. We then retrain the same network from scratch but with y = ? fixed throughout all training iterations.
The resulting models are evaluated via the reconstruction error and the maximum mean discrepancy (MMD) between the aggregated posterior (z) - E. (=(2)(())) Makhhani et al. 2016) and the prior p(z) = (O, I). If too few latent dimensions are removed by swamping the appropriate channels with noise following the prior (i.e., under-regullarization)) then we would expect go (z) be
confined near a low-dimensional manifold in I" and the MMD to be much larger. Note that for ideal generative modeling performance via an autoencoder architecture, is required that
1 Σ (z((((t)))
2J
(o(2)z)) (dæ) = p(=),
(10)
meaning the MMD from . (0, I) is ideally zero Makhzani et al. 2016). With manifold data this only possible if an optimal sparse representation is produced by the VAE or autoencoder-basee
analogue [Tolstikhin et al. 2018).
Results displayed in Figure 4(a) where expected the reconstruction errors nearly identical, but the learnable case leads to much lower MMD values, indicative of a better local solution with
reduced under-regularizztioo. We also plot the evolution of the gradient magnitudes dC(8. dz in Figure 4(b) (other gradients are similar). When is learned, the gradient increases slowly; however, with fixed y = 7", there exists a large gradient right from the start since 7 is small but the reconstruction error is high. This contributes
worse final solution per the results in Figure 4(a) Additionally, examples of using a learnable y to improve generated sample quality based on these principles can be found in DDai and Wipf 2019].
We close this section with one notable caveat: Although learning 7 can be beneficial for the reasons we have given, it is not panacea and in certain situations there can be unintended consequences. For

---

350
300
Leamable Y
Mranient
Magnitut
250
Fix
200
CelebA
150
Rec. Err.
MMD
Learnable y
352.8
93.3
100
Fix 7 7
349.9
291.8
50
0
100
200
300
400
Iterations (x100)
Figure 4: (a) Reconstruction error and MMD between (z(2) and (N I) on CelebA (128 x 128 resolution). We first train VAE with learnable y and obtain the optimal value y Then we fix 7 and re-train the same network from scratch. While the final reconstruction errors are almost the same, the MMDs between (=(2) and the prior (O, I) are significantly different. (b) The
Evolution of the gradient dC(0.6) Although both curves end up with similar final values, the
dz 2
large initial gradient with fixed y is disruptive to the final solution.
example, if particular VAE model experiences posterior collapse during training, then it may be necessary to place an upper bound on to help reduce the collapse risk.
5
Conclusion
It is not uncommon to learn the VAE decoder variance parameter in situations where the training data has a noise component that we are unable or do not wish to model. By doing so we can avoid tuning trade-off parameter while allowing the model to adapt to the data. However, with sufficient capacity networks and relatively clean data, the risk of unbounded gradients when training Y has frequently been raised as potentially problematic phenomena. We nonetheless provide formal justification for this choice (even in cases where does tend to zero) on two primary fronts:
We prove that unbounded gradients are in fact necessary for guaranteeing that global minima of canonical AE architectures will coincide with optimal sparse representations, meaning high fidelity reconstruction of the training data using the minimal number of informative latent dimensions. Hence there no obvious alternative if this form of parsimony is our goal. Furthermore, given the value of such representations to numerous downstream tasks as described in Section 2.2, our analysis suggests that heuristic modifications to constraints on the VAE energy function may be ill-advised, and large gradients should be accommodated to the extent possible (e.g., reduced step size, checks for oscillating gradient sign patterns, etc.).
We present compelling evidence that by learning y, large gradients away from global minimizers, as well as at least some bad local minimizers, can be mitigated or smoothed within the VAE loss surface. This helps to explain observed successes learning 7 in situations where the optimal value turns out to be small or near zero. Note that as mentioned in Section [ it is already known that fixing y too high posterior collapse [He et al., 2019 Lucas et al., (2019 Razavi et al.] 2019). In a similar vein, we have demonstrated the complementary yet underappreciated fact that prematurely fixing too low, even to what may ultimately be the optimal value near zero, can steer convergence towards under-regularized local minima and the inadvertent wasteful deployment of latent degrees-of-freedd ffreeddom......mmmmmmmm
lead over-regulaiization and the stuely-studied phenomena of
And finally, although not our focus, our results herein naturally relate to more flexible VAE models with Gan-Gassiian latent posteriors Kingma et al., 2016 Rezende and Mohamed, 2015) or adapt- able/trainable priors Bauee and Mnih) 2019 Tomezak and Welling 2018). While these types of enhancements can be useful tools for favoring (=) p(=), they do not circumvent the infinite gradients that will occur around optimal sparse representations.
10

---

Funding Transparency Statement
Some initial components of this work were conceived while Bin Dai was an intern and David Wipf was FTE Microsoft Research in Beijing. Additionally, K. Wenliang contributed as intern and David Wipf as an FTE at the AWS Shanghai Al Research Lab. There are no other sources of funding to report.
References
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2:1-18, 2015.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang. and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint arXiv:1802.014300 2018.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In International
Conference on Artificial Intelligence and Statistics, 2019.
Alexandre Belloni, Victor Chernozhukov. and Lie Wang. Square-root lasso: Pivotal recovery of sparse signals via conic programming. Biometrika, 98(4):791 1906,,, 2011.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3798) 2013.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foun- dations of Computational mathematics, 96):717-772, 2009.
Rick Chartrand and Wotao Yin. Iteratively reweighted algorithms for compressive sensing. In International Conference on Accoustics, Speech, and Signal Processing, 2008
Yichen Chen, Dongdong Ge, Mengdi Wang, Zizhuo Wang, Yinyu Ye, and Hao Yin. Strong NP- hardness for sparse optimization with concave penalty functions. International Conference on Machine Learning, 2017.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint 170.:0514... 2018.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In International Confernece on Machine Learning, 2020.
D.L. Donoho. Compressed sensing. IEEE Trans. Information Theory, 52(4):128- 2006.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. American Statistical Association, 96(456):134--13 2001.
Partha Ghosh, Mehdi Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12438 2019.
Irina Gorodnitsky and Bhaskar Rao. Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions signal processing, 45(3):600 616 1997.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick Lagging inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2019.

---

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. 5-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Yue Hu, Sajan Goud Lingala, and Mathews Jacob. A fast majorize- algorithm for the recovery of sparse and low-rank matrices. IEEE Transactions on Image Processing, 21(2)7742-733, 2012.
Diederik Kingma and Max Welling. Auto-cncoding variational Bayes. In International Conference Learning Representations, 2014.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems. 2016.
Xiaopeng Li and James She. Collaborative variational autoencoder for recommender systems. In International Conference on Knowledge Discovery and Data Mining, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In International Conference on Computer Vision, 2015.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior
collapse in generative latent variable models. In International Conference on Learning Representa- tions, Workshop Paper, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2016.
Pierre-Alexandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variables 2018.
models. arXiv preprint
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems. 2018.
Andrew Ng. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
Jason Palmer, David Wipf, Kenneth Freutz-Delgadee and Baskar Rao. Variational EM algorithms for non Gau-Siass latent variable models. In Advances in Neural Information Processing Systems, 2006.
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv: 104.210444.844 2021.
Bhaskar Rao, Kjersti Engan, Shane Cotter, Jason Palmer, and Kenneth Kelgzz-D Subset selection in noise based on diversity measure minimization. IEEE Trans. Signal Processing, 51(3): 760-770, 2003.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with 5-VAEs. In International Conference on Learning Representations, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: 2015.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv preprint arXiv:1810.00530 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In International Conference on Neural Networks, 1993.
12

---

Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations. 2018.
Jorge Silva, Jorge Marques, and João Lemos. Selecting landmark points for sparse manifold learning. In Advances in Neural Information Processing Systems 18, 2006.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student-
t variational autoencoder for robust density estimation. In International Joint Conference Artificial Intelligence, 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
llya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
Jakub Tomezak and Max Welling. VAE with VampPrior. In International Conference Artificial Intelligence and Statistics, 2018.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, 2016.
David Wipf and Yi Wu. Dual-space analysis of the sparse linear model. In Advances in Neural Information Processing Systems, 2012.
Haowen Xu, Wenxiao Chen, Nengwen Zhao Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In International World Wide Web Conference, 2018
Li Xu, Shicheng Zheng, and Jiay Jia, Unnatural lo sparse representation for natural image deblurring. In Computer Vision and Pattern Recognition, 2013.
13


Reviews:
[{"review_id": "ZwchPyuT_F", "reviewer": "Reviewer_VPWz", "summary": "This paper studies the infinite gradient of training VAEs, namely when the decoder's variance is a learnable parameter, the variance will converge to zero, resulting in unbounded gradient. Although previously this is seen as an undesirable property, the authors prove that unbounded gradients are necessary for obtaining optimal spare representations. The authors also try to argue that learning the variance is better than fixing it to an arbitrarily small value.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper address an important issue in VAE training. It is widely observe that when the decoder variance is learnable, it will converge to a small number near zero. This paper make a nice connection between this phenomenon and optimal sparse representations, which is arguably the goal for learning an AE or VAE. By proving that it is inevitable to avoid the unbounded gradient while minimizing a generalization of AE loss, the adaptive decoder variance is justified.\n\nI have several questions regarding the paper.\n\n1. When you study the learning of decoder variance, do you consider the effect of initialization? Typically when we learn the decoder variance, it is initialized as 1. However, what if we use other initializations, for example, initialize it at the convergence value, but still set it to be learnable? Maybe it is interesting to study different initializations using the dynamics in figure 4b.\n\n2. I have a question regarding your statement that the using learnable gamma is important for improving sample quality. However, in my understanding, since learnable gamma will result in very small value of gamma, in the end the VAE model is still somewhat under regularized, and directly sample from it should not lead to good sample quality. Previously this is resolved by training another generative model on the aggregated posterior, but I think the latent model is something else, and strictly speaking the sample obtained by two-stage models are not samples from the VAE. Maybe the authors can discuss a bit on this.\n\n3. Regarding the OOD detection experiments, since the log likelihood (or equivalently BPD) of the VAE is largely dominanted by the reconstruction loss, and you show that learning gamma and fixing gamma at a small value leads to similar recon loss, do you observe similar performance of OOD detection when gamma is fixed at a small value? Intuitively they should behave similar here.\n\n\nAfter rebuttal:\nThanks for addressing my concern, I will keep my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "MZJ-TAsICJ2", "reviewer": "Reviewer_f5mJ", "summary": "The paper investigates a specific work regime for VAE characterized by close to infinite gradients. The authors states that this phenomenon occurs for small values of a regularization power $\\gamma$ and cannot be avoided, if one wants a perfect data reconstruction.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Clear analysis of presented phenomenon with a theorem, nice illustrations, profound investigation.\n\n- Present explanation of a well-known effect during the VAE training.\n\nCons:\n- The paper provides only one possible reason for large gradients. However, in practice it can be the case, that infinite gradients phenomenon occurs due to different reasons, so there is no guarantee that the paper finally solves the problem.\n\n- Optimization of gamma refers to Bayesian problem statement, thus a full Bayesian discussion of the problem at hand can provide more intuition on what is going on, and how can we avoid the infinite gradient effect with a proper prior on parameters.\n\n- Formula (6) – why the objective function has h in the first term, as usually we don’t use any h() for the training of AE?\nMaybe there exists another criterion for AE learning, that doesn’t require assumptions of Theorem 4 to get the perfect reconstruction with at most r nonzero rows of Z (for example, we impose only some regularisation on z in the second term in (6)?)? \nUnder which conditions on data we get that $x^{(i)} = \\mu_x(z^{(i),\\theta})$ requires that $\\|z_k\\|_2>0$ for at least r<d rows?\n\n- Does the infinite gradients phenomenon occur in other types of models? What makes this statement unique? What is with GMM or Kernel PCA or bayesian regression or PCA? E.g., in case of kernel PCA we can also require the data interpolation property. Will we observe the same problem?\n\n- Assumption in formula (4) seems unrealistic given typically noisy settings for VAE and for statistics in general. It can lead to bad solutions making further arguments invalid. A discussion is needed.\nDo we observe the infinite gradients phenomenon in case we do not require the interpolation property?\n\n- The present work uses insights from \nB. Dai and D. Wipf. Diagnosing and enhancing VAE models. In ICLR, 2018.\nI would expect some discussion on the relation of the new results and the results from that paper.\n\n- I understand that in case of sufficiently clean data, lying on the manifold, the interpolation property is a reasonable requirement. According to the authors, we have to tune gamma to achieve it. Are there any specific algorithms for tuning gamma, not just general advices?\n\n- The optimization problem in (6) follows from the VAE optimization problem (3) only if sigma_z is constant and gamma is small. Theorem 4 are proved for the optimization problem (6). I am OK to transfer the conclusions of Theorem 4 to the VAE case when gamma is small. But the requirement of constant sigma_z is not obvious. What does happen if we model sigma_z with some neural network and tune its parameters? Will we observe the same effects? Maybe adaptive sigma_z will compensate the instability?\n\nIn general, the paper is interesting, but the assumptions sometimes look too narrow or results are not formally connected, I need more clarifications.\n\nMinor comments:\n- Line 103 – should it be minimal instead of maximal?\n\n- Line 325 - \"spare representations\". Maybe \"sparse\"?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.0"}, "extra_text": {}}, {"review_id": "LjLygXgsFlf", "reviewer": "Reviewer_69AE", "summary": "Summary. This paper tries to analyze the regularization trade-offs for variational auto-encoding (VAE) models, in particular, explaining the infinite gradient phenomena in the training of VAE models. The author(s) defines some concepts to carry out the analyses. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===== After Rebuttal =====\n\nPlease refer to my \"After Rebuttal\" post.\n\n=====================\n\n~~This is clearly not a NeurIPS level submission. I know it is quite frustrating to receive a rejection, so I have patiently read this manuscript until page 5, trying my best to prepare as much feedback as possible to help the author(s) improve this work, then I gave up. Its contents are simply not technically sound. I scrolled down to the Experiment section and found out it is not adequate as well, so here I am writing this review. The language issues in writing (sentences too long, grammar issues, bad expressions, etc.) I can tolerate them, but there are factual errors in this paper, so my evaluation is a firm no. The author(s) have some fundamental misunderstanding of variational inference models, as reflected in the definitions, analysis, and examples presented in this work. This work will be of limited value to the community. See more detailed comments in the limitation section below.~~", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Infinite gradients or overfitting? It turned out the infinite gradient issue the author(s) heavily focused on is caused by letting the precision parameter of the Gaussian noise model be trainable, which always explodes to infinity. Well, it is well known this is simply the overfitting in the finite sample scenario, and of course, it will drive the gradient to infinity. \n- The PCA example is inconsistent with the optimal sparse representation definition, as its latent representation has no uncertainty at all. \n- Reconstruction error is just an intuitive explanation of likelihood loss under the Gaussian observation model. For a proper theory of variational inference, the concepts should be instead defined by conditional likelihoods. \n- Please use vector formats instead of bitmaps in the Figures.\n- The experiments need to be substantially enriched. \n- I do not feel the author(s) have shed real valuable insights to guide practice. What does the statement \"this result suggests that heuristic modifications to or constraints on the VAE energy function may be ill-advised, and large gradients should be accommodated to the extent possible.\" even mean? Such confusing. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "E387FJU3qpl", "reviewer": "Reviewer_ATkA", "summary": "**Update after the authors' response**: I appreciate the detailed authors' response, and while it clarifies some points the paper remains a borderline case to me unless the applicability of the findings in the paper was clearly acknowledged and the breadth of the main claims was reduced accordingly. \n\n**Update after further engagement with authors**: We have managed to clear up some misunderstanding / misinterpretation from both sides, and I have raised my score accordingly from a 5 to a 6.\n\nThe paper investigates Autoencoders with a Gaussian likelihood model, where the variance is a trainable parameter. As has been observed before, this often leads to learned variances approaching zero (particularly for high-capacity models and noise-free data), which in turn leads to exploding gradients. The paper argues that these gradients and the collapse of the variance towards very low values should be embraced, rather than being seen as a pathology of the model. In particular the paper argues that under Gaussian likelihood and a fixed latent-space prior, if the goal is to learn a lossless representation of the dataset (which implies a high-fidelity generative model) near unbounded gradients are necessary to avoid underfitting (i.e. using unnecessary latent dimensions). Finally the paper proposes a scheme to mitigate optimization issues with unbounded gradients and illustrates the scheme on CelebA\n\n**Contributions:**\n\n1) Analyzing the origin of unbounded gradients of Gaussian likelihood (V)AEs with learnable (co-)variance. Significance: unsurprisingly, the source of the problem is the Gaussian likelihood term, which is unbounded for the variance approaching zero. The latter has been pointed out various times in the VAE literature and is a well known pathology when fitting Gaussian (mixture) models. \n\n2) A definition of optimal sparse representations as allowing for zero reconstruction error while only using a minimally required amount of dimensions (lossless compression). Significance: I think the definition can be useful for some particular applications (most notably training a high fidelity generative model), but in practice I would argue there are many clear cases where zero reconstruction error is clearly undesirable for an optimal representation (because the data contains noise or other task-irrelevant variations that should not be captured by the representation). This is well discussed in the literature on lossy compression and representation learning (rate distortion theory and information bottleneck literature and their modern application to neural networks and auto-encoder-like architectures).\n\n3) Showing that, under said definition, “optimal sparse representations” are only attainable via near-unbounded gradients for any AE type architecture. Significance: I think this claim is at best overstated, at worst wrong: since Gaussian likelihoods are the root of the gradient explosion, the findings do not apply for other likelihood models that do not suffer from the pathology of unboundedenss (under a certain parametrization), such as e.g. Bernoulli likelihoods. The claim thus only applies to Gaussian likelihoods and other likelihood models with unbounded values, but not any AE architecture. Please correct me if my understanding (on the reliability of the findings on Gaussian likelihoods) is wrong.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality, Quality, Clarity, Correctness:**\n\nThe paper follows up on a previously pointed out pathology, and shows that this pathology is unavoidable and rather than working around it (by limiting gradient magnitude), it should be incorporated into the optimization process (leading to a variance-smoothing scheme). The technical part of the paper is well written, but important conceptual assumptions are not discussed (many works would disagree that optimal representations are lossless), and some claims are made w.r.t. arbitrary AE models where, as far as I can tell, all the claims strictly only apply to Guassian-likelihood models (and other likelihoods that grow unbounded when fitting noise-free data) with fixed latent-space prior distribution (i.e. no aggregate posterior). Some of the claims (please correct me if I’m wrong) no longer apply when either of these assumptions is not met. The experiments shown are insightful, but since only a single experiment with the proposed variance-smoothing scheme is shown, it remains somewhat unclear how well the  corresponding results generalize and lead to claimed improvements - particularly since comparisons to other methods are missing and the sparsity of the solution obtained is not computed.\n\n**Verdict:**\n\nOverall I am on the fence with this paper. The analysis shown in the paper seems correct, but the findings are not very surprising, given that the likelihood model is Gaussian. Some of the claims (no optimal representation with any AE without near-unbounded gradients) sound very surprising, but only because they are stated without the explicit assumptions under which they hold. My two main concerns (see more details below) are: one, the analysis and results seem not to apply to well behaved likelihood models - this is never stated in the paper and should be clearly spelled out. Two, the definition of optimal representations (as lossless compressions) is too broad - it might be appropriate in some tasks, but it is clearly inappropriate in other tasks (where lossy compressions are desirable, e.g. for generalization, or under noisy data). This needs to be discussed thoroughly. I am therefore leaning towards rejection, noting that these issues can potentially be sufficiently addressed in the rebuttal. However since these main concerns regard multiple passages throughout the paper it is hard to judge whether a revised version of the manuscript would address the criticism sufficiently, without seeing the actual revised version. Either way, I am happy to revise my final verdict based on the other reviews and the authors’ response.\n\n**Pros:**\n * Thorough analysis of the main problem and pointing out that under a number of assumptions the main problem cannot be avoided\n * Proposal on how to incorporate the problem into a more numerically stable optimization process.\n * Concrete definition of the goal (optimal sparse lossless representations)\n\n**Cons:**\n * Claims stated in very general terms, despite depending on fairly specific assumptions (which, admittedly, are standard modelling assumptions in large parts of the VAE literature).\n * Main pathology results from unbounded likelihoods which has been identified as problematic before. Discussion of these previous works falls a bit short.\n * Experimental findings for the proposed variance-smoothing look OK but there is not a lot of experimental evidence to back them up, and the single experiment that was run does not have multiple repetitions such that the stat. significance of the empirical findings remains unclear (though I do expect most findings to be significant, the margins look fairly good).\n\n**Improvements:**\n\n1) I disagree with Definition 1 (in the broadest sense): optimal sparse representations for many tasks are produced via **lossy** compression, implying non-zero construction error. This includes tasks like representation learning, disentanglement, outlier detection with noisy data - in fact any task that assumes that the data is not noise-free (hence each datapoint contains some irrelevant information which should be removed via lossy compression). There are numerous papers the topic of what constitutes good representations, perhaps most notably\n * The Information bottleneck method, Tishby et al. 2000\nBut also:\n * Emergence of Invariance and Disentanglement in Deep Representations, Achille et al. 2018\n * Fixing a broken ELBO, Alemi et al. 2018.\n * Deep Variational Info Bottleneck, Alemi et al. 2017\nThis needs to be discussed in at least a paragraph, and the definition needs to be weakened to “optimal lossless representations for noise-free data” or similar (I’m not one of the authors, so certainly not asking for citations but would like to see a proper discussion).\n\n1a) Clearly point out in which applications the lossless compression is desirable, and in which applications it is not desirable. Section 2.2 is currently biased towards one side.\n\n1b) Outlier detection with low \\gamma (and the illustration in Fig. 1) assumes (almost completely) noise-free data that lies on a smooth manifold perfectly captured by the model (i.e. perfect inter- and extrapolation). This rarely holds in practice and should be discussed in a more nuanced fashion.\n\n2) Please (throughout the paper) make it clear that the main claims only apply to models with Gaussian likelihood (and trainable variance, but the latter is made sufficiently clear in the paper). Some of the claims sound like they would apply to any AE with arbitrary likelihood models. This is not meant to reduce the significance of the paper (after all I would say that the paper addresses the most widely used VAE model, hence it is of high significance) but to sharpen the focus and generalizability of the claims and results. I personally also think that the community should think more carefully about the standard modeling assumptions and search for alternative likelihood models whenever appropriate. The latter message is missing completely from the paper.\n\n3) If the goal is to learn a high-fidelity image generator (or other generative model) or density modeling of noise-free data then a reconstruction error of 0 is indeed desirable, however I personally would question whether variational autoencoders are the right choice in these situations (admittedly they have been used in the literature in this context). It should be fairly obvious that a Gaussian likelihood with learnable variance is a wrong modelling choice in such a situation (I would even argue that having a fixed prior is often a wrong modelling choice, particularly when underfitting is an issue). Alternatives should be discussed (e.g. flow-based models, Bernoulli likelihoods, “aggregate posteriors” as priors, …). Please make an effort here to point out alternatives, I am convinced that readers will greatly benefit from that and it will help put the paper into wider context.\n\n4) Sparsity as discussed in Definition 1 (ii) (line 105) is achieved by pushing the posterior towards the prior for non-informative dimensions. It has been discussed before (info bottleneck literature, fixing a broken ELBO) how this can be achieved by implementing a proper lossy compression scheme (following rate distortion theory) where, crucially, the prior is not fixed but given by the marginal or “aggregate posterior”. Famously, this leads to unneeded dimensions not being used (their distribution is the same as the prior which is the aggregate posterior). I therefore disagree with Line 327: “Hence there is no obvious alternative if this form of parsimony is our goal” - sparsity can also be achieved by using an aggregate posterior. This is also what Tomzcak and Welling 2018 do (but also e.g. Deep Variational Info Bottleneck which uses a variational approximation to the marginal). Please discuss/clarify.\n\n\n**Comments:**\n\n * Line 345: I only partially agree. Adaptable/trainable priors should help with getting sparse representations - if they use Gaussian likelihoods they will not prevent infinite gradients of course (see also Improvement 4) above).\n * Line 219-220: Given my comments above (in the improvements) I think this statement needs to weakened or it needs to be clarified under what assumptions the statement holds (Gaussian likelihoods, fixed piors, and the precise definition of sparse optimal representation as Definition 1, but not for other, e.g. lossy optimal representations).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please add a paragraph with a focused discussion of the limitations of the proposed method (and perhaps reiterating the limited scope of the claims / theoretical analysis).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {"ethical_concerns": "No immediate ethical concerns. It would be nice to add a short statement regarding ongoing ethical debates with high-fidelity generative models, to which this work contributes (mostly through analysis though)."}}]

Author rebuttal:

Meta-review: The authors show that infinite gradients are required to recover optimally sparse representations of data in autoencoder models using a Gaussian likelihood with a learned variance.

The results in the paper appear both correct and interesting in the context of autoencoders, but the reviewers found some of the claims insufficiently precise and at times overstated, which is a serious issue for a theoretical work. Most notably, they were not convinced by the claims of the applicability of the results to VAEs (e.g. the zero reconstruction error as one of the assumptions). The back-and-forth with the reviewers provided some necessary clarifications and explanations, and generated some excellent suggestions for improving the paper. Unfortunately implementing the changes required would amount to a major revision and thus is not feasible without resubmission. The paper would also really benefit from a section covering prior work on infinite gradients, including a detailed discussion of the relevant results in Dai and Wipf (2019) and their relationship to those in this paper.
Decision: Accept (Spotlight)