On the Value of Infinite Gradients in Variational Autoencoder Models
Authors: Bin Dai
Li Kevin Wenliang
David Wipf
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: variational autoencoders
sparse representations
latent variable models

Abstract: A number of recent studies of continuous variational autoencoder (VAE) models have noted, either directly or indirectly, the tendency of various parameter gradients to drift towards infinity during training.  Because such gradients could potentially contribute to numerical instabilities, and are often framed as a problematic phenomena to be avoided, it may be tempting to shift to alternative energy functions that guarantee bounded gradients.  But it remains an open question: What might the unintended consequences of such a restriction be?  To address this issue, we examine how unbounded gradients relate to the regularization of a broad class of autoencoder-based architectures, including VAE models, as applied to data lying on or near a low-dimensional manifold (e.g., natural images).  Our main finding is that, if the ultimate goal is to simultaneously avoid over-regularization (high reconstruction errors, sometimes referred to as posterior collapse) and under-regularization (excessive latent dimensions are not pruned from the model), then an autoencoder-based energy function with infinite gradients around optimal representations is provably required per a certain technical sense which we carefully detail.  Given that both over- and under-regularization can directly lead to poor generated sample quality or suboptimal feature selection, this result suggests that heuristic modifications to or constraints on the VAE energy function may at times be ill-advised, and large gradients should be accommodated to the extent possible.

On the Value of Infinite Gradients in Variational Autoencoder Models
Bin Dai Institue for Advanced Study
K. Wenliang
Gatsby Computational Neuroscience Unit University College London
Tsinghua University
caibo@plyysicc@@hoomall...coo
kevinli@gatsly ucl. ac uk
David Wipf
Shanghai Research Lab
Amazon Web Services
davidwipf@ggail.comm com com
Abstract
A number of recent studies of continuous variational autoencoder (VAE) models have noted, either directly indirectly, the tendency of various parameter gradients to drift towards infinity during training. Because such gradients could potentially contribute to numerical instabilities, and are often framed as problematic phe- nomena to be avoided, it may be tempting to shift to alternative energy functions that guarantee bounded gradients. But it remains an open question: What might the unintended consequences of such restriction be? To address this issue, we examine how unbounded gradients relate to the regularization of a broad class of autoencoder-basee architectures, including VAE models, applied to data lying on or near a oww-iimesioonl manifold (e.g., natural images). Our main finding is that, if the ultimate goal is to simultaneously avoid over-regularization (high reconstruction errors, sometimes referred to as posterior collapse) and under- regularization (excessive latent dimensions an autoencoder- baseed energy function with infinite gradients around optimal rep- resentations is provably required per certain technical sense which carefully detail. Given that both over- and under-regularization can directly lead to poor generated sample quality suboptimal feature selection, this result suggests that heuristic modifications to or constraints on the VAE energy function may at times be ill- advised, and large gradients should be accommodated the extent possible.
not pruned from the model), then
Introduction
Suppose we have access to continuous variables x that are drawn from ground-truth measure Mgt- This measure assigns probability mass Pge (dxz) to the infinitesimal dæ residing within C Rd such that we have [x Myt (dx)
This formalism allows us consider data that may lie on or near an d), capturing the notion of low-dimensional
r-dimensional manifold embedded in Rd (implying structure relative to the high-dimensional ambient space.
Because of the possibility of unknown latent manifold, it is common to approximate the corre- sponding ground-trute measure via a density model parameterized as
(x = f po
(1)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this expression A are trainable parameters and x E RR serves as low-dimensional latent repre- sentation, with fixed prior p(z) (O I) and ideally x 2 r. If some 8 were available such that [A po- (ar (a)d z [A Mgg( (dx) for any measurable A C x. then the model would adequately reflect the intrinsic underlying distribution. Of course we will generally not know in advance the value of 0*, but in principle we might consider minimizing - log Ps (a:) averaged across a set of training samples
(x(() Ji=1 drawn from Mgt, i.e., minimize +/4 - -log [po (a())) ~ J -llog [po(2)] Ugg (dar)
over 0. Unfortunately though, the marginalization required to produce pe (ax(1) is generally in- tractable for models of sufficient representational power. To circumvent this issue, the variational
autoencoder (VAE) [Kingma and Welling 2014, Rezende al., 2014] instead optimizes the tractable
variational bound 6) -
n -log po (= 12
E Ba (2)200)
[logpo po [z) + KL [0(() M[(())
1
(2)
Here (=(z))) represents a variational approximation to pa (z|xx) with additional parameters ó gov- erning the tightness of the bound. It is commonly referred to as an encoder distribution since
quantifies the mapping from æ to the latent code z. For analogous reasons, po(a)22) is labeled as the decoder distribution. When combined, the data-dependent factor - (2(0)) [log po (a/|²] can be viewed as instantiating form of stochastic autoencoder (AE) structure, which attempts to assign high probability to accurate reconstructions of each æ;; if 9s (z|zz) is Dirac delta function, then regular deterministic AE emerges with loss dictated by the decoder negative log-likelihood
- log pu ae ))) Beyond this, KL [1o (z (x)) [[x(2)] serves as a regularization factor that pushes the
encoder distribution towards the prior. The bound (2) can be minimized over {0, Φ) using SGD and simple reparameterization trick [Kingma and Welling 2014 Rezende et al., (014).
The latter requires that we assume specific functional form for the encoder distribution. In this regard, it is common to select Ia (z .x) =>(11t [
where the Gaussian moment vectors H. and a= are functions of model parameters Φ and the random variable æ. i.e., H4= = f (æ; Φ), and o= = o= (x;; Φ). Similarly, for continuous data the decoder model is conventionally parameterized as Pe (z) 2) = N(x p I), with mean defined analogously as H4, = 4 (z; 0) and scalar variance parameter 7 > O. The functions μı (#; og Φ), and n4, (z; are all instantiated using deep
neural network layers. Given this definitions, can be expressed in the more transparent form
(() =
1
E as (2(22(()) [1 (1x((0) Me (z; 0) [13 dloog
(3)
1=1
+ o= (z)(0) )12 log diag [o2 (a of .)
+ | (2(0;; b0) ||.}
Although VAE models have been successfully applied to a variety of practical problems Li and She 2017, [chott al. 2018 Walker al.) 2016). at times they exhibit potentially problematic behavior that has not been fully investigated. For example, a number of recent works have mentioned that if a trainable decoder variance parameter y is included within a VAE as in (3), then the optimal value may converge to zero resulting in infinite or unbounded gradients and potential instabilities (Dai and Wipi Mattei and Frellsen 2018 Rezende and Viola 2018 Takahashi et al.) 2018). And we emphasize that this phenomena can occur even within the confines of the stated Gaussian assumptions and inevitable regularizution effects of the KL term. While these unbounded gradients may indeed be troublesome from optimization perspective, in this work we will reframe such gradients as an integral part of successful autoencoder- based energy functions designed to model (in a sense that will be precisely quantified below) continuous data arising from low-dinensional manifold.
To accomplish this, our analysis is split into three parts. First, in Section 2 we detail how unbounded gradients contribute to an optimal, balanced form of regularization, allowing the VAE to capture oow-dimensional manifold structure via maximally parsimonious (and lossless) latent representation. Such representations turn out to be critical for tasks such as generating non-blum non-blurry samples that resemble the training data (Tolstikhin et al. 2018), or for using autoencoder-bel based models in general to robustly screen outliers [An and Cho 2015 Xu et al.) 2018). Of course it is natural to consider whether these same goals could not be achieved using an alternative energy function with strictly bounded gradients.
2

---

The second and primary component of our contribution answers this question in the negative. More concretely, our main result from Section B proves that canonical autoencoder-basee architectures will necessarily require unbounded gradients to guarantee the type of maximally parsimonious latent representation mentioned above. Thirdly, in Section we elucidate the benefits of learning 7 during training, even in situations where we know that the optimal value will be at or near zero and contribute to arbitrarily-Iarge gradients. In particular, we argue that (at the very least) learning 7 localizes troublesome unbounded gradients to narrow regions around minima of (3), while simultancously smoothing the VAE objective across optimization trajectories prior to convergence.
Overall, our contribution can be viewed as complementary to the wide body of work analyzing what commonly-refere to as posterior collapse in VAE models [He et al. 2019] Razavi et al. 2019].
The latter can be related to the situation where is too large (either implicitly [Dai et al. 2020
or explicitly [Lucas et al. 2019) and along all or most latent dimensions the posterior do x x0))
collapses to the prior N NO, leading to high reconstruction errors. In contrast, we direct our attention herein to the opposite condition whereby y is arbitrarily small and unbounded gradients invariably ensue. In this regime, the resulting latent representations obtained from bad local minimizers can potentially be under-regularized in sense that will be described in subsequent sections.
Optimal Low-Dimensiee
Structure via Unbounded VAE Gradients
As alluded to previously, the VAE objective will experience unbounded gradients if y 0 as has sometimes been observed (at least approximately) during training. But perhaps counter-intuitively, this phenomena nonetheless serves a critical purpose in the context of modeling data with low- dimensional manifold structure. To quantify this assertion, Section 2. will first precisely define what type of now-dimensional or sparse latent representations will be considered optimal for our present analysis; later in Section 2.2
we link this definition to practical VAE/AE applications.
2.1 Optimal Sparse (Lossless) Representations
An autoencoder- architecture (VAE or otherwise) with decoder the (:;6), con- straint E O, and arbitrary encoder μtg component produces optimal sparse representation of a training set X w.r.t. if the following two conditions simultaneously hold:
Definition
(i) The reconstruction error is zero, meaning
x (3) Mo [22 (0; :0: =0
(4)
(ii) Conditioned on achieving perfect reconstructions per criteria (i) above, the number of latent dimensions such that μzz (z(()) Φ)); 0 for all is maximal across any O E ② and any encoder function μ... A j-th latent dimension so-defined provides no benefit in reducing the reconstruction error and could in principle be removed from the model.
Remark Conceptually, this definition is merely describing the most parsimonious latent repre- sentation of the training data (conditioned on the available capacity of the decoder) that nonetheless allows us to obtain perfect reconstructions. And when combined with the low-dimensional manifold assumption from Section [] it readily follows that an optimal sparse representation of X will gen- erally involve x uninformative dimensions, assuming k 2 and adequately parameterized decoder family As an illustrative example, for data lying on a low-dimensional linear subspace,
'The encoder μ, function is allowed to be unconstrained here since, unlike the decoder, it does not contribute over-fitting (in principle
infinite capacity encoder be used). Additionally. the VAE encoder from 3 will also have variance component; however, does not directly play a role Definition [ We will address the relationship between a and optimal sparsity in subsequent discussion.
*Obviously the decoder requires sufficient flexibility to capture the manifold structure; however, there finite sample regime, the decoder is allowed be arbitrarily complex, then in principle just a single nonzero latent dimension will always be sufficient to achieve zero reconstruction error regardless of the actual data structure. This form of degenerate VAE over over-fitting has
one additional nuance worth mentioning here.
been previously quantified in (Dai et al. 2018)
3

---

the corresponding optimal sparse representation obtainable via a linear decoder will be defined by the smallest subspace containing all of the data variance, i.e., the standard PCA solution.
Remark 2 Although Definition T may appear to involve overly restrictive assumptions, it nonetheless well-approximates practical situations of broad interest. For example, as has been quantified in recent study [Pope et al. 2021]. natural images do indeed have a very low intrinsic dimension relative to the high-dimensional pixel space. Hence these images can in principle be reconstructed almost exactly using oow-dimeniional representations. Moreover, many classical under-determined inverse problems have been framed in terms of obtaining perfect reconstructions of observed measurements subject to some minimal measure of parsimony (Candès and Recht 2009).
The particular lossless notion of optimality are adopting here is not meant preclude alternatives that may be tailored for different scenarios. Rather, the proposed definition is merely selected showcase class of VAE/AE usage regimes whereby infinite gradients can play an influential role. Consequently, lossy conceptions of optimal parsimony [Alemi et al. (016 Tishby al. 2000]. while useful in their own right, are laregly outside the scope of this work.
Remark 3
Remark Although the encoder is generally stochastic, prior analysis from (Dai and Wipf 2019 has revealed that the VAE global minimum is nonetheless capable of achieving something analogous to Definition]] | More concretely, for unneeded latent dimensions the posterior is pushed to the prior to optimize the KI. regularizer, i.e., 2o (zj (1) -0, 1) for all i, which amounts to uninformative noise that will be filtered by the decoder so as not to impact reconstructions. In contrast, for
informative dimensions the posterior variance satisfies d= (x();)) ;) + 0 for all i. Collectively, this allows the VAE global minima to achieve
E 90 (2)2)1)
ua [z: 0
pb p4= (2)
(5)
while relying on the fewest number of active latent dimensions, such that both criteria (i) and (ii) of Definition ean be simultaneously satisefied.
This capability requires that the VAE avoid both over- or under-regularivzaion of the latent repre- sentations. To be more precise, VAE over-regularizzaion (sometimes loosely referred to as latent posterior collapse [He et al, 2019 Razavi et al. 2019]) occurs when too many latent dimensions are uninformative (i.e., the latent posterior along these dimensions is close to the uninformative prior) such that the reconstruction error is high and criteria (i) is violated. In contrast, with under-regularized solutions criteria (i) may be satisfied, and yet in reducing the reconstruction error towards zero, an excessive number of latent dimensions are informative in violation of criteria (ii).
In avoiding both of these suboptimal scenarios, it can be shown that the VAE explicitly relies on - 0 and the attendant unbounded gradients that follow (Dai and Wipf 2019]. From an intuitive standpoint, might expect that achieving criteria (i) would require an unbounded gradient given that, if we minimize (3) over y in isolation, the optimal value satisfies
[|
Σ E 1E (2(00())
pr [2; 0
(6)
we then plug this value back into the dl log y term from 3 the result, as well as the corresponding gradients of other model parameters, is unbounded from below as the reconstruction error goes to zero (note also that in this instance, the original 1/7 data term becomes a constant, so there is no counteracting effect). Of course to actually achieve near-zero reconstruction errors, least some dimensions of a₂ must be pushed towards zero as mentioned previously, which can also lead to infinite gradients within the KL-divergence factor.
2.2 Relevance to Typical VAE Usage Regimes
Obtaining minimalist latent representations as distilled by Definition can serve a variety of practical downstream applications, such as feature extraction [Bengio et al. 2013 Ng 2011]. compression
[Ballé et al.) 2018 Donoho 2006 Minnen et al., 2018), manifold learning Silva et al.) 2006), corruption removal DDai et al.] 2018], or even the generation of realistic samples. With respect to
the latter, it has been shown in Dai and Wipi 2019) that what we have above defined as an optimal
sparse representation can be viewed as a necessary (albeit not sufficient) condition for generating

---

ambient space
inlier, train
inlier, test
2D principal subspace containing data
o outlier test
1D inlier manifold
Figure I: The importance of optimal sparse representations in screening outliers, In this example, the simple 2D principal subspace obtainable by PCA can perfectly reconstruct the inlier manifold shown in red. But this requires using two separate informative dimensions, allowing both inliers and outliers to be reconstructed with zero error within this subspace. In contrast, it is only by recovering the curved ID inlier manifold, which relies on single informative dimension, that inliers and outliers can be differentiated. Please see supplementary for practical example using real data.
samples using continuous-space Gaussian VAE that match the training distribution. In principle, deterministic AE architecture capable of producing optimal sparse representations can also be leveraged to generate realistic samples; this would simply involve first discarding the uninformative dimensions and then applying the same analysis from [Dai and Wipf. 2019]. In fact, variants of this strategy have been previously considered in Ghosh et al. 2019 Tolstikhin et al. 2018).
And as a final motivational example, any AE-based architecture capable of producing optimal sparse representations can naturally be applied to screening outliers by squeezing the latent space to the minimal number of informative dimensions needed for reconstructing inliers. In doing so, we reduce the risk that outlier points x. (out) can be accurately reconstructed by exploiting the superfluous latent flexibility. Here we are assuming that (out) ~ Pout F Hgt for some outlier distribution Hout- Figure contains an illustration of the basic rationale. The only exception to this line of reasoning would be adversarial outliers that follow the exact same low-dimensional structure as the inliers, meaning yout and Pgt both apply all of their probability mass to the same low-dimensional manifold. In this scenario, we would need to exploit differences between Pout and jgt within the manifold to reliably screen outliers, regime in which Definition is not directly applicable. That being said, differentiating Hout and Mgg once a shared low-dimensional manifold has been modeled is far easier than doing so in the original ambient space.
Additionally, in the supplementary we demonstrate that indeed, the inlier data (in this case Fashion MNIST samples) come from fow-dimensional manifold, outlier points (MNIST samples) can be reliably differentiated, provided that λ 2 r and the VAE has sufficient capacity and the learned y can converge to near zero. And because of the VAE's propensity to find optimal sparse representations where possible, even as x is raised such that x >> r, unneeded dimensions are shut off to reduce the risk of outliers masquerading as inliers (see supplementary).
2.3 Implications for 3-VAE models
The 3-VAE (Higgins al. 2017
represents a commonly-adopeee modification of the original VAE objective, whereby the KL term is rescaled by some fixed parameter 3 > O. For Gaussian VAE models (which our focus), this scale factor effectively makes no difference if a fixed decoder variance is adopted. In this situation, 33 can just be directly absorbed into y, and the dlo 7 normalization factor from (3) can be viewed as an irrelevant constant. However, if y is learned then 8 * will make non-trivial difference because of the imbalance introduced w.r.t. the now critical Gaussian normalization factor. In particular,
is too large (specifically / > d/r, where is the data dimension and is the manifold dimension), then optimal sparse representations will generally be impossible to achieve even while learning 7-
This is because, as can be inferred from the analysis in (Dai and Wipi 2019]. the VAE loss (when granted sufficient capacity) scales as (d-r) log y around optimal sparse representations as 7 becomes small. In this expression, the d log 7 factor is derived from the Gaussian normalization mentioned above, while the -r log 7 factor originates from the KI. term. However, if we scale the KI. term by B such that 8 > d/r, then (d - Br) log tends towards positive infinity as y becomes small, and the
5

---

VAE will instead sacrifice the reconstruction error such that optimal sparse representations are not possible.
3 Can we Reliably Obtain Optimal Sparse Representations without Unbounded Gradients?
As discussed in Section 2 given data originating from a now-dimensional manifold, optimal sparse representations are a necessary requirement (at least approximately) for various tasks such as gener- ating non-blurry samples aligned with the ground-truth distribution alternatively, screening for outliers. We have also discribed how the divergent gradients associated with y - O, allow VAE global minima to achieve such optimal sparse representations. But what about alternatives that circumvent such unbounded gradients altogether? For example, could we not consider a regularized AE model that, while encouraging sparse latent representations [Ng 2011]. explicitly relies on energy function terms with bounded gradients, e.g., as may be derived from a family of sparse penalty functions with bounded gradients [Chen et al. [017 Fan and Li 2001 Palmer al. [006] ? Despite this conceptual possibility, per the analysis that follows, the answer turns out to be unequivocally no within the stated context. Or more specifically, if we wish to guarantee an optimal sparse representation without additional assumptions on the decoder model and observed data, then even arbitrary AE-based objectives will necessarily require penalty terms with infinite gradients around optimal solutions.
3.1 A Generic AE-based Objective for Optimal Sparse Representations
Consider the constrained objective function
())
-
9
(0) Mx (20)
+ h 1111)) 1=1
dn
s.t. (1) = Mz (0) Vi, E θ,
where Z - [=10)] E R and z denotes the k-th row of Z. This expression can be viewed as characterizing a typical regularized AE with a generic penalty functions 9 R + R and h R+ + I the reconstruction error and the norm across training samples of each latent dimension, respectively, Additionally, the constraint 0 E θ included in 7 can, among other things, serve to prevent the trivial solution Z - O, which could occur if each x (t) is pushed to zero while the decoder n includes an unconstrained compensatory factor that grows towards infinity such that the error (+) ua (z (z()) f) 1l, can still be minimized to zero for all i. Any regularized AE must include such constraints to avoid trivial solutions, or else additional penalty terms on 0 that serve a similar purpose. Note also that if we happen to choose g = h, the provided multipliers 1/ /n,, 1/d, and 1/(dn) induce a form of proportional regularization within energy functions composed of multiple penalty factors of varying dimension designed to favor sparsity (Wipf and Wu 2012). The square-root Lasso can be viewed as a special case of this strategy that emerges when & is a square-root function [Belloni al. 2011]. However, for arbitrary selections of g and f (with any tunable trade-off parameter absorbed within), (7) reflects a broad family of AE architectures.
We can also relate 77 to various VAE instantiations. Define Ioe an indicator function satisfying Ioe (u) = 3o for u 7 0 and zero otherwise. We then have the following:
Lemma 2 Let ph_ (z; Wz + for some W E Bdxx and b E Rd, and a (æ; x) for any arbitrary 8 R=, Then the limit y + 0, the VAE loss from (3) is such that ming : (w;6) L (0, Φ) = ming (0, 6) reduces to 7 with g(-) = I-)(.) and h(.)) = Tog(?), excluding irrelevant constant factors.
Lemma 3 For any arbitrary μe (z;8) and O O if we enforce a= (x; o) + f all x and apply a log transformation to each ||2, then the VAE loss from (3) collapses to (7) with g(-) = h(-)) log(.), excluding irrelevant constant factors.
Collectively, these results point to close affiliation between 7 and the VAE loss, especially given
that 7 - 0 and o² (æx; Φ) O along many dimensions are characteristics of VAE global optima [Dai
and Wipf, 2019]. Hence it is natural to consider more general selections of g and h in the context of optimal sparse representations.

---

3.2 On the Difficulty Avoiding Unbounded Gradients
Given generic AE architecture as in (7. this section examines what possible functions g and h are such that global minimum of L. (6, @) is capable of producing an optimal sparse representation. This can be addressed as follows:
Theorem For any functions g R R and h IR+ E with bounded gradients, and any o Ji=t <Rdx and
dimension set {d, r) that order as d 2 BRRT >> 0, there exists data X = {x
decoder {μz, (z; ; E }} (with the capacity to reconstruct x lying within some parameterized family of k-dimensional manifolds) which satisfy the following:
(a) - rows and zero elsewhere.
- 1r [0 1 = 0 for some 0 E ④ and Z E R with | > 0 0.
(b) Minimizing C (0, 6) over 9 and any possible encoder produces either solution with
n -ا=1(=00)) Me (2))) 0 2 0 (i.e., imperfect reconstruction), or where || 0 for strictly more than r rows of Z (i.e., not maximally sparse).
1
This result effectively implies that, to guarantee every global minima corresponds with an optimal sparse reconstruction per our definition, irrespective of the decoder and observed data, the constituent penalty functions must have an unbounded gradient (at least around zero; see further intuitions below). This can be viewed as necessary, albeit not sufficient condition, for optimal sparsity, sufficiency requires additional care taking limits around zero, e.g., 7 - 0 in the case of the VAÉ. Consequently, we cannot simply replace VAE model with standard AE architecture to somehow guarantee optimal sparse representations devoid of infinite surrounding gradients (unless further assumptions on the data and decoder are introduced).
3.3 High-Level Intuition Behind Theorem 4
While the proof is predicated on a nuanced counterexample designed with a specific technical purpose in mind (see supplementary file), we can nonetheless loosely convey the basic idea through toy illustration shown in Figure 2 Here are assuming that the data points {2))) Ji=1 lie ID manifold embedded in 2D ambient space; the extension to higher dimensions is straightforward. Moreover, we stipulate that this manifold is tightly squeezed within a small non-negative e x square near zero, represented by the blue curve on the lefthand side of Figure Now consider sample point x'' [ x/2] taken from somewhere along the stated 1D manifold. We represent this point using two candidate decoder functions, both assumed to be within the capacity of p as displayed in the middle of Figure
For the simple decoder case, which is just the identity function (z, 0) = z, the values 21 = 2 and 22 = 2/2 needed for perfect reconstruction will both be small, i.e., {=1, x/2} S E by design. In contrast, the optimal decoder only requires that a single dimension of x, namely 21, be nonzero. However, the optimal value actually needed for perfect reconstruction, denoted 27, can be arbitrarily large in controlling where along the extended, labyrinthine manifold pathway x' is located (for case of presentation will assume 2 is also positive). Hence we can easily have that
z
max (z(,=6)
Because of this, ensure that =" [2] is preferred over the z' alternative, we require concave penalty function i on each encoder dimension such that any infinitesimal movement away from zero incurs arbitrarily- modest additional cost (see the green curve on the righthand side of Figure 2). From this it follows that any movement of z and z² away from zero, no matter how small, will be such that we can guarantee that the penalties on z" and z' will satisfy
cost, while increases originating from points away from zero incur only
((+) + h(0) = (xfz) ~ M((2) RJ 1(22)) < h(22) + h(22) ~ 21h(2]) - k(0),, (9)
and so z. is preferred. The righthand side Figure 2 motivates this relationship. Note also that we were to explicitly bound the slope of h around zero, then we could always select an sufficiently small such that the inequality in (9) reversed; hence an unbounded slope required to achieve the stated result.
7

---

Two candidate decoders:
h(x)) h(x)) = h((2)
simple
h(2)
sample point from 1D menrfold (bive curvej in small regian 20 ambient
optimal sparse
Mx
representation
21
2
Figure 2: 2D illustration of the intuition behind Theorem See Section
3.3 for details.
To large extent, the intuition here mirrors the basic scenario from Figure I and is emblematic of broader situations that naturally arise in practice. For example, if we run PCA on MNIST data, we find that only a 100 or so principal components are needed to achieve highly accurate reconstructions. But VAE model with only around 15 informative latent dimensions can accomplish something similar [Dai et al. 2018 by closely approximating an optimal sparse representation using a nonlinear decoder. Of course unless have an objective function with a strong preference for lower- dimensional structures, as instantiated through large gradients around optimal sparse representations, then the network may well favor or converge to a simpler, higher-dimennionall alternative (e.g.. resembling a PCA solution).
Mitigating Unbounded Gradients via 7-Dependent Smoothing
While we have argued that unbounded gradients may serve a useful purpose in obtaining optimal latent representations, they may nonetheless pose challenges from an optimization standpoint. In addressing this concern, it is worth acknowledging that energy functions involving infinite gradients and/or unbounded regions are already indispensable across wide range of structured regression and sparse estimation problems [Gorodnitsky and Rao 1997 Rao et al., 2003). This history implies that when training VAE or other related AE model, we may borrow appropriate tools designed to mitigate the risk of converging to bad local solutions or regions of instability. In this vein, one effective strategy involves partially minimizing what amounts to smoothed version of the original objective function. The degree of smoothness is then gradually reduced as the optimization trajectory moves towards optimum. Within the domain of underdetermined linear inverse problems, this procedure is frequently used to find maximally sparse representations with minimal reconstruction error [Chartrand and Yin 2008 Hu et al.) 2012 Xu al. 2013
The VAE automatically accomplishes something similar when we choose to iteratively estimate y dur- ing training rather than merely setting its value to near zero may be theoretically optimal (assuming we know that there exists sufficient network capacity to achieve negligible reconstruction errors). Initially, when the reconstruction cost is still high because encoder/decoder parameters have not converged, the learned 7 will be larger and the overall VAE energy will be relatively smooth, devoid of
many deep local minimizers. It is only later as the data fit Σ E go (2(
[|ae(() pa (z; 0)]
becomes small that y will follow suite, and by this point it is more likely that we have already approached a basin attraction capable of producing optimal sparse reconstructions. Additionally, unlike fixing Ry 0 for all training iterations, in which case gradients will be unbounded right from beginning, by learning > we will likely only encounter large gradients a narrow neighborhood around minimizing solutions. This implies that in practice, we only need accommodate such gra- dients when the reconstruction error becomes small, at which point stability countermeasures can be deployed if/when necessary, e.g., reduced step size, checks for oscillating gradient sign patterns Riedmiller and Braun 1993]. etc.
To help visualize these points, in Figure 3 we have plotted 1D slices through the objective function of a simple VAE model involving a single layer for both encoder and decoder, applied data from random low-dimensional subspace. We vary 7 E {10-3, 10-2 2 10 .1), which exposes the increasing gradients and multi-modal nature of the objective function as 7 becomes smaller. Dashed vertical lines indicate the minimal value of the respective curve for each y. Additionally, we have explicitly designed the model underpinning this visualization such that there will exist an optimal sparse representation at zero on the z-axis. Consequently, we can readily observe that becomes
8

---

00001
normalized VÁE enargy
0.5
0011
0.1
normalized VAE energy
0.5
=001
=011
100
0.5
-0.5
-1
-0.5
-1
-0.5
0.5
1D through parameter space
1D slice through parameter space
(a)
(b)
Figure 3: Plots (a) and (b) show two sets of representative 1D slices through the VAE objective function (3) as the value of is varied. Dashed vertical lines indicate the r-axis location of the minimal value of each respective slice and y setting. And for both plots (a) and (b) the 1D slices are set such that an optimal sparse representation would occur at zero on the r-axis when y - 0. It can be observed that disconnected local minima only occur when y small.
sufficiently small, the minimizing value of the VAE energy increasingly aligns with an optimal sparse representation desired. However, as y is reduced the energy is less smooth and disconnected local minima appear in both ID slices. And local minima of the VAE loss surface can at times be risk points for under-regularized representations.
To further explore the implications of this 7-ddeeedennt smoothing effect, we empirically compare practical scenario whereby learning y may be better than fixing an arbitrarily small value. To this effect, we first train VAE model on CelebA data [Liu et al. 2013) and learn an appropriate small value of y denoted 7 (note that 7 need not be exactly zero since with real data and limited capacity the network will generally display some nonzero reconstruction errors). Please see the supplementary for network and training details. We then retrain the same network from scratch but with y = ? fixed throughout all training iterations.
The resulting models are evaluated via the reconstruction error and the maximum mean discrepancy (MMD) between the aggregated posterior (z) - E. (=(2)(())) Makhhani et al. 2016) and the prior p(z) = (O, I). If too few latent dimensions are removed by swamping the appropriate channels with noise following the prior (i.e., under-regullarization)) then we would expect go (z) be
confined near a low-dimensional manifold in I" and the MMD to be much larger. Note that for ideal generative modeling performance via an autoencoder architecture, is required that
1 Σ (z((((t)))
2J
(o(2)z)) (dæ) = p(=),
(10)
meaning the MMD from . (0, I) is ideally zero Makhzani et al. 2016). With manifold data this only possible if an optimal sparse representation is produced by the VAE or autoencoder-basee
analogue [Tolstikhin et al. 2018).
Results displayed in Figure 4(a) where expected the reconstruction errors nearly identical, but the learnable case leads to much lower MMD values, indicative of a better local solution with
reduced under-regularizztioo. We also plot the evolution of the gradient magnitudes dC(8. dz in Figure 4(b) (other gradients are similar). When is learned, the gradient increases slowly; however, with fixed y = 7", there exists a large gradient right from the start since 7 is small but the reconstruction error is high. This contributes
worse final solution per the results in Figure 4(a) Additionally, examples of using a learnable y to improve generated sample quality based on these principles can be found in DDai and Wipf 2019].
We close this section with one notable caveat: Although learning 7 can be beneficial for the reasons we have given, it is not panacea and in certain situations there can be unintended consequences. For

---

350
300
Leamable Y
Mradient
Magnitut
250
Fix
200
CelebA
150
Rec. Err.
MMD
Learnable y
352.8
93.3
100
Fix 7 7
349.9
291.8
50
0
100
200
300
400
Iterations (x100)
Figure 4: (a) Reconstruction error and MMD between (z(2) and (N I) on CelebA (128 x 128 resolution). We first train VAE with learnable y and obtain the optimal value y Then we fix 7 and re-train the same network from scratch. While the final reconstruction errors are almost the same, the MMDs between (=(2) and the prior (O, I) are significantly different. (b) The
Evolution of the gradient dC(0.6) Although both curves end up with similar final values, the
dz 2
large initial gradient with fixed y is disruptive to the final solution.
example, if particular VAE model experiences posterior collapse during training, then it may be necessary to place an upper bound on to help reduce the collapse risk.
5
Conclusion
It is not uncommon to learn the VAE decoder variance parameter in situations where the training data has a noise component that we are unable or do not wish to model. By doing so we can avoid tuning trade-off parameter while allowing the model to adapt to the data. However, with sufficient capacity networks and relatively clean data, the risk of unbounded gradients when training Y has frequently been raised as potentially problematic phenomena. We nonetheless provide formal justification for this choice (even in cases where does tend to zero) on two primary fronts:
We prove that unbounded gradients are in fact necessary for guaranteeing that global minima of canonical AE architectures will coincide with optimal sparse representations, meaning high fidelity reconstruction of the training data using the minimal number of informative latent dimensions. Hence there no obvious alternative if this form of parsimony is our goal. Furthermore, given the value of such representations to numerous downstream tasks as described in Section 2.2, our analysis suggests that heuristic modifications to constraints on the VAE energy function may be ill-advised, and large gradients should be accommodated to the extent possible (e.g., reduced step size, checks for oscillating gradient sign patterns, etc.).
We present compelling evidence that by learning y, large gradients away from global minimizers, as well as at least some bad local minimizers, can be mitigated or smoothed within the VAE loss surface. This helps to explain observed successes learning 7 in situations where the optimal value turns out to be small or near zero. Note that as mentioned in Section [ it is already known that fixing y too high posterior collapse [He et al., 2019 Lucas et al., (2019 Razavi et al.] 2019). In a similar vein, we have demonstrated the complementary yet underappreciated fact that prematurely fixing too low, even to what may ultimately be the optimal value near zero, can steer convergence towards under-regularized local minima and the inadvertent wasteful deployment of latent degrees-of-freedd ffreeddom......mmmmmmmm
lead over-regulaiization and the stuely-studied phenomena of
And finally, although not our focus, our results herein naturally relate to more flexible VAE models with Gan-Gassiian latent posteriors Kingma et al., 2016 Rezende and Mohamed, 2015) or adapt- able/trainable priors Bauee and Mnih) 2019 Tomezak and Welling 2018). While these types of enhancements can be useful tools for favoring (=) p(=), they do not circumvent the infinite gradients that will occur around optimal sparse representations.
10

---

Funding Transparency Statement
Some initial components of this work were conceived while Bin Dai was an intern and David Wipf was FTE Microsoft Research in Beijing. Additionally, K. Wenliang contributed as intern and David Wipf as an FTE at the AWS Shanghai Al Research Lab. There are no other sources of funding to report.
References
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2:1-18, 2015.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang. and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint arXiv:1802.014300 2018.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In International
Conference on Artificial Intelligence and Statistics, 2019.
Alexandre Belloni, Victor Chernozhukov. and Lie Wang. Square-root lasso: Pivotal recovery of sparse signals via conic programming. Biometrika, 98(4):791 1906,,, 2011.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3798) 2013.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foun- dations of Computational mathematics, 96):717-772, 2009.
Rick Chartrand and Wotao Yin. Iteratively reweighted algorithms for compressive sensing. In International Conference on Accoustics, Speech, and Signal Processing, 2008
Yichen Chen, Dongdong Ge, Mengdi Wang, Zizhuo Wang, Yinyu Ye, and Hao Yin. Strong NP- hardness for sparse optimization with concave penalty functions. International Conference on Machine Learning, 2017.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint 170.:0514... 2018.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In International Confernece on Machine Learning, 2020.
D.L. Donoho. Compressed sensing. IEEE Trans. Information Theory, 52(4):128- 2006.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. American Statistical Association, 96(456): 2001.
Partha Ghosh, Mehdi Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12438 2019.
Irina Gorodnitsky and Bhaskar Rao. Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions signal processing, 45(3):600 616 1997.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick Lagging inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2019.

---

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. 5-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Yue Hu, Sajan Goud Lingala, and Mathews Jacob. A fast majorize- algorithm for the recovery of sparse and low-rank matrices. IEEE Transactions on Image Processing, 21(2)7742-733, 2012.
Diederik Kingma and Max Welling. Auto-cncoding variational Bayes. In International Conference Learning Representations, 2014.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems. 2016.
Xiaopeng Li and James She. Collaborative variational autoencoder for recommender systems. In International Conference on Knowledge Discovery and Data Mining, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In International Conference on Computer Vision, 2015.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior
collapse in generative latent variable models. In International Conference on Learning Representa- tions, Workshop Paper, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2016.
Pierre-Alexandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variables 2018.
models. arXiv preprint
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems. 2018.
Andrew Ng. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
Jason Palmer, David Wipf, Kenneth Freutz-Delgadee and Baskar Rao. Variational EM algorithms for non Gau-Siass latent variable models. In Advances in Neural Information Processing Systems, 2006.
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv: 104.210444.844 2021.
Bhaskar Rao, Kjersti Engan, Shane Cotter, Jason Palmer, and Kenneth Kelgzz-D Subset selection in noise based on diversity measure minimization. IEEE Trans. Signal Processing, 51(3): 760-770, 2003.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with 5-VAEs. In International Conference on Learning Representations, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: 2015.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv preprint arXiv:1810.00530 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In International Conference on Neural Networks, 1993.
12

---

Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations. 2018.
Jorge Silva, Jorge Marques, and João Lemos. Selecting landmark points for sparse manifold learning. In Advances in Neural Information Processing Systems 18, 2006.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student-
t variational autoencoder for robust density estimation. In International Joint Conference Artificial Intelligence, 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
llya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
Jakub Tomezak and Max Welling. VAE with VampPrior. In International Conference Artificial Intelligence and Statistics, 2018.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, 2016.
David Wipf and Yi Wu. Dual-space analysis of the sparse linear model. In Advances in Neural Information Processing Systems, 2012.
Haowen Xu, Wenxiao Chen, Nengwen Zhao Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In International World Wide Web Conference, 2018
Li Xu, Shicheng Zheng, and Jiay Jia, Unnatural lo sparse representation for natural image deblurring. In Computer Vision and Pattern Recognition, 2013.
13


Reviews:
[{"review_id": "_hupuBO3W9t", "reviewer": "Reviewer_9vZA", "summary": "The author(s) challenge the conventional notion that infinite gradients are bad. Specifically, they examine infinite gradients in the context of auto-encoders and VAEs. Their theoretical findings suggest infinite gradients are necessary at global optima to achieve their definition (1) of an \"optimal sparse representation.\"", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n- Section 4 of this work is most similar to Dai and Wipf (2019). Dai and Wipf (2019) experimentally showed the benefit of training $\\gamma$ (beyond fixing it at a low value). However, Dai and Wipf (2019) focus more heavily on ensuring a valid ancestral sampling process via a two-stage VAE. This article attempts to explain *why* optimizing  $\\gamma$ is better than fixing it at an arbitrarily low value. Figure 3 is incredibly informative at showing this benefit. Thus, I feel the author(s) are well-enough distinguished from Dai and Wipf (2019). Although, I might consider moving their citation forward in this section from the last sentence since to the best of my knowledge Dai and Wipf (2019) were the first to recognize the unappreciated benefit of learning $\\gamma$, which the author(s) \"$\\gamma$-dependent smoothing\"\n\n### Quality:\n- Overall, I think the theoretical findings are well presented.\n\n### Clarity: \n- This paper is exceptionally well written.\n\n- Line 47: \"The latter requires that we assume specific functional forms for the encoder and decoder distributions.\" This is only true for the encoder. The decoder distribution can be any distribution (selecting a distribution with reparameterization gradients is unnecessary since you don't differentiate any MC-integrated quantity w.r.t. decoder distribution)\n\n- I think the author(s) need explain $\\theta \\in \\Theta$ better in the main article. In particular, $\\Theta$ is used before it is defined. Can the author(s) please elaborate what is $\\Theta$ (or provide an example) somewhere before the supplement?\n\n- Equation 13 drops $\\frac{1}{n}$ between lines. This scaling reappears in equation 14.\n\n### Significance: \n\n-  Theorem 4 is significant to those working on disentanglement based on flavors of the $\\beta$-VAE.\n\n- This article explains why Dai and Wipf's (2019) proposal to optimize  $\\lambda$ is better than fixing it at an arbitrarily low value. While this explanation (section 4) lacks  the the theoretical scaffolding of the earlier sections, it benefits from well illustrated examples.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the author(s) discuss in their checklist, the main limitation of their theoretical findings assumes that the global minima can be found. I agree that their work has no obvious way to negatively impact society.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1KlSoIJzBG", "reviewer": "Reviewer_9TAc", "summary": "This is a nice theoretical work, with direct practical implications. It could be considered as a follow-up of a recent paper appeared in ICLR 2019 (properly cited), in that it expands the preliminary considerations related to the effects of unbounded gradients (due to an appropriate consideration of the variance of the log-likelihood model adopted in traditional VAEs) on the \"optimality\" of sparse representations learned by variational autoencoders (VAE).\n\nFirst, this paper proceeds with a definition of an optimal representation, which amounts to one inducing a zero reconstruction error, and a minimum number of informative latent variables. Here, the assumption is that the input data lies on a d-dimensional space, the learned representation $z$ on a k-dimensional space, and the data distribution on a $r$-dimensional manifold, with $r \\leq k < n$.\n\nAt this point, the key contribution of the paper is a theorem, that indicates that irrespectively of the functional form of a VAE, optimal sparse representations go hand in hand with infinite gradients. This is inevitable under the modeling assumption of a Gaussian likelihood with $\\gamma$ scalar variance times the identity matrix.\n\nA second contribution is to observe that although the intuition would suggest to aim for $\\gamma \\to 0$, it is useful to consider an annealing schedule throughout the VAE training. This naturally arises if we consider $\\gamma$ as a learnable parameter. The practical implication is that $\\gamma$ should neither be prematurely set too high (as usually done in the literature) or too low (as the intuition form the aforementioned analysis would suggest): by an appropriate learning of $\\gamma$ the resulting latent representation becomes optimal, with visible benefits on standard metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the prose and narrative of this paper, which nicely (and somehow pedagogically) explain an important phenomenon that has been studied in the literature extensively, albeit with a different set of tools.\n\nI must confess I had troubles starting from sec.2 of this work, as my understanding of an optimal representation related to prior works that are more closely related to information theory: \"Information Dropout: Learning Optimal Representations Through Noisy Computation\", by Achielle, Soatto https://arxiv.org/abs/1611.01353v3, and \"The information bottleneck method\" by Tishby et al., https://arxiv.org/abs/physics/0004057. \nSo, condition ii) in Def.1 sounds really strange to me: \"the number of latent dimensions of $z$ containing no information about $X$ should be maximal\". This is somehow clarified in the comments to the definitions, but honestly I find this phrasing very difficult to digest. My interpretation is: if $k > r$, then we want all of the $k-r$ components to behave as the prior.\n\nAnother point of confusion is the way that eq(3) and eq(5) are discussed, below eq(5). When you plug back eq(5) in eq(3), you do not only consider the $\\log \\gamma$ term, but there also is a $1/\\gamma$ term to take into account. \n\nThat being said, here is a question.\n\n* Q1: what about situations in which the likelihood is not Gaussian? I assume similar considerations to those made in this work would somehow hold, but it would be nice to discuss this more general case.\n\nSection 3 reads well, and did not pose any major challenges to be appreciated, especially thanks to sec 3.3 which is helpful (I did not check the full proof in the appendix).\n\nSection 4 is again straightforward to appreciate, and the implications are clear. What would be interesting to understand is in the following question.\n\n* Q2: what is the convergence rate of learning $\\gamma$? That is, how fast does it reaches the optimal solution $\\gamma \\to 0$ compared to the variational parameters of the encoder, and the parameters of the decoder? Is there any potential instability in the process? Would it be possible (as illustrated in fig.3) that when $\\gamma$ reaches small values, the VAE loss gets stuck into a bad local minimum?\n\nIn conclusion, I find this a good paper, which nicely extends existing considerations from the literature. It has some practical implications, but an extensive experimental campaign is already contained in previous work from ICLR 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't think the authors clearly point out any technical limitations of their work. In my opinion, one point to address is to discuss  about a more general setting whereby the likelihood is not Gaussian. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-fyg6Vvu1bG", "reviewer": "Reviewer_e4Tf", "summary": "This paper has two main contributions:\n- It shows that when training (continuous gaussian) Variational Autoencoder, growing gradients are to be expected. It relates those infinite gradients to the fact that one of the goal of such models is to achieve an optimal sparse representation, and that obtaining such representations require having a variance of the observation model going to zero, which in turns imply growing gradients.\n- It shows that the property of incurring such infinite gradients is not specific to Variational Autoencoder, but still holds for any kind of Autoencoder with a sparsifying regularization.\n- It provides evidence that ad-hoc methods usually used to mitigate those growing gradients, such as fixing the observation model variance, have adverse effect on learning. More specifically, since the case of fixing a high variance has already been covered in the litterature, it empirically shows that fixing the variance low initially (even to a value close to the optimal one, as obtained from a run training jointly the variance, the encoder and the decoder) has an adverse effect on performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper at hand is clear, novel, and provides interesting theoretical and practical insights. The presentation of the paper is good, and notably the introduction is a very natural and well polished exposition of Variational Autoencoders.\n\nI have two mild concerns about the paper:\n\n- I think the way Theorem 4 is stated could be made more formal, which would ease the comprehension and avoid confusions. Currently, given the proof, I think the formal statement that holds is the following:\nFor any (concave, non decreasing) h with bounded gradients near 0, there exists a dataset X and a class of decoders \\mu_x(z, \\theta) such that, for an arbitrarily expressible encoder, the global optimum of the loss L_h(\\theta, \\phi) does not achieve an optimal sparse representation.\nWith the formulation that is given in the text, I think it could be misinterpreted that the statement holds for all class of decoders, e.g. that the statement would be:\nFor any (concave, non decreasing) h with bounded gradients near 0, and for all class of decoders \\mu_x(z, \\theta) there exists a dataset X such that, for an arbitrarily expressible encoder, the global optimum of the loss L_h(\\theta, \\phi) does not achieve an optimal sparse representation.\n\n- While it is clear, given prior works, that obtaining an optimal sparse representations requires that \\gamma should go to 0, it would be useful to make the relationship between \\gamma going to zero and gradients becoming unbounded clearer. Notably, it might be useful to strengthen the fact that when \\gamma goes to 0, not only the gradient of the loss w.r.t. to \\gamma goes to \\infty, but also potentially the gradient of the loss w.r.t. the parameters of the decoder, as the numerator of this gradient will typically scale like \\sqrt{\\gamma}, while the denominator will scale like \\gamma. While it may not be possible to provide a formal statement on that, I think this would still strengthen the claim on unbounded gradients, as having unbounded gradients only on \\gamma could be viewed as an easily fixable problem (e.g. by specifically normalizing this gradient), but having potentially unbounded gradients on the decoder parameters could prove much harder to deal with.\n\nI think those two concerns are easily fixable, and given the interesting insights provided by the paper, I recommend acceptance of the paper. I will increase my ratings if my concerns are alleviated after rebuttal from the authors.\n\n# Post rebuttal edit:\nThe authors have addressed my two concerns. Given their response, I am raising my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are, to the best of my understanding, addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong paper that all reviewers felt should be accepted to the conference; a viewpoint I concur with myself.  Some constructive criticisms have though been raised and I encourage the authors to incorporate these suggestions for the final version of the work.

A few extra comments from my end to help update the paper for the camera-ready version:
- I'm not sure "sparse" is the right term in Definition 1 (i.e. in the term "optimal sparse representation") and could be a bit misleading.  Sparse representations usually refer to cases where *different* sets of latent dimensions will be "activated" for different inputs, rather than the same latents always being redundant.  I would suggest using some other term like "minimal".
- It should be made clear in Eq 6 that this is now an optimization w.r.t Z.
- Even with the discussed generalization of Theorem 4, I think some care is needed in discussing the generality of the results and impossibility of AE frameworks that can satisfy Definition 1 without infinite gradients as even with this generalization, it is still a particular form of loss function rather than any possible AE framework.  As such, some of the higher level assertions made need slightly more careful quantification to avoid overclaiming / misunderstanding.
- I feel that the introduction could be tightened up a little: though nicely written at a low level, it somewhat meanders at the moment and one has to go through quite a lot of basic material before getting to the point of the paper.  In particular, it is only really in the second half of page 2 that the content becomes specific to the paper; before that it is a very generic overview of the area, which is wasteful.  In other words, though (as a reviewer noted) this is a nice introduction to VAEs, I don't think such a comprehensive introduction to them at a general level is needed and this both slightly slows down the early part of the paper and uses up space that could maybe be better utilized.
Decision: Accept (Spotlight)