Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias
Authors: Frederic Koehler
Viraj Mehta
Chenghui Zhou
Andrej Risteski
Venue: ICLR 2022 Poster (2022)
Keywords: variational autoencoders
encoder
optima
stability
low-dimensional manifold

Abstract: Variational Autoencoders (VAEs) are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on a lower dimensional manifold. Recent work by Dai and Wipf (2020) proposes a two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. They gave partial support for this conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics.  In this paper, we show that for linear encoders/decoders, the conjecture is true—that is the VAE training does recover a generator with support equal to the ground truth manifold—and does so due to an implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.

Published as a conference paper at ICLR 2022
VARIATIONAL AUTOENCODERS IN THE PRESENCE OF LOW-DIMENSIONAL DATA: LANDSCAPE AND IMPLICIT BIAS
Frederic Koehler*
Viraj Mehta"?,
Chenghui Zhou*3, and Andrej Risteski
Depaarment of Computer Science, Stanford University. focehler@stan.ord.cco edu Robotics Institute, Camegie Mellon University, virajme cs cmu edu Machine Learning Department, Carnegie Mellon University, {chenghuz, ari aristesk Jandkkea@aneee cmw.
ABSTRACT
Variational Autoencoders (VAEs) are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on lower dimensional manifold. Recent work by Dai and Wipf (2020) proposes two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with variance which is correctly supported on the ground truth manifold. They gave partial support for this conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics. In this paper, we show that for linear encoders/decoders, the conjecture is truc-that is the VAE training does recover a generator with support equal to the ground truth manifold- and does due to implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is superset of the ground truth manifold.
INTRODUCTION
Variational autoencoders (VAEs) have recently enjoyed revived interest, both due to architectural choices that have led to improvements in sample quality (Oord et al. 2017 Razavi et al. 2019b Vahdat & Kautz 2020) and due to algorithmic insights (Dai et al. 2017 Dai & Wipf 2019). Nev- ertheless, fine-grained understanding of the behavior of VAEs is lacking, both on the theoretical and empirical level.
In our paper, we study a common setting of interest where the data is supported on ciw-dimensional manifold- often argued to be the setting relevant to real-world image and text data due to the man- ifold hypothesis (see e.g. Goodfellow et al. (2016)). In this setting, Dai and Wipf (2019) proposed conjecture that for standard VAE training with such data distributions: (I) the generator's 's covariance will converge to 0, (2) the generator will learn the correct manifold, but not the correct density on the manifold (3) the number of approximately 0 eigenvalues in the encoder covariance will equal the intrinsic dimensionality of the manifold (see also Dai et (2017)). Formally, they showed that some optima of the VAE loss satisfy this conjec- ture, but they did not attempt to analyze the training dynamics.
two-stage training process for VAHs, based
In this paper, we revisit this setting and explore the behaviour both the VAE loss, and the training dynamics. Through combination of theory and experiments we show that:
In the case of the data manifold being linear (i.e. the data is Gaussian, supported on a linear nubspace--quivaalentyy it is produced as the pushforward of Gaussian through a linear map), and the encoder/decoder being parametrized as linear maps, we show that: a) the set of optima includes parameters for which the generator's support is strict superset of the data manifold; b)
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2022
the gradient descent dynamics are such that they converge to generators with support equal to the support of the data manifold. This provides a full proof of the conjecture in Dai & Wipi (2019), albeit we show the phenomenon is combination of both the location of the minima of the loss as well as an implicit bias of the training dynamics.
In the case of the data manifold being nonlinear (i.e. the data distribution is the pushforward of the Gaussian through a nonlinear map f II"" - R2, S d), the gradient descent dynamics from a random start often converges to generators underlying data distribution, while driving reconstruction error to O and driving the VAE loss to -00. This shows that the conjecture in Dai & Wipf (2019) does not hold for general nonlinear data manifolds and architectures for the generator/encoder.
whose support strictly contains the support of the
We will provide an informal overview of our findings in Section 3 The rigorous discussion on the VAE landscape are Section and on the implicit bias of gradient descent in Section 5
Organization:
2 SETUP
We will study the behavior of VAE learning when data lies a low-dimensional manifold more precisely, we study when the generator can recover the support of the underlying data distribution. In order to have a well-defined "ground truth", both for our theoretical and empirical results, we will consider synthetic dataset that are generated by a "ground truth" generator as follows.
Data distribution:
To generate a sample point x for the data distribution, we will sample ~
(0, ), and output = f(z), for suitably chosen f. In the linear case, f(z) Az, for some matrix A E __d""", In the nonlinear case, fz) will be a nonlinear function f R - BR We will consider several choices for f.
Parameterization of the trained model: For the model we are training, the generator will sample (O, I.) and output x ~ [f((2), e)), for trainable f, c; the encoder given input will output 2 ~ N(g(z), D), where D E R"x" is a diagonal matrix, and g. D are trainable. In the linear case, f. will be parameterized as matrices A, B; in the nonlinear case, several different paffferenrrrammeeriiiaaaan parameterizations
will be considered. In either case the VAE Loss will be denoted by L(-), see (3).
OUR RESULTS
Linear VAEs: the correct distribution is not recovered. Recall in the linear case, we train a
linear encoder and decoder to learn Gaussian distribution consisting of data points ~ N(, Σ)
equivalently, the data distribution the pushforward of a standard Gaussian Z ~ (0, I,) through Az with AAT E; see also Section 2 above. In Theorem I of Lucas) jet al. (2019), the authors proved that in a certain probabilistic PCA setting where is full-rank, the landscape of the VAE loss has no spurious local minima: at any global minima of the loss, the VAE
a linear generator
decoder exactly matches the ground truth distribution, i.c. AAT + +e2 E.
We revisit this problem in the setting where y has rank less than so that the data lies on the lower- dimensional manifold/subspace spanned by the columns of A or equivalently Σ, denoted span(A). We show empirically (i.e. via simulations) in Section 6 that when Σ is rank-degenerate the VAE actually fails to recover the correct distribution. More precisely, the recovered A has the correct column span but fails to recover the correct density confirming predictions made in Dai & Wipf (019)) We then explain theoretically why this happens, where it turns out we find some surprises.
Landscape Analysis: Linear and Nonlinear VAE. Dai & Wipi (2019) made their predictions on the basis of the following observation about the loss landscape: there can exist sequences of VAE solutions whose objective value approaches -oo (i.e. are asymptotic global minima), for which the generator has the correct column span, but does not recover the correct density on the subspace. They also informally argued that these are all of the asymptotic global minima of loss landscape (Pg 7 and Appendix I in Dai & Wipf 2219)), but did not give a formal theorem proof of this claim.
We settle the question by showing this is not the case namely, there exist many convergent se- quences of VAE solutions which still go to objective value oo but also do not recover the correct
"They also argued this would hold in the nonlinear case, hut our simulations show this is generally false in that setting, even for the solutions chosen by gradient descent with standard initialization see Section 6
2

---

Published as a conference paper at ICLR 2022
column span instead, the span of such A is strictly larger subspace. More precisely, obtain a tight characterization of all asymptotic global minima of the loss landscape:
fixed matrices such that A ABA and suppose that #(i A, 0} 3. Suppose that A, B are d. i.e. the number of zero columns of Å is strictly larger than r d. Then there exists & - and positive diagonal matrices Dr such that lime I(A, B, Dr, (c) -C0. Also, these are all of the asymptotic global minima: any convergent sequence of points (Ae, Br, De; along which the loss L goes -0o satisfies A. A. + with A ABA such that #(i: A. 03 > rr-d. d.
Theorem 1 (Optima of Linear VAE Loss, Informal Version of Theorem
To interpret the constraint #(i : A; 0} > d. observe that if the data lies on a lower- dimensional subspace of dimension r* < (i.e. r" the rank of Σ), then there exists a generator which generates the distribution with r- r* > r d zero columns by taking an arbitrary low- rank factorization I.I." = Σ with I, d r" and defining A d x r by A = [T. Odxr-r'). The dimension r* and the ambient dimension d, the more
larger the gap is between the manifold/intrinee flexibility we have in constructing asy asymptotic global minima of the landscape. Also, we note there is no constraint in the Theorem that d 2 O: the assumption is automatically satisfied if d.
To summarize, the asymptotic global minima satisfy A = ABA, the column span of Á contains that of A, but in general it can be a higher dimensional space. For example, if d, r 2 r² + 2 and
Ir.. 0)
o 0 then Ā I, T,
0 and B T 0
and the ground truth generator is is a perfectly valid asymptotic global optima of the landscape, even though decoder À generates a different higher-dimensional Gaussian distribution N (o, I,-+1
0
0 0
0 ( 0 0) than the ground truth. In
the above result we showed that there are asymptotic global minima with higher dimensional spans even with the common restriction that the encoder variance is diagonal; if we considered the case where the encoder variance is unconstrained, as done in Dai & Wipf (2019). and/or can depend on its input x, this can only increase the number of ways to drive the objective value to -oc.
We also consider the analogous question in the nonlinear VAE setting where the data lies on a low- dimensional manifold. We prove Theorem 6 that even in a very simple example where we fit a VAE to generate data produced by 1-layer ground truth generator, there exists bad solution with strictly larger manifold dimension which drives the reconstruction error to zero (and VAE loss to -oo). The proof of this result does not depend strongly on the details of this setup and it can be adapted to construct bad solutions for other nonlinear VAE settings.
We note that the nature both of these result is asymptotic: that is, they consider sequences of solu- tions whose loss converges to -00 - but not the rate at which they do so. In the next section, we will consider the trajectories the optimization algorithm takes, when the loss is minimized through gradient descent.
Linear VAE: implicit regularization of gradient flow. The above theorem indicates that studying the minima of the loss landscape alone cannot explain the empirical phenomenon of linear VAE training recovering the support of the ground truth manifold in experiments; the only prediction that can be made is that the VAE will recover a possibly larger manifold containing the data.
We resolve this tension by proving that gradient flow, the continuous time version of gradient de- scent, has an implicit bias towards the low-rank global optima. Precisely, we measure the effective rank quantitatively in terms of the singular values: namely, if a. o. (A) denotes the k-th largest singular value of matrix Á, we show that all but the largest (iinssaan/ A) singular values of Á decay at an exponential rate, as long as: (1) the gradient flow continues to exist] and (2) the gradient flow does not go off infinity, i.e. neither Á or é go to infinity (in simulations, the decoder A converges to a bounded point and غ + 0 so the latter assumption is true). To simplify the proof, we work with a slightly modified loss which "eliminates" the encoder variance by setting to its optimal value:
"We remind the reader that the gradient flow on loss L(x) is a differential equation dx/dt - ((x).. Unlike discrete-timg gradient descent, gradient flow in some cases (e.g. dr /dt = 2²) has solutions which exist only for finite time (e.g. = 1/(1 +)), which "blows up" at f 1), so we need to explicitly assume the solution exists up time T.
3

---

Published as a conference paper at ICLR 2022
(, ming B, 2); this loss has simpler closed form, and we believe the theo- rems should hold for the standard loss as well. (Generally, gradient descent on the original loss L will try to optimize D in terms of the other parameters, and if it succeeds to keep D well-tuned in terms of A, B, E then will behave like the simplified loss L2.)
Theorem 2 (Implicit Bias of Gradient Flow, Informal version of Theorem 5 Let A d x r be ar- bitrary and define W to be the spun of the rows of A, let @ = (A(0), (0), E(0) be arbitrary initialization and define the gradient flow equation (ODE)
(t) (A(t), B(t), ē(t)) by the ordinary differential
dō(t) = (()
(1)
dt
with initial condition Og If the solution to this equation exists on the time interval [0, ']] and satisfies max max, max,ll((A) (l) 221 S K, then for all E [0, T] we have
d
(A((() S C(A,A)
I dim
where C(A, A) == || A (0) and Pw= is the orthogonal projection onto the orthogonal complement of W.
Together, our Theorem 1 and Theorem 2 show that if gradient descent converges to a point while driving the loss to -00, then it successfully recovers the ground truth subspace/manifold span A. This shows that, in the linear case, the conjecture of Dai & Wipf (2019) can indeed be validated provided we incorporate training dynamics into the picture. The prediction of theirs we do not prove that the number of zero entries of the encoder covariance D converges to the intrinsic dimension; as shown in Table in a few experimental runs this does not occur - in contrast, Theorem Bimmlies that Á should have the right number of nonzero singular values and our experiments agree with this.
Nonlinear VAE: Dynamics and Simulations. In the linear case, we showed that the implicit bias of gradient descent leads the VAE training to converge to a distribution with the correct support. In the nonlinear case, we show that this does not happen- even in simple cases.
Precisely, in the setup of the one-layer ground truth generator, where we proved (Theorem 6 there exist bad global optima of the landscape, we verify experimentally (see Figure D that gradient descent from a random start does indeed converge to such bad asymptotic minima. In particular, this shows that whether not gradient descent has a favorable implicit bias strongly depends on the data distribution and VAE architecture
More generally, by performing experiments with synthetic data of known manifold dimension, make the following conclusions: (1) gradient descent training recovers manifolds approximately containing the data, (2) these manifolds are generally not the same dimension as the ground truth manifold, but larger (this is in contrast to the conjecture in Dai & Wipf (2019) that they should be equal) even when the decoder and encoder are large enough represent the ground truth and the reconstruction error is driven to O (VAE loss is driven to -co), and (3) of all manifolds containing the data, gradient descent still seems to favor those with relatively low (but not always minimal) dimension. Further investigating the precise role of VAE architecture and optimization algorithm, as well as the interplay with the data distribution is an exciting direction for future work.
3.1 RELATED WORK
Implicit regularization. Interestingly, the implicit bias towards low-rank solutions in the VAE which we discover is consistent with theoretical and experimental results in other settings, such as deep linear networks/matrix factorization (e.g. Gunaseear et al. (2088); Li et al. (2018); AAooa al. (201)): Li et al. (2020); Jacot et al. 2021)) although it seems to be for a different mathematical reason unlike those settings, initialization scale does not play major role. Similar to the setting of implicit margin maximization (see e.g. Di & Telgarsky (20rskk)((11r) (2018): Schapire & F'reund (2013); [Soudry al. (2018), in our VAE setting the optima are asymptotic (though approaching a finite point, not off at infinity) and the loss goes to -oc. Kumar & Poole (2020); Tang & Yang (202I) also explore some implicit regularization effects tied the Jacobian of the generator and the covariance of the Gaussian noise.
4

---

Published as a conference paper at ICLR 2022
Architectural and Algorithmic Advances for VAEs. There has been recent surge in activity with the goal of understanding VAE training and improving its performance in practice. Much of the work has been motivated by improving posterior modeling to avoid problems such "posterior
collapse", see e.g. (Dai et al. 2020 Razavi et al. 2019a; Pervez & Gavves 2021 Lucas et al. 2019)
He al. 2019 Oord al. 2017] Razavi et al. 2019b Vahdat & Kautz (020). Most relevant the
current work are probably the works Dai & Wipf (211 Lucas et al. (2019) discussed earlier. A
relevant previous work to these is Dai al. (2017); one connection
the current work is that they
also performed experiments with ground truth manifold, in their case given as the pushforward of a Gaussian through ReLU network. In their case, they found that for a certain decoder and encoder architectures that they could recover the intrinsic dimension using a heuristic related to the encoder covariance eigenvalues from Dai Wipi (2019); our results are complementary in that they show that this phenomena is not universal and does not hold for other natural datasets (e.g. manifold data on a sphere fit with a standard VAE architecture).
VAE LANDSCAPE ANALYSIS
In this section, we analyze the landscape of a VAE, both in the linear and non-linear case.
Preliminaries and notation. We use VAE to model datapoint x E RR the pushforward of 2 ~ 4.). We have the following standard VAE architecture:
p(z|2) = ((
q(>|x) = ((g()), D)
where e² > 0 is the decoder variance, a diagonal matrix with nonnegative entries, and f. D, are all trainable parameters. (For simplicity, our D) does not depend on x; this is the most common setup in the linear VAE case we will primarily focus on.) The VAE objective (see Appendix A for explicit derivation) is to minimize:
L(f.g, D,e) Ee B(o....))
|| 2
f(g(z) + -1
dlog(e) Tr(D)/2 EEogg log Du.
(3)
We also state a general fact about VAEs for the case that the objective value can be driven to -o0, which was observed in (Dai & Wipt 2019): they must satisfy e + and achieve perfect limiting reconstruction error. The first claim in this Lemma is established in the proof of Theorem 4 and the second claim Theorem 5 in Dai & Wipf (019). For completeness, we include self-contained proof in Appendix B.
Lemma (Theorems 4 and 5 of Dai & Wipf 2019)). ) Suppose ft. 9t. De. e for 2 1 are a sequence such that lim L( Lt, Dr, E = oc.
Then: 1) lim = 0 and 2)
lim Exap- B Be-N(o, (x) D 2 21) O.
In fact, the reconstruction error and are closely linked in simple way:
Lemma 2. If J,g. D are fixed, then the optimal value of to minimize L is given by V Emmp B [|| [(((x) + D1/22/)113.
4.1 LINEAR VAE
Setup: In the linear VAE case, we assume the data is generated from the model x = As with A E [dxrr' and z ~ NO, I.)). We will denote the training parameters by À E gdxr, B E RRa D E R' and ē > 0. where r 2 is fixed hyperparameter which corresponds to the latent dimension in the trained generator, and we assume D is a diagonal matrix. With this notation in place, the implied VAE has generator/decoder ă ~ NN2,,, 1,, ea and encoder 2 ~ ((xx, D). The VAE objective as a function of parameters Ô = (Ā, B, D,&) is (see Appendix Ab:
1 T.(0) = 272 A ABA 1
BA + d log + Σ (Bull D. - log D:s)) (4)
Our analysis makes use of simplified objective L1, which "eliminates" D out of the objective by plugging in the optimal value of for a choice of the other variables. We use this as a technical tool when analyzing the landscape of the original loss L.
5

---

Published as a conference paper at ICLR 2022
Lemma 3 (Deriving the simplified loss L1). Suppose that A, B, are fixed. Then the objective is minimized by choosing for all that Dis
8² 1A,
where A is column of A, and for
LA(A, B, == min; L(A, B, D,2) it holds that
1 + log
[|4,12 + 22)
L,
27 |A ABA + =11BBA1 + (d rd-
Σ
(5)
2
Taking advantage of this simplified formula, we can then identify (for the original loss L) simple suf- ficient conditions on A. B which ensure they can be used to approach the population loss minimum by picking suitable εc. D, and prove matching necessary conditions.
are fixed matrices such that A ABA and suppose that # A. 0} > i.e. the number of zero columns of À is strictly larger than r d. Then for any sequence of positive Et - 0 there exist a sequence of positive diagonal matrices Dj such that:
Theorem 3. First, suppose that Á r, B
1. For every such that As # O, iee column of Á nonzero, we have (De)st -
2. lime I(A, B, Dr, ()) -00
Conversely, suppose that that Ae, Bi, De an arbitrary sequence such that lim L(A,, Br, De, ēt) -00 Then 0o, we must have that:
1. é + 0 and I|A B.AB -
2. max, (De) a1l (A.)
where (At); denotes the i-th column of Å..
3. For any 5 > O, lim inf # [1( < 8) r d, i.e. asymptotically Å, has strictly more than r d columns arbitrarily close to zero.
2i) converge to a point (Å, B, D, 2) then = 0. A ABA, D. = 0 for every such that A; + O, and #
In particular, if (A,, B,, &.)
0} > rrd d
For the sufficiency direction, we observe that in ( the first term is zero if A = ABA and the sum of the last two terms goes to -0 if ē + 0 and enough columns of Å are zero. Based upon similar reasoning combined with Lemma we show necessity. The full proof is in the Appendix.
4.2 NONLINEAR VAE
In this section, we give a simple example nonlinear VAE architecture which can represent ground truth distribution perfectly, but has another asymptotic global minimum where it outputs data lying on a manifold of larger dimension (r* + 8 instead of r* for any 2 1). The ground truth model is one-layer network ("sigmoid dataset" in Section 6 and the bad decoder construct outputs standard Gaussian y* + dimensions padded with zeros.
Theorem 4 (Theorem in Appendix). Let s 2 be arbitrary and consider the sigmoid setup from Section 6 There exists AI, A2, B s.t. for é, + there exists D s.t. (1) the VAE loss L(Ai, Ae, B, E1, -o;; (2) The output of the decoder ã Á + a (A2)), z N(O, I.) is standard Gaussian in the first r" + coordinates and zero in the remaining ones.
Thus, the generator constructed has as support a manifold of larger dimension (r* + s). Moreover, in Section 6 we show that this is not merely theoretical possibility: we show through simulations that gradient descent from a random initialization often converges to similar minima.
IMPLICIT BIAS OF GRADIENT DESCENT IN LINEAR VAE
In this section, we prove that even though the landscape of the VAE loss contains generators with strictly larger support than the ground truth, the gradient flow implicitly biased towards low-rank solutions. We prove this for the simplified loss (A, B,²) ming [(Ā, B,ē D), which makes

---

Published as a conference paper at ICLR 2022
the calculations more tractable, though believe our results should hold for the original loss L as well. The main result we prove is as follows:
Theorem 5 (Implicit bias of gradient descent). Let : dxr be arbitrary and define W to be the span of the rows of A, let 6(0)= = (A(O), B(O), E(0)) be an arbitrary initialization and define the gradient flow 6(t)
(A(t), E(t), by the differential equation (I. with initial condition Og. If the solution to this equation exists on the time interval [o, T] and satisfies max) max, [l))))) ||2 +
(2) S K, then for all E [0, T] we have
(A(()) S Se IP
(6)
k=dim(W)+L
where Pw is the orthogonal projection onto the orthogonal complement ofW.
Towards showing the above result, we first show how to reduce to matrices where A has d dim (oon(soaaan(A)) A)) rows that are all-zero. To do this, we observe that the linear VAE objective is invariant to arbitrary rotations in the output space (i.e. x-space), so the gradient descent/flow trajec- tories transform naturally under rotations. Thus, we can "rotate" the ground truth parameters as well as the training parameters.
This is formally captured Lemma the Appendix. Recall that by the singular value decomposi- tion A = USV for some orthogonal matrices U, V and diagonal matrix S, and rotation invariance in the z-space lets us reduce to analyzing the case where U I, i.e. = SVT. This matrix has a zero row for every zero singular value.
Analysis when A has zero rows. Having reduced our analysis to the case where A has zero rows, the following key lemma shows that for every such that row of A (denoted A is zero, the gradient descent step VL or L1 will be negatively correlated with the corresponding row A
Lemma 4 (Gradient correlation). If row i of A is zero, then
OL AU
θL AU Σ
Σ
Σ J=1 14,12
DAij
OAij
1=1
The way we use it is to notice that since the negative gradient points towards zero, gradient descent will shrink the size of A00, Since the size of the matrix Á stays bounded, this should mean that for small step sizes the norm of row i of À shrinks by a constant factor every step of gradient descent on loss L1. We formalize this in continuous time for the gradient flow, i.c. the limit of gradient descent step size goes to zero: for the special case of Theorem in the zero row setting, the corresponding rows of Á decay exponentially fast.
Lemma (Exponential decay of extra rows). Let be arbitrary, and let 0(0) be an arbitrary initialization and define the gradient flow ²(t) = (Â(t), B(t), to be solution of the differential equation T with initial condition @ If the solution exists on the time interval [o, I`I and satisfies maxee (.77 max, ]Il(A(())) E(t)²] < K for some K > 0, then for all i such that row of A is zero we have [[(0 S | (0)ll for all l E [0,T].
B(O), 2(0)
SIMULATIONS
In this section, we provide extensive empirical support for the questions addressed theoretically. In particular we investigate the kinds of minima VAES converge to when optimized via gradient descent over the course of training.
Linear VAFs: First, we investigate whether linear VAEs are able to find the correct support for a distribution supported over a linear subspace. The setup is as follows. We choose a ground truth linear transformation matrix A by concatenating an r* x r" matrix consisting of iid standard Gaussian entries with a zero matrix of dimension (d (d-r*) x r"; the data is generated as Az, 2 ~ NO Thus the data lies in a y dd-mmessiionll subspace embedded in a d-dimensional space. We ran the experiment with various choices for r", d well the latent dimension of the trained decoder (Table Every result is the mean over three experiments run with the same dimensionality and setup but a different random seed.
7

---

Published as a conference paper at ICLR 2022
Intrinsic Dimension
3
6
9
Ambient Dimension
12
20
12
20
12
20
20
Mean #0's in Encoder Variance
3.3
3.7
6
9.3
9
12
Mean # Decoder Rows Nonzero
3
6
9
9
12
Mean Normalized Eigenvalue Error
0.44
0.71
0.49
0.47
0.30
0.45
0.42
Optima found by training linear VAE on data generated by linear generator (i.e. a linearly trans- formed standard multivariate gaussian embedded in a larger ambient dimension by padding with zeroes) via gradient descent. The results reflect the predictions of Theorem the number of nonzero rows of the decoder always match the dimensionality of the input data distribution with no variance while number of nonzero dimensions of encoder variance is greater than or equal to the nonzero rows. All VAEs are trained with a 20-dimensional latent space. Clearly, the model fails to recover the correct cigenvalues and therefore has a substantially wrong data density function.
Table I:
Results: From Table we can see that the optima found by gradient descent capture the support of the manifold accurately across all choices of d, r, with the correct number nonzero decoder rows. We also almost always see the correct number of zero dimensions in the diagonal matrix corresponding to the encoder variance.
However, gradient descent is unable to recover the density of the data on the learned manifold in the linear setting - in sharp contrast to the full rank case (Lucas et al. 2019). We conclude this by comparing the eigenvalues of the data covariance matrix and the learned generator covariance matrix. In order to understand whether the distribution on the linear subspace has the right density, we compute the eigenvalue error by forming matrices X, X with n rows, for which each row is sampled from the ground truth and learned generator distribution respectively. We then compute the vector of eigenvalues Å for the ground truth covariance matrix AAT and empirical covariance matrix (1/n) X Å respectively and compute the normalized eigenvalue error || - AlI/||A||. In no case does the density of the learned distribution come close to the ground truth.
Nonlinear Dataset In this section, we investigate whether VAEs are able to find the correct support in nonlinear settings. Unlike the linear setting, there is no "canonical" data distribution suited for nonlinear VAE, so we explore two setups:
Sphere dataset: The data are generated from the unit sphere concatenated with zero padding at the end. This can be interpreted as unit sphere embedded in a higher dimensional space. We used layers of 200 hidden units to parameterize our encoder and decoder networks.
To measure how well the VAE has learnt the support of the distribution, we evaluate the average of (1|::001)llzz 122 - 1)², where are generated by the learnt generator. We will call this quantity manifold error. We have also evaluated the padding error, which is defined as ||ā
Sigmoid Dataset: Let NN I,), the sigmoid dataset concatenates with a(aa, 22) where a" E IR" is generated according to (0, I,). We add additional zero paddings to embed the gen- erated data higher dimensional ambient space. The decoder parameterized by nonlinear function f=) = A2 + (C2) and the encoder is parameterized by a linear function g(z) Bz. The intrinsic dimension of the dataset is
To measure how well the VAE has learnt the support of the distribution, of (a((a", (a())
evaluate the average 12, where ğ are generated by the learnt decoder. We will call this quantity manifold error. The padding error is defined as similarly as the sphere dataset.
Results: Due to space constraints, results for the sphere dataset (Table 3 and Figure 3 are in Ap- pendix D In both of the nonlinear dataset experiments, we see that the number of zero entries in the diagonal encoder variance is less reflective of the intrinsic dimension of the manifold than the linear dataset. is, however, at least as large as the intrinsic dimension (Table 3 2). We consider a coordinate to be O if it's less than 0.1. We found that the magnitude of each coordinate to be well separated, i.c. the smaller coordinates tend to be smaller than 0.1 and the larger tend to be bigger
than 0.5. Thus the threshold selection is not crucial. We did not include padding error in the tables because it reaches zero in all experiments
We show the progression of manifold error, decoder variance and VAE loss during training for the sphere data in Figure 3 and for the sigmoid data in Figure 2 which are included in Appendix [ due to space constraints. Datasets of all configurations of dimensions reached close to zero decoder
8

---

Published as a conference paper at ICLR 2022
1.5-
250
1.0
200
0.5
150
0.0-
100
-0.5
-1.0-
0.2
0.4 0.6 0.8
1.0
-5.0
Figure I: demonstration that in the nonlinear setting (both types data padded with zeroes embed in higher ambient dimension, see Setup in Section 6 VAE training does not always recover distribution with the correct support. Left figure: A histogram of the norms of samples generated from the VAE restricted to the dimensions which are not zero, which shows many of the points have norm less than 1. (The ground-truth distribution would output only samples of norm 1.) The particular example here is Column in Table Right figure: Two-dimensional linear projection
data output by VAE generator trained on our sigmoid dataset. The z-axis denotes (a", Z..) and the y y-axis is ž, the blue points are from the trained VAE and the orange points are from the ground truth. In contrast to the ground truth data, which satisfies the sigmoidal constraint 2.41 o (((a", x.r)). the trained VAE points do not and instead resemble standard gaussian distribution. This case that closely resembles the example provided in Theorem Also similar to Theorem VAE model plotted here (from Column 6 Table 2 achieves nearly- reconstruction error, approximately 0.001.
Intrinsic Dimensions
3
3
5
5
7
7
Ambient Dimensions
17
11
22
15
28
VAE Latent Dimensions
8
10
16
13
24
Mean Manifold Error Mean #O's 's in Encoder Variance
0.09
0.13
0.23
0.24
0.18
0.28
3
3.6
6.3
7.3
8
Table 2: Optima found by training a VAE on the sigmoid dataset. The VAE training consistently yields encoder variances with number of 0 entries greater than or equal to the intrinsic dimension.
variances, meaning the VAE loss is approaching -c.. To demonstrate Theorem 6 from both datasets to visualize their output.
we took examples
For the sphere dataset, we visualize the data generated from the model, with 8 latent dimensions, trained on unit sphere with 2 intrinsic dimensions and 16 ambient dimensions (Column 2 in Table 3). Its training progression is shown as the orange curve in Figure We create a histogram of the norm of its first 3 dimensions (Figure and found that more than half of the generated data falls inside of the unit sphere. The generated data has one intrinsic dimension higher than its training data, despite its decoder variance approaching zero, which is equivalent its reconstruction error approaching zero by Lemma
In the sigmoid dataset, the featured model has 24 latent dimension, and is trained 7-dimensional manifold embedded in a 28-dimensional ambient space. We produced a scatter plot given 1000 generated data points -,-- from the decoder. The z-axis in the Figure is (a" , and the y-axis is Fr In contrast to the groundtruth data, whose scatter points roughly form a sigmoid function, the scatter points of the generate data resemble gaussian distribution. This closely resembles the example provided in Theorem 6 Hence, despite its decoder variance and reconstruction error both approaching zero and loss consistently decreasing, the generated data do not recover the training data distribution and the data distribution recovered has higher intrinsic dimensions than the training data. We also investigated the effect of lower bounding the decoder variance as a possible way to improve the VAE performance (details are given in Appendix E.. This enabled the VAE to recover the correct manifold dimension in the sigmoid example, but not the sphere example; methods of improvements to the VAE's manifold recovery is important direction for future work.

---

Published as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32:7413-7424, 2019.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv: 2019.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint arXiv: 1a0:::100.05555 05148, 2017.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? reassessing blame for vae posterior PMLR, 2020.
collapse. In International Conference Machine Learning, pp.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Sre- bro. Implicit regularization in matrix factorization. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-10. IEEE, 2018.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Berg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv: 1901.0554, 2019.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Arthur Jacot, François Ged, Franck Gabriel, Berfin Şimşek, and Clément Hongler. Deep linear networks dynamics: Low-rank biases induced by initialization scale and 12 regularization. arXiv preprint arXiv:2106. 15933, 2021.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv: /00.:://0..00000 2018.
Abhishek Kumar and Ben Poole. On implicit regularization 3-vaes. International Conference on Machine Learning, pp. 5480-5490. PMLR, 2020.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in ver-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pp. 2-44 47. PMLR, 2018.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09833 2020.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't 't blame the elbo! linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32:9408-94 9918,, 2019.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint uxXiv1111.000337 2017.
Adeel Pervez and Efstratios Gavves. Spectral smoothing unveils phase transitions in hierarchical variational autoencoders. In International Conference Machine Learning, pp. 8536-8545. PMLR, 2021.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint arXiv:1901.03410 2019a.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pp. 14866-14876, 2019b.
Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Kybernetes. 2013.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822-287)) 2018.
10

---

Published as a conference paper at ICLR 2022
Rong Tang and Yun Yang. On empirical bayes variational autoencoder: An excess risk bound. In Conference on Learning Theory. pp. 4068-448440 4125. PMLR, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
A DERIVATIONS OF VAE LOSSES
We have (for some constants C1, C2, O3):
log p(xl() 1
22 || f(=)]l= log((c) C
logp(z) = -11=11//22 +2 C2
log (=)l)) - (----9()) g(x), D= 112-g(a)) g(x))
log d D C3
where the first line uses that log dott 2 log V 2d = d log(c). The VAE obeeetivve is to maximize
the expectation of log p(z|2) + log p(z) log ((2(a)) ()) for z from the data p" and ~ g(2[x)) This means that explicitly the objective is to maximize
Ex Ex (x<1(())) + log log 1og((>/>)]- C
Eaap' Beng(=(x)
2e x - f())11 - dllo(e)) - ||=1|//2 /2 z (0(); D- (2 - g(x)) + log det D
2c || f(g(x) /2/222211 dllo(() (19(2)) + 11/22/112/2
2 (5', >²) + log det D
which simplifies to (up to additive constant)
Exep (,
- 1- f(g(s) + 22/22111 [l((2))1 dloo(e)) Tr(D)/2+ 2 Σ log Da
1
2
and converting this to minimization form gives the VAE Loss
Linear VAE derivation. Plugging in the linear VAE parameters into the loss function, we get
L(Ā, B, D, = Expp Beno-BBennn (e.
222 |
A(Br + 11/220)112 |/rl
(7)
dlog(E) Tr(D)/2
log D.s
(8)
We can write out the expectation as:
E E..N
A A(BA= A= 2//22/)112 + ||BA=||///2 /2]
22
-E,NO.,)
(IA - ĀBA)= AD1/2212 + |BAx
2
||A ABA +
AADV2112 + BAI
272
27
where used that z are independent and the identity E M2212 (MMT,1) | [| Next, we can observe that
ADD DalAAN
where À; is the i-th column of the matrix Á. Therefore recover
11

---

Published as a conference paper at ICLR 2022
B DEFERRED PROOFS FROM SECTION 4
B.I GENERAL FACTS
Proof of Lemma | For completeness, we include the proof of these claims; they are similar to the proof's of Theorems 4 and 5 in Dai & Wipf (2019).
First, consider the objective for fixed f.g, D, e and omit the subscript We have
Expp (2
22 22212 f(g()) D1/22/112 (1)(())) =
and
Tr(D)/2-
Σ 2
log D
log D.:) 2 r/2
from the inequality logæ 2 for 2 O. Since these terms are both bounded below, the only way the objective goes to negative infinity dl d'log + -00 which means c + 0.
Now that we know 6 + 0, we claim that lim Ex-p- E 1,,, (x( (x)+ Suppose otherwise: then this for infinitely many this quantity is lower bounded by some constant >> 0, hence the objective for those é is lower bounded by c/é² + d loo(() + r/2 and this goes to +o0 as e + 0, instead of -o0.
20)2 1) 0.
Proof of Lemma
Taking the partial derivative of 3 with respect to c and setting to zero gives
1=---pas BewNO.. -(g(a) >
and solving for gives the result.
B.2 LINEAR VAE
Proof of Lemma which means
Taking the partial derivative with respect to D. gives -1/1///211-1//
1
22
D 1|/,11/22 +1
1|4:11 4
hence
2²
Du 4.11//1//22 + D. log D. =1 - log
|| + 22
follows that the objective value at the optimal D is
(A, =
2/2 IIA
1
ABA
+
1 +1 dlogē ē +
Σ
1- log
22
2
2
1|A::12 + E2
||1
ABA
+ 2
1 , BA + (d- rog +
1
Σ
1 1 log
22
2
|| + 22)
Proof of Theorem 3 First we prove the sufficiency direction, i.e. that if A = ABA and there exists
i such that A, = 0 then we show how to drive the loss to -0. By Lemma if we make the optimal choice of D (which clearly satisfies the conditions D described in the Lemma) the objective simplifies to
L (A,
IA ABAIE
|BAI + (d r) log é Σ (1 (1+ log (1A.11++)) x))
2
1 = + (d r) logg E + Σ [1 + log (11,,11 22)
12

---

Published as a conference paper at ICLR 2022
where in the second line we used the assumption A ABA. Note that for each zero column A, 0
have (1/2) log(II-A,)) + 2²) log é so the objective will go to provided (d + #
0 this is equivalent to asking d - + # A. 0} > which is exactly the assumption of the Theorem.
A, 0}) log - -oo. Since
Next prove the converse direction, i.c. the necessary conditions. Note: split the first item in the lemma into two conclusions in the proof below (so there are four conclusions instead of three). The first conclusion follows from the first conclusion of Lemma The second conclusion of Lemma tells us that
0 = lim t-000
lim || 1A-AAB,A1+++111-D1 t-000
which gives us the second and third conclusions above. For the fourth conclusion, since (Ae, Br, D.) & L(Ae, Be, De, Ex) we know that lim (Ae, Br, D.) -oo and recalling
1 |A ABA & BAB + (d - r) log غ Σ (1+1og log (1A7)) + +²)) 2
L L1 2
we see that, because the first two terms are nonnegative, this is possible only if the sum of the last two terms goes 00. Based on similar reasoning to the sufficiency case, this is only possible strictly more than r of the columns of (Ae) become arbitrarily close zero; precisely, if there exists such that at most r - d of the columns of Å, have norm less than ó, then
(d - r) log + 2 Σ (1+ log log (1A:112+ + +2²)
Σ (1+ log (11,112 + 2+)
2 2
A|1A.1126
2 Σ
(1 log (82 2))
$:
which does not go to -o as ē + 0 (and the other terms of Lj are nonnegative).
B.3 NONLINEAR VAE
We give the full details of the construction of the bad nonlinear VAE optimum and prove that it is an asymptotic global minimum. (Note: in the notation of Section 6 we are considering a" with 0/1 entries, but the proof generalizes straightforwardly for arbitrary a" with the correct support.)
Suppose s 2 1 is arbitrary and the ground truth x E R with > r" + s is generated the [ I,,), 2,+41 = a -...+ for arbitrary nonlinearity a, and 4r-42 = ... = Ta = 0. Furthermore, suppose the architecture for the decoder with latent dimension r" 1 is
Setup:
following way: (ZI,
(x) = Aiz+o (A2²)
where o--) is applied as entrywise nonlinearity, and the encoder is linear, g(x) := Br.
Observe that the ground truth decoder can be expressed by taking Åg to have a single nonzero row in position r + with entries (1, 1.0, ........) 0),
A = 4--
B 1,- 0
0
where a ground truth encoder which achieves perfect reconstruction.
On the other hand, the following VAE different from the ground truth achieves perfect reconstruc- tion:
1 1.++s 0
1,++1 0
A =
Ag = 0,
B =
(9)
0 0
0
13

---

Published as a conference paper at ICLR 2022
Ty- No 0). which means is strietly higher- 0
The output of this decoder is a Gaussian N
0,
dimensional than the ground truth dimension r (This also means that if we drew the corresponding plot to [ for this model, we would get something that looks just like the experimentally obtained result.) We prove in the Appendix that this is an asymptotic global optima:
Theorem 6. Let s 2 1 be arbitrary and the ground truth and VAE architecture is as defined as above. For any sequence & - O, there exist diagonal matrices Dr such that:
1. the VAE loss L(A1, A2, B, De, Ei) -0 where A A2, B are defined by
2. The number of coordinates of D, which to zero equals r + 8.
Proof. We show how to pick D, as function of & and that if & + 0, the loss goes to -c0. From now on we drop the subscripts.
With these parameters, the VAE loss is
Egpo (2(
| f(g(x) 1'/22/111 + +log())+IIr(D)))))
Σ log D
2
+111 = (112222 Σ Dis + Famp' | [P/22] + ddlog(e) + Tr(D)/2
Σ log Duc
Taking the partial derivative with respect D for S s and optimizing gives (1/2²) + 1/D i.e.
2
DH 1 1/22
2-1
and plugging this into the objective gives
(1/23²)
Σ
Das + Expp' 10110+1111//2 + dlog(e)+ + Tr(D)/2 - - Σ log D.s
1=1
(112)
+ Eanp' |100004111 (d (d-r" 8) log(E)
22
+
1x +1 + Tr(D)/2+
log(1 + + +
Σ log D...
2
2 :=r++2
Setting the remaining D to 1, we see that using d > r" + s that the loss goes to -co provided 0, proving the result.
C DEFERRED PROOFS FROM SECTION 5
First, we formalize the rotation invariance of the objective.
Lemma 6 (Rotational Invariance of Gradient Descent on Linear VAE), Let LA B, D, 2) denote the VAE population loss objective (4. Then for an arbitrary orthogonal matrix U, we have LA (Ā, B, D, LU D.
Furthermore,
UV (A, = VUALu BUT D.E)
and
(V D. T VUGLU BUT D.E). As consequence, for any 1j 2 0 define (Āi, Bi, DI,ēi) (A, B. D,E) LL D, then
( D,,,) = (U',, TU D.ē) T7 (UA, .D,a) Lu a (U,, BU" D.
i.e. gradient descent preserves rotations by U. The same result holds for the gradient flow (i.e. continuous time gradient descent), or replacing everywhere the loss L by the simplified loss La.
14

---

Published as a conference paper at ICLR 2022
Proof of Lemma 6 We give the proof for L as stated, but it is exactly the same for the simplified loss L1-
From the objective function and U U-1 observe that
LUA(U A, BUT, D.
1
IUA AABBU-AII + - -1IBU--UIBB- + dlog + 2/2 Σ (DiIIAAI// + Di. log D(,)
22
2 11
ABAI + - 1BA|1 + dog & + Σ (DullA,11//2 + Dua llg Dis)
LA,(A, B, E)
Then from the above and the multivariate chain rule have
V AAL B, D, AuTAAUU BUT _D.)) - (VVVUVVVVAAAAA (VuALus(UA, BU-1, (D)))
so multiplying both sides on the left by U and using U = U gives the second claim, and similarly V LA B, D, V a LU BUT D,ē) (V Buz Bur a BUT E))) gives the third claim. Then the gradient descent claim follows immediately.
Proof of Lemma 4 First prove the conclusion for the original loss I. Since (ABA)at Ej Aig Are have that
0|A ABAI?
Σ
Ave
E
Auy By -
=
Aue
E BykAne/
) ( BynAne
aA.j
OAi
j',
and if we know the corresponding row in is zero then this simplifies to
ale ABAII
)
Σ
Σ j',k
Aige BynAnd
Σ Bye
aAi
which means that
2
Au Lle ABAI?
Au 0A
Σ2
Σ j.k
ByrAee =1
where the notation ( denotes row i of matrix A. row ( has nonnegative dot product with row A
Thus, for this term the gradient with respect to
Also,
0 (1/2) oA,
Σ 1 14,1/2 -
and so
Σ Au oA:
(1/2) > D, 1, = DigARi1e
which gives the first result. For the second result with the simplified loss L1, observe that
24g
log(
| +2)
AAi
|1
22
so
8 AA.
2
Σ
A.
Σ
+22)
|
+
and the other terms in the loss behave the same the case L. Including the factor of 1/2 from the loss function gives the result.
15

---

Published as a conference paper at ICLR 2022
Proof of Lemma
From Lemma 4 we have that for any such row i,
d dt [1 (000=)))))((000)000 d dt
(1))
(A(t))
= (t), - V Aauo (1()) - II((()))) 12 S -1/KK]] (t)l
which by Gronwall's inequality implies ( - e t/k (0) as desired.
Proof of Theorem Before proceeding, we observe that the first inequality in (6) consequence of the general min-max characterization of singular values, see e.g. Horn & Johnson (2012). We now prove the rest of the statement.
As explained at the beginning of the section, we start by taking the Singular Value Decomposition = USVT where S is the diagonal matrix of singular values and U, are orthogonal. We assume the diagonal matrix S is sorted so that its top-left entry is the largest singular value and its bottom- right is the smallest. Note that this means the first dim(WW rows of U are an orthonormal basis for W. Note that for any time t, P (+) = Σ Li=dim(W)+ lI(U ((U)(tt)))ll* because the rows (Umm(W)+::-... Ua) are an orthonormal basis for W L, Therefore we have that
I AT())
Σ
((U
d=dim(W))
I <e-tKK 1/K
E
((U(())))II"
|tKK (0))
proving the result, provided we justify the middle inequality. Define A' := = = SVT, which has a zero row for every zero singular value of A, and apply Lemma (using that the definition K is invariant to cef--multiplication of Á by an orthogonal matrix) and Lemma 6 to conclude that the rows of U A(t), i.c. the columns of U UA(t)", corresponding to zero rows of A shrink by a factor of e -1/K This directly gives the desired inequality, completing the proof.
D DEFERRED FIGURES AND PLOTS FROM SECTION 6
Eigenvalues of Linear Data. As we've discussed, in our linear setting the VAE does not recover the ground truth data density. Since generative process for ground-truth data is z Ae for matrix A and z normally distributed, we can characterize the density function by the eigenvalues of the true or estimated covariance matrix. We give figures for the normalized emror of these eigenvalues between the learned generator and the ground truth in Table T A concrete example of eigenvalue mismatch for a problem with nonzero dimensions is ground-truth set of covariance eigenvalues
[0.001 0.156
1.54
5.06
9.55
16.4]
while the trained linear VAE distribution has covariance eigenvalues
À = |0.035
0.166
1.49
4.24
5.97 7.85).
Here, the VAE was easily able to learn the support of the data but clearly is very off when comes to the structure of the covariances.
E EXPERIMENTS WITH DECODER VARIANCE CLIPPING
As was suggested by anonymous ICLR 2022 reviewer, one potential way to evade the results in our paper is to restrict the decoder variance from converging to In this section, we examine (empirically) the impact of clipping the decoder variance during training. We caveat though, that our paper does not analyze the landscape of the resulting constrained optimization problem, so our results don't 't imply anything about this regime.
16

---

Published as a conference paper at ICLR 2022
Nanitoid Enter
Decocer
WAT
150
6:0
2.0
Pertic cin 12 arao kuar dn
6-12
Sieres andin
820
60
20 510
10
-100
Figure 2: VAE training on 6 datasets with different choices of dimensions for sigmoidal dataset (see Setup in Section 6). The r-axis represents every 5000 gradient updates during training. The left-most figure the manifold error (see Sctup Scction 6). The middle and right figure confirms that the decoder variance approaches zero and the VAE loss steadily decreasing during the finite training time.
Nantel Errr
Decoder Verience
WA
600
Foreei kanker liter
620-
516
Fioeei cin ana wnninn boir can
enidim linen
805
Figure 3: VAE training on 5 datasets generated by appending zeros to uniformly random samples from unit sphere to embed in a higher dimensional ambient space. The z-axis represents each iteration of every 5000 gradient updates. The left-most figure is the manifold error see Setup in Section 6). The middle and right ligure confirms that the decoder variance approaches and the VAE loss steadily decreasing during the finite training time.
Intrinsic Dimensions
2
4
Ambient Dimensions
6
16
10
21
14
VAE Latent Dimensions
8
16
Mean Manifold Error Mean #0's in Encoder Variance
0.02
0.14
0.04
0.06
0.03
5
6
Table 3: Optima found by training VAE on data generated by padding uniformly random samples from a unit r-sphere with zeroes, so that the sphere is embedded in higher ambient dimension. We evaluated the manifold error described in the setup. The VAE training this dataset has consistently yielded encoder variances with number of entries greater than the number of intrinsic dimension.
17

---

Published as a conference paper at ICLR 2022
We conduct the same nonlinear experiments described in Section erated from spheres and linear sigmoid functions. The only change is to clip the decoder variance when it goes below certain threshold. In the figures below, the chosen threshold is e z 0.018, though we tried also e-2 and e-3, with similar outcomes. We initialize the decoder variance with e-3 for this set of experiments, the optimization still can decrease it.
where where fit VAEs to data gen-
With this change, the optimization process on the sigmoid dataset does yield encoder variances with their number of zeros reflective of their intrinsic dimensions as in Table For the sphere experiment, this still does not happen, as in Table [ In fact, the model consistently recovers one more dimension than the true intrinsic dimension of the manifold and the smaller encoder variances can be as large as 0.1. We also provide figure (Figure 4 in the same style as Figure []] We see that training with a clipped decoder variance allows the model better capture the general shape of the sigmoid function, though the variance of the generated points is high for both of the sphere and sigmoid datasets. Other training details, such as the general trend of manifold error, encoder variance and VAE loss, can be referred in Figure 5 5ad
Overall, the benefit of clipping the decoder variance during training is inconclusive as we see in- consistent results in the sphere and sigmoid datasets. Designing more algorithms to improve the ability of VAE's to recover data supported on low dimensional manifold is an important direction for future work- both empirical and theoretical.
1.00
200
3.5
150
330
080
50
0.00 03.00 1,00
1.25
1.50
2.75
-20.0 -1.5 +0.0
-2.5
Figure 4: A demonstration of how the data points generated by the model trained with clipped decoder variance is distributed. Left figure: A histogram of the norms of samples generated from the VAE restricted to the dimensions which not zero, which shows many of the points have norm less than (The ground-truth distribution would output only samples of norm 1.) The particular example here is Column 2 in Table The data points that do not fall on the sphere tend to on both sides whereas those generated without decoder variance clipping tend to lie inside the sphere as in Figure Right figure: Two-dimensional linear projection of data output by VAE generator trained on our sigmoid dataset. The z-axis denotes (a" ....) and the y-ax y-axis is 1, the blue points from the trained VAE and the orange points from the ground truth. The generated data points roughly capture the shape of sigmoid function.
Manifold Enor
Decoder Verience
VIE Loss
dirr 17 dine latant
nit rus dire 23 ano sok are 12 lalutt cim
nor dire ama din latree din
10
25 26
Figure 5: VAE training 6 datasets with different choices of dimensions for sigmoidal dataset (see Setup Section 6) The r-axis represents every 5000 gradient updates during training. The left-most figure is the manifold enror (see Setup in Section 6). The middle and right figure shows that as the decoder variance is bounded below, the VAE loss stops decreasing further.
18

---

Published as a conference paper at ICLR 2022
Variance
230
ner dini ammoen
300
lntent
calir dire latten sin.
337
Figure 6: VAE training on 5 datasets generated by appending zeros to uniformly random samples from unit sphere to embed in higher dimensional ambient space. The z-axis represents each iteration of every 5000 gradient updates. The left-most figure is the manifold error see Setup in Section (6) The middle and right ligure shows that as the decoder variance bounded below, the VAE loss stops decreasing further.
Intrinsic Dimensions
3
3
5
5
7
7
Ambient Dimensions
17
22
15
28
VAE Latent Dimensions
6
8
10
16
13
24
Mean Manifold Error Mean #0's in Encoder Variance
0.15
0.15
0.23
0.23
0.24
0.24
3
5
5
7
7
Table 4: Optima found by training VAE sigmoid dataset. with number of 0 entries equal to the intrinsic dimension.
training yields encoder variances
Intrinsic Dimensions
2
4
6
Ambient Dimensions
6
16
10
21
14
VAE Latent Dimensions
6
8
10
16
13
Mean Manifold Error Mean #'. in Encoder Variance
0.03
0.03
0.03
0.02
0.02
3
5
Table 5: Optima found by training VAE on data generated by padding uniformly random samples from unit r-sphere with zeroes, that the sphere is embedded in higher ambient dimension. We evaluated the manifold error described in the setup. The VAE training this dataset has consistently yielded encoder variances with number of 0.1 entries greater than the number of intrinsic dimension.
19


Reviews:
[{"review_id": "eZPFv_J-YSc", "reviewer": "Reviewer_KKdc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The work tackles previous work (Dai & Wipf, 2019), which is very interesting.\n- Theorems and proofs are formally provided.\n\nWeakness\n- The problem statement or the motivation is not properly provided.\n- The paper seems to be revised, and written in formal way.\n- Linear VAE is barely used. Instead of linear VAE discussion, it would be better to discuss more on the non-linear VAE case, which is more widely used.\n- No analysis in benchmark or real-world dataset. The authors only deal with toy datasets. ", "summary_of_the_paper": "The authors study further on the conjectures of Dai & Wipf (2019). For the linear case, they provide the proof that the conjecture is true. For the non-linear case, the paper disagrees with the conjecture, and they argue that the VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.  \n", "summary_of_the_review": "I'm negative on this paper. The paper needs to be revised in formal way. The motivation should be clearly introduced (rather than citing the paper), and the conclusion should be also provided. The paper seems not ready to be published, however, the work is interesting and I'm looking forward to have the paper (with additional experiment on benchmark or real-world) in the revised version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SeV7vOa3wa", "reviewer": "Reviewer_ayh8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that a deeper insight into the training behavior of VAEs is a very important research area. The paper provides further insight into possible degenerate optima that can be obtained by VAEs, both linear and nonlinear, during training.\n\nHowever, I have several concerns regarding the accuracy of the paper. This is my current understanding, and I am fully willing to change my score if I have misunderstood anything.\n\n1. Though the authors heavily cite and build on Dai & Wipf's paper, they do not explore whether the two-stage VAE proposed by Dai & Wipf remedies any of the problems posed in this paper. Without a central theorem or a clear theme that ties together the provided theorems, the current work feels somewhat disjointed.\n\n2. I take issue with the way this paper positions itself with respect to Dai & Wipf. Namely, it misinterprets the theoretical results provided in the cited paper. \nIn the abstract, it is implied that Dai and Wipf’s result is on the convergence properties of the VAE during training, and that the present work expands on these convergence properties. See: \n“Recent work by Dai and Wipf (2019) suggests that on low-dimensional data, the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. In this paper, via a combination of theoretical and empirical results, we show that the story is more subtle. Precisely, we show that for linear encoders/decoders, the story is mostly true and VAE training does recover a generator with support equal to the ground truth manifold, but this is due to the implicit bias of gradient descent rather than merely the VAE loss itself.”\nHowever, it is my understanding that all of Dai & Wipf’s results do not consider any convergence behavior in training. Instead, they only analyze *optimal* behavior of the VAE. They do construct sequences of encoder/decoder functions that converge to these optima. But these sequences are unrelated to the training behavior of the VAE. Moreover, Dai & Wipf’s results only imply the generator only converges to a solution with 0 variance if the variance of the decoder goes to zero.\n\n3. The first bullet point in Lemma 1, a restatement of Theorems 4 and 5 in Dai & Wipf, does not seem correct. Translating the notation in Dai and Wipf to that of this paper, Theorem 4 in Dai & Wipf implies the existence of a sequence of $f_t, g_t, D_t, \\gamma_t$ such that $\\gamma_t \\rightarrow 0$ and $L(f_t, g_t, D_t, \\gamma_t) \\rightarrow -\\infty$. Lemma 1 appears to state the converse, i.e. all sequences $f_t, g_t, D_t, \\gamma_t$ s.t. $L(f_t, g_t, D_t, \\gamma_t) \\rightarrow -\\infty$ have that $\\gamma \\rightarrow 0$.\nMoreover, in both parts of Lemma 1, $f_t, g_t, D_t$ are allowed to be any sequence such that $\\mathcal{L} \\rightarrow -\\infty$, whereas in Dai & Wipf they are fixed w.r.t. $t$ and optimal w.r.t. the loss $\\mathcal{L}$.\n\n\nStyle:\n\nPage 6. \"There exists $\\tilde{A}_1, \\tilde{A}_2, \\tilde{B}$ s.t. for $\\tilde{\\epsilon}_t \\rightarrow 0$ there exists $\\tilde{D}_t$ s.t.\" This setup for Theorem 4 was difficult to parse\n\nPage 8. $(\\sigma(\\langle a^∗, \\tilde{x}_{:r} \\rangle) − \\tilde{x}_{r+1})^2$ ...\n\n\nshould $\\tilde{x}_{r+1}$ be instead $\\tilde{x}_{r+1:}$?\n", "summary_of_the_paper": "This paper builds on a recent theoretical work by Dai & Wipf [1]. Dai & Wipf analyze the *optimal* behavior of VAEs, when applied to manifold-valued data. This work analyzes the *training* behavior of VAEs, when applied to manifold-valued data. Both consider the non-trivial case where the manifold’s intrinsic dimension is lower than the ambient dimension. Dai & Wipf considers VAEs with arbitrarily complex encoders and decoders. The present work considers VAEs with linear encoders and linear/single-hidden layer nonlinear decoders.\n\n[1] Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789,\n2019.", "summary_of_the_review": "While the paper provides insight in a very exciting area of theoretical research in machine learning, I currently do not recommend acceptance due to inaccuracies and the lack of a cohesive thesis.\n\nPros:\n+ Novel theoretical analysis of VAEs during training and at global optima\n+ Theoretical observations are corroborated by empirical results\n\nCons:\n- Theory only considers linear encoders and linear/1-hidden layer nonlinear decoders\n- The convergence behavior during training is only provided for linear VAEs\n- Work feels disjointed, writing could be clearer (e.g. Theorem 4 has very little discussion)\n\n**AFTER REBUTTAL**\n\nMy understanding of the work has been improved by the rebuttal. However, I do not feel that the stance taken in the manuscript adequately represents the stance taken in the rebuttal. I would be more willing to recommend acceptance if the misunderstandings were properly addressed in the manuscript---however, the manuscript, as it stands, has not been edited in such a way. (See my reply to the rebuttal.) I still stand by most of my points. Here is the updated summary:\n\nPros:\n+ Novel theoretical analysis of VAEs during training and at global optima\n+ Theoretical observations are corroborated by empirical results\n\nCons:\n- Theory only considers linear encoders and linear/1-hidden layer nonlinear decoders\n- Work feels disjointed, writing could be clearer (e.g. Theorem 4 has very little discussion)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NubLWIwenFA", "reviewer": "Reviewer_U2NF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Original Review\n\nThe results and informal discussions in Dai and Wipf (2019) can be misinterpreted in a few ways, and clarifications are valuable to the general ML audience.\n\nThat being said, most results here are arguably unsurprising, and largely stem from an unfortunate definition of \"asymptotic optimality\": it includes any (trajectory of) parameters which have loss tending to infinity, without accounting for the rate of divergence.  It is clear from e.g. Dai and Wipf (2019, p. 7) that any (suitable) VAE models parameterizing a $\\hat{r}<d$ manifold will have its loss tend to infinity, as its leading term is $-\\frac{d-\\hat{r}}{2}\\log \\gamma$, where the decoder variance $\\gamma$ is assumed to converges to zero.  It is also clear from the same source that if we lower bound $\\gamma$ by a *small positive constant* $\\gamma_0$, then (among the parameter sets considered in Dai and Wipf (2019) and this work) the global optima should have as small a dimension as possible, because the leading term of the loss will be $-\\frac{d-\\hat{r}}{2}\\log \\gamma_0$.  Dai and Wipf (2019) stated that their practical recommendations start from this form of the leading term of loss (last sentence of Section 3).  The result is relevant because in practice the optimal decoder variance will be bounded by the reconstruction error of the AE (Dai et al., 2020), although the theoretical picture is far from clear.\n\nThere are nuances around the informal discussions in Dai and Wipf (2019): one of the most important limitations is their restriction to a particular set of parameters (decoder variance is either 1 or near zero along each axis).  They also ignore errors from estimation, approximation and optimization.  However, the fact that there exists different parameter (trajectories) leading to model distributions with \nsupport having different dimensions does not appear to be one of them.\n\nThe interesting part is the analysis of the implicit regularization effect from optimization, and the nonlinear simulation here reveals one surprising limitation of previous work that may have real consequences.  It would greatly enhance the paper if the authors experiment with simple fixes for this issue: e.g. clipping the decoder variance from below.  Regardless of whether the fixes work, the results will provide more insight as well as possible guidance for practitioners.\n\nMinor comments:\n\n1. In Theorem 4, $\\tilde z$ is not defined; immediately below Theorem 5, it will be helpful to clarify the rotation is applied to both the model parameters and ground truth.\n\n2. You may want to cite the following two papers, which improve the understanding of ELBO landscape over Dai and Wipf (2019):\n    1. Kumar and Poole (2020), On Implicit Regularization in β-VAEs, ICML 2020.\n    2. Tang and Yang (2021), On Empirical Bayes Variational Autoencoder: An Excess Risk Bound, COLT 2021.\n\n### Post-rebuttal Update\n\nMy original concern is that the main results here appear irrelevant in light of the informal discussions in Dai and Wipf.  While I'm not sure if these discussions are not obvious, they do rely on strong assumptions and do not account for the optimization process.  More importantly, the findings in the new experiments provide contrary evidence to the discussions in Dai-Wipf.  This is interesting and demonstrates the relevance of the results in this work.  For this reason, I change my score to acceptance.", "summary_of_the_paper": "This work revisits the population loss analysis for VAE ELBO (Dai and Wipf, 2019) and note that undesirable \"asymptotic global optimas\" exist, where the support of the model distribution $p_{model}(dx)$ has higher dimensions than the true data manifold.  Additionally, it shows that, for linear VAE, such optimas are excluded due to the implicit bias of gradient descent, but empirically nonlinear VAEs often stuck in such optimas.", "summary_of_the_review": "(edited)\n\nStrengths\n\n+ The cautions against misinterpretation will be useful to the broad ML community.\n+ Findings about the implicit bias are interesting.\n\nLimitations\n\n+ Further efforts are needed to understand the discrepancy between the results here, which do not provide rate of divergence, and the informal calculations in Dai and Wipf, which implicitly places assumptions that do not appear consistently supported by experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AaVYVJ5tPs6", "reviewer": "Reviewer_MVZZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper are: \n- Combination of loss landscape analysis with gradient flow analysis sounds sensible and validates the claim by (Dai & Wipf; 2019) for the linear case. \n- The sigmoid dataset example is simple enough to see VAE may overestimate the intrinsic dimension. \n\nThe weaknesses are: \n1) the organization of the paper could be improved, and \n2) the paper lacks any tips for practitioners. \n\n1: Section 3 explains the general ideas while Sections 4 and 5 give rigorous discussions about the loss and optimization dynamics, respectively.  \nI noticed this structure only after reading through the three sections, which was confusing to me. \nExplaining the structure of the discussion at an appropriate place (at the end of Introduction or at the beginning of Section 3) may improve the clarity. \n\n2: Most practitioners use nonlinear VAEs, but the paper provides a negative result in that the VAE objective can induce a larger intrinsic dimension and larger support of data distribution. \nIs there any wisdom or tip from the theory discussed in the paper for practitioners? \nSuch information will strengthen the importance of the paper. \nFor example, the impact of the choice of latent dimensionality $r$ or a way to choose $r$ can help VAE users. \n\nQuestion about gradient flow analysis: can we extend to stochastic gradient setup? \nIn the linear case, the expectation over latent variable $z$ in the VAE objective is analytic. \nOn the other hand, we need a Monte-Carlo approximation for typical nonlinear cases using the reparameterization trick, etc. ", "summary_of_the_paper": "This paper examines the conjecture given in (Dai & Wipf; 2019) on the support of distribution that VAE learns. \nThe contributions are listed as follows: \n- For the linear case where the data is Gaussian with rank-degenerate covariance, and encoder/decoder are both linear, this paper proves that VAE captures the intrinsic dimension of data distribution correctly by analyzing the objective and its gradient-flow dynamics. \n- For nonlinear cases, the paper shows a counterexample to the conjecture in (Dai & Wipf; 2019) where the support of VAE generators is a superset of that of data distribution. \n- Numerical experiments are presented to support the theory. ", "summary_of_the_review": "The paper gives a sensible discussion, but a lack of practical aspect limits the broader importance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes the behavior of VAEs in modeling data lying on a low dimensional manifold. It formally proves some of the conjectures/informal-statements in an earlier work by Dai and Wipf (2019) in the case of linear VAE and linear manifold, and disproves the same for the nonlinear case. In particular, it proves, by analyzing the objective and its gradient-flow dynamics, that VAE captures the intrinsic dimension of data distribution correctly. For nonlinear cases, the paper shows a counterexample to the conjecture in (Dai & Wipf; 2019) where the support of VAE generators is a superset of that of data distribution. 

Two of the reviewers had raised following specific concerns -- (i) Theory only considers linear encoders and linear/1-hidden layer nonlinear decoders, (ii) The convergence behavior during training is only provided for linear VAEs, (iii) Some statements in the introduction/abstract misrepresent the results in (Dai & Wipf; 2019). However the authors have adequately addressed (i) and (ii) in their response -- the paper shows that the correct manifold will only be recovered in the linear case; in the nonlinear case, even for simple manifolds (1-hidden layer) the correct manifold is not recovered as shown by the counterexamples. Authors have also promised to modify the statements in the abstract and introduction to address the concern in (iii). Other two reviewers are largely positive about the paper. The paper makes an important contribution to the VAE literature in further clarifying VAEs' behavior while modeling low dimensional manifolds, and will be a good addition to the conference program.
Decision: Accept (Poster)