Learning Manifold Dimensions with Conditional Variational Autoencoders
Authors: Yijia Zheng
Tong He
Yixuan Qiu
David Wipf
Venue: NeurIPS 2022 Accept (2022)
Keywords: Conditional Variational Autoencoder
Deep Generative Model

Abstract: Although the variational autoencoder (VAE) and its conditional extension (CVAE) are capable of state-of-the-art results across multiple domains, their precise behavior is still not fully understood, particularly in the context of data (like images) that lie on or near a low-dimensional manifold. For example, while prior work has suggested that the globally optimal VAE solution can learn the correct manifold dimension, a necessary (but not sufficient) condition for producing samples from the true data distribution, this has never been rigorously proven.  Moreover, it remains unclear how such considerations would change when various types of conditioning variables are introduced, or when the data support is extended to a union of manifolds (e.g., as is likely the case for MNIST digits and related).  In this work, we address these points by first proving that VAE global minima are indeed capable of recovering the correct manifold dimension.  We then extend this result to more general CVAEs, demonstrating practical scenarios whereby the conditioning variables allow the model to adaptively learn manifolds of varying dimension across samples.  Our analyses, which have practical implications for various CVAE design choices, are also supported by numerical results on both synthetic and real-world datasets.

Learning Manifold Dimensions with Conditional
Variational Autoencoders
Yijia Zheng Tong He Yixuan Qiu
David Wipf
Department of Statisties, Purdue University
2 AAmazon Web Services School of Statistics and Management, Shanghai University of Finance and Economics Theng7090purdde edu, (htong, daviwipf @amazon com, qiuyixuanOsufe edu.
Abstract
Although the variational autoencoder (VAE) and its conditional extension (CVAE) are capable of sfate-of-the-att results across multiple domains, their precise behavior is still not fully understood, particularly in the context of data (like images) that lie on or near low-dimensional manifold. For example, while prior work has suggested that the globally optimal VAE solution can learn the correct manifold dimension, a necessary (but not sufficient) condition for producing samples from the true data distribution, this has never been rigorously proven. Moreover, it remains unclear how such considerations would change when various types conditioning variables are introduced, or when the data support is extended to union of manifolds (c.g., as is likely the case for MNIST digits and related). In this work, address these points by first proving that VAE global minima are indeed capable of recovering the correct manifold dimension. We then extend this result more general CVAEs, demonstrating practical scenarios whereby the conditioning variables allow the model to adaptively learn manifolds of varying dimension across samples. Our analyses, which have practical implications for various CVAE design choices, are also supported by numerical results on both synthetic and real-world datasets.
Introduction
Variational autoencoders (VAE) 6 and conditional variants (CVAE) 17 powerful generative
models that produce competitive results in various domains such as image synthesis [3]
13
21],
natural language processing [16). time-series forecasting p9 19] and trajectory prediction S As
a representative example, when equipped with an appropriate deep architecture, VAE models have recently achieved state-of-the-arr performance generating large-scale images [IT]. And yet despite
this success, there remain VAE/CVAE behaviors in certain regimes of interest where we lack a precise understanding or supporting theoretical foundation.
In particular, when the data lie on or near a low-dimensional manifold, as occurs with real-world images [12.. it is meaningful to have model that learns the manifold dimension correctly. The latter can provide insight into core properties of the data and be viewed as a necessary, albeit not sufficient, condition for producing samples from the true distribution. Although it has been suggested in prior work 3 4 that a VAE model can learn the correct manifold dimension when globally optimized, this has only been formally established under the assumption that the decoder is linear or affine 2 And the potential ability to learn the correct manifold dimension becomes even more nuanced when conditioning variables introduced. In this regard, of discrete conditions (e.g., MNIST image digit labels) may correspond with different "slices" through the data space, with each
Work completed during internship at the AWS Shanghai AI Labs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

inducing a manifold with varying dimension (intuitively, the manifold dimension of images labelled "1" is likely smaller than those of "5"). Alternatively, is possible to have data expand fully in the ambient space but lie on a low-dimensional manifold when continuous conditional variables are present. Such a situation can be trivially constructed by simply treating some data dimensions, or transformations thereof, as the conditioning variables. In both scenarios, the role of CVAE models remains under-explored.
Moreover, unresolved CVAE properties in the face of low-dimensional data structure extend to practical design decisions as well. For example, there has been ongoing investigation into the choice between fixed VAE decoder variance and a learnable one [3 4 10 [5] 18), an issue of heightened significance when conditioning variables are involved. And there exists similar ambiguity regarding the -odmoney-adopped strategy of sharing weights between the prior and encoder/posterior in CVAES 17]. Although perhaps not obvious at first glance, both cases these considerations are inextricably linked to the capability of learning data manifold dimensions.
Against this backdrop our paper makes the following contributions:
ii) In Section 2.T we provide the first demonstration of general conditions under which VAE global minimizers provably learn the correct data manifold dimension.
(ii) We then extend the above result in Section 2.2 to address certain classes of CVAE models with either continuous or discrete conditioning variables, the latter being associated with data lying on union of manifolds.
(iii) Later, Section 3 investigates common CVAE model designs and training practices, including the impact of strategies for handling the decoder variance as well as the impact of weight sharing between conditional prior and posterior networks.
(iv) Section 4 supports our theoretical conclusions and analysis with numerical experiments on both synthetic and real-world datasets.
2
Learning the Dimension of Data Manifolds
In this section we begin with analysis that applics to regular VAE models with no conditioning. We then later extend these results to more general CVAE scenarios.
2.1 VAE Analysis
We begin with observed variables x X' R4, where X' is the ambient data space equipped with some ground-truth probability measure Wgt. Hence the probability mass of an infinitesimal dæ on X is (dz) and Sx (dz( = VAE models attempt to approximate this measure with parameterized distribution (1)):) instantiated as marginalization over latent variables in p=(x)= J po po(1|2 p))p()((()))))))()))))) (dz Here p(z) N (z|0, f) is a standardized Gaussian prior and pe represents a parameterized likelihood function that is typically referred to as the decoder.
IR* as
To estimate decoder parameters 0, the canonical VAE training loss is formed as a bound on the average negative log- likelhhood given by
((0,0)= = - (og =
(Ll[ 2 f/ log po (x))
(I)
where the latent posterior distribution qa (2|2) (or the VAE encoder) controls the tightness of the bound via trainable parameters ó. Borrowing from [4) we package widely-adopted VAE modeling assumptions into the following definition:
Definition 1 (s-simple VAE) A re-smmle VAE is a VAE model with dim[z] = k latent dimensions, the Gaussian encoder 9o (z(x)) N(=lu, (x; Φ), (o;;; (x;)),)), the Gaussian decoder po (x|2)) N (z))pz (z; 0), rI), and the prior p(2) = N (z I). Here 7 > 0 is trainable scalar included within while the mean functions μt (z: Φ) and pz (2; 8) are arbirrarily-comple L-Lipschitz continuous functions; the variance function 0²( Φ) can be arbitrarily complex with no further constraint.
Our goal this section will be to closely analyze the behavior of simple VAE models when trained on data restricted to Now-dimensione manifolds defined as follows:

---

Definition 2 (Data lying on manifold) Let and d denote two positive integers with < d. Then M, is a simple r-Riemannian manifold embedded in F when there exists a diffeomorphism ip between M, and IRr Specifically, for every E Mr, there exists u .p(x) E IR", where up is invertible and both yp and yp1 are differentiable.
As pointed out in 4 when training k-simple VAEs such manifold data, the optimal decoder variance will satisfy Y + O (i.e, unbounded from below). And as we will soon show, one effect of this phenomena can be to selectively push the encoder variances along certain dimensions of towards zero as well, ultimately allowing these dimensions to pass sufficient information about x through the latent space such that the decoder can produce reconstructions with arbitrarily small error. To formalize these claims, we require one additional definition:
Definition 3 (Active VAE latent dimensions) Let {67, ) denote globally-optine parameters of k-sipple VAE model applied 4 a function of an arbitarr fixed % Then a dimension j E {L....,k) K} of latent variable z is defined as an active dimension (associated with sample x) if the corresponding optimal encoder variance satisfies a (x; 27,3 + 7 O.
We now arrive at the main result for this section:
Theorem (Learning the data manifold dimension using VAFs) Suppose X = M, with r < d. Then for all 2 r, any globally-optimal k-simple VAE model applied to ( satisfies the following:
(i) L00 02) (d - r) log + O(1)
(ii) The number of active latent dimensions almost surely equals r, and
(iii) The reconstruction error almost surely satisfies Fa,,, (x|=²) [|| - [=(=:;0)11]]
While all proofs are deferred to the appendices, we provide high-level sketch here. First, we prove by contradiction that there must exist at least active dimensions with corresponding encoder variances tending
rate of O(r)- If this is not the case, we show that the reconstruction term will grow at a rate of O 2),, leading to an overall loss that is unbounded from above. Next, we obtain upper bound and lower bounds I both of which scale (d r)) log r + O(1) when the number of active dimensions is r. And lastly, we pin down the exact number of active dimensions by showing that the inclusion of any unnecessary active dimensions decreases the coefficient of the log y scale factor, i.c., the factor (d - r) uniquely achieves the minimal loss.
Overall, Theorem I provides number of revealing insights into the VAE loss surface under the stated conditions. First, we observe that although the loss can in principle be driven to minus infinity via sub-optimal solutions 7 - 0, globally-optimal solutions nonetheless achieve optimal rate (i.e., largest possible coefficient on the logoy factor) as well as the minimal number active latent dimensions, which matches the ground-truth manifold dimension Moreover, this all occurs while maintaining a reconstruction error that tends to zero as desired. Additionally, while we have thus far treated y as manually controlled parameter tending to zero for analysis purposes, when transition to a trainable setting, similar intuitions are still applicable. More specifically, around global L2 minimizers, the corresponding optimal > scales as d [x; 1', where (2; 0" 1:: denotes the r optimal latent posterior standard deviations associated with active dimensions. Hence both the decoder variance and the latent posterior variances of active dimensions converge to zero in tandem at the same rate; see the proof for more details. In contrast, along the remaining inactive dimensions we show that lim.- (1; (i)) = ((1), which optimzes the KI. term without compromising the reconstruction accuracy.
In closing this section, we note that previous work 44 has demonstrated that global minima of VAE models can achieve zero reconstruction error for all samples lying on a data manifold. But it was not formally established in a general setting that this perfect reconstruction was possible using minimal number of active latent dimensions, and hence, it is conceivable for generated samples involving a larger number of active dimensions to stray from this manifold. In contrast, to achieve perfect reconstruction using the minimal number of active latent dimensions, as we have demonstrated here under the stated assumptions, implies that generated samples must also lie on the manifold. Critically, the noisy signals from inactive dimensions are blocked by the decoder and therefore cannot produce deviations from the manifold.
3

---

2.2 Extension to Conditional VA
In this section, we extend our analysis to include conditional models, progressing from VAEs to CVAEs. For this purpose, we introduce conditioning variables drawn from some set C. with associated probability measure Pyt such that Sc ( = 1. Moreover, for any c E C there exists a subset X. S X with probability measure wg: satisfying xx. whe (dx) = Collectively we also have Sc Jx. (ge(dx))go(de) = 1. Given these definitions, the canonical CVAE loss is defined as
[(0,6) = Jel2. (Hle,)) (2)2,, + [, wig (dx))
2 J.l. - log Po (d))
which forms an upper bound on the conditional version of the expected negative log-liklihhod. We may then naturally extend the definition of the s-simple VAE model to the conditional regime as follows:
Definition 4 (N-simple CVAE) A k-simple CVAE is an extension of the k-simple VAE with the revised conditional, parameterized prior pa pz(z]c) = (2luuz(c; (); (c: 0)}, the Gaussian encoder (o(z|z, c) = N (1, c; Φ), diag (z c; o)}), and the Gaussian decoder po(x c) c; f), 7)). The encoder/decoder mean functions jtz M2(2,c; 6) and pz(2,, c; f) are arbitrarily- complex L-Lipschitz continuous fanctions, while the prior mean Mz(c; 8) and variance o² (c; 8) Fand functions
(2, (x]
encoder variance o2 c; Φ) can all be arbitrarily-comp functionns with no further constraint.
Likewise, we may also generalize the definition of active latent dimensions, where we must explicitly account for the modified conditional prior distribution which controls the relative scaling of the data.
Definition 5 (Active CVAE latent dimensions) Let f 02) denote globally-optimal parameters of K-simple CVAE model applied to 2 as a function of an arbitrary fixed 7/. Then a dimension j E
(1,...,k) K3 of latent variable is defined as an active dimension (associated with sample pair (x, c)) if the corresponding j-th optimal encoder/prior variance ratio satisfies a (2, c, /025//2(e55 (7)00) + 0
as 7 0.
Note that a CVAE (or VAE) with a standardized (parameter-fie free) Gaussian prior, the prior variance equals one. Hence it follows that Definition is special case of Definition 5 where 0. (c; 8) = I.
Before proceeding to our main result for this section, there one additional nuance to our manifold assumptions underlying conditional data. Specifically, if e follows a continuous distribution, then it can reduce the number of active latent dimensions needed for obtaining perfect reconstructions. To quantify this effect, let M₁ C M, denote the subset of M, associated with X.. Intuitively then, depending on the information about x contained in c, the number of active latent dimensions within MF may be less than r. We quantify this reduction via the following definition:
Definition 6 (Effective dimension of conditioning variable) Given an integer E {0, (0....,r), r). let C, denote subset of C with the following properties: (i) There exists a function g C, - R as well as t dimensions of ((x) denoted as ((x) such that g(c) = sp(x); for all pairs {(c,z) : c E Cr E M where sp is diffeomorphism per Definition 2 and (ii) there does not exist such function g for 1 + 1. We refer to I as the effective dimension of any conditioning variable e E C..
Loosely speaking, Ciainition@jnniiaaaa dimensions of x within M,. Incidentally, if = r, then this definition implies that xr degenerates to a deterministic function of c. Given these considerations, Theorem [ can be extended to CVAE models conditioned on continuous variables as follows:
that any C. can effectively be used to reconstruct t S
Theorem 2 (Learning the data manifold dimension using CVAEs) Suppose C = C, and X. = M, with r 2 1, t 2 and r 2 t. Then any globally-optine K- k-simple CVAE model applied to the loss with ι 2 r, satisfies the following:
Whiile the parameters of the prior mean and variance functions are labeled as 0, this is merely to follow standard convention and group all parameters from the generative pipeline together under the same heading; it is meant to imply that the decoder and prior actually share the same parameters.

---

(i) C( 62) = (d-r+t) tor--toogg--- + O(1), and
(ii) The number of active latent dimensions almost surely equals r - t, and
(iii) The reconstruction error almost surely satisfies Eg,; (z]z,e) [|| =2( O(7).
This result indicates that conditioning variables can further reduce the CVAE loss (by increasing the coefficient on the log y term 7 >>> O) around optimal solutions. Moreover, conditioning can replace active latent dimensions; intuitively this occurs because using to reconstruct dimensions of æ, unlike dimensions of z, incurs no cost via the KL penalty term. Additionally, is worth mentioning that even if the observed data itself is not strictly on a manifold (meaning r = d), once the conditioning variables e are introduced, manifold structure can be induced on Xx, i.e. with t > it follows that d 1" + t > O and the number of active latent dimensions satisfies r - d.
2.3 Adaptive Active Latent Dimensions
Thus far our analysis has been predicated on the existence of single r-dimensional manifold underlying the data z, along with a conditioning variable e that captures t degrees-of-freedees within this manifold. More broadly though, it is reasonable to envision scenarios whereby the data instead lie on a union of manifolds, each with a locally-defined value of r and possibly for continuous conditioning variables. In such instances, both Theorems [and can be naturally refined to reflect this additional flexibility.
While we defer a formal treatment to future work, it is nonetheless worthwhile to consider CVAE behavior within two representative scenarios. First, consider the case where e is now discrete random variable taking a value in some set/alphabet (ox k= such as the label of an MNIST digit 23 (whereby m = 10). It then becomes plausible to assume that r = f(axs) for some function f. meaning that the manifold dimension itself may vary conditioned on the value of (e.g., the space of digits "I" is arguably simpler than the space of digits "8"). In principle then, a suitably-designed CVAE model trained on such data should be able to adaptively learn the dimensions of these regional manifolds. Later in Section 4.4 we empirically demonstrate that when we include a specialized attention layer within the CVAE decoder, which allows it to selectively shut on and off different latent dimensions, the resulting model can indeed learn the underlying union of oow-dimensional manifolds under certain circumstances. From conceptual standpoint, the outcome is loosely analogous to separate, conditoanti VAE being trained on data associated with each Og.-
As second scenario, we may also consider the case where & varies for different values of a continuous conditioning variable c. Extrapolating from Theorem 2 we would expect that the number of active latent dimensions t will now vary across regions of the data space. Hence appropraite CVAE architecture should be able to adaptively compress the active latent dimensions so as to align with the varying information contained in Again, Section 4.4 demonstrates that this is indeed possible.
3 On Common CVAE Model Design Choices
In this section, we review CVAE model designs and training practices that, while adopted in various prior CVAE use cases, nonetheless may have underappreciated consequences, especially within the present context of learning the underlying data manifold dimensionality.
3.1 On the Equivalence of Conditional and Unconditional Priors
Per the canonical CVAE design, it is common to include parameterized, trainable prior (2(2)c) within CVAE architectures [0 7 8 20 24. However, the strict necessity of doing so is at least
partially compromised by the following remark:
Remark (Converting conditional to unconditional priors) Consider a k-simple CVAE model with prior po(z)c), encoder (e(2) c) and decoder po px( c). We can always find another k-simple CVAE model with prior p(2) ~ No, T), encoder qa (2)x0 c) and decoder po (z12, c), such that C(O,0) Φ) = L(O', σ') and pe pz(c) = po (x(c).
Remark [ lindicates that, at least principle, a parameterized, conditional prior is not unequivocally needed. Specifically, detailed in the appendix,
always explicitly convert an existing
5

---

simple CVAE model with conditional prior po(zle) into another k-simple CVAE model with fixed prior p(z) (=( without sacrificing any model capacity in the resulting generative process Por (xlcc) po ())())))(())))))))))))))) c)p(()dz;; essentially the additional expressivity of the conditional prior is merely absorbed into a revised decoder. Even so, there may nonetheless remain differences in the optimization trajectories followed during training such that achievable local minima may at times lack this equivalence.
3.2 The Impact y Initialization on Model Convergence
As emphasized previously, VAE/CVAE models with sufficient capacity applied to manifold data achieve the minimizing loss when y - However, directly fixing y near zero can be problematic for reasons discussed in [31, and more broadly. performance can actually be compromised when is set to any fixed positive constant noted in 10 15 18).
Even so, it remains less well-understood how the initializaton of a learnable y may impact the optimization trajectory during training. After all, the analysis have provided is conditioned on finding global solutions, and yet it is conceivable that different y initializations could influence a model's s ability steer around bad local optima. Note that the value of y at the beginning of training arbitrates the initial trade-off between the reconstruction and KL terms, as well as the smoothness of the initial loss landscape, both of which are factors capable of influencing model convergence. We empirically study these factors in Section 4.5
3.3 A Problematic Aspect of Encoder/Prior Model Weight Sharing
Presumably to stabilize training and/or avoid overfitting, one widely-adopted practice is to share CVAE weights between the prior and posterior/encoder modules [8] 17 19]. For generic, fixed-sized input data this may take the form of simply constraining the encoder as (2(2),, More commonly though, for sequential data both prior and encoder are instantiated as some form of recurrent network with shared parameters, where the only difference is the length of the input sequence g 19]. i... the full sequence for the encoder or the partial conditioning sequence for the prior.
p=(2le) [17].
More concretely with respect to the latter, assume sequential data x = {m:) where is a time index (for simplicity here we will assume fixed length n across all samples, although this can be easily generalized). Then associated with each time point xy within a sample x, we have a prior conditioning sequence ej = I<t - (-, |- The resulting encoder and prior both define distributions over a corresponding latent z via qs (zt (<))) and pa(2) (<))) respectively. Along with the decoder model po (z1|=1, I<t), the revised, sequential CVAE training loss becomes
/ X 1=1
C(0,0) 6)
{1- (ssleec))) Do ( Fi x<]] + KIL
(dx),
where we observe that there is now an additional summation over the temporal index This mirrors the fact that reconstruct (or test time generate) sample x, the CVAE will sequentially produce each time point z conditioned on previously reconstructed (or generated) values. We now analyze one of the underappreciated consequences of encoder/prior weight-sharing within the aforementioned sequential CVAE context.
Theorem (Weight sharing can compromise the performance of sequential CVAEs) Assume sequential data with ground-truth measure Wgt defined such that the probability mass of xı conditioned on xzt lies on a manifold with minimum dimension r >> (this excludes strictly deterministic sequences). Moreover, given k-simple sequential CVAE model assume that the prior is constrained to share weights with the encoder such that po(2)(<<1) go (2)(0e)))) Then the corresponding loss from (3) satisfies L(O, Φ) = (11 for any 0 and Φ.
Given the rigidity of this lower bound, Theorem indicates that the sequential CVAE model is unable to drive the the loss towards minus infinity required to produce active latent dimensions (per Definition 5) and correctly learn oow-dimensional manifolds when present. Intuitively, this relates to subtleties of the differing roles the encoder and prior play in sequence models.
This is defined analogously to the s-mimple CVAE from Definition 4

---

In particular, it follows that the encoder distribution (=(=)(<<1) at time l is in every way identical to the prior at time 1 1+1 1 given that po [<(++]) (0 [#(++1) ) = (0(2)+1 per the adopted weight sharing assumption. But these distributions are meant to serve two very different purposes: (i) The encoder is meant to push the variances of active dimensions to zero so as to learn the underlying manifold, while the remaining/superfuour dimensions merely output useless noise that is filtered by the decoder. (ii) In stark contrast, the role of the prior is not to instantiate reconstructions, but rather to inject the calibrated randomness needed to match the conditional uncertainty in the ground-truth distribution. Therefore, unless the observed data sequences are deterministic, in which case 21 can be predicted exactly from I<t, it is impossible achieve both (i) and (ii) simultaneously. And indeed, we verify these insights via numerical experiments conducted in Section 4.6
Experiments
In this section we first corroborate our previous analysis in controllable, synthetic environment; later we extend to real-world datasets to further support our conclusions 4 The VAE/CVAE model architectures used for our experiments described in Section A of the appendix.
4.1 Datasets and Metrics
Synthetic Data We begin by generating dinnnimnniional data u E R' from a Gaussian mixture model with 5 equiprobability components, each with mean μ4 and transformed variance log ou drawn from a standardized Gaussian distribution. Next, to produce the high-dimensional data manifold X' = M, in Rd, we compute = sigmoid(Gzu), where G2 Raxr is randomly initialized projection matrix. For conditional models, we also compute c = G.ula where Ge E Rt is another random projection. Given these assumptions, can then construct training datasets with varying values of r, d, and l. For all experiments with synthetic data, the training set size is 100,000. We also perform tests using synthetic sequential data described in Section 4.6
Real-world Data We also investigate model behaviors via MNIST 23 and Fashion MNIST 22
These two datasets involve image samples with clear-cut low- low-dimensional manifold structure, while at the same time, they are not so complex that more intricate architectures and training designs are required that might otherwise obfuscate intended message (e.g., more complex models may fail at times to learn correct low-dimensional structure simply because of convergence issues).
By convention, we report the loss from 1 for VAEs and (2) for CVAEs; these losses are often referred to as the negative evidence lower bound (-ELBO) (note though that these values are not always directly comparable across different testing scenarios). We also include auxiliary metrics to diagnose model behavior, including the number of active dimensions (AD), the reconstruction error (Recon), the KL-divergence (KL), and the learned decoder variance 7
Metrics
4.2 Learning Manifold Dimensions with VAEs
We begin by exploring the VAE's ability to learn the correct manifold dimension in support of Theorem Specifically, using the synthetic dataset, we vary the ambient dimension d E {10, 20, 30} and the trouh manifold dimension r (2, 8, 10} and compare with the estimated number of active dimensions with x = (5, 20). Please see Section B of the appendix for details regarding how active dimensions are computed in practice (while in numerical experiments encoder variances will not converge to exactly zero, we can nonetheless observe these variances closely clustering around either 0 or 1 as expected).
Results are shown in Table [ where we observe that when k 2 r, the VAE can generally align the active dimensions with the value of expected once 7 has converged contrast, when K is too small (i.c., = 5), it is no longer possible to learn the manifold dimension (not surprisingly, the -ELBO and reconstruction error are also much larger as well).
small value. In
Complementary results on MNIST and Fashion MNIST are reported in Table While we no longer have access to the ground-truth value of r, we can still observe that increases, the number of active dimensions saturates as expected.
1 'Code available at httt::/ggihuh.co///hnnggyi/ http://wwww.ooooooooii/iiaooln/e
'NNIST is under the CC BY-SA 3.0 license, and Fashion MNIST is MIT-Licensed.
7

---

Table 1: Aligning learned VAE active dimensions (AD) with the ground-truth manifold dimension When surplus of latent dimensions) the VAE largely succeeds as 7 the reconstruction error (Recon), and -ELBO converge to relatively small values; however, when K < r this is not possible.
d
AD
Recon
KL
-ELBO
3x10-1
18.31
1.625x10-
-58.26
4
2.6x10-3
24.22
5.654x10
-29.83
10
9.2x10-3
24.14
3x10-4
-17.39
8
1.27x10-2
27.91
1.4x10-3
-10.38
10
5.99x10
16.39
2.5x10
-6.40
1.6x10-3
17.98
5.052x10-9
-114.52
4
1.75x10-2
23.11
2x10
-60.90
20
3.09x10
28.96
6x10-4
-43.75
8
3.42x10-2
33.83
1.2x10-3
-36.82
10
10
4.74x10-
35.81
11x10-
-28.44
2.6x10-3
18.42
x.221x10-3
-176.74
4
2.73x10
24.60
2x10-4
-00.28
30
4.74x10
31.89
9x10-4
-76.46
8
5.68x10-2
37.28
1.6x10-3
-65.66
10
10
1.13x10-1
35.13
2.5x10-3
47.00
6
1.299x10-1
22.53
211x10-3
-36.97
20
8
3.719x10-1
16.618
8,8x10
-22.60
10
3.564x10-1
15.966
1.113x10
-16.96
Table 2: VAE latent compression real datasets in further support of Theorem
Dataset
AD
-ELBO Recon
5
14.899
-842.286
MNIST
16
9.749
-1065.83
32
13
7.469
-1224.37
5
13.163
-935.127
Fashion MNIST
16
9
9.026
-1216.68
32
9
7.820
-1327.26
4.3 Learning Manifold Dimensions with CVAES
Turning to CVAEs, we provide empirical support for Theorem using synthetic datasets with varying the effective dimension of the conditioning variable For this purpose, set to be the first t dimensions of u by equating G. to a t-dim identity matrix 4t and let d = к = 20, and r 10. across all values of t.
Table shows the results, whereby the CVAE correctly learns that AD
Meanwhile, on MNIST and Fashion MNIST we train CVAEs with the class label of each image as the conditioning variable, and the results with 32 are in Table 4 By comparing with Table we observe that the CVAE model for MNIST has lower AD and -ELBO, suggesting that when conditioned labels, the model can more easily confine the resulting representations to lower- dimensional manifold. In contrast, for MNSHionMNIST the CVAE/VAE results are more similar, indicating that the class labels provide marginal benefit, possibly because their visual complexity and manifold structure is more similar across classes. We also provide an example of the CVAE encoder variances produced on the MNIST dataset in Section B of the appendix.
4.4 Adaptive CVAE Active Dimensions within a Dataset
Continuing with CVAE experimentation, we now tum verifying aspects of Section 2.3 namely, the ability adaptively learn active dimensions that vary regionally within single dataset composed of union of low-dimente din-dmmnsiooal ground-truth manifolds. To this end, we choose discrete indicator

---

Table 3: CVAE latent compression showing AD = - exactly in support of Theorem Here c is the first dimensions of u, i.e. Ge 4t 20, and 10.
t
ELBO
Recon
IKLL
7
AD
1
-31.41
x.61x10-2
33.26
2.4x10-3
3
-36.67
4.66x10-3
27.78
2.4x10-3
7
5
-42.78
4.86x10-2
20.81
2.6x10-3
5
7
-52.39
4.29x10-2
13.72
2,2x10-3
3
9
-62.25
3.84x10-2
6.07
2x10-3
Table 4: CVAE latent compression in real datasets with
32. AD is averaged over all the classes.
AD
Recon
KL
7
-ELBO
MNIST
12
6,044
81.672
0.0063
-1489.42
Fashion MNIST
9
8.773
54.552
0.0102
-1239.09
of each source manifold, 5 in total with E 5) latent dimensions within each respectively. Table 5) shows the results using d = 20 and K 40, noting that both the AD and -ELBO values ctas-connitiioal within a single dataset. Also, we report performance both with and without special attention layer in the decoder that helps to selectively determine which dimensions should be active on sample-by-sample basis. These results indicate that, when equipped with suitably flexible decoder network, the CVAE has the ability to adaptively learn AD values that correctly align with data lying on a union of manifolds.
To further explore adaptive active dimensions, we also consider the case where c is continuous variable with & {2, 4, 6, 8, 10} varying from sample-to-sampll (this is achieved by using the first {2, 4, 6, 8, 10} dimensions of as the conditioning variable). To maintain a constant size of E R lo we apply zero-padding. Using
12, 20, and 20, results are shown in Table again with and without the attention layer that exploits to correctly learn the optimal AD across all values of t.
4.5 The Impact of Initialization on Model Training
We next investigate how the initialization of y impacts VAE/CVAE model convergence as related to the discussion in Section 3.2 Results with synthetic data and = 20, k 20, r = 10, and for the CVAE, are shown in Table as the initial 7 is varied. Here we observe that for the VAE model, the correct number of active dimensions is only learned when is initialized sufficiently small. In contrast, for the CVAE we include results both with and without a parameterized prior. However, given adequately-sizer decoder, both models perform similarly would be expected (note that we used the same decoder for both models, so the parameter-free prior model was technically not strictly equivalent per Remark TT. And with the exception of the log 7 20 case when the parameterized CVAE prior model falls into local minima, they are both able correctly learn the number of active CVAE dimensions as AD
t 5, consistent with Theorem 2 In comparison with the VAE, the CVAE models are likely less sensitive to the initial 7 because the number of active latent dimensions that need to be learned is half that of the VAE (5 versus 10).
Table Adaptively learning CVAE active dimension involving data lying on a union of 5 manifolds with E .l....,3 5}. and d = 20, x = 40. A discrete e labels each manifold/class, and the AD values and -EBOO are computed on a cass-conditional basis.
True AD
AD without attention
AD with attention
ELBO without attention
-ELBO with attention
1
-102225
-114.22
2
-62.42
-99.81
3
3
-60.13
-74.28
4
-28.06
-50.36
5
5
5
-7.58
-59.25

---

Table 6: Adaptively learning active dimensions involving continuous and associated t
(2. 4, 6, 8, 10) varying within single dataset; also r 12, d 20, n. 90. The AD values
and -ELBO are computed on a class-conditional basis.
True AD
AD without attention
AD with attention
EEBO without
-ELBO with attention
attention
10
10
-9.69
-41.49
4
8
10
-33.71
-20.52
6
-27.10
-73.26
4
10
-50.70
-80.64
10
2
-61.77
-55.14
Table Learned active dimensions as initial 7 varied, with 20, 10,
5, 20.
Init log 7
VAE
CVAE p(2)
CVAE po(2/c)
AD
-ELBO
AD
-ELBO
AD
-ELBO
-20
10
-28.99
5
-41.20
5
-40.72
-10
9
-28.57
5
-44.53
5
-45.25
-27.56
5
-44.38
5
-45.2
10
3
-13.89
5
-43.22
5
-43.66
20
-1.7
5
-45.22
4
-37.85
4.6 Performance Degradation Using Shared Weights between Encoder and Prior
Finally, we provide empirical support for Theorem E relating the potential impact of weight-sharing within sequential models. We generate sequences via an autoregresive-mooin--vveees
(ARMA)
process, and we define the conditioning variable c at time / as the concatentation of the previous timepoints within each sequence. Results training CVAE model to generate analogous sequences shown in Table We observe that when the encoder and prior share parameters (first row) the -ELBO cannot be significantly reduced, largely because the reconstruction eтror remains high. In contrast, without shared weights (second row) the model successfully drives the reconstruction error and -ELBO to much smaller values as might be expected based on Theorem 3] Of course larger, more complex models there could be other mitigating factors, such overfitting risks, that might allow shared weights to at times perform relatively better.
Table 8: Impact of weight sharing between encoder and prior on sequential data.
Shared Weights
-ELBO
Recon
KL
True
-2.49
0.374
18.09
0.012
False
-45,015
x
175.99
7.252x10-1
5 Conclusion
this paper we provide several insights into the behavior VAEs and CVAEs applied to data lying on low-dimensional manifolds. This includes formal demonstration that both VAE and CVAE global minima can learn manifold dimensions underlying the data, including those manifolds that have been modulated by conditioning variables. We also explore common CVAE design choices that can have practical implications when applying to various specialized applications, such as sequential data or data composed of union of manifolds.
Limitations Our primary theoretical contributions are predicated on properties of VAE/CVAE global optima assuming sufficient model capacity to represent ground-truth manifolds. However, we have spoken relatively little about strategies for steering optimization trajectories away from bad local solutions that may have poor, unpredictable properties. Similarly, beyond that attention mechanism we mentioned for learning region-specift active latent dimensions, we have not thoroughly explored what types of decoder inductive biases might best align with real-world manifold data.
10

---

References
[1] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In Interna- tional Conference on Artificial Intelligence and Statistics, 2019.
[2] Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint arXiv: 08, 2018
[3] Bin Dai, Li Wenliang, and David Wipf. On the value of infinite gradients in variational autoencoder models. Advances in Neural Information Processing Systems, 34, 2021.
[4] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
[5] Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez. and Aaron C. Courville. Pixelvae: A latent variable model for natural images. In 5th International Conference on Learning Representations, 2017.
[6] Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In International Confer- ence Learning Representations, 2014.
[7] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel Neural relational inference for interacting systems. In International Conference on Machine Learning, pages 2688 2697. PMLR, 2018.
[8] Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajectory prediction. Advances in Neural Information Processing Systems, 34, 2021.
[9] Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Learning to infer causal graphs from time-series data. In Proceedings of the First Conference on Causal Learning and Reasoning. PMLR, 2022.
[10] Gerre-Alexandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. Advances in Neural Information Processing Systems, 31, 2018.
[11] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10775-10784, 2021.
[12] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations, 2021.
[13] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances neural information processing systems, 32, 2019.
[14] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
[15] Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv preprint arXiv:1810.00590 2018.
[16] Tulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[17] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28, 2015.
[18] Hiroshi Takahashi, Tomoharu Iwaaa, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student-t variational autoencoder for robust density estimation. International Joint Confer- ence on Artificial Intelligence, 2018

---

[19] Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592-23660. 2021.
[20] Jakub Tomezak and Max Welling. VAE with a VampPrior. International Conference on Artificial Intelligence and Statistics, 2018.
[21] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[22] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint nen:::7000074444 2017.
[23] Corinna Cortes Yann LeCun. THE MNIST DATABASE of handwritten digits. http://yann
lecun. com/ ete///xdbmm/ modd//mnit//
[24] Jun- Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances neural information processing systems, 30, 2017.
12

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section and Appendix.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're
using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "JYa-FgO2UDh", "reviewer": "Reviewer_pxuR", "summary": "This paper aims to understand the relationship between the true intrinsic dimension $r$ of the data, which is assumed to be smaller than the ambient space dimension $d$ on which the data lives (i.e. the paper assumes the manifold hypothesis holds, $r < d$), and the latent variable dimension $\\kappa$ in Gaussian VAEs. The authors aim to prove that, when $\\kappa \\geq r$, the optimal ELBO achieved by a Gaussian VAE with fixed decoder variance $\\gamma$ behaves like $(d-r)\\log \\gamma$, and that the number of activate latent dimensions (i.e. those whose posterior variance goes to 0) is equal to $r$. The authors argue that this insight can be used to estimate the intrinsic dimension $r$. The authors then aim to extend this result to the setting of conditional VAEs (CVAEs). In a final result, the authors aim to show that the practice of sharing weights between the prior and the approximate posterior, which is commonly used in CVAEs, cannot achieve the same optimal ELBO value as could be achieved without the parameter sharing from their earlier result. The authors then claim that this means the practice of sharing these parameters should not be used.\n\nUnfortunately, the paper is very poorly written, to the point where even mathematical definitions cannot be understood in a precise manner. While I detail why I think this is the case below, I highlight that I am giving this paper a soundness score of 2 not because I necessarily think the results are wrong, but presentation is so unclear that it's very hard to judge. Similarly, I give this a contribution score of 2 not because I think the contribution itself would be bad (I actually think a properly re-written version of the paper could be quite strong), but because once again, it's very hard to judge. My confidence score of 5 aims to reflect the fact that I feel very confident that the paper has very poor exposition.\n\n-------------------------------------------------\nUPDATE AFTER REBUTTAL\n-------------------------------------------------\n\nThe authors have significantly addressed the clarity issues present in their original submission. While I still believe the contribution over [4] on the active dimensions being used at optimality is somewhat oversold, this is not the only contribution made by the paper, and I believe the insights presented about conditioning and intrinsic dimension will be valuable to the community. I have thus increased my score accordingly.", "questions": "I have the following questions:\n\n16. It is claimed that \"this has never been rigorously proven\" in the abstract, and that \"this has only been proven under an assumption that the decoder is linear or approximately linear [2,4]\" when talking about the number of active dimensions estimating the intrinsic dimension. Could you please explain why this is the case? I do not think [4] uses neither a linear nor an approximately linear assumption, and it is also not clear how exactly the results in [4] (particularly the discussion after equation 9 of the arxiv version of [4]) are not rigorous.\n\n17. In Table 4, it is not clear how the AD values are obtained: were these averaged over different $c$'s?\n\n18. How relevant is the VAE setup here? For example, [B] obtains very similar results about the value of the optimal log-likelihood to those of Theorem 1, in a general context of density estimators rather than just VAEs. Similarly, [C] extends the result of [A], which is quite related to the results in the paper being reviewed, from VAEs to density estimators in general. It feels like the ideas presented here about conditional models might be extended to the general setting of density estimators as well. Could you comment on this? To be clear, I am aware that [B] and [C] are very recent and I do not see them not being discussed or compared against in the submitted version as a problem, although I do believe some discussion should be added when the paper is updated.\n\n\n[B] LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood; Tempczyk, Michaluk, Garncarek, Spurek, Tabor & Golinski (2022)\n\n[C] Diagnosing and Fixing Manifold Overfitting in Deep Generative Models; Loaiza-Ganem, Ross, Cresswell & Caterini (2022)", "limitations": "I find it hard to properly assess the limitations of this work in its current state given its poor presentation. If my doubts are clarified during the review process, particularly point 8 about definition 4, and question 16 about the differences with [A], I will update this section later on. That being said, part of the point of this paper is that intrinsic dimension can be estimated with VAEs, yet no comparison is carried out against other methods of estimating intrinsic dimension, e.g. [D].\n\n[D] Maximum Likelihood Estimation of Intrinsic Dimension; Levina & Bickel (2004)", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI believe this paper considers a relevant question in VAEs, whose answer will be of interest to the community. I also believe that the explicit consideration of the interaction between conditioning and how this affects intrinsic dimension to not only be relevant, but also understudied. The authors have correctly spotted a poorly understood area, and I think this type of research to be valuable, when properly communicated.\n\n## Weaknesses\n\nAs I mentioned in my summary, I believe the biggest weakness of this paper is poor exposition. Below I give a list of points that are not properly explained in the mathematical exposition paper (in order of appearance):\n\n1. In definition 1, $\\gamma$ is called a \"tunable scalar\", this really makes it unclear whether or not it is a parameter of the VAE or not. It only becomes clear later, after continuing to read the paper, that the authors mean a fixed hyperparameter.\n\n2. Again, in definition 1, functions being assumed to be \"sufficiently complex\" is not a precise mathematical assumption and should be dropped from formal definitions.\n\n3. In definition 2, $\\gamma_0$ is assumed to be \"arbitrarily small\" satisfying $\\gamma_0 \\leq \\gamma$. This is again highly confusing, as $\\gamma$ was previously \"defined\" as being part of the definition of a $\\kappa$-simple VAE, so this phrasing makes it seem like the VAE does not depend on $\\gamma_0$. Yet, the requirement that $\\sigma_z(x)^2_j = O(f(\\gamma_0))$ would be nonsensical if $\\sigma_z(x)^2_j$ did not depend on $\\gamma_0$. In other words, either $\\sigma_z(x)^2_j$ depends on $\\gamma_0$, in which case we just have a $\\kappa$-simple VAE with $\\gamma_0$ and the requirement that $\\gamma_0 \\leq \\gamma$ makes no sense; or $\\sigma_z(x)^2_j$ does not depend on $\\gamma_0$, in which case it's $\\sigma_z(x)^2_j = O(f(\\gamma_0))$ that doesn't make sense. The notation from this paper is clearly inspired by that of [A], and I would recommend the authors take this one step further and make the dependence of the involved neural networks on their parameters more explicit, as in [A]. For example, changing $\\sigma_z(x)^2$ to $\\sigma_z(x; \\phi)^2$ would make statements much more mathematically precise, e.g. $\\sigma_z(x; \\phi^*_{\\gamma_0})^2_j = O(f(\\gamma_0))$ is much clearer.\n\n4. Again, there is yet more ambiguity in definition 2: the phrasing \"where $f$ is any function that goes to $0$ when $\\gamma_0$ goes to zero\" does not make sense. Requiring that $\\sigma_z(x)^2_j = O(f(\\gamma_0))$ for any such $f$ is equivalent to requiring that $\\sigma_z(x)^2_j = 0$. I believe the authors meant that there exists one such $f$, not that this holds for any $f$. Furthermore, even if I am correct and the authors meant that such an $f$ exists, I believe this would simply be equivalent to requiring that $\\sigma_z(x; \\phi^*_{\\gamma_0})^2_j \\rightarrow 0$ as $\\gamma_0 \\rightarrow 0^+$, which would be a much clearer way to state the requirement (big-O notation is really meant to bound convergence rates, which do not seem to be of relevance in the definition if $f$ is merely required to exist).\n\n5. Theorem 1 says \"the global optimal...\", which implicitly assumes the solution is unique in $\\theta$ and $\\phi$, which I do not believe holds if the neural networks involved are \"sufficiently complex\".\n\n6. Statement $(i)$ in Theorem one provides $\\int_\\mathcal{X} (d-r) \\log \\gamma_0 + O(1) \\omega_{gt}(dx)$, an integral which does not seem to depend on $x$, and which I believe can be written simply as $ (d-r) \\log \\gamma_0 + O(1) $. It would also be good to be more precise as to what is considered a constant and what is not in the $O(1)$ notation. Finally, similarly to the point above, it is not clear what is meant by \"can be uniquely achieved\", and I do not believe this holds: for example, if a particular VAE configuration achieves the optimal ELBO, applying an orthogonal transformation to the input of the decoder would leave $p(x|z)$ unchanged, and thus achieve the exact same optimum using a different function (and changing the encoder appropriately).\n\n7. In definition 3, the prior is missing from the definition of $\\kappa$-simple CVAE. While this is not relevant for VAEs as the prior is just assumed to always be an isotropic Gaussian, this is relevant to the ensuing discussion about the prior being learned as well in the context of CVAEs.\n\n8. Definition 4: This is where it really gets incomprehensible. What is meant by subindexing a vector with $t$? That is, $\\varphi(x)$ is, by definition, a vector in $\\mathbb{R}^r$. In the manuscript, subindexing with an integer has been used before to denote coordinates (e.g. $\\sigma_z(x)^2_j$ is the $j^{th}$ coordinate of $\\sigma_z(x)^2$, which by the way is actually defined as a matrix in definition 1, but at least this abuse is understandable), so it seems that $\\varphi(x)_t$ should be the $t^{th}$ coordinate of $\\varphi(x)$, but the authors go on to say that $\\varphi(x)_t \\in \\mathbb{R}^t$ without explaining exactly what $\\varphi(x)_t$ means. Is it the first $t$ coordinates of $\\varphi(x)$? Furthermore, even if the previous questions were answered, what is $x$ here? The definition involves $g(c) = \\varphi(x)_t$, but it is never explained what $x$ is: the left side seems to depend on $c$ but not $x$, and the right side on $x$ but not $c$.\n\n9. Definition 5: Same issues as in definition 2.\n\n10. Theorem 2: Same issues as in Theorem 1.\n\n11. Corollary 2.1: $\\theta^*$ and $\\phi^*$ are defined as optimal solutions to a CVAE, but given the loss defined for CVAEs in equation 2, these optimal solutions should depend on $c$, and they do not. Usually the objective being minimized for CVAEs is not really equation 2, but rather an expectation of equation 2 over some prior $p(c)$, so it is not clear exactly what $\\theta^*$ and $\\phi^*$ mean, as they do not seem to depend on $c$.\n\n12. Theorem 3: Once again, the notation for the loss on equation 3 is sloppy and not too clear, as the actual loss should not really depend on $l$, but the RHS does. Is there a missing summation over $l$ somewhere? Or should $x_l$ be replaced by $x_{\\geq l}$?\n\nNow I move on to other, non-mathematical, issues I have with the presentation of the paper:\n\n13. The authors write down some issues, (i) through (iv) in the introduction and keep referring to them, and the readers keeps having to go back to see what the point was. It would be clearer to avoid this.\n\n14. In line 161, the prior in CVAEs is defined as $p_\\theta(z|c)$, which is rather confusing notation as $\\theta$ is already used to denote the parameters of the decoder. This notation obfuscates the discussion about parameter sharing, and becomes even more confusing when using $q_\\theta(z|x,c)$ in line 162.\n\n15. Several experiments are carried out on simulated data, but not enough details are given about how this was generated. For example, section 4.3 should really refer to a detailed appendix explaining the setup in more detail.\n\nFinally, the paper has several grammatical errors, typos, and minor points which are technically incorrect; all of which I would encourage the authors to try to fix. While I see these points as very minor compared to the other issues, I list some below:\n\n- line 32: \"each could like on a manifold\"\n- line 33: \"manifold dimension of an image of 1 ...\": a single image has intrinsic dimension 0, it is the manifold itself which has intrinsic dimension $>0$.\n- lines 35-36: \"Such an example ... as the conditioning variable\" is pretty hard to understand as a sentence.\n- line 128: \"Theorem.1\"\n- line 182: \"The cost of VAE balances the three modules with the two terms\" is also hard to understand.\n- Citations are also sloppy: [4] and [5] are the same, and at least one paper is cited as an arxiv preprint rather than as a conference paper, namely [13], which appeared in ICLR 2021. Note that [13] is simply the one I noticed, but I did not extensively check this, and would not be surprised if this happened more than once.\n\n\n[A] Diagnosing and Enhancing VAE models; Dai & Wipf (2019)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FCZg4spwdDb", "reviewer": "Reviewer_R6JP", "summary": "In this work, the authors theoretically analyze the properties of Variational Autoencoders (VAEs) and Conditional VAEs (CVAEs) when the data lies on a low-dimensional manifold. Notably, they demonstrate, under assumptions, VAEs will converge to a solution that will correspond to using the same number of latent dimensions (active dimensions) as the number of dimensions in the data manifold; therefore, allowing for the identification of the number of manifold dimensions. They also extend the analysis when the data lies on a mixture of low-dimensional manifolds. Further, the authors analyze several active practices in their framework and provide theoretical insights into why they may or may not work. Finally, the authors offer some experimental support to their claims. ", "questions": "Please refer to the Strengths and Weaknesses section. ", "limitations": "I think the authors captured the limitation reasonably well in their last comment in the paper. I agree with the authors and believe their insights must be tested thoroughly in different empirical settings. As is, the take-aways from the paper seem minimal if the experiments do not follow the results. Please refer to the Strength and Weaknesses section for more detailed comments. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n### Theoretical Analysis. \nIn this paper, the authors propose theoretical insights into the behavior of VAE and Conditional VAE when the data lies on low dimensional manifolds. Under assumptions, the converged solutions can also be used to determine the manifold dimensions. The authors also analyze several active practices in their framework and provide theoretical insights into why some of them may or may not work. I think there are several interesting and important theoretical insights in this paper. \n\n\n# Weakness\n### Theorem 1 \nI am unclear about what it means to find an optimal solution under $\\gamma = \\gamma_0$. From definition 1, it seems that $\\kappa$-simple VAE is defined under the assumption of $\\gamma$ being a tunable parameter. Can the authors comment on what I am missing here? \n\n### Definition 4\nWhat is $\\psi(x)$ here? Is it the same as L52? Are these the first $t$ dimensions of that vector? Some $t$ dimensions? It is a bit unclear to me. \n\n### Choosing active dimensions \nI am a bit confused about how the active dimensions are being determined. What does $O(\\gamma_0)$ mean in practice in Theorem 1 and 2 mean? Can't there be a huge constant with big-O notation? How do you decide that a dimension is active? I think this crucial to the entire paper, and I could not find any discussion on this.\n\n### Qualitative experiments and visualizations \nCan the authors comment on the lack of visual toy problems? It is often easier to argue with a running example for manifold learning papers. However, I believe the authors can exploit this when discussing the different active practices and how they can harm the final learned model; clear visual benefits for hypothesized techniques can be compelling. \n\nFurther, there are no qualitative experiments in the paper. It would be better if there were visual comparisons alongside the numbers. It is difficult to put the numbers by themselves in perspective without a visual yardstick. \n\n### Synthetic Experiments \nIn L217, the authors say that the generated results are controlled by parameters $r, d, t,$ and $\\kappa$. I am unsure how $\\kappa$ influences the synthetic data generation process. Isn't $\\kappa$ a parameter of the inference? \n\nFurther, I am unsure if the authors report ELBO or negative ELBO under the NLL column (L226). Also, why not report it as ELBO? If you want to claim NLL, probably do other estimation techniques like Importance Weighted ELBO or as Annealed IS-based estimator. \n\nAlso, I think the NLL values are not really informative as they cannot be compared across the rows? In each row, the data changes, and therefore there is a different target NLL value? I think a more informative experiment would be where you vary $\\kappa$ and demonstrate how using more or less capacity changes the NLL, KL, and Recon for the same data? \n\nFurthermore, if I am correct, the last block of rows should be comparable with the second block rows. So why is the reconstruction error lower when we have less number of AD than $r$? \n\n### Real World experiments\nIn the introduction, L32, the authors mention how the different digits can lie on a different manifold. It would have been interesting if the authors analyzed their learned manifolds in this light. \n### Writing\nThe writing of the introduction builds a lot of suspense about what is being addressed. While this may be nice sometimes, providing a summary of the contribution can also be beneficial without forcing the reader to go through the entire section to understand what was done and how. \n\nThe captions to tables should be more informative than they are now. For instance, in Table 3, r = 10 is implied or hidden in the text. Further, it should be mentioned which results do the tables correspond to. For example, it seems to me that Table 2 and 4 are the only real-world results and the rest are all synthetic experiments. The experimental section goes back and forth, and clear descriptions can be helpful. Another example is in Table 7. What is the true number of manifold dimensions?\n\n# Minor things\n- L32. \"each could **like** on.\" \n- L34. \"expand fully in the ambient space **but become on a** low-dimensional manifold when conditioned.\"\n- Several other typos. Please, do a thorough re-read.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BDpk3S0cezf", "reviewer": "Reviewer_ZAHe", "summary": "This paper studies the behaviour of variational auto-encoders (VAEs) and conditional variational auto-encoders (CVAEs) in the presence of data supported on a low-dimensional manifold.\nThe authors demonstrate that VAEs are able to learn the intrinsic manifold dimension of the data at optimality.\nThey also show that a similar result exists for CVAEs, and that effective conditioning should reduce the loss function at optimality.\n\nThe authors then probe some common design choices in VAEs and CVAEs, noting the following:\n1. Conditioned and unconditioned priors are theoretically equivalent.\n2. Learning the decoder variance should result in better performance.\n3. A common weight-sharing technique in autoregressive models should be avoided.\n\nThe experiments section then directly addresses their claims, particularly on the synthetic dataset with known groundtruth intrinsic manifold dimension.\n\n### EDIT AFTER REBUTTAL\n\nI am happy with the discussion that we've had and believe the clarity of the paper can be easily increased beyond the recent revisions which have themselves already increased the clarity. Thus, I am raising my score to a *Weak Accept*.", "questions": "These are the main questions I have of the authors. There are others listed above as well but they are more minor.\n1. Can the authors provide an explanation of the link between active latent dimensions and some notion of intrinsic dimensionality of the generative model in data space?\n2. How do we reconcile the simple Riemannian manifold assumption with the idea of using conditioning to learn \"unions of manifolds\"?\n3. How can we expect to learn $\\gamma$ at all in light of the discussion in Section 3.2?\n4. How valid is maximizing likelihood in the presence of manifold-supported data?\n5. How can we say that the results on MNIST and Fashion MNIST are consistent with intuitive visual complexity?\n6. Can we get $t$ in practice for realistic conditioning variables, and if so can we get more practical understanding of intrinsic dimension from this?", "limitations": "I think the authors have done a reasonable job of addressing the practical limitations of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Overall Assessment\n\nAlong the standard review dimensions of __quality__, __clarity__, and __significance__, we have *both* strengths and weaknesses, while on the dimension of __originality__ we have only strengths.\nAltogether, I believe the weaknesses outweigh the strengths to some degree, and thus I would weakly recommend rejection at this time.\nIf the authors can adequately answer my questions about the paper overall and perhaps address some of the issues with __clarity__, I would be inclined to increase my recommendation.\nI'll elaborate below.\n\n## Strengths\n\n1. I consider part (ii) of __Theorem 1__ to be a major strength of the paper. The utility of this result is clear: VAEs which are well-optimized have the ability to estimate intrinsic manifold dimension, under the assumption that we have a reliable method to estimate the number of active latent dimensions.\n2. I found the paper overall to be a fairly original work. Of course there are quite strong connections to previous work -- namely the _Diagnosing and Enhancing VAEs_ paper -- but the analysis provided here is sufficiently novel and comes with further useful insights.\n3. I found the introduction and motivation of the paper to be very clear, and the general storyline to be straightforward to follow.\n4. __Theorem 2__ part (i) is quite sensible and intuitive: adding good conditioning variables can reduce the loss function.\n5. I find Section 3.1 to be quite interesting.\n6. I think pursuing ideas around \"unions of manifolds\" to be quite intuitive and worthwhile. I appreciate that this paper is bringing this idea to the table.\n7. __Theorem 3__ is also an interesting result that appears to shed light on an issue with a common practical technique.\n8. Table 1 is a strong result - it is nice to see that $r$ = AD in almost all cases where $\\kappa \\geq r$.\n9. Table 3 is also a strength - we can clearly see AD = $r - t$ here.\n10. Table 8 is a strong result.\n\n## Weaknesses\n\nThere are some major weaknesses to which I would first like to devote individual sections, and then I'll comment on minor issues later on.\n\n### On the Relationship Between Active Latent Dimensions and Manifold Dimension\n\nTheorem 1 shows that a VAE trained to optimality will have its number of active latent dimensions exactly equal to the intrinsic manifold dimension of the data.\nWe would ideally also have some result that directly relates the number of active latent dimensions to the intrinsic dimensionality of the overall generative model, so that we could say samples from the generative model have dimensionality approximately equal to the true data dimension.\nYet I cannot easily see such a link between the two quantities, which reduces the significance of the result.\nIt is stated in the text that active latent dimensions are those \"used for reconstruction\"; is it then the case that the \"inactive\" latent dimensions do not provide any dimensionality to the generative model?\n\nAdmittedly, it is possible that I have not fully understood Definition 2 in-depth, but I would also argue that the Definition 2 (and Definition 5 by extension) is itself not very clear.\nI am left lacking intuition on what active dimensions actually _mean_: simply stating that the encoder variance goes to zero with $\\gamma$ does not provide much insight (and by the way, how does $\\gamma_0$ factor in here? It appears that we are replacing $\\gamma$ in the $\\kappa$-simple VAE definition with $\\gamma_0$, but then why do we need to say $\\gamma_0 \\leq \\gamma$?).\n\nSome clarity and further discussion would be appreciated around both of these points here.\n\n### Significance of Theorem 2 / Corollary 2.1 and CVAE Analysis\n\nAs far as I understand, Theorem 2 part (ii) and Corollary 2.1 (by the way, I think there is an error in Corollary 2.1: shouldn't the active dimensions be $r-t_1$ and $r-t_2$, respectively?) are statements about the mechanics of the encoder in latent space.\nYet I am again left wondering how this relates to the dimensionality learned by the generative model in data space, as I see no direct correspondence between the two.\nWhy should changes in effective dimensionality in the latent space correspond to changes in intrinsic dimensionality in the data space?\nHow does this result in \"different data manifolds\"?\n\nI am also a bit confused about how the proposed capacity of CVAEs to learn unions of manifolds relates to the original assumption from L61 that states the data lives on a simple Riemannian manifold with constant dimension and a single chart. \nHow do we reconcile these two seemingly contradictory points?\n\n\n### Minor Weakness, Issues, and Calls for Clarification\n\n1. I appreciate that Eq. (1) is still a lower bound on the average log-likelihood of the density $p_\\theta$ over the data distribution $\\omega_{gt}$. However, I am curious about the authors' opinion of the validity of the maximum likelihood objective for a full-dimensional density in the presence of manifold-supported data? In particular, we can no longer interpret the maximum likelihood objective as minimizing the KL divergence from $\\omega_{gt}$ to the distribution induced by $p_\\theta$ over $\\mathbb R^d$, as $\\omega_{gt}$ is not absolutely continuous with respect to $p_\\theta$'s distribution.\n2. Along the same vein, it is inappropriate to compare NLL/ELBO values across data with different manifold dimensions and thus dominating measures; yet the experiment section appears to do this a few times. Some care here would be prudent.\n3. I find Section 3.2 difficult to understand. How can we expect $\\gamma$ to adaptively balance the two terms in the loss function when the optimal learning behaviour is simply to make $\\gamma$ as small as possible? Based on this analysis, I find it remarkable that learning $\\gamma$ can succeed at all.\n4. The headline of Section 3.3 does not accurately represent the scope of Theorem 3, which strictly relates to conditional autoregressive models. While Theorem 3 is indeed interesting, the section purports to make a more general statement about weight sharing that does not appear.\n5. What is meant by \"sufficiently complex\" in L70?\n6. [4] and [5] are the same citation.\n7. L97: It is claimed that \"the ratio of the norm term in reconstruction and $\\gamma$ is constant, i.e. the data dimensions $d$.\" Why wasn't this ever checked in practice?\n8. Parentheses appear to be missing from the integrand in lines 85, 133, and 172\n9. What is the role of $\\kappa$ in Theorems 1 and 2 beyond simply $\\kappa \\geq r$? Why doesn't $\\kappa$ appear in the bounds?\n10. In definition 4, is $\\phi$ the same $\\phi$ as in L62?\n11. Why does the prior variance take $x$ as an argument in definition 5?\n12. L153-155 is written in a very unclear way.\n13. L219-221 is a very weak and nebulous discussion.\n14. In L249 it is stated that $c = u_{1:t}$, but earlier we have $c = h'(u_{1:t}) = G'u_{1:t}$. Which one is correct? Did you just take $G' = I_t$? \n15. In L239-240 it is stated that \"the number of active dimensions equals to data's intrinsic dimension\" and L256-257 it is stated that \"This characterization of dataset complexity is consistent with their intuitive visual complexity\". I would have to strongly disagree with this: in my opinion, MNIST is \"less complex\" than Fashion-MNIST, so I don't quite understand why it ends up with lower active dimensions and thus lower intrinsic dimension. Would the authors clarify what was meant here?\n16. Why is $\\kappa = 90$ so large in the Continuous Condition experiment? Is this a typo? Seems very strange.\n17. I generally find the details around Table 5 and Table 6 to be lacking. How many datapoints from each class? How is the reconstruction error for each class, i.e. has the CVAE learned each of the manifolds or just the dimensionalities? Is $t$ considered to be $0$-dimensional in the case of Table 5?\n18. What is the correct number of AD in Section 4.4?\n19. Is there some gap between the theory (Theorem 3) and the result in Table 7 about the equivalence of $p(z)$ and $p(z \\mid c)$? It appears that the conditional prior does a better job in the more difficult case of $\\log \\gamma = 20$ initially. Is it perhaps easier to learn with the conditional prior?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept