A Geometric Perspective on Variational Autoencoders
Authors: Clément Chadebec
Stephanie Allassonniere
Venue: NeurIPS 2022 Accept (2022)
Keywords: Variational autoencoders
latent space modeling
Riemannian geometry

Abstract: This paper introduces a new interpretation of the Variational Autoencoder framework by taking a fully geometric point of view. We argue that vanilla VAE models unveil naturally a Riemannian structure in their latent space and that taking into consideration those geometrical aspects can lead to better interpolations and an improved generation procedure. This new proposed sampling method consists in sampling from the uniform distribution deriving intrinsically from the learned Riemannian latent space and we show that using this scheme can make a vanilla VAE competitive and even better than more advanced versions on several benchmark datasets. Since generative models are known to be sensitive to the number of training samples we also stress the method's robustness in the low data regime.

A Geometric Perspective on Variational Autoencoders
Clément Chadebec
Stéphanie Allassonnière Université Paris Cité, INRIA, Inserm, SU Centre de Recherche des Cordeliers stephanie allaassoniieeeiniii..
Université Paris Cité, INRIA, Inserm, SU Centre de Recherche des Cordeliers clement. chadebec@inria.c
Abstract
This paper introduces new interpretation of the Variational Autoencoder frame- work by taking fully geometric point of view. We argue that vanilla VAE models unveil naturally a Riemannian structure in their latent space and that taking into consideration those geometrical aspects can lead to better interpolations and an improved generation procedure. This new proposed sampling method consists in sampling from the uniform distribution deriving intrinsically from the learned Riemannian latent space and we show that using this scheme can make vanilla VAE competitive and even better than more advanced versions on several bench- mark datasets. Since generative models are known to be sensitive to the number training samples we also stress the method's 's robustness in the low data regime.
Introduction
Variational Autoencoders (VAE) 25 43 are powerful generative models that map complex input data in much lower dimensional space referred to as the latent space while driving the latent variables to follow given prior distribution. Their simplicity
in practice has made them very attractive models to perform various tasks such as high-fidelity image generation 41), speech modeling [5),
clustering [52] data augmentation
g
Nonetheless, when taken their simplest version, it was noted that these models produce blurry samples on image generation tasks most of the time. This undesired behavior may be due to several limitations of the VAE framework. First, the training a VAE aims at maximizing the Evidence Lower BOund (ELBO) which is only lower bound on the true likelihood and so does not ensure that we are always actually improving the true objective 6 may be simplistic
13 533 Second, the prior distribution [14] leading poor data generation and there exists guarantee that the actual distribution of the latent code will match a given prior distribution inducing distribution mismatch [1]. Hence, trying to tackle those limitations through richer posterior distributions [46 42
better priors (50) represents a major part of the proposed improvements over the past few years. However, the tractability of the ELBO constrains the choice in distributions and finding a trade-off between model expressiveness and tractability remains crucial. In this paper, we take a rather different approach and focus the geometrical aspects vanilla VAE is able capture in its latent space. In particular, we propose the following contributions:
We show that VAEs unveil naturally a latent space with a structure that can modeled as a Riemannian manifold through the learned covariance matrices in the variational posterior distributions and that such modeling can lead to better interpolations.
We propose a natural sampling scheme consisting in sampling from a uniform distribution defined on the learned manifold and given by the Riemannian metric. We show that this procedure improves the generation process from vanilla VAE significantly without complexifying the model nor the training. The proposed sampling method outperforms more advanced VAE models terms of Frechet Inception Distance 200] and Precision and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Remark 2.1. In practice, po(x)2) 2)) is chosen depending on the modeling of the input data but is often taken as a simple distribution (e.g fixed variance Gaussian, Bernoulli ...) and a weight 3 can be applied to balance the weight of the KL term [21]. Hence, the ELBO can also be seen Recall 45
Scores on four benchmark datasets. We also discuss and show that it can benefit more recent VAES as well. An implementation is available on github
We show that the method appears robust to dataset size changes and outperforms even more strongly peers when only smaller sample sizes are considered.
We discuss the link of the proposed metric to the pull-back metric.
2 Variational autoencoders
Considering that we are given IRP set of data points deriving from an unknown distribution
p(z),, a VAE aims at inferring with a parametric model {pe, / E )) using a maximum likelihood estimator. A key assumption behind the VAE is to assume that the generation process involves latent variables living in lower dimensional space such that the generative model writes
e~ple p(2)
I H~ p=(2)))
where p is a prior distribution over the latent variables often taken as standard Gaussian and po (z|=) is referred to as the decoder and is most of the time taken parametric distribution the parameters of which are estimated using neural networks. Hence, the likelihood po writes:
pe( po(x12))(=)d22.
(1)
2
As this integral is most of the time intractable is P=(2(z),, the posterior distribution. Hence, Variational Inference [23] is used and simple parametrized variational distribution (2(2)z) is introduced approximate the posterior (z(xx). (2(2)) is referred to as the encoder and, in the vanilla VAE, gs is chosen as a multivariate Gaussian whose parameters Pe and Eg are again given by neural networks. unbiased estimate po of the likelihood (o(x) can then be derived using importance sampling with (2(22) and the ELBO objective follows using Jensen's 's inequality:
log po (a)) log Ee--gs [Po] 2 Ezzgo [lo po] Ezzgg log po(po())))) KT (=|=)))))))))) C (2)
The ELBO is now tractable since both (=(x))) and (zrl)) are known and so can be optimized with respect to the encoder and decoder parameters.
two terms objective 18]. The first one is a reconstruction term given by Pe (x(==) while the second one is a
regularizer given by the KL between the variational posterior 96 and the prior p. For instance, in the case of a fixed variance Gaussian for pe p0(x(2) we have
LRFC
||x 10c--191 (=) BREG = B -L)(o(s))))lp()).
(3)
3 Related work
natural way to improve the generation from VAEs consists in trying to use more complex priors [22] than the standard Gaussian distribution used in the initial version such that they better match true distribution of the latent codes. For instance, using Mixture of Gaussian [35] 16] or Variational Mixture of Posterior (VAMP) (50) as priors was proposed. In the same vein, hierarchical latent variable models [48 (66] or prior learning IT 2 have recently emerged and aimed at finding the best suited prior distribution for a given dataset. Acceptancccrejecciion sampling method was also proposed to try to improve the expressiveness of the prior distribution [4. Some recent works linking energy-based models (EBM) and VAEs [5]] or modeling the prior an EBM 38 have demonstrated promising results and are also worth citing.
On the ground that the latent space must adapt to the data well, geometry-awant latent space modelings as hypershpere [15), torus [17 or Poincaré disk 34 or discrete latent representations
41 were proposed. Other recent contributions proposed to see the latent space as a Riemannian manifold where the Riemannian metric is given by the Jacobian of the generator function 10 47
This metric was then used directly within the prior modeled by Brownian motions 24] Others

---

proposed to learn the metric directly from the data throughout training thanks to geometry normalizing flows 9 or learn the latent structure of the data using transport operators [12. While these geometry- -aaeed methods show interesting properties of the learned latent space they either require the computation of a time consuming model-dependent function, the Jacohian, or add further parameters to the model to learn the metric or transport operators adding some computational burden.
Arguing that VAES are essentially autoencoders regularized with Gaussian noise, Ghosh et al. [18] proposed another interesting interpretation of the VAE framework and showed that other types of regularization may be of interest as well. Since the generation process from these autoencoders is no longer relying on the prior distribution, the authors proposed to use ex-pos ox-post density estimation by fitting simple distributions such as a Gaussian mixture the latent space. While this paves the way for consideration of other ways to generate data, it mainly reduces the VAE framework to an autoencoder while we believe that it can also unveil interesting geometrical aspects.
Another widely discussed improvement of the model consists in trying to tweak the approximate posterior in the ELBO so that it better matches the true posterior using MCMC methods [46] or normalizing flows 42 For instance, methods using Hamiltonian equations in the flows to target the true posterior were proposed.
Finally, while discussing the potential link between PCA and autoencoders some intuitions arose on the impact of both the intrinsic structure of the variance of the data [40] and the shape of the covariance matrices in the posterior distributions 44 on disentanglement in the latent space. We also believe that these covariance matrices indeed play a crucial role in the modeling of the latent space but in this paper, we instead propose to see their inverse as the value of a Riemannian metric.
Proposed method
In this section, we show that vanilla VAE unveils naturally Riemannian structure in its latent space through the learned covariance matrices in the variational posterior distribution. We then propose new natural generation scheme guided by this estimated geometry and consisting in sampling from a uniform distribution deriving intrinsically from the learned Riemannian manifold.
4.1 A word on Riemannian geometry
First, we briefly recall some basic elements of Riemannian geometry needed in the rest of the paper. A more detailed discussion on Riemannian manifolds may be found in Appendix A. A d-dimensional manifold M is manifold which locally homeomorphic to a d-dimensional Euclidean space. the manifold M is further differentiable it possesses tangent space T, at any E M composed of the tangent vectors of the curves passing by z. If M is equipped with a smooth inner product g : (-1-)= defined on its tangent space T3 for any manifold and g is the associated Riemannian metric. Then, a local representation of g at any 2 E M is given by the positive definite matrix Gzz) (See Appendix A). If M is connected, a Riemannian distance between two points 21, 22 of M can be defined
M then M is called Riemannian
distc 22) = inf / Vi(U (10)G(())))))))) = inf L(7) s.t. 21 7()) 22 = (b), 7
iinf
where L is the length of curves y R + M traveling from to 22. Curves minimizing L and parametrized proportionally the are length are geodesic. The manifold M is said to be geodesically complete if all geodesic curves can be extended to R. In an Euclidean space, G reduces to Ta and the distance becomes the classic Fuclidean one. A simple extension of this Fuclidean framework consists in assuming that the metric is given by a constant positive definite matrix Σ different from lg. In such case the induced Riemannian distance is the well-known Mahalanobis distance
disty (2,, 22) (22 21) Σ 22 21)
4.2
The Riemannian Gaussian distribution
Given the Riemannian manifold M endowed with the Riemannian metric G and a chart z, an infinitesimal volume element may be defined on each tangent space T2 of the manifold M as follows
d.M. det G(z)dz,
(5)
3

---

with da being the Lebesgue measure. This defines canonical measure on the manifold and allows to extend the notion of random variables Riemannian manifolds whose density can be defined with respect to that Riemannian measure (see Appendix A). Hence, a Riemannian Gaussian distribution
on M can be defined using the Riemannian distance of Eq. + instead of the Euclidean one.
distt (istea(z,1)) 1)) (2
(2, 20
Nriom (zlo, p)) T exp
( 20
C
exp
JM,, (6)
where dd2 is the volume element defined in Eq. ( Thus, multivariate normal distribution with covariance matrix Σ is only specific case of the Riemannian distribution with o = 1 and defined on
the manifold M (IR G) where G is the constant Riemannian metric G(2) ---, 222 M.
4.3 Geometrical interpretation of the VAE framework
Within the VAE framework, the variational distribution qo (2(2c) is often chosen as a simple multivari- ate Gaussian distribution defined on Rd with being the latent space dimension. Hence, as explained in the previous section, given input data point I2, the posterior (=(2) (N(((()), E(x;)) can also be seen as a Riemannian Gaussian distribution where the Riemannian distance is simply the distance with respect to the metric tensor Σ Σ- (x:)) Hence, the VAE framework be seen as follow: As with an autoencoder, the VAE provides a code u(x;) which is lower dimensional representation of an input data point z;. However, it also gives tensor Σ- (xx) depending on 2ς which can be seen as the value of a Riemannian metric G at p(xz) i.e.
G(u(z)) = =I= (x))
This metric is crucial since it impacts the notion of distance in the latent space now seen as the Riemannian manifold M = (IRd, G) and so changes the directions that are favored in the sampling from the posterior distribution 1s (2(z;)) Then, a sample = is drawn from a standard (i.e. a = 1 in Eq. (6) Riemannian Gaussian distribution and fed to the decoder. Since we only have access to finite number of metric tensors Σ (zz). as a first approximation the VAE model assumes that the metric is locally constant close to u(z;;) and so the Riemannian distance reduces to the Mahalanobis distance in the posterior distribution. This drastically simplifies the training process since now Riemannian distances have closed form and are easily computable. Interestingly, the VAE framework will impose through the ELBO expression given in Eq. (3) that & gives a sample x ~ p=(x=2) close to I; when decoded. Since z has probability density function imposing higher probability for samples having the smallest Riemannian distance to jr, the VAE imposes in a way that latent variables that are close in the latent space with respect to the metric G will also provide samples that are close in the data space X' in terms of L22 distance as noticed in Remark. 2.1 Noteworthy is that the latter distance can be amended through the choice of the decoder pe (z]22) This is an interesting property since allows the VAE to directly link the learned Riemannian distance in the latent space to the distance in the data space. The regularization term Eq. (3) ensures that the covariance matrices do not collapse to Og and constraints the latent codes to remain close to the origin easing optimization. Finally, at the end of training, we have a lower dimensional representation of the training data given by the means of the posteriors μu(z;) and a family of metric tensors (G; = Σ '(z))) corresponding to the value of a Riemannian metric defined locally on the latent space. Inspired from [19, we propose to build a smooth continuous Riemannian metric defined on the entire latent space as follows:
N G(z)
disty: - (a,) ((z:)) ((n))e
E Σ 1=1
(z:) -w (2) e -11:12. La. wi(z) exp
(7)
p(x;)) is the Riemannian distance between and ((;;;) with respect to the locally constant metric (p((z)) -xx(x) Since the sum in Eq. 77 is made on the total number of training samples N, the number of centroids (J((z;)) and so of reference metric tensors can be decreased for huge datasets by selecting only A < N element: J and increasing p to reduce memory usage. We provide an ablation study on the impact of X, the number of centroids and their choice along with a discussion on the choice for p in Appendix F. The parameter is only there ensure that the volume of (Rd G) is finite, property that is needed in Sec. 4.5 and its value can be set as close as desired to zero so the norm of does not influence the metric
where ((:) p(z;))²
This may be performed with k-medoids algorithm.

---

close to the centroids. In practice, it is set below computer precision (i.e. T 2 0). Rigorously, the metric defined in Eq. 7 should have been used during the training process. Nonetheless, this would have made the training longer and trickier since it would involve i) the computation of Riemannian distances that have no longer closed form and so make the resolution of the optimization problem in Eq. (4) needed, ii) the sampling from Eq. (6) which is not trivial and iii) the computation of the regularization term. Moreover, for small values 3 in Eq. (3), the samples generated from the variational distribution ((((x;) [x;)) can be assumed to be concentrated around μ(x;) and so we have the following first-order Taylor expansion around μ((x;)
G(=) +
Σ E (x;) ()(u(zi)) + +- (2rs) Ja (u(ax))( ((s;)), (8) $=1
where Ju.; (f((2;)) is the Jacobian of the interpolant w; evaluated at (xx;;)) Note that we have further
assumed small enough and À to neglect the influence of the other E(r;) in Eq. 77. Hence by approximating the value of the metric during training by its value at µ((;;) (i.e. Σ -x))),). the VAE training remains unchanged, stable and computationally reasonable since Riemannian Gaussians become multivariate Gaussians in (=(z)) as explained before. Noteworthy is the fact that following the discussion on the role of the KL loss in the VAE framework and the experiments conducted in [18) in our vision of the VAE, the prior distribution is only seen as regularizer though the KL term and other latent space regularization schemes may have been also envisioned. the following, we keep the proposed vision and do not amend the training.
4.4 Link with the pull-back metric
It has been shown that a natural Riemannian metric on the latent space of generative models can be the pull-back metric given by G(z) = J (z) J Jz(z)
[ and induced by the decoder mapping
12d + D outputing the parameters of the conditional distribution pe pe(z]2). Actually, there exists strong relation linking the metric proposed in this paper to the pull-back metric. Indeed, assuming that samples from the variational posterior 2 ~ 90(2[z) = (f((z), E(x) remain close x(x) (e.g. by setting small / in Eq. (3)) allows consider an approximation of the log density h(2) = log (x|²) 2) next to ((x) for a given x 28
1 12 u(x)) H, (n(z))) -m(x))
h(2) h(p(z)) +
m(=))
where Jn, (m(x)) is the Jacobian and (u(x)) is the Hessian of h. Using this and remarking that Ex Exng.
(2) 10] =0 and E Exrgo (2= 14) (n) - p)] (H)
makes the ELBO Eq. 3 write:
L h((()) + 1 + TTHE (T(gg())))ll()))
Assuming standard Gaussian prior, Kumar and Poole 28 showed that Σ maximizing the ELBO is
(x) (La H, (u(z)))
(10)
and if we further assume some regularity on the neural networks used for the decoder mapping (e.g. piece-wise linear activation functions) we have
Exx) = ((o)) -
1
(11)
where (z(o((x))) is the Hessian of log (o; g(2)). A standard case for the VAE is to assume that pe (2]22) ((po(z), a Ip) and so gives (g(((()))) = - Ip If we further set o 1
Eq. [11 gives a relation between the pull-back and the metric we propose
(x) ((((z)) J (x(x))) Ia-
Hence, the proposed metric is closely linked to the pull-back metric and may be useful to approximate it (at least close to the ((z)) and avoid the computation of potentially costly function.
5

---

Affine
Our
Affine
O
O
O
Figure I: Top left: Visualization and interpolation in 2D latent space learned by VAE trained with binary images of rings and disks. The log of the metric volume element det G(z) (proportional to the log of the density we propose to sample from) is shown in gray scale. Top middle and right: Riemannian distance from starting point (color maps). The dashed lines affine interpolations between two points in the latent space and the solid ones are obtained by solving Eq. (12). Bottom: Decoded samples along the interpolation curves.
4.5 Gearetry-aware sampling
Assuming that the VAE has learned latent representation of the data in space seen Riemannian manifold, we propose to exploit this strong property to enhance the generation procedure. natural way to sample from such latent space would consist in sampling from the uniform distribution intrinsically defined on the learned manifold. Similar to the Gaussian distribution presented in Sec. the notion of uniform distribution indeed be extended to Riemannian manifolds. Given
a set C M having a finite volume, a Riemannian uniform distribution on A writes (39
1a(2) Vol(A) JM Ia(3)dM.
La(2)
P^(2)
This density taken with respect to dM2,, the Riemannian measure but using Eq. ( and coordinate system allows to obtain a pdf defined with respect to the Lebesgue one. Moreover, since the volume of the whole manifold M (Rdd, G) finite, we can now define a uniform distribution on M
det G(2)
Ukiem (2) SRd V det G(2)dz
Since the Riemannian metric has closed form expression given by Eq. 7 sampling from this distribution is quite easy and may be performed using the HMC sampler [36] for instance. Now we are able to sample from the intrinsic uniform distribution which is natural way of exploring the estimated manifold and the sampling is guided by the geometry of the latent space. A discussion on
practical outcomes can be found in Appendix B. Noteworthy is the fact that this approach can also be easily applied to more recent VAE models having a Gaussian posterior (e.g. 6 29 [5) 32)). We detail this and show that the proposed method also benefit these models in Appendix G.
4.6 Illustatioon on a toy dataset
The usefulness of such sampling procedure can be observed in Figure where a vanilla VAE was trained with a toy dataset composed of binary images of disks and rings of different size and thickness (example inspired by (8). On the left presented the learned latent space along with the embedded training points given by the colored dots. The log of the metric volume element is given in gray scale.

---

In this example, we clearly see a geometrical structure appearing since the disks and rings seem to wrap around each other. Obviously, sampling using the prior (taken as a (O, Iz)) in such case is far from being optimal since the sampling will be performed regardless of the underlying distribution of the latent variables and so will create irrelevant samples. To further illustrate this, we propose to interpolate between points in the latent space using different cost functions. Dashed lines represent affine interpolations while the solid ones show interpolations aiming at minimizing the potential V(2) = (Vdel G(=))-+ all along the curve ie. solving the minimization problem
inf ! ((((t))dt s.t.
7(0) = 21, 7(1) = 22
(12)
In Figure are presented the decoded samples all along the interpolation curves. Thanks to those interpolations we can see that i) the latent space seems to really have specific geometrical structure
since decoding all along the interpolation curves obtained by solving Eq. [22 leads to qualitatively satisfying results, ii) certain locations of the latent space must be avoided since sampling there will produce irrelevant samples (see red frames and corresponding red dashes). Using the proposed sampling scheme will allow to sample in the light-colored areas and ensure that the sampling remains close to the data iee. where information is available and so does not produce irrelevant images when decoded while still proposing relevant variations from the input data.
5 Experiments
In this section, we conduct a comparison with other VAE models using other regularization schemes, more complex priors, richer posteriors, ex-post density estimation or trying to take into account geometrical aspects. In the following, all the models share the same auto-encoding neural network architectures and we used the code and htper-parameteen provided by the authors if available" See Appendix D for models descriptions and the comprehensive experimental set-up.
5.1 Generation with benchmark datasets
First, we compare the proposed sampling method to several VAE variants such as a Wasserstein Autoencoder (WAE) 44), Regularized Autoencoders (RAEs) 18, vamp-prior VAE (VAMP) (50), a Hamiltonian VAE (HVAF) 7 a geometry-aware VAE (RHVAF) p and an Autoencoder (AE). We elect these models since they different ways generate the data using either the prior or ex-post density estimation. For the latter, we fit a 10-component mixture of Gaussian in the latent space after training like [8]
Figure 2 shows a qualitative comparison between the resulting generated samples for MNIST 30
and CELEBA BiI, see Appendix C for SVHN [37] and CIFAR 10 (7). Interestingly, using the non-prior based methods seems to produce qualitatively better samples (rows 7 to end). Nonetheless, the resulting samples seem even sharper when the sampling takes into account geometrical aspects of the latent space as we propose (last row). Additionally, even though the exact same model is used, clearly see that using the proposed method represents strong improvement of the generation process from a vanilla VAE when compared to the samples coming from a normal prior (second row). This confirms that even the simplest VAE model actually contains a lot of information in its latent space but the limited expressiveness of the prior impedes to access to it. Hence, using more complex priors such as the VAMP may be tempting idea. However, one must keep in mind that the ELBO objective in Eq. must remain tractable and so using more expressive priors may be impossible.
These observations are even more supported by Table D where we report the Frechet Inception Distance (FID) and the precision and recall (PRD) score against the test set to assess the sampling quality and diversity. Again, fitting a mixture of Gaussian (GMM) in the latent space appears to be an interesting idea since it allows for a better expressiveness and latent space prospecting. For instance, on MNIST the FID falls from 40.7 with the prior to 13.1 when using GMM. Nonetheless, with the proposed method we are able to make it even smaller (8.5) and PRD scores higher without changing the model and performing post processing. This can also be observed on the other datasets. Impressively, in almost all cases, the proposed generation method can either compete or outperform peers both in terms of FID and PRD scores.
We also perform wider hyper-parameed search on MNIST and CELEBA for each model in Appendix C
7

---

MNIST
CELEBA
AB-N -
7
7
5
4
4
VAE N
0
3
6
3
7 3
Q
WAE
J
4
9
6
1
8
3
9
6
0
4
5 7
VAMP
9
0
3
4
0
6
HVAE
RHVAE
4
8
/
2
9
6 4 3
9
7
0
6
6
6
8
AE GMM
VAB-GMM
2
2
0
5
8
5
RAE
2
6
4
4
9
VAE Ours
5
/
7
3
8
4
Figure 2: Generated samples with different models and generation methods. Results with RAE variants are provided in Appendix C.
Near.
Near. rec.
Near.
Near.
Near,
Near.
Near.
Near.
Gen.
rEc.
5 5
5
9
9
9
3
3 3
2
2
2
reconstruction vs. generation
MNIST 11.27
CELEBA 30.12
FID
Figure 3: Left: Nearest train image (near. train) and nearest image in all reconstructions of train images (near. rec.) the generated one (Gen.) with the proposed method. Note: the nearest reconstruction may be different from the reconstruction of the nearest train image. Right: The FID score between 10k generated images and 10k reconstructed train samples.
Finally, check if the proposed method does not overfit the training data and is able produce diverse samples by showing the nearest neighbor in the train set and the nearest image all the reconstructions of the train images to a generated image in Figure 3 (left). We also provide the FID score between 10k generated samples and 10k train reconstructions in Figure (right). These experiments show that the generated samples are not only resampled train images and that the sampling prospects quite well the manifold. To support even more this claim we provide in Appendix analysis in a case where only two centroids are selected in the metric. This also shows that the generated samples are not only interpolation between the k selected centroids since some generated images contain attributes that are not present in the images the decoded centroids.
The outcome of such an experiment that using post training latent space processing such as ex-post density estimation adding some geometrical consideration the model allows to strongly improve the sampling without adding more complexity to the model. Generating 1k samples on CELEBA takes approx. 5.5 min for our method vs. min for 10-component GMM on GPU V100-16GBB
5.2 Investigating robustness in low data regime
We perform comparison using the same models and datasets as before but we progressively decrease the size the training set to see the robustness Despite rarely performed in most generative models related papers, this setup appeared to us relevant
methods according to the number of samples.

---

Table FID (lower better) and PRD score (higher is better) for different models and datasets. For the mixture of Gaussian (GMM), we fit 10-component mixture of Gaussian in the latent space.
MNIST (16)
SVHN (16)
CIFAR (32)
CELEBA (64)
MODEL
PRD1
FID 4
PRD
FID
PRD 1
FID 4
1
AE (O, 1)
46.41
0.86/0.77
119.65
0.54/0.37
196.50
0.05/0.17
64. 64
0.29/0 42
WAE
20.71
0.93/0.88
49.07
0.80/0.85
132.99
0.24/0.52
54.56
0.57/0.55
VAE-N(O., 1)
40.70
0.83/0.75
83.55
0.69/0.55
162.58
0.10/0.32
64.13
0.27/0.39
VAMP
0.83/0.88
91.98
0.55/0.63
198.14
0.05/0.11
0.09/0.10
HVVE
15.54
0.97/0.95
98.05
0.64/0. 68
201.70
0.13/0.21
52.00
0.38/0.58
RHVAB
36.51
0.73/0.28
121.69
0,55/0.41
167. .1
0.12/0.22
55.12
0.45/0.56
AE GMM
9.60
0.95/0.90
54, 21
0.82/0.85
130.28
0, 0,35/0,58
56 07
0,32/0 48
RAE (GP)
0.97/0.98
61.43
0.79/0.78
120.32
0.34/0.58
0.28/0.49
RAE (L2)
9.89
0.97/0.98
58.32
0.82/0.79
123.25
0.33/0.54
54.45
0.35/0.55
RAE (SN)
11.22
0.97/0.98
95.
0,53/0.65 63
114.59
0.32/0.53
55.04
0.36/0 56
RAE
11,23
0.98/0.98
66.20
0.76/0.80
118.25
0.35/0.57
53.29
0.36/0.58
VAE GMM
13.13
0.95/0.92
52.32
0.82/0.85
138.
0,29/0.53
55.50
0,37/0.49 49
VAE OURS
8.53
0.98/0.97
46.99
0.84/0.85
93.53
0.71/0.68
48.71
0.44/0.62
MNIST
CIFAR 10
300
MAB
200
200
RAE-G
100
VAE-W/3L31
50
150
KIAE G504
BARGT-CMM
RAB CMX
20
RAD-N-
VAT-CMM
10
100
0.1
0.5
1.0
5.0
0.1
0.5
1.0
5.0
-00
X10*
Figure 4: Evolution of the FID score according to the number of training samples.
since 1) is well known that it may be challenging for these models, 2) in day-to-day applications collecting large databases may reveal costly if not impossible (e.g. medicine). Hence, we consider MNIST, CIFARIO and SVHN and use either the full dataset size, 10k, 5k 1k training samples. For each experiment, the best retained model is again the one achieving the best ELBO on the validation set the size of which is set as 20% of the train size. See Appendix D for further details about experiments set-up. Then, we report the evolution of the FID against the test set in Figure E Results obtained on SVHN are presented in Appendix E. Again, the proposed sampling method appears quite robust to the dataset size since it outperforms the other models' FID even when the number of training samples is smaller. This is made possible thanks to the proposed metric that allows to avoid regions of the latent space having poor information. Finally, our study shows that although using more complex generation procedures such as ex ex-post density estimation seems to still enhance the generation capability of the model when the number of training samples remains quite high (2:5),, this gain seems to worsen when the dataset size reduces as illustrated on CIFAR. In addition. we also evaluate the model data augmentation task with neuroimaging data from OASIS (33 mimicking a day-to-day scenario where the limited data regime is very common in Appendix C.3.
6 Conclusion
In this paper, we provided geometric understanding of the latent space learned by VAE and showed that it can actually be seen as a Riemannian manifold. We proposed a new natural generation process consisting in sampling from the intrinsic uniform distribution defined on this learned manifold. The proposed method was empirically shown to be competitive with more advanced versions of the VAES using either more complex priors, ex ex-post density estimation, normalizing flows or other regularization schemes. Interestingly, the proposed method revealed good robustness properties in complex settings such as high dimensional data or low sample sizes and appeared to benefit more recent VAE models as well. Future work would consist in trying to use this method to perform data augmentation in those challenging contexts and compare its reliability for such a task with state of the art methods trying this metric to perform clustering in the latent space.

---

Acknowledgments and Disclosure of Funding
The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' d'avenni" program, reference ANR-19 ANR-19-P31A-000 P31---00000 (PRAIRIE 31A Institute) and reference ANR- 10-1ATHU-06 (Agence Nationale de la Recherche-10-ee
Institut Uniiiiaaoooivvessaarr---/ This work was granted access to the HPC resources of IDRIS under the allocation ADO1 1013517 made by GENCI (Grand Equipement National de Calcul Intensif).
10

---

References
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint urXiv:1612.00410, 2016.
[2] Jyoti Aneja, Alexander Schwing. Jan Kautz, and Arash Vahdat. NCP-VAE: Variational autocncoders with noise contrastive priors, arXiv:2010.02917 [cs, stat), 2020.
[3] Georgios Arvanitidis, Lars Kai Hansen, and Sören Hauberg. Latent space oddity: On the curvature of deep generative models. In 6th International Conference Learning Representations, ICLR 2018, 2018.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autocncoders, The 22nd International Conference on Artificial Intelligence and Statistics, pages 66-75. PMLR, 2019.
[5] Merlijn Blaauw and Jordi Bonada. Modeling and transforming speech using variational autoencoders. Morgan N, editor Interspeech 2016; 2016 Sep 8-12; San Francisco, CA.[place unknown]: ISCA; 2016. 1770-4., 2016. Publisher: International Speech Communication Association (ISCA).
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. arXiv: 559.009990099 fcs, stat], 2016.
Importance weighted autoencoders.
[7] Anthony L Caterini, Arnaud Doucet, and Dino Sejdinovic. Hamiltonian variational auto-encoder. In Advances in Neural Information Processing Systems, pages 8167 8877, 2018.
[8] Clément Chadebec, Elina Thiteau-Sutre, Ninon Burgos, and Stéphanie Allassonnière. Data augmentation in high dimensional low sample size setting using geometry-hased variational autoencoder. IEFE Transactions on Pottern Analysis and Machine Intelligence, 2022.
[9] Clément Chadebee, Clément Mantoux, and Stéphanie Allassonnière. Geometry- aware hamiltonian varia- tional auto-encoder. arXiv: 1518, 2020.
[10] Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1540-1550. PMLR, 2018.
[11] Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Picter Abbeel. Variational lossy autocncoder. arXiv preprint 1rXiv:1611.027555 2016.
[12] Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In International Conference Artificial Intelligence and Statistics, pages 2359-2367. PMLR, 2021.
[13] Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In International Conference on Machine Learning, pages 1078-1086. PMLR, 2018.
[14] Bin Dai and David Wipf. Diagnosing and Enhancing VAE Models. In International Conference on Learning Representations, 2018.
[15] Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational auto-encoders. 34th Conference on Uncertainty Artificial Intelligence 2018, UAI 2018, pages 856-865. Association For Uncertainty in Artificial Intelligence (AUAT), 2018
[16] Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnel,, Matthew C. H. Lec, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv:161 /tt.:6611.0265 [cs, stat), 2017.
[17] Luca Falorsi, Pim Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, Taco Cohen. Explorations in homeomorphic variational auto-encoding. arXiv:1807.0446 [cs, stat), 2018.
[18] Partha Ghosh, Mchdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In 8th International Conference Learning Representations, ICLR 2020, 2020.
[19] Søren Hauberg. Oren Freifeld, Michael Black. Geometric take Metric Learning. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012, URL https:: http::///oooceedeeneeeeee neur wwwurips.cc//e ccppa/ Paper. pdf

---

[20] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge local nash equilibrium. In Advances in Neural Information Procexsing Systems, 2017.
[21] Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-VAF, Learning basic visual concepts with a constrained variational framework. ICI.R, 2(5)66 2017.
[22] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, page 2, 2016.
[23] Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction variational methods for graphical models. Machine Learning. 37(2):183225 1999.
[24] Dimitrios Kalatzis, David Eklund, Georgios Arvanitidis, and Soren Hauberg. Variational autoencoders with riemannian brownian motion priors. In International Conference on Machine Learning, pages 5053-5066 PMLR, 2020.
[25] Diederik P. Kingma and Max Welling. Auto-cncoding variational bayes. arXiv:1312.6114 [cs, stat), 2014.
[26] Alexej Klushyn, Nutan Chen, Richard Kurle, and Botond Cseke. Learning Hierarchical Priors in VAEs. Advances in neural information processing systems, page 10, 2019.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Abhishek Kumar and Ben Poolc. On implicit regularization in 8-vaes. In International Conference on Machine Learning, pages 5480 5490. PMLR, 2020.
[29] Anders Boesen Lindho Larsen, Swren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In International conference on machine learning, pages 1558-1566. PMLR. 2016.
[30] Yann LeCun. The MNIST database of handwritten digits. 1998.
[31] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2015.
[33] Daniel S. Marcus, Tracy H. Wang, Jamie Parker, John G. Csernansky, John C. Morris, and Randy L.. Buckner. Open access series of imaging studies (OASIS): Cross-sectional MRI data in young, middle aged, nondemented. and demented older adults. Journal of Cognitive Neuroscience, 1998-14498-507, 2007.
[34] Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee whye Teh. Continuous hierarchical representations with poincaré variational outo-encoders. Advances neural information processing systems, pages 12565 12576, 2019.
[35] Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent gaussian mixtures. In NIPS Workshop on Bayesian Deep Learning, volume 2, page 131, 2016.
[36] Radford M Neal. Hamiltonian importance sampling. In talk presented at the Banff International Research Station (BIRS) workshop on Mathematical Issues in Molecular Dynamics, 2005.
[37] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[38] Bo Pang. Tian Han, Erik Nijkamp, Song- Chun Zhu, and Ying Nian Wu. Learning latent space energy- prior model. Advances in Neural Information Processing Systems, 33, 2020.
[39] Xavier Pennee. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 25(1):127-154, 2006. ISSN 0924-9907, 1573-7683. doi: 10. 10.1000//8851-000 1-0000811-00006660 6228-4.
[40] Alexander Rakowski and Christoph Lippert. Disentanglement and local directions of variance. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 19-34. Springer, 2021.
12

---

[41] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Processing Systems, 2020.
[42] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages
PMLR, 2015.
[43] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wienstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International conference machine learning, pages 1278 1286. PMLR, 2014.
[44] Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 12400,1 2019.
[45] MSM Sajjadi, O Bachem, M Lucie, O Bousquet, and S Gelly. Assessing generative models via precision and recall. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), pages 5228-
5237, 2019.
[46] Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International Conference on Machine Learning, pages 1218 1226, 2015.
[47] Hang Shao, Abhishek Kumar, and P. Thomas Fletcher. The riemannian geometry of deep generative models. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), doi: 10.1 D..100////...2008.00
pages 428 4288. IEEE, 2018. ISBN 978-
[48] Casper Kaae Sønderby, Tapani Raiko, Lars Maalće, Sören Kaae Sønderby, and Ole Winther. Ladder variational autoencoder. In 29th Annual Conference Neural Information Processing Systems (NIPS 2016). 2016.
[49] Tolstikhin, O Bousquet, S Gelly, and B Schölkopf. Wasserstein auto-encoders. In 6th International Conference on Learning Representations (ICLR 2018), 2018.
[50] Jakub Tomezak and Max Welling. Vae with vampprior. In International Conference Artificial Intelligence and Statistics, pages 1214-1223, PMLR, 2018.
[51] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- models. In International Conference Learning Representations, 2020.
[52] Linxiao Yang. Ngai-Man Cheung, Jiaying Li, and Jun Fang. Deep clustering by gaussian mixture variational autoencoders with graph embedding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6440-6448 6449, 2019.
[53] Cheng Zhang, Judith Bütepage. Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE Transactions on Puttern Analysis and Machine Intelligence, 41(8):2008-200 2018.
13


Reviews:
[{"review_id": "L0CPOO9o1DI", "reviewer": "Reviewer_9492", "summary": "The paper takes a geometric view on the classic VAE model by reinterpreting the latent space through the lens of Riemannian manifolds. It proposes a new sampling technique utilizing the learned manifold and Riemannian metric and empirically shows the method quantitatively improves sample quality on several datasets over various standard VAE baselines.", "questions": "**Suggestions**:\n- 1, line 22-23: perhaps change ‘inducing over-regularization’ with ‘manifold/distribution mismatch’\n- 4.3, line 182-185: it is stated that the Beta term to weigh the influence of the KL divergence on the total loss is predominantly seen as a reguralizer, similar to [18]. And that ‘the proposed vision [is kept] and [not amended] for training’. It would increase clarity to briefly touch upon the fact that while the ultimate outcome of the KL-loss as a regularizar is similar, the justification of [18] is very much different.\n- 5.1: given the existence of appendix D, perhaps transfer the description of baseline models and benchmark datasets here to focus the section on results discussion. Splitting up the discussion between paragraphs separated by a line of white space would likely make it easier for the reader to grasp the main experimental take-aways.\n\n**Possible Typos / formatting suggestions**:\n- Abstract, line 2: *we indeed argue..* → we argue\n- 1, line 13: *map into* → map to\n- 1, line 17: *these models produce .. samples* → these models produce blurry samples on image generation tasks most of the time\n- 1, line 35: *improves significantly…* → improves the generation process from a vanilla VAE significantly\n- 1, line 36: *the training ..* → the training. The proposed sampling method outperforms more advanced VAE …\n- 4.3, line 133: *voluntarily chosen* → (often) chosen?\n- 4.3, line 134: *Hence, the … as follows.* → Hence, the .. as follow:\n- 4.3, line 144: *As first approximation and since … the VAE model* → Since …, as a first approximation the VAE model\n- 4.3, line 147: *This simplified drastically* → This drastically simplified\n- 4.3, line 182: *that, likewise [18],* → that like [18]\n- 4.6, line 202: *may be easily appended in Figure* → can be observed/seen in Figure\n- 4.6, line 231: *below are presented….* → refer to figure number\n- 6, line 309: *Then, we proposed* → We proposed\n- 6, line 311: *It showed to be competitive* → The proposed method was empirically shown to be competitive\n", "limitations": "The authors are careful to highlight certain practical ‘shortcuts’ of their method where it deviates from theory and the consequences, e.g. section 4.3 starting line 170. They also spent considerable effort in the various appendices, e.g. appendix B and F, to further provide practical implementation details/considerations of their method. While the potential negative societal impact of this specific work seems small, it is not entirely clear why the authors highlight appendix C as answering this question. The only line that seems relevant here is C.3. 100-102?", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- [**quality**] the elaborate background on Riemannian geometry and extensive appendices nicely introduce and illustrate the proposed solution. The technical derivations are also clearly presented and well written. The empirical results show improvements over classic VAE baselines.\n- [**originality**] while there are increasingly many works around enriching the VAE framework by extending it to use more complex distributions, e.g. normalizing flows, or adding distributions defined on manifolds with a non-trivial geometry, e.g. hyperspherical/hyperbolic, the angle proposed in this work adds a Riemannian geometry perspective in a novel and empirically useful manner.\n\n**Weaknesses**:\n- [**clarity**] the at times non-standard English makes this paper sometimes challenging to read. Other formatting choices could likely strengthen readability as well, e.g. section 5. (see next section for suggestions on possible sentences, subsections that could perhaps be improved upon).\n- [**significance**] as noted by the authors in the introduction, one of the major reasons for the classic VAE’s sustained appeal for modern machine learning tasks is its simplicity and ease of use. This surely contributes to it often still having an edge over more ‘recent methods’, e.g. diffusion based models, with higher quality results. As such, the complexity of the proposed method is likely to limit its adaptation, especially if it can’t be shown to perform on par with state of the art methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BYffrmrtmUk", "reviewer": "Reviewer_7Rkz", "summary": "This paper proposes to use Riemannian geometry to analyze the latent space of variational autoencoders (VAE).  Specifically, the authors define a Riemannian Gaussian distribution on the learned latent space. Based on this, a novel sampling process is proposed in which samples from the uniform distribution are defined on the learned Riemannian latent space. Experiments show that the proposed sampling method achieves competitive results on several benchmark datasets.", "questions": "1. It would be interesting to see some discussion about how the proposed method can benefit other more recent VAE-based methods rather than only the vanilla VAE. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is well written and easy to follow. The authors clearly explained the mathematics behind the geometrical interpretation of the VAE framework. It is very helpful that the authors give a brief review of Riemannian geometry and the Riemannian Gaussian distribution.\n\n2. Several experiments with different metrics show that the proposed method is effective and robust. \n\n\nWeaknesses:\n\n1. While I understand that this work focuses on the sampling process for vanilla VAE without modifying the training procedure, I think this sampling process can be seen as a second stage for estimating the learned latent distribution of vanilla VAE. For comparison, in two-stage VAE [1],  they achieve 34 for FID on CelebA with the second stage of training which is approximating the learned latent distribution by vanilla VAE.  Some discussion/comparison between these methods would be helpful.\n\n[1] Dai, Bin, David Wipf, and Minhao Jiang. \"Diagnosing and Enhancing VAE models (ICLR 2019).\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "74AFhRuQC_h", "reviewer": "Reviewer_AaFY", "summary": "Authors identify a problem with the standard sampling process of a VAE. They then interpret the mean and variance predicted by a vanilla VAE encoder as the tangent space of a Riemannian Manifold. Under this interpretation, the authors then sample uniformly from the space.\n \nUnfortunately, I do not have a good understanding of Riemannian manifolds, so I cannot evaluate the validity of the method described by the authors (if the U_Riem is indeed a valid distribution, for example).\n \nHowever, assuming the validity of this method, it is important to measure empirically how good the samples derived from this method are under several metrics and datasets. The authors perform several experiments to this end in the Experiments section.\n", "questions": "\nHave experiments been performed with pre-trained models that already perform comparably with SOTA models, and does the new sampling method work better on top of those results?\n", "limitations": "\nA better sampling method for image generation would have wider implications on misinformation in the public sphere. However, I believe the empirical results are not strong enough to substantiate this as a worry for this particular work at the moment. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\nI am not fully certain of this, but to my knowledge, this is a novel way of interpreting the latent space of the VAE, and leveraging the encoder predictions for improving sampling.\n\nWeaknesses:\n\nThe authors would make a better case of their method if they took an _existing_ trained model that already does well, and show that their method of sampling achieves better samples.\n\nRelatedly, the baselines appear to be implemented by the authors. While I understand this is for an “apples to apples” comparison against equivalent latent dimension sizes and architectures, it is hard to know for sure that the baselines will do better under some hyperparameter tuning, thereby weakening the strength of the results\n\nMinor: the language could use a second pass, some grammatical errors:\n- Line 17: “Nonetheless, when taken in their simplest version, it was noted that these models produce blurry samples most of the time.”\n- Line 20: “Second, the prior distribution used…” run on sentence. -\n- Line 34: “We show that this procedure significantly improves the generation..”\n- Line 261: “This supports the fact that even the simplest VAE model”\n- List is non-exhaustive\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept