GM-VAE: Representation Learning with VAE on Gaussian Manifold
Authors: Seunghyuk Cho
Juyong Lee
Dongwoo Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: VAE
Representation learning
Hyperbolic space
Distribution on Riemannian manifold
Information geometry

Abstract: We propose a Gaussian manifold variational auto-encoder (GM-VAE) whose latent space consists of a set of diagonal Gaussian distributions. It is known that the set of the diagonal Gaussian distributions with the Fisher information metric forms a product hyperbolic space, which we call a Gaussian manifold. To learn the VAE endowed with the Gaussian manifold, we first propose a pseudo Gaussian manifold normal distribution based on the Kullback-Leibler divergence, a local approximation of the squared Fisher-Rao distance, to define a density over the latent space. With the newly proposed distribution, we introduce geometric transformations at the last and the first of the encoder and the decoder of VAE, respectively to help the transition between the Euclidean and Gaussian manifolds. Through the empirical experiments, we show competitive generalization performance of GM-VAE against other variants of hyperbolic- and Euclidean-VAEs. Our model achieves strong numerical stability, which is a common limitation reported with previous hyperbolic-VAEs.

Under review as a conference paper at ICLR 2023
VAE: REPRESENTATION LEARNING WITH VAE ON GAUSSIAN MANIFOLD
Anonymous authors Paper under double-blind review
ABSTRACT
We propose Gaussian manifold variational auto-encoder (GM-VAE) whose latent space consists set of diagonal Gaussian distributions. is known that the set of the diagonal Gaussian distributions with the Fisher information metric forms a product hyperbolic space, which we call a Gaussian manifold. To learn the VAE endowed with the Gaussian manifold, we first propose a pseudo Gaussian manifold normal distribution based on the Kullback-Leibhe divergence, a local approxima- tion the squared Fisher-Rao distance, to define a density over the latent space. With the newly proposed distribution, we introduce geometric transformations at the last and the first of the encoder and the decoder of VAE, respectively to help the transition between the Euclidean and Gaussian manifolds. Through the empir- ical experiments, we show competitive generalization performance of GM-VAL against other variants of hyperbolic- and Euclidean-VAN Our model achieves strong numerical stability, which is a common limitation reported with previous hyperbolic-VAEs
INTRODUCTION
The geometry of latent space in generative models, such as the variational auto-encoders (VAE) Kingmaa & Welling 2013) and generative adversarial networks (GAN) (Goodfellow et al.) 2020). reflects the structure of the representation of the data. Mathieu al. (2019); Nagano al.) (2019); Cho et al. (2022) show that employing hyperbolic space the latent space improves in preserving the hierarchical structure of the data the latent space. The expanded geometry is not just limited to the hyperbolic space, the space can be other types of Riemannian manifolds, such as spherical manifolds (Xu Durreti 2018 Davidson et al. 2018) and the product Riemannian manifolds with mixed curvatures (Skopek al. 2019).
Meanwhile, it is known that univariate Gaussian distributions equipped with Fisher information metric (FIM) form a Riemannian manifold, sharing the manifold with Poincaré half- half-plane which is one of the four isometric hyperbolic models. This statistical manifold is known to have metric tensor akin to that of the Poincaré half-plane (Costa al., 2015), providing possibility of viewing it as hyperbolic space. Furthermore, the diagonal Gaussian distributions form a product of Riemannian manifolds showing the presence of extended statistical manifold.
Based on the connection between hyperbolic spaces and statistical manifolds, in this work, we add an alternative perspective on hyperbolic VAEs with a viewpoint from the information geometry. Previously proposed hyperbolic VAEs rely on the distributions defined over the hyperbolic space. Riemannian normal and wrapped normal are commonly used as prior and variational distributions over the hyperbolic space. Unlike the Gaussian distribution in Euclidean space, these distributions suffer from numerical instability (Mathieu et al., 2019 (Skoeek et al., 2019). addition, the Riemannian normal requires performing rejection sampling. which often generates too many unwanted samples.
From the information geometric perspective of the hyperbolic space, we introduce a new distribution, named a pseudo Gaussian manifold normal distribution (PGM normal). The Gaussian manifold, here, refers to the statistical manifold with univariate Gaussian distributions. The newly proposed distribution the KL divergence manifold. Since the KL divergence approximates the squared Riemannian distance of the statistical manifold, derived from FIM, the proposed distribution follows the geometric property of the Gaussian
statistical distance between two distributions in the Gaussian

---

Under review as a conference paper at ICLR 2023
distributions. We show that the PGM normal is easy sample, and the KL divergence between two PGM normals can be computed analytically.
With the PGM normal as prior and variational distributions, define Gaussian manifold VAE (GM-VARE whose latent space is defined over the Gaussian manifold. Nevertheless, the data points still assumed to be defined over the Euclidean space. To correct the mismatch between the data space and the latent space, we introduce a transformation from Euclidean to hyperbolic space at the last and the first layers of the encoder and decoder, respectively.
Empirical experiments with multiple datasets show that GM-VAE can achieve a competitive general- ization performance against existing hyperbolic VAžs. During the experiments, PGM normal is robust in terms of sampling and computation of the KL divergence, compared to the commonly-used hyperbolic distributions; we briefly explain the reason why others are numerically unstable. Analysis of the latent space exhibits that the geometrical structures and probabilistic semantics of the dataset can be captured in the representations learned with GM-VAE.
observe that the
We summarize our contributions as follows:
We propose a variant of VAE whose latent space is defined on a statistical manifold formed by diagonal Gaussian distributions.
We propose new distribution called pseudo Gaussian manifold normal distribution, which is easy to sample and has closed form Kl.-divergence, train the VAE on the manifold.
We propose new encoder and decoder structures to support the proper transition between Euclidean (data) space and the statistical manifold.
We empirically verify that the newly proposed model performs similarly to existing hyper- bolic VAEs while achieving stable training without numerical issues.
PRELIMINARIES
In this section, we first review the fundamental concepts of the Riemannian manifold. We then explain the commonly-used distributions the Riemannian manifolds and visit the concepts of Riemannian geometry between statistical objects.
2.1 REVIEW OF RIEMANNIAN MANIFOLD
A n-dimensional Riemannian manifold consists of a manifold M and metric tensor g : . - RR*n which smooth map from each point x M to symmetric positive defnitt matrix. The metric tensor g(x) defines the inner product of two tangent vectors for each point of the manifold (x))x: : T&M x T&M + R, where TxM is the tangent space of x.
A Riemannian manifold can be characterized by the curvature of the curves defined on it. The curva- ture of Riemannian manifold can be computed at each point of the curves, while some manifolds have curvature of constant value. For example, the unit sphere S has constant positive curvature of +1, and the Poincaré half-plane 2/ has constant negative curvature of 11. The hyperbolic models Among the hyperbolic space, the Klein model, the Poincaré disk model, the Lorentz (Hyperboloid) model, and Poincaré half-plane model are known to be isometric and have the same value of curvature -1 (Nickel & Kiela 2018 Gulcehre al. 2018 Tifrea et al.) 2018).
The metric tensor induces basic operations of the Riemannian manifold such as a geodesic, exponential map, log map, and parallel transport. Given two points x, y MM, geodesic 7x [0, M is a unit speed curve on M being the shortest path between (0) x and 7(1) = y. This can be interpreted as the generalized curve of a straight line in the Euclidean space. The exponential map expx T + M is defined as (1), where 7 is geodesic starting from x and 70(0)
where tangent vector v E M. The log map logx M + T&M the inverse of the exponential map, v. The parallel transport PTx-ay TxMM + Ty.M moves the tangent vector
i.e., log, (expx(v)) v along the geodesic between x and y. The distance function dm y) can be induced from the metric tensor follows: dm(x,y) = f. 1(1(0),(00))c
(1)
2

---

Under review as a conference paper at ICLR 2023
2.2 DISTRIBUTIONS OVER RIEMANNIAN MANIFOLD
Given a squared distance function dM M x M - R>0 of Riemannian manifold M, the probability density function of the Riemannian normal distribution can be computed by:
_d 2
Pu.o (z)
Z.M exp
(-
(2)
where μ E M the Fréchet mean of the distribution, and E R>o is the dispersion parameter and ZM is the normalizing factor. This known be preserving the maximum entropy property of the Gaussian distribution (Pennec (006). Note that the distribution requires computing the integral shown in Equation I which often does not have an analytic solution. In some special cases, one can compute the distance analytically but the computation is intractable in general. Mathieu et al. (2019) propose a rejection sampling method of the Riemannian normal defined on the Poincaré disk model, which we call Poincaré normal distribution.
An alternative to Riemannian normal is the wrapped normal distribution. The wrapped normal distri- bution is constructed by transforming sample from Gaussian distribution via parallel transportation and an exponential map:
exp (PT, ~~ N(O,
(3)
where μ E M is the mean vector the distribution, 0, the origin of M. f() maps a Euclidean vector tangent vector of 0,, and v is sample obtained from Euclidean normal with the zero mean and covariance The probability density of the sample can be computed by using the change of variable technique. Note that f(-) is well-defined in hyperbolic spaces. For example, in the Lorentz model, we concatenate zero at the first dimension of the vector, and in the Poincaré disk model, is an identity function. (Nagano et al. 2019) propose wrapped normal distribution hyperbolic space, and we call it hyperbolic wrapped normal distribution.
2.3 STATISTICAL MANIFOLD
The parameter manifold M of the probability distributions pa X + RR where 0 E M, equipped with the Fisher information metric (FIM) forms a Riemannian manifold (Rao 1992). The FIM is defined as:
9ig 0 2 (logpo(z) 0 log po (x
(a( dr.
00,
00;
In the parameter space of univariate Gaussian distributions {14, σ) TuE R. R the FIM can be simplified as two-dimensional diagonal matrix o 2 diag(1 (1, 0.5) (Costa et al. 2015). The diagonal form of the FIM implies that the Riemannian manifold with {(,,) has the same set of points as the manifold of the Poincaré half-plane, but with different curvature of value -0.5.
The parameter space of the n-dimensional diagonal Gaussian distributions becomes the product of manifolds of the parameter space of univariate Gaussian distributions. The operations on the product of the Riemannian manifolds X " M, are defined manifold wise. For example, an exponential map
wise..
applied point (p, (p=))_
Mr, with tangent vector 2 Mp for each i {1,..., ,n).
can be represented as (expp, (ve) ) 1=1
2.4 STATISTICAL DISTANCE
The statistical distance is the distance, but may not be a metric, between two statistical objects such as random variables and probability density function. The statistical distance can provide similarities between two probability density functions.
On a statistical manifold equipped with FIM, a statistical distance called the Fisher-Rao distance can be well-derived. The Fisher-Rao distance of the statistical manifold is the Riemannian distance induced from the Fisher information metric using Equation I For example, the Fisher-Rao distance in the statistical manifold with the univariate Gaussian distribution can be easily induced using the Riemannian distance of the Poincaré half-plane model, where the Riemannian metric is similar (Costa)
al. 2015).
3

---

Under review as a conference paper at ICLR 2023
Kullback-Leibler (KL) divergence is another widely-used statistical distance, which is defined as DKL (p(x) || g(z)) S2x p(x) log p(a) da for distributions p(z), q(x) in the same statistical manifold. For example, the KL divergence for two univariate Gaussian distributions, N ( oi) and
q(a)
(p2, o2), can computed as:
DEL (M(11 ((2,, log " of (11 112)² 01
1
20
2
For the n-dimensional diagonal Gaussians, the KL divergence is calculated by summing the KL divergence of the univariate Gaussians for each dimension. One notable property of KL divergence is that it can locally approximate the squared Fisher-Rar distance.
3 METHOD
In this section, we first derive reparameterization of the Gaussian distribution to form statistical manifold with an arbitrary curvature. We then propose Pseudo Gaussian manifold (PGM) normal distribution. Finally, we suggest new variant of the variational auto-encoder, whose latent space is defined over the statistical manifold.
3.1 MANIFOLD WITH ARBITRARY CURVATURE
As shown in Section 2.3 the univariate Gaussian distributions form a statistical manifold with negative half curvature, whose manifold is the same the manifold of Poincaré half-planc. Previous studies on hyperbolic spaces emphasize the importance of having an arbitrary curvature (Skopek et al., 2019 Mathieu al. 2019). These works empirically show that the generalization performances of hyperbolic VAËs can be improved with varying curvatures.
We show that the statistical manifold of univariate Gaussian have arbitrary curvature by reparameterizing the Gaussian distribution properly. Let Let (V22cu, or) be the reparameterized Gaus- sian distribution with additional parameter > 0. The reparameterization leads to the FIM of diag (1, c) showing that the curvature of the statistical manifold is -c
With the arbitrary curvature, we also verify that the KI. divergence still approximates the Riemannian distance
DKL (W(V2c(p dp), a + do) [[ 1 2
dp do
T
o
du do
+ ((do)²).
2c
where the first term is the squared Riemannian norm of the vector (d4,, da) in the manifold, which approximates the squared Fisher-Rao distance between (H, o) and (μ du, do).. The derivation of FIM and KL divergence with the reparameterized normal is described in Appendix IIn A.2 We call the statistical manifold with Gaussian distributions having a curvature of the Gaussian manifold Ge-
3.2 PSEUDO GAUSSIAN MANIFOLD NORMAL DISTRIBUTION
We propose pseudo Gaussian manifold normal distribution (PGM normal) defining a distribution over the Gaussian manifold. Let (n,,) o) E & be a point the Gaussian manifold. Inspired by the Riemannian normal, we define the probability density function of PGM normal distribution with KL-divergence as:
Ka(u, 3,, = (0/3)3 7(c exp
(DRulN(VVee u. IIN((V2. 3) (V2c.7)²
(5)
where a, B, and 2 are the parameters of the distribution, and -c is the curvature. As shown in the previous section, the KL divergence approximate the Fisher-Rao distance from Gaussian distribution N( a, 8) to A ((22c p.. Therefore, the PGM normal accounts for the geometric structure of the Guassian distributions.
4

---

Under review as a conference paper at ICLR 2023
2
2
Enc
d0p
Dec
Ui
P22
E
C
E Ege
C
Input image
Reconstructed image
Figure 1: An examplar architecture of GM-V The illustration shows the architecture of GM-VAL which sets the latent space to a three-dimensional diagonal Gaussian manifold. The encoder outputs parameters of the PGM normal, which the points of the Gaussian manifold. The gray line refers to the sampling process. The decoder reconstructs data from the samples.
The factorization of the probability density function in Equation 5 mmultippied with the square root of the determinant of the metric tensor shows the advantages of the PGM normal, which can be written as:
Kolls, a,,,,,,,,7, det((g) = Nu; ar, 8222) Gamma (02:
1 4cy2 4c,22,3
where Gamma (2; b) = F(aj r(aj2* 20 exp (-bz) and is the Fisher information metric of the Gaussian manifold. Note that the factorization has the same form the well-known conjugate prior to the Gaussian distribution. that sense, the PGM normal incorporates the geometric structure into the prior distribution explicitly. Thanks to the properties of Gaussian and Gamma distribution, the PGM normal easy to sample and has closed-form KL divergence. The detailed derivation is available in Apppendix B
8
3.3
GAUSSIAN MANIFOLD VAE
We propose Gaussian manifold VAE (GM-VAE) whose latent space is defined over the Gaussian manifold. To be specific, we place a PGM normal prior over the latent space of the VAE and add a proper geometric transformation at the last layer of the encoder and the first layer of the decoder for the conversion between the Euclidean space and Gaussian manifold.
The evidence lower bound (ELBO) of the GM-VAE be formalized with the Gaussian-manifold {1,, Σ) R², E E R5p} as:
E
(px)) /dol(g)))))o)))))))))))) det(g) [log (x
E)] DKL (Ro( x) dee((g) p(H E)- vdet(g) (7)
where (x μ. Σ) is the decoder network, (p,, E x) is the encoder network and p(u, E) is the prior. The variational distribution is set to go (1e, Σ x) = K( (x), (x),),, ), (2) (x)), where og (x(x) e R" and (x)x), (x) E IR%. and the prior is set to p(u, Σ) KC I, /) in our experiments. The training for the parameters of GM-VAL VAE (0 and Φ) is to maximize the ELBO.
Algorithm 1 Encoder
Algorithm 2 Decoder
Input Input data x, Encoding layers Enc(-) Output Parameter (o., (3 E Ge, 22 E Ryo
Input Sample (M, a) ~ IC(-), Decoding layers Dec(.) Output Reconstruction x'
I: v, , Enc(x)
D DVEE
I: v = T- (2,
D (M, E Gos v E L2
2: v = expo (f(i))
DVE C3
2: V logg (v)
DVEE
(a, = (v)
(a. g.
3: x' = Dec(v)
4: return (ax, 7"
4: return x'
5

---

Under review as a conference paper at ICLR 2023
Geometric transformations GM-VAE Mathieu et al. (2019) propose transformation from Euclidean space to the Poincaré disk to define a latent space over the Poincaré disk. We propose a novel tramsformation in VAE from a Euclidean space the Gaussian manifold and vice versa.
For a numerically stable transformation between two spaces, we adopt operations defined on the Lorentz model, which is isometric to the half-plane manifold (Nickel & Kiela 2018). The isometry T. : 2 + Ge between the two-dimensional Lorentz model with curvature manifold with curvature -c can be defined as:
C and the Gaussian
Te((t, 2,),
y (c(t=x) Vdlt x²)
and the inverse is as:
((x,))) (1 cT2 +2²
-1 + cT² + y²
2/y
2
In the encoder, we convert the output of the last layer, which is in the Euclidean space, to the Lorentz model using the exponential map at the origin and then convert it to the Gaussian manifold using Te. In the decoder, we convert the input of the first layer, which is in the Gaussian manifold, to the Lorentz model using the inverse of the transformation T- and then convert it to the Euclidean space using the log map at the origin of the Lorentz model. Figure iillusttate the architecture of GM-VAE and the pseudo code for the encoder and decoder are shown in Algorithm 1 and Algorithm 2
Remark Unlike a typical VAE, where the latent space consists of samples from a Gaussian distribution, the latent space of GM-VAE consists of set of Gaussian distributions. With this aspect, GM-VAE can be considered as a hierarchical VAE with an additional prior over the Gaussian prior. However, instead of sampling another latent variable from the latent distribution, we directly transform the latent distribution itself to x in the decoder network via transformation from hyperbolic Euclidean space. From this perspective, GM-VAE can also be considered as variant of Poincaré VAE (Mathieu et al. 2019), whose latent space can be interpreted via the Gaussian manifold.
4 RELATED WORK
Information geometry on VAE Focusing on the virtue of bridging probability theory and differ- ential geometry, the adaptation of information geometry to the deep learning framework has been investigated in various aspects (Karakida et al., 2019 Bay & Sengupta | Gomes et al. 2022). Having said that, Han et al. (2020) show that the training process of VAE can be seen minimizing the distance between the two statistical manifolds: manifolds with the parameters of the decoder and the encoder. Not only can the parameters but the outputs from the VAE decoder be modeled as probability distributions. |Avanitiiis et al. (222T) suggest a method of using the pull-back metric defined with arbitrary decoders on latent space. Our work focuses more on the statistical manifolds lying on the outputs of the encoder with the benefits from the information geometry.
VAE with Riemannian manifold latent space The latent space of VAE reflects the geometrical property of the representations of the data. The efficacy of setting the latent space be hyperbolic space (Mathieu et al. 2019 Nagano et al. [2019; Cho et al. 2022) or spherical space (Xu & Durrett 2018 Davidson al. 2018) has been verified for various datasets. [Skopek al. (2019) further extends the approach to enable the latent space to be the product of Riemannian manifolds with
Table I: A comparison of the PGM normal (ours) with the commonly used distributions the hyperbolic space: Hyperbolic wrapped normal and Poincaré normal. Our method enables the easy sampling and computation of closed-form KL, with the utilization of the information geometry.
Easy sampling
Information geometry
Closed-form KL
Hyperbolic wrapped normal
X
Poincaré normal PGM normal (ours)
X
O

---

Under review as a conference paper at ICLR 2023
different learnable curvatures. On top of these arts, explore the method of setting the latent space to be a diagonal Gaussian manifold, which is isometric to the product of the hyperbolic space, providing novel viewpoint on prior work with information geometry.
Distributions on the hyperbolic space Defining a distribution in the hyperbolic space with easy sampling is challenging, Nagano et al. (2019) suggests hyperbolic wrapped normal distribution from the observation that the tangent space is Euclidean space. Leveraging operations defined on the tangent spaces, e.g., parallel transport, enables an easy sampling algorithm. Mathieu et al. (2019) propose sampling method for the Riemannian normal defined on the Poincaré disk model using rejection sampling. This method rejects the pathological samples and enables accurate sampling from the distribution, but this demands high amount of time complexity. These distributions are applied in
many cases (Cho et al. 2022 Skopek al. 2019 Mathieu & Nickel 2020) but suffer from stability issues because of the absence of closed-form KL divergence. Our proposed distribution, however, not only share the common merits but also has overcome the stability problem with closed-fomm KL divergence. Table 1 summarizes the properties each distribution.
EXPERIMENTS
In this section, we compare the performance of VAE with the three baselines: Euclidean VAE, hyperbolic wrapped normal VAE (HWN VAE), and Poincaré VAE. The Euclidean VAE is the standard VAE with Euclidean latent space. The HWN VAE uses the product of two-dimensional Lorentz models as latent space and uses the hyperbolic wrapped normal to model the prior and variational distributions. The Poincaré VAE uses the product two-dimensional Poincaré disk models as a latent space and uses the Poincaré normal model the prior and variational distributions. The Euclidean VAE, HWN VAE, and Poincaré VAE are denoted as E-VA, L-VAE, P-VAE, respectively in the following results.
5.1 DENSITY ESTIMATION
first conduct density estimation task to check the generalization ability different models. We three datasets: binarized-MNNN MNIST (engg] 2012). binarized Oaniiglot (Lake et al.) 2015), and the images from Atari 2600 Breakout with binarization (binarized- Breekkout (Nagano et 2011)) The binarized- Breakout are collected from plays with pre-trained Deep Q-Network (Mnih et al. (015). The size of images are 28 x 28, 28 x 28, and SO x 80 for binarized-MNIS' and binarized- Breakout, respectively. The value of the threshold for binarization set to 0.5, 0.5, and ainaized--reakoou..
Ainarize--Breakout
0.1 for binarized-Mi
binarized-Omniglle and binarized-Bei -BeekduttrBkott respectively; the threshold
Table 2: Density estimation on real-world datasets. denotes the latent dimension. We report the negative test log-likelihoods of average 10 runs for binarized-MNIST and binarized-Ommi an average runs for binarized- Breakout with the 95% confidence interval. N/A in the ligeliklddoodd indicates that the results are not available due to the failure of all runs, and N/A in the standard deviation indicates the results are not available due to failures of some runs. The best results are bolded.
and
Bhaarized-BBeakout
GM-VAE (c= 1)
GM-VAE (c= 1/2)
GM-VAE (c= (c=3/2)
E-VAE
L-VAE
P-VAE
10
79.60
79.95
80.52
80.344.30
80.10
80.38 +.18
MNIST
20
74.48+.46
73.67+.32
72.95+.11
73.27
73.31+.29
73.28+.19
73.80+.07
73.464.23
72.944.10
73.494..1
73.35+.16
73.55
136.534.30
136.254.36
134.95
134.01
135.204.21
133.79+.300
Omniglot
20
121.18 18
119.95+.40
117.79+.13
118.794..3
118.73+.99
119.03
30
118.67 ..07
117.164.48
115.094.66
117.97 2..3
117.70+.47
117.95
24
50.37 4:46
50.82
N/A
49.35
50.824.92
49.88
Breakout
28
48. 15..07 2.2
48. 75.74NIA
N/A
47.01
48.31
46.82+.77
32
48. 18.006+.18
48. 48.64NAA
N/A
47.044.23
48. 48.134.30
46.88
7

---

Under review as a conference paper at ICLR 2023
Table Ablation study on the geometric transformations of GM-VAE. Vanilla denotes the models without the geometric transformations, and Geo denotes the models with the geometric transforma- tions. The geometric transformations enhance the generalization performance most cases.
1/2
3/2
d
Vanilla
Geo
Vanilla
Geo
Vanilla
Geo
10
80.28
80.344.30
80.28+.19
80.10
80,38
80.38+.18
MNIST
20
+.22
4.594
73.31 1.29
75.74
73.28
30
74.63
73.49+.13
74.48+.50
73.35+.16
75.05+.48
73.554.22
10
135. 125
134.01
2..2474..24
135.20
134.694..3
133.79.2.00
Omniglot
20
112.224:.6
118.79+.53
121.10+.59
118.73+.39
122.44+.66
119.03+.57
30
120.85
1.35
119.48
117.70
121.23
117.95
24
51.69
49.35-a.67
51.044.72
50.82
51.57
49.88
Breakout
28
49.564.69
47.01
49.514.61
48.31
19.455..3
46.82
32
49.454.29
47.04+.23
49.03+.35
48.13
49.07+.54
46.88+.25
is determined to visualize the components clear. The other details on the implementation and experimental setups are described in/Appendix D
for
The results are reported at Table 2
In Sinarized-MNIS and binarized- binarized-Omnigloo Omniglot, the models learned on the product hyperbolic space and the Gaussian manifold mostly outperform the Euclidean VAE, In binarized-Be Brrekkut, the GM-VAE with curvature values I and 3/2 outperform the baselines while the Poincaré VAE fails to run in all the settings and the HWN VAE fails to run in some of the settings due to numerical issues, which we further investigate in details.
Numerical stability We conduct analysis of the numerical stability of the PGM normal dis- tribution compared to the HWN and Poincaré normal. During the density estimation experiment, the HWN VAE and Poincaré VAE are often shown be numerically unstable and fail to run in Baaarized-Breakouu Similar observations have been reported in several previous works (Mathieu al. 2019 Chen et al., 2021 Skopek al. 2019).
The hyperbolic wrapped normal uses the exponential map when transforming the output of the encoder to the Lorentz model and during the sampling, as described in inEquuation 3] The overlapped Lorentz model exponential map often causes an overflow. In the training of Poincaré VAF, the KI. divergence between the variational distribution and the prior distribution needs be approximated by the bog-probability of the samples due to the absence of closed-form KL divergence in Poincaré normal. To compute the log probability of a given sample, the distance between two Poincaré disk model points, which are the sample and the Fréchet mean of the distribution, needs to be calculated, where the denominator term is numerically unstable. The PGM normal, on the other hand, is free
(a) I-SNE visualization.
(b) Latent traversal of representations.
Figure 2: Analysis of the learned latent space of GM-VAE with Smarized-MNISST (a) t-SNE visualization the representation with respect to the class labels. (b) Increasing the value of B, along the gray line, results in an increasing degree of uncertainty in the reconstructed images.
8

---

Under review as a conference paper at ICLR 2023
(a) Latent traversal of representations.
(b) Hierarhey the data.
Figure Analysis of the latent space learned from GM-VAE with the binarized -rarzeedddBeakout.. (a) Reconstructing the representations, along the gray line, shows a similar hierarchical structure, where (b) the hierarchy between the images is expressed the dotted line.
from instability with the help of stability when using log-covariance. Please check the detailed arguments with equations Appendix E
Geometric transformations We conduct an ablation study on the geometric transformations of GM-VAE. We compare the setting of GM-VAEs incorporating the geometric transformations to the setting of GM- VAEs using only exponential function send the output the encoder to the Gaussian manifold but no additional geometric transformation at the first layer of the decoder. The results are Table 3 We can see that the geometric transformations enhance the performance the GM-VAE, except for two results but with similar performance.
5.2 LATENT SPACE ANALYSTS
To check whether the latent representation coincides with the known labels, we first plot the latent spaces of binASized-MNIST via t-SNE (van der Maaten & Hinton 2008) visualization, with represen- tations from all dimensions. The visualization shown in Figure 2a palpresents that the label semantics are well clustered the learned latent space. We also analyze the changes in the reconstructed images along the geodesic of the latent space. Figure 2b]shows the reconstructed images from a geodesic interpolation between two latent representations, with a fixed value of a. The interpolation of the latent space is performed within one dimension while fixing the value of representations in other dimensions. As f increases, the reconstructed images become ambiguous, matching our intuition on the role of variance. Reconstruction images with fixed value of is available at Appendix F
Figure shows the analysis with Bnharized-Breakone The images the binarized- Enaaiiee-BBrekkout possess hierarchy as the cumulative rewards and the amount of the breakout bricks, the portion of blank space in the image, are highly correlated (Nagano et al., 2019). We observe that there is high correlation between B and the hierarchy. For example, shown in Figure 3al increasing reconstructs a more general image in the hierarchy. The highest Pearson correlation between the values and the negative cumulative reward is 0.655. Again, as B represents the variance, conjecture the increasing variance induces a more general image in the dataset.
CONCLUSION
In this work, we propose a novel method of representation learning with GM-VAE, utilizing the Gaussian manifold for the latent space. With the newly-proposed PGM normal distribution defined over Gaussian manifold, which shows better stability and ease of sampling compared to the commonly- used ones, verify the efficacy of our method on several real-world datasets. Our analysis of latent space and representations exhibits that GM-VAE is beneficial for capturing both the geometrical structures and probabilistic semantics. We believe that the connection between the statistical manifold and hyperbolic spaces provides a new insight to the research community and hope to see more interesting connections and analyses the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Georgios Arvanitidis, Miguel González-Duque, Alison Pouplin, Dimitris Kalatzis, and Søren Hauberg. Pulling back information geometry. arXiv preprint Sttv:2006.05366, 2021.
Alessandro Bay and Biswa Sengupta. 2eoseq2seq: Information geometric ggueence-t--equenae networks. arXiv preprint
2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.000 wtwp:000900055000 00099, 2015.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. arXiv preprint arXiv:2105. /4686, 2021.
Seunghyuk Cho, Juyong Lee, Jaesik Park, and Dongwoo Kim. A rotated hyperbolic wrapped normal distribution for hierarchical representation learning. arXiv preprint arXiv:2205.1337/ 2022.
Sueli IR Costa, Sandra A Santos, and Joao Strapasson. Fisher information distance: geometrical reading. Discrete Applied Mathematics, 197:59 69, 2015.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational nuto-encoders. arXiv preprint /
2018.
Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 291)):141114,, 2012. 2012.
Eduardo Dadalto Camara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. Igeood: An information geometry approach to out-o--distriiuttio detection. arXiv preprint arXiv:2203.07799, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1444144139 2020.
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. Hyperbolic attention networks. arXiv preprint 1 1805.09786 2018
Tian Han, Jun Zhang, and Ying Nian Wu. From em-projections to variational auto-encoder. 2020.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference Artificial Intelligence and Statisties, pp. 1032-1041. PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14t..6112999 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 35066266) 2015. doi: 10. 126
science.aab3050. URL htttp:://wwwtt http://wwww.iiee.... http:////www......e.o///////////
aab3050
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 33:2503-2515, 2020.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Contin- uous hicrarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5177550):5295533 2015.
10

---

Under review as a conference paper at ICLR 2023
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient-bad based learning. In International Conference on Machine Learning, pp. 4693 PMLR, 2019.
Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pp. 3779-3788. PMLR, 2018.
Xavier Pennec. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements. 2006.
Journal of Mathematical Imaging and Vision,
C. Radhakrishna Rao. Information and the Accuracy Attainable in the Estimation of Statis- fical Parameters, pp. 235-247. Springer New York, New York, NY, 1992. ISBN 978-1- URL
4612-0919-5. doi: 1111007997-114612-09111
978-1-4612-0919-33
Ondrej Skopek, Octavian-Fugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. arXiv preprint arXiv:1911.08411 2019.
Alexandru Tifrea, Gary Bécigneul, and Octavian Oetavian-Eugen Ganea. Poincar I'e glove: Hyperbolic word embeddings. arXiv preprint
006666,, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma-
chine Learning Research, 12579926605. 2008. URL hRtp://://www jmlr. wwog/papers/ie /
wannderaattenUs..cee
Jaacheng Xu and Greg Durrett, Spherical latent spaces for stable variational autoencoders. arXiv preprint ar 1808 10805, 2018.
A GAUSSIAN MANIFOLD
A.1 CURVATURE OF THE GAUSSIAN MANIFOLD
We construct Riemannian manifold ((,,a) μ E R diag(1/o², 1/(oo²)) which we will name Gaussian manifold. We need to show the value of the curvature.
Ry0} with the metric tensor
First, we need to compute the Christoeffel symbols of the Gaussian manifold defined as:
1 ggght
[ōg,t 0ga
0gk
r
(8)
0g. 0g,
0g.
where gij is the (i,j) element of metric tensor and g the (i,j) element of the inverse of metric tensor.
The Christoeffel symbols of the Gaussian manifold are:
r
(9)
r 0
-1 -
(10)
Then, the sectional curvature of the space Ki is computed as: Rm 0, u.) Ki det g
(11)
where Rm is the Riemannian curvature which is computed as:
@rm Öy do
ar
Rm(p, t,t, (1) 91m
THE Ip
(12)
By putting the metric tensor and the Christoeffel symbols together, the curvature of the Gaussian manifold is computed as:
Rm(u,,,, 0.
Kg det
(13)
11

---

Under review as a conference paper at ICLR 2023
A.2 GAUSSIAN MANIFOLD WITH KL-DIVERGENCE
Between two univariate Gaussian distributions. N (M1, of) and and.N (N2,, oī), we can compute the KL divergence as:
of + (M) - u2) -1). o (KI, [1 02)) = log
(14)
03
We extend the KI. divergence for an arbitrary curvature of the Gaussian manifold as:
9it (11((11)0))) (12, 02)) DRu(W(V2en, on) (VV222222
(15)
2c
Now, we show that the extended KI. divergence still approximates the Riemannian distance of the manifold as:
0²
(a - do)² + 2c(dp)2
(ku + du + do), (k,0))
log
-1
(16)
2 2c
(o + do)
-2 1 do
)
2odo (do)2
20(du))
2c
0²
o²
(17)
do
(do)²
20do + (do)²
2c(du)²
2
+((()))
2c
0
20²
(18)
dp
1
0
dd
+ ((((o)3).
(19)
2
do
0
do
c
B PSEUDO GAUSSIAN MANIFOLD NORMAL DISTRIBUTION
In this section, propose pseudo- Gaundd-a - manifold normal distribution for the Gaussian manifold
as:
B,,7) 7(7) f3 exp (- (R,, (a, ())
σ²
(20)
72
The given probability density function needs to satisfies the following condition:
1 Kolm,o,,, V Vo(u,)))pppp
(21)
where Vlu, o))d(p )) is the probability measure over the Gaussian manifold induced with the Lebesgue measure d(j,, a) and the Lchesgue-Radon-Niikon theorem. We can find the normalizing
12

---

Under review as a conference paper at ICLR 2023
factor (7) as:
3
g ((L, σ), (a, (a,
1
(7)
exp
Voo² du
72 g Rii σ), (a, 3))
VCB3
σ exp
7/²
du
00
(8
/.00 σ. 1-(022 (00)
Ve83
-20
exp
400?
So
exp
4c82g2 do
00
(11 a)2
exp
2B2₂2 dp
1
1
len
2 1
eex
exp
r
2 cB3
40y2
4
4c7?
(11
Gamma
0²:
1
do²
(
Nu (83)
4eo7
4c,222
V2T
2/c2 7
Texp
4coy2
4ry2
40y?
Finally, the logarithm of the normalizing factor is computed as:
1
1
1 Acg2 (1 log(4c72)). (22)
log ((7) log(2 log c - lloo + 2 log 72 + log
2
4og2
C OPERATIONS OF THE HYPERBOLIC SPACES
C.I ISOMETRIES
In this section, we derive the isometries between the two-dimensional hyperbolic models, the Lorentz model, the Poincaré disk model and the Gaussian manifold with arbitrary curvatures. Isometry between the Poincaré disk model and the Lorentz model TC.+Pe Le - Pe is computed as:
CC.-P.((t.2 u))
+1'
+v++
y
and the inverse is:
+ (x² + y²)
2y
(c.,) Vel1- -x21-(x2 y2) 1 - (x² + y²) - (x² + +22cc/
(23)
Isometry between the Gaussian manifold and the Poincaré disk model Tp Pe Ue is computed as:
-2y
1- (x² + +2 y²)c
Tp. (x, (iver 1)2 + y (Ver - 1)2 + yー
and the inverse is:
(
Vcx² + (y² - 11) cx2 + (y 1)2
22x cx2 + (y 1)3
Tp. (r.1)
(24)
Finally, the isometry between the Gaussian manifold and the Lorentz model T2.+g. can be derived by composing Te.+P. and To. as:
- 1
Te. (1,x,
(val1 - x)) Vell - x)
and the inverse is:
T=l C. T (x. ( 1 cx2 +2² -1 cx + y2 x
(25)
2//c
2/0y
13

---

Under review as a conference paper at ICLR 2023
We then empirically show that the isometries preserve the distance between the points when trans- formed to other models. We randomly sampled 1,000 pairs of Gaussian manifold points with range of μ E [-100, 100] and [0, 100]. We report the average difference in the distance for each pair before the transformation and after the transformation. We vary the curvature value from 0.25 to 2. For the Gaussian manifold, we use the following distance function for arbitrary curvature:
log Velai x2)² + (11 + y2)² + Vo(21 x2)² + (0) y2)
du, ((x1, (22, = vc Vc(zi x2)² + (v1 + 32)² v /(xx x2)² + (y) 12)²
lable 4 shows that the proposed isometries well-presend the distances.
Table 4: Validation of the proposed isometries between the hyperbolic models.
TP.
Tg.
Tg.
0.25
5.72e-12 13
1.50--13
4.76c- 13
0.50
8.41e-13
3.10c-13
6.23e-13 13
1.00
7.84e
4.22e-13
4.88e 13
1.50
3.22e-12
2.5le-12
8.30e- 13
2.00
1.89e 12
1.20--12
8.86c-13
C.2 THE LORENTZ MODEL OPERATIONS
The n-dimensional Lorentz model with curvature is C where the manifold is {x g=+1 (x, c. = } where (x, c. is the Lorentzian product computed as (), -XoYo EL x,y, The exponential map of the Lorentz model defined as:
exp< (v) = (a)x + sinh(a) v a
(26)
and the log map of the Lorentz model is defined as:
log (y) cosh ((9)
3/221 (y - 3x),
(27)
where a Vlly, V)r. and 3 = - c(x,
D IMPLEMENTATION DETAILS
In this section, we introduce the implementation details for the density estimation experiment.
For the encoder and the decoder of Sinarized-MNIS and binarized-Omnille we use a two-layer fully connected neural network, where the dimension of the hidden units is 200 with the hyperbolic tangent activation, following the setting used in the importance weighted VAE O. For binarized- Breakout, the encoder is four-layer convolutional neural network with leaky ReLU activation followed by a fully connected layer. The decoder consists of fully connected layer followed by three transposed convolutional layers with ReLU activation. For both the encoder and decoder used in the Kinarized-Breakoutt, we place the batch normalization layer at the end of all layers. Since the datasets are binarized, we use Bernoulli distribution as the output of decoders for all tasks.
For training, we use Adam optimizer (Kingma & Ba 2014) with constant learning rate 1e-3 and set the batch size to 100. We train the VAΣs for 300 epochs on Simarized-MNIST and binarized- and 200 epochs on binarized-Bress -rrekkout. The trained VAEs are evaluated by the log-likelihood on the test set for each task with sepportance-eiightee sampling (Burda et al., 2013). The models learned from binarized-MNIIST and binarized- binarized-Ommigloot use 500 samples for the importance weighted sampling and the models learned from binarized Breakout use 50 samples for the importance weighted sampling.
14

---

Under review as a conference paper at ICLR 2023
E NUMERICAL STABILITY
We conduct an analysis of the numerical stability of the PGM normal distribution compared to the
HWN and Poincaré normal. During the density estimation experiment, the HWN VAE and Poincaré VAE are often shown to be numerically unstable and fail to run in Smmarized-Breakow Similar
observations have been reported in several previous works (Mathieu et al. 2019 Chen al. 2021 Skopek et al. (019)).
The hyperbolic wrapped normal uses the exponential map when transforming the output of the encoder to the Lorentz model and during the sampling, as described in Eqquatin 3] The overlapped Lorentz model exponential map often causes an overflow due to the hyperbolic functions in the exponential map such as cosh and sinh. Note that the hyperbolic functions exponentially grow with the positive input value.
In the training of Poincaré VAE, the KL divergence between the variational distribution and the prior distribution needs to be approximated by the log-probabilityy of the samples due to the absence of closed-form KL divergence in Poincaré normal. To compute the log probability of a given sample, the distance between two Poincaré disk model points, the sample and the Fréchet mean of the distribution needs to be calculated, where the distance function of the Poincaré disk model is defined as:
1 Vc
cosh -1 (1 (1+ 2c (1
X y/12
de (x)
(28)
c
c|x (1 cly 12)
where - is the Euclidean norm. The denominator term is unstable when ||x|| or ||y|| is close to value v which is occured when x and y are near the border the Poincaré disk.
PGM-normal, on the other hand, the KL divergence between an arbitrary PGM normal and (O, 7, I), which is the only operation used during the training of GM-VAE, can be stably computed using the oog-covriaance For example, the KL divergence between univariate Gaussian distribution M(,, o) and the prior distribution mentioned above written as Equation 14 can computed with log o². The KI divergence between two Gamma distributions, Gamma (a1
b1) and Gamma(az, b2), written as:
bi I(a,) (a1 a2))((1) (1
b2
DKL (Gamma(aj, b1) || Gamma(a1 b1)) a2 log
Im
01
b2 l((a2)
b
(29)
where dé is the digamma function, be stablly computed using log bj when dy is large due small (3 and in the factorization Equation 6
F LATENT SPACE ANALYSIS
In this section, we provide additional visualizations of the learned representations of GM- VAE. Figure 4]shows the t-SNE visualization of the Buclidean VAE learned on binarized-MNIST Figure shows the interpolation results of the binarized-MNIST representations along the axis, by changing the value of a on three dimensions and fixing all the other values including 3. Figure and Figure show the latent representations from all the dimensions with binarized -NIIST and -Brarized-BBeakout respectively.
15

---

Under review as a conference paper at ICLR 2023
Figure 4: t-SNE visualization with representations learned from Euclidean- VAE.
3
7
7
7
4
5
8
8
3
9
9
9
;
5
C
0
O
Figure 5: Interpolation results of the representations learned from GM-VAE along the μ axis.
16

---

Under review as a conference paper at ICLR 2023
Figure 6: The binarized- Sinarize--Breakout representations learned from GM-VAE
Figure 7: The binarized-MNISY representations leamed from -AE.
17


Reviews:
[{"review_id": "9Oeb4Nl89-", "reviewer": "Reviewer_R7Dy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper defines PGM-normal, a distribution over the family of diagonal Gaussian distributions, which forms a manifold. With this distribution, they propose a variant of VAE whose latent space is this manifold. They propose two transformations for the VAE to map between Euclidean space and the manifold. Empirically, they show that their method matches the performance of existing hyperbolic VAEs.", "summary_of_the_review": "The theoretical part is largely known. The motivation of the paper is weak. The experiments only show the proposed method works. ", "strength_and_weaknesses": "Weakness: \n\n(1) The hyperparameter $c$ only changes the scaling of $\\mu$ and $\\alpha$. It does not influence the representation capability of the VAE, and can be absorbed into the encoder/decoder model. The motivation that \"generalization performances of hyperbolic VAEs can be improved with varying curvatures\" is weak. ---- by introducing hyperparameters, one always have room for improving the generalization performance. \n\n(2) Although well-motivated from manifold theory, the distribution over the manifold $\\mathbb{R} \\times \\mathbb{R}_+$  reduces to a Normal distribution and a Gamma distribution. Instead, the properties of the manifolds are only used in the exponential map of the Lorentz model and the isometry between Lorentz space and the Gaussian manifold, the former being a known result in the previous paper.\n\n(3)  It is not well-motivated to model the latent space as a Gaussian distribution. The connection of the proposed VAE to hierarchical VAE is weak.\n\n(4) The experiments. From Table 2 I cannot see any benefit of the proposed method. It only works on Breakout dataset. Besides, why do you need to do binarization instead of modelling the original data?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4pIsSFFGGF6", "reviewer": "Reviewer_G6cG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces and studies a new type of hyperbolic latent space called Gaussian manifold in the context of variational auto-encoder in order to tackle numerical and sampling issues with existing hyperbolic latent spaces. To make this latent space works in practice, the authors designed a geometric transformation at the last encoder step and the first decoder step; as well as a novel distribution over Gaussian manifold called a pseudo Gaussian manifold normal distribution that makes use of information geometry, easy to sample from, and easy to compute KL divergence. Finally, comprehensive testing on empirical datasets are carried out to highlight the effectiveness of the scheme. \n\n\n\n\n", "summary_of_the_review": "Despite the paper containing some interesting and novel ideas on how to make the Gaussian manifold suitable for VAE setting, I find that the theoretical motivation for said ideas is heuristical at best and there is not much empirical evidence to demonstrate the strength of the proposed method. Therefore, I recommend rejection. ", "strength_and_weaknesses": "Strengths\n\n1. The proposed method is novel, interesting and makes use of clever engineering details that are rather intuitive.\n2. Empirical results clearly indicate that the proposed method is superior in terms of numerical stability to existing hyperbolic latent space methods, over certain datasets (such as binarized-Breakout). In fact, in my opinion, the main selling point of the method should be in terms of numerical stability and the authors may consider bringing the discussion in Appendix E into the main text. \n3. Empirical results in terms of negative test likelihood do not suffer, compared to other methods. \n4. The introduction to hyperbolic latent space is comprehensive and self-contained. Overall I find the language clear and the flow easy to follow, despite small typos. \n\nWeaknesses\n\n1. Some notations are not clearly defined. In figure 1, the calligraphic E space (I suspect is Euclidean space) is not defined and the calligraphic L^2 space is only defined in passing in subsection 3.3. I suggest that the authors at least make the notation in the figures and its caption self-contained.  What does the triangle in Table 1 mean? How is it different from a circle?\n2. Although a lot is mentioned about information geometry and the incorporation of closed form KL, the paper does not clearly mention (or hint at) specific mathematical guarantees for the proposed model. As a result, a lot of the justifications for the proposed method are heuristical and do not form concrete theoretical backings. \n3. Empirical results are rather limited in quantity. There are only a total of 3 datasets and the authors only demonstrate better stability in 1 of these 3 datasets. The proposed methods also do not seem to scale as well as competing methods when increasing the latent space dimension.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4jvwLNUmn7_", "reviewer": "Reviewer_k6FC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a pseudo Gaussian manifold normal distribution is proposed as the prior of latent representation of data, and the latent representation of data is learned in the framework of VAE, so as to realize a Gaussian manifold learning.", "summary_of_the_review": "The manuscript presents an interesting idea, but the details and rationality of the hypothesis are not clear, and the experimental results are difficult to support and explain the hypothesis.", "strength_and_weaknesses": "Strength：\n\nThe idea to learn a new manifold for data is interesting.\n\n\nWeaknesses:\n\n1. The motivation of the methods in the manuscript is not clear and most hypotheses lack explanation. For example, why use diagnal Gaussian distributions.\n\n2. The authors only introduce different methods by plain descriptions and isolated formulas, and does not analyze or summarize the characteristics of different methods in manifold learning. For example, what scenario does the Riemannian Gaussian distribution apply to, and what scenario does the wrapped normal distribution apply to？\n\n3. The authors directly give the distribution expression of PGM in (5) without explaining its geometric meaning.\n\n4. How to get the  conclusion of ”With this aspect, GM-VAE can be considered as a hierarchical VAE with an additional prior over the Gaussian prior“？\n\n5. The experimental results are not convincing. In Table 2，the proposed method does not perform as well as P-VAE and the authors do not make any anlysis between P-VAE and GM-VAE.\n\n6. For the results of visualization of latent space such as in Figure 2 and Figure 3, how to evaluate them?\n\n7. What's the difference between the proposed GM-VAE and some Gaussian-process-based manifold learning methods, such as GPLVM or VAE-DGP?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject