Mixed-curvature Variational Autoencoders
Authors: Ondrej Skopek
Octavian-Eugen Ganea
Gary Bécigneul
Venue:  (2020)
Keywords: variational autoencoders
riemannian manifolds
non-Euclidean geometry

Abstract: Euclidean space has historically been the typical workhorse geometry for machine learning applications due to its power and simplicity. However, it has recently been shown that geometric spaces with constant non-zero curvature improve representations and performance on a variety of data types and downstream tasks. Consequently, generative models like Variational Autoencoders (VAEs) have been successfully generalized to elliptical and hyperbolic latent spaces. While these approaches work well on data with particular kinds of biases e.g. tree-like data for a hyperbolic VAE, there exists no generic approach unifying and leveraging all three models. We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature is fixed or learnable. This generalizes the Euclidean VAE to curved latent spaces and recovers it when curvatures of all latent space components go to 0.

Published as a conference paper at ICLR 2020
MIXED-CURVATUR VARIATIONAL AUTOENCODERS
Ondrej Skopek! Octavian-Fugen Ganea & Gary Bécigneul!?
Department of Computer Science, ETH Zürich
2 Computer Science and Artificial Intelligence Laboratory. Massachusetts Institute of Technology askopek@oskopek. com, octêmit eedmmiteedul gary. perr.beci.n www.bbecciggii.... gneul@inf. @inf. www.ffettz.ccc
ABSTRACT
Euclidean geometry has historically been the typical "workhorse" for machine learning applications due its power and simplicity. However, it has recently been shown that geometric spaces with constant non-zero curvature improve rep- resentations and performance Consequently, generative models like Variational Autoencoders (VAEs) have been successfully generalized elliptical and hyperbolic latent spaces. While these ap- proaches work well on data with particular kinds of biases e.g. tree-like data for a hyperbolic VAE, there exists no generic approach unifying and leveraging all three models. We develop Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product constant curvature Riemannian manifolds, where the per-component curvature is fixed learnable. This gener- alizes the Euclidean VAE to curved latent spaces and recovers it when curvatures of all latent space components go to O.
variety of data types and downstream tasks.
INTRODUCTION
Generative models, growing area of unsupervised learning, aim model the data distribution p(æ) over data points æ from a space X, which is usually nigh-dimensiioal Euclidean space R". This has desirable benefits like naturally definable inner-product, vector addition, or closed- form distance function. Yet, many types of data have a strongly non-Fuclidean latent structure (Bronstein et al. 2017). e-g. the set of human-inter inteerrrrrtee images. They are usually thought to live "natural image manifold" (Zhu et al.) 2016). continuous lower-dimensional subset of the space in which they are represented. By moving along the manifold, one can continuously change the content and appearance of interpretable images. As noted in Nickel & Kiela (2017). changing the geometry of the underlying latent space enables better representations of specific data types compared to Euclidean spaces of any dimensions, e.g. tree structures and scale-free networks.
Motivated by these observations, a range of recent methods learn representations in different spaces of constant curvatures: spherical or elliptical (Batmanghelich al. 2016), hyperbolic (Nickel & Kiela 2017] Sala et al. 2018: Tifrea et al.) 2019) and even in products of these spaces (Gu et al. 2019 Bachmann al. 2020). Using combination of different constant curvature spaces, Gu
2019 aim to match the underlying geometry of the data better. However, an open question remains:
how to choose the dimensionality and curvatures of each of the partial spaces?
A popular approach to generative modeling is the Variational Autoencoder (Kingma & Welling 2014). VAEs provide way sidestep the intractability marginalizing a joint probability model of the input and latent space p(æ, 2), while allowing for prior p( 2) on the latent space. Recently, variants of the VAE have been introduced for spherical (Davidson et al. 2018 Xu & Durrett 2018)
and hyperbolic (Mathieu et al. 2019 Nagano et al. 2019) latent spaces.
Our approach, the of constant curvature $paces[ It has the advantage of a better reduction in dimensionality, while maintaining efficient optimization. The resulting latent space a "non-constantly" curved manifold that is more flexible than a single constant curvature manifold.
Variational Autoencoder, is generalization of VAEs to products
Cooee available GitHub http:////ggthhtb...c////////pppp

---

Published as a conference paper at ICLR 2020
Our contributions are the following: (i) we develop principled framework for manipulating rep- resentations and modeling probability distributions in products of constant curvature spaces that smoothly transitions acToss curvatures of different signs, (ii) generalize Variational Autoencoders to learn latent representations on products of constant curvature spaces with generalized Gaussian- like priors, and (iii) empirically, our models outperform current benchmarks on synthetic tree dataset (Mathicu et al. (211)) and on image reconstruction on the MNIST (LeCun) 1998). Omniglot (Lake et al. 2015), and CIFAR (Krizhevsky 2009) datasets for some latent space dimensions.
2 GEOMETRY AND PROBABILITY IN RIEMANNIAN MANIFOLDS
To define constantly curved spaces, we first need to deñnee the notion of sectional curvature (TT,) of two linearly independent vectors in the tangent space at a point x E M spanning two-dimensional plane Tn [Berger 2012). Since we deal with constant curvature spaces where all the sectional curvatures are equal, we denote a manifold's curvature as K. Instead of curvature K, we sometimes use the generalized notion of a radius: R = 1/ VK].
There are three different types of manifolds M we can define with respect to the sign of the curva- ture: a positively curved space, a "flat" space, and a negatively curved space. Common realizations of those manifolds are the hypersphere Sx. the Euclidean space E, and the hyperboloid lil
{a R
(x,x)) 1/K}, for K
for K > 0
M
E.
R
for K < 0 where (: is the standard Euclidean inner product, and (1, :) is the Lorentz inner product,
H = fx E IR+1 (z. x)²2 = 1/K),
(xx, v) --191 + T.lh Vz, E
We will need to define the exponential map, logarithmic map, and parallel transport in all spaces we consider. The exponential map in Euclidean space is defined as expx (v) = + u, for all x E and v E E". Its inverse, the logarithmic map is log (y) = y æ. for all æ. y E E". Parallel transport in Euclidean space is simply an identity PT PT (v) v, for all x. y E and v E T=B". An overview of these operations in the hyperboloid H6 and the hypersphere 8% can be found in For more details, refer Petersen al. (2006). Cannon al. (197), or Appendix A
Table T
2.1
STEREOGRAPHICALL PROJECTED SPACES
The above spaces are enough to cover any possible value of the curvature, and they define all the necessary operations we will need train VAEs in them. However, both the hypersphere and the hyperboloid have an unsuitable property, namely the non-convergence of the norm of points as the curvature goes O. Both spaces grow as K 0 and become locally "flatter", but do that, their points have to go away from the origin of the coordinate space 0 to be able to satisfy the manifold's definition. A good example of a point that diverges is the origin of the hyperboloid (or a pole of the
hypersphere) μ40 (L/vIK) (K) 0, ... 0)T In general, we
that || 1/K K-0, too.
Additionally, the distance and the metric tensors of these spaces do not converge to their Euclidean variants K - O, hence the spaces themselves do not converge Rdd This makes both of these spaces unsuitable for trying to learn sign-agnostic curvatures.
Luckily, there exist well-defined non-Euclidean spaces that inherit most properties from the hyper- boloid and the hypersphere, yet do not have these properties namely, the Poincaré ball and the projected sphere, respectively. We obtain them by applying stereographic conformal projections, meaning that angles are preserved by this transformation. Since the distance function on the hyper- boloid and hypersphere only depend on the radius and angles between points, they are isometric.
We first need to define the projection function Px- For a point (E: "T E g2++1 and curvature K R. where & R. x, y E R
1
K
1
2
2nT
T
pK((E: x TTT)
PK -1 (g) =
1
KTT
K
K
|1
+
K
1
2

---

Published as a conference paper at ICLR 2020
The formulas correspond the classical stereographic projections defined for these models (Lee) 1997/. Note that both of these projections map the point 040 (L/v|K), 0) in the original space to p4 = O in the projected space, and back.
Since the stereographic projection is conformal, the metric tensors of both spaces will be conformal. In this case, the metric tensors of both spaces are the same, except for the sign of K: g = g _x (A#) )gg, for all in the respective manifold (Ganea al. 2018a), and 0y I for all y E. The conformal factor A is then defined as X 2/(1 + K | ( Among other things, this form of the metric tensor has the consequence that we unfortunately cannot define a single unified inner product in all tangent spaces at all points. The inner product at a E M has the form of (u, v)_
)2 (> )2 (u, 2)2 for all u, v E T.
We can now define the two models corresponding to K > 0 and K < 0. The curvature of the projected manifold is the same as the original manifold. An n-dimensional projected hypersphere
(K > 0) is defined as the set D - pk \ { (-=) = R", where μ40 - (1// [1/vK]] ................... ,0 E k along with the induced distance function. The n--dimnniinnal Poincaré bull IPR (also called the Poincaré disk when n = 2) for a given curvature K < is defined as PR = PK (K()) =
( R": :: x)2 < k} with the induced distance function.
2.2 GYROVECTOR SPACES
An important analogy to vector spaces (vector addition and scalar multiplication) in non-Euclidean geometry is the notion of gyrovector spaces (Ungar) (008). Both of the above spaces D3 k and Px (jointly denoted as MK) share the same structure, hence they also share the following definition of addition. The Möbius addition θ of x; E MM (for both signs K) is defined as
(1 2K (2r, y) K 1 j + (1 K 1l
x D y
2K (x. v) + K2 ||0|| 1
We can, therefore, define "gyrospace distances" for both of the above spaces, which are alternative curvature-aware distance functions
2
dr (x, y)
VK tan - - vll2), de (z,
V-K tanh (V--K K θ B112).
These two distances equivalent to their non- non- gyrooypsopcce variants dm( y) dgn (z. y), as is shown in Theorem A.4 A.4 and its hypersphere equivalent. Additionally, Theorem A.5 shows that
dMgg (I. y) K-0, 2 ||= -
which means that the distance functions converge to the Euclidean distance function as K 0.
We can notice that most statements and operations in constant curvature spaces have dual statement or operation in the corresponding space with the opposite curvature sign. The notion of duality one which comes up very often and in our case is based on Euler's formula e cos(s) + i sin(2) and the notion of principal square roots V ivK: K. This provides a connection between trigonometric, hyperbolic trigonometric, and exponential functions. Thus, we convert all the hyperbolic formulas above to their spherical equivalents and vice-versa.
Since Ganea et al. 20188) and Tifrea al. (2099) used the same gyrovector spaces to define exponential map, its inverse logarithmic map, and parallel transport in the Poincaré ball, we can reuse them for the projected hypersphere by applying the transformations above, as they share the same formalism. For parallel transport, we additionally need the notion of gyration (Ungar 2008
gyr(æ, y/v ex (x @x y) @D (æ 0K (y @K v)).
Parallel transport in the both the projected hypersphere and the Poincaré ball then is PT (T) (XK / /X )gyr(y, =2/0 for all x, y E MM and v Ta Using curvature aware definition scalar products ((x, z) K (x, y), if K 2 0, (x, y) c if K < 0) and of trigonometric functions
sin K 0 sinh
cos if > 0 COS cosh
tan K>
sink
tank tanh
if K <
if K < 0
if K <<<
can summarize all the necessary operations in all manifolds compactly in Table and Table
3

---

Published as a conference paper at ICLR 2020
Table Summary of operations in Sx and Hg.
Distance
d(æ, y)
cosk ((K) (zz, 3) x)
V|Ky
Exponential map
K expx (v)
coSk ( [VK]] |1/llxx)))
sink (vIK) |K| [|011x)
v
//K ||01
cos (K (ar, ())k)
Logarithmic map
log (y)
(y - (xc, U) x)
sink (cos (K (xe, 3)) k)
K (y, v) K
Parallel transport
(xc + y) PT (Tz-y(D) (v) = v + K (x. y)
Table 2: Summary of operations in projected spaces Dx and Pk-
1
2K | 2/12
Distance
d(xr y)
COSK
1
v
K 2
(1 + K |0 2 (1 + K 11
Gyrospace distance
dgyr(x, y)
tan
( (VlK)
[
1 K 8112)
K
K
K |
Exponential map
expx (v) x 0K
tank VIK|
2
ViKi
|
Logarithmic map
log (y) V
tank
(VIK -1-20 + 8/12)
S K y
1
Dx 2112
A PTK x-y (D)
Parallel transport
gyrlly.-aa
AR
2.3 PRODUCTS OF SPACES
Previously, our space consisted of only one manifold of varying dimensionality and fixed curvature. Like Gu et al. 2019), we propose learning latent representations in products constant curvature spaces, contrary to existing VAF approaches which are limited to a single Riemannian manifold.
Our latent space M' consists of several component spaces M' X"= (=1 Mi; where n; is the di- mensionality of the space, K; is its curvature, and M E (E, S, D1, HH,, P] is the model choice. Even though all components have constant curvature, the resulting manifold M' has non-constant curva- ture. Its distance function decomposes based its definition (x, y) Σi= d? dwnz (x y
where (i) represents vector in M K. corresponding to the part of the latent space representa- tion of x belonging to M All other operations we defined on our manifolds are element-wist wise. Therefore, we again decompose the representations into parts x (1), apply the operation on that part
(+)) = - (()))(a)())) (x)(1) and concatenate the resulting parts back ă = O :=1 (+))
The signature of the product space, i.e. its parametrization, has several degrees of freedom per component: (i) the model M, (ii) the dimensionality n;, and (iii) the curvature K,- We need to select all of the above for every component in our product space. To simplify, we use a shorthand notation for repeated components: (M*) K, y = X"=
Euclidean spaces, the notation is redundant. For 7I....,, 72g. Z, such that Σ n, n Z, it holds that the Cartesian product of Euclidean spaces E" is E" X E.: However, the equality does hold for the other considered manifolds. This is due to the additional constraints posed on the points in the definitions of individual models of curved spaces.
4

---

Published as a conference paper at ICLR 2020
2.4 PROBABILITY DISTRIBUTIONS ON RIEMANNIAN MANIFOLDS
To be able to train Variational Autoencoders, we need to chose a probability distribution p as a prior and corresponding posterior distribution family q. Both of these distributions have to be differentiable with respect to their parametrization, they need to have a differentiable Kullback- Leiber (KL) divergence, and be "reparametrizable" (Kingma Welling 2014). For distributions where the KL does not have a closed-fome solution independent on 2, or where this integral is too hard to compute, we can estimate it using Monte Carlo estimation
((2(0) 1 Σ log p(2(0) L
1(2(1)
DKL (g)l =
if log
((z
where =0)) ~ for all 1 1= = 1, 1,... L. The Fuclidean VAE uses a natural choice for a prior on its latent representations the Gaussian distribution (Kingma & Welling 2014). Apart from satisfying the requirements for VAE prior and posterior distribution, the Gaussian distribution has addi- tional properties, like being the maximum entropy distribution for given variance (Jaynes 1957). There exist several fundamentally different approaches to generalizing the Normal distribution to Riemannian manifolds. We discuss the following three generalizations based on the way they are constructed (Mathieu et al. (019)..
Wrapping This approach leverages the fact that all manifolds define a tangent vector space at every point. We simply sample from a Gaussian distribution in the tangent space at po with mean 0, and use parallel transport and the exponential map to map the sampled point onto the manifold. The PDF can be obtained using the multivariate chain rule if we can compute the determinant of the Jacobian of the parallel transport and the exponential map. This is very computationally effective at the expense of losing some theoretical properties.
Restriction The "Restricted Normal" approach is conceptually antagonal instead of expanding point to a dimensionally larger point, we restrict a point of the ambient space sampled from Gaussian to the manifold. The consequence is that the distributions constructed this way are based on the "flat" Euclidean distance. An example of this is the von Mises-Fisher (vMF) distribution (Davidson et al. 2018). A downside of this approach is that vMF only has a single scalar covariance parameter K, while other approaches parametrize covariance in different dimensions separately.
Maximizing entropy Assuming a known mean and covariance matrix, we want to maximize the entropy of the distribution (Pennec) 2006). This approach is usually called the Riemannian Nor- mal distribution. Mathieu et al. (2019) derive for the Poincaré ball, and Hauberg (2018) derive the Spherical Normal distribution on the hypersphere. Maximum entropy distributions resemble the Gaussian distribution's 's properties the closest, but it is usually very hard to sample from such distribu- tions, compute their normalization constants, and even derive the specific form. Since the gains for VAE performance using this construction of Normal distributions over wrapping is only marginal, as reported by Mathieu et al. (2019), we have chosen focus on Wrapped Normal distributions.
To summarize, Wrapped Normal distributions are very computationally efficient to sample from and also efficient for computing the log probability of a sample, as detailed by Nagano et al. (2019). The Riemannian Normal distributions (based on geodesic distance in the manifold directly) could also be used, however they are more computationally expensive for sampling, because the only methods available are based on rejection sampling (Mathieu et al. 2019).
2.4.1 WRAPPED NORMAL DISTRIBUTION
First of all, we need to define an "origin" " point on the manifold, which we will denote as μi e Mg. What this point corresponds to is mannifld-specifie: in the hyperboloid and hypersphere corresponds to the point #4 (1/ / |K], .) and in the Poincaré ball, projected sphere, and Euclidean space it is simply #4 = O, the origin of the coordinate system.
Sampling from the distribution WM Σ) has been described in detail in Nagano al. (0019) and Mathieu et al. (2019), and we have extended all the necessary operations and procedures to arbitrary curvature K. Sampling then corresponds to:
N Σ) MR; PTK PTKk- (v) TyMMK, ==xpp exp K (u) E MR.
5

---

Published as a conference paper at ICLR 2020
The log- probability of samples can be computed by the reverse procedure:
u = log (z) E TM.MK, w= PT m (2) E THo
0f
log WW Σ) log Σ) log det
where = cxp o PT poー The distribution be applied all manifolds that have intro- duced. The only differences are the specific forms of operations and the log-determinant in the PDF The specific forms of the log-PDF for the four spaces HI, S. DD, and P are derived in Appendix B All variants of this distribution are reparametrizable, differentiable, and the KI. can be computed using Monte Carlo estimation. As consequence of the distance function and operations convergence the- orems for the Poincaré ball |A.5 (analogously for the projected hypersphere), [A.11 A.18 and A.20 we see that the Wrapped Normal distribution converges to the Gaussian distribution as K -
3 VARIATIONAL. AUTOENCODERS IN PRODUCTS SPACES
To be able to learn latent representations in Riemannian manifolds instead of in Rd as above, only need to change the parametrization of the mean and covariance in the VAE forward pass, and the choice of prior and posterior distributions. The prior and posterior have to be chosen depending on the chosen manifold and are essentially treated as hyperparameters of our VAE. Since we have defined the Wrapped Normal family of distributions for all spaces, we can use WN o² I²I) as the posterior family, and W.N I) as the prior distribution. The forms of the distributions depend on the chosen space type. The mean is parametrized using the exponential map exp pmo as an activation function. Hence, all the model's 's parameters live in Euclidean space and can be optimized directly.
K
In experiments, we sometimes use vMF (14, x) for the hypersphere S% (or a backprojected variant of vMF for D&x with the associated hyperspherical uniform distribution U($)) as prior (Davidson al. 2018). or the Riemannian Normal distribution RN o²) and the associated prior R&N RNNPo
for the Poincare ball P (Mathieu et al. 2019).
3.1 LEARNING CURVATURE
We have already approaches to learning VAEs in products of spaces of constant curvature. However, we can also change the curvature constant in each of the spaces during training. The individual spaces will still have constant curvature each point, we just allow changing the constant in between training steps. To differentiate between these training procedures, we will call them fixed curvature and learnable curvature VAEs respectively.
The motivation behind changing curvature of non-Euclidean constant curvature spaces might not be clear, since it is apparent from the definition of the distance function in the hypersphere and hyperboloid d(x 3) = R f that the distances between two points that stay at the same angle only get rescaled when changing the radius of the space. Same applies for the Poincaré ball and the projected spherical space. However, the decoder does not only depend on pairwise distances, but rather on the specific positions of points in the space. It can be conjectured that the KI. term of the ELBO indeed is only "rescaled" when we change the curvature, however, the reconstruction process is influenced in non-trivial ways. Since that is hard to quantify and prove, we devise a series of practical experiments to show overall model performance is enhanced when learning curvature.
fixed curvature VAEs, all component latent spaces have a fixed curva- ture that is selected priori and fixed for the whole duration of the training procedure, as well as during evaluation. For Euclidean components it is O, for positively or negatively curved spaces any positive or negative number can be chosen, respectively. For stability reasons, we select curvature values from the range [0.25, 1.0], which corresponds to radii in [1.0, 2.0. The exact curvature value does not have significant impact performance when training fixed curvature VAE, motivated by the distance rescaling remark above. In the following, we refer to fixed curvature components with constant subscript, e.g. Hy.
Fixed curvature VAFS

---

Published as a conference paper at ICLR 2020
Learnable curvature VAEs In all our manifolds, the curvature K. This enables us to treat K as a parameter of the model and lear it using gradient- based optimization, exactly like we learn the encoder/decoder maps in VAE.
differentiate the ELBO with respect to
Learning curvature directly is badly conditioned - we are trying to learn one scalar parameter that influences the resulting decoder and hence the ELBO quite heavily. Empirically, we have found that Stochastic Gradient Descent works well to optimize the radius of a component. We constrain the radius be strictly positive in all non-Euclidean spaces by applying a ReLU activation function before we use it in operations.
Universal curvature VAEs However, we must still a priori select the "partitioning" of our latent space - the number of components and for each of them select the dimension and at least the sign of the curvature of that component (signature estimation).
The simplest approach would be to just try all possibilities and compare the results on a specific dataset. This procedure would most likely be optimal, but does not scale well.
To eliminate this, we propose an approximate method we partition our space into 2-dimensional components (if the number of dimensions odd, one component will have dimensions). We initialize all of them as Euclidean components and train for half the number of maximal epochs we are allowed. Then, we split the components into 3 approximately equal-sized groups and make one group into hyperbolic components, one into spherical, and the last remains Euclidean. We do this by changing the curvature of a component by a very small c. We then train just the encoder/decoder maps for a few epochs to stabilize the representations after changing the curvatures. Finally, allow learning the curvatures of all non-Fuclidean components and train for the rest of the allowed epochs. The method is not completely general, it never uses components bigger than dimension 2, but the approximation has empirically performed satisfactorily.
We also do not constrain the curvature of the components specific sign in the last stage of training. Therefore, components may change their type of space from a positively curved to a negatively curved one, vice-versa.
Because of the divergence of points as K + 0 for the hyperboloid and hypersphere, the universal curvature VAE assumes the positively curved space is ID and the negatively curved space is P. In all experiments, this universal approach is denoted as U?..
4 EXPERIMENTS
For our experiments, we use four datasets: (i) Branching diffusion process (Mathicu et al.
2019
BDP) synthetic tree-like dataset with injected noise, (ii) Dynamically MNIST digits
[eCCun
1 1998)- we binarize the images similarly to Buuud et al.] (2016); Salakhutdinov & Murray 2008 the training set binarized dynamically (uniformly sampled threshold per-sample bin(
(0. [0.TIP = x > 1), E x S [0, 1), and the evaluation set is done with a fixed binarization mnaamically-binarize
Omniglot characters (Lake et al. 2015) downsampled to 28 x 28 pixels, and (iv) CIFAR- 10 (Krzhevsky)) 2009).
(a 0.5), (iii)
All models in all datasets are trained with early stopping on training ELBO with a lookahead of 50 epochs and a warmup of 100 epochs (Bowman et al. 2016). All BDP models are trained for 1000 epochs, MNIST and Omniglot models are trained for 300 epochs, and CIFAR for 200 epochs. We compare models with given latent space dimension using marginal log-likelihood with importance sampling (Burda et al. 2016) with 500 samples, except for CIFAR, which uses 50 due to memory constraints. In all tables, we denote it as LL. We run all experiments at least 3 times to get an estimate of variance when using different initial values.
In all the BDP, MNIST, and Omniglot experiments below, we use simple feed-forwart encoder and decoder architecture consisting of a single dense layer with 400 neurons and element- ReLU activation. Since all the VAE parameters {0.Φ} live in Euclidean manifolds, standard gradient- based optimization methods. Specifically, we use the Adam (ingmma & Ba 2015) optimizer with learning rate of 10 and standard settings for 3, = 0.9, B2 0.999, and c 10
use
For the CIFAR encoder map, we use a simple convolutional neural networks with three convolutional layers with 64, 128, and 512 channels respectively. For the decoder map, we first dense layer
7

---

Published as a conference paper at ICLR 2020
Table 3: Summary of results (mean and standard deviation), latent space dimension 6, spherical covariance, on the BDP (left) and MNIST (right) datasets.
Model
LL
Model
LL
sq De E6
- 55581+0038
St
96.7110011
- -55.78+0.07
vMF Sf D
-97.03=0.14
-5..28.0.56
-9..21+0.23
(H²,)²
-5.008+0.22
E
-97.16:10.15
H 1
- -56.18+0.32
HIS
-97.10=#044
(P22)
55.988+0.62
(IP2 )as
-97.56=2000
p6
56.7410.55
(R. P2 )
-9..5410.199
(RWV P2,
54.9911012
($2)3
-96.46:10.12
(82)3
-56.05-0.21
$6
-96.72+0.15
(D²)3
-53.06=t0.36
vMF S
-9..72#0.18
(H2)3
-5.80++0.2
ID6
-97.72++0155
(P2y3
-56.29+0.00
(1
-0..37
(RW 3223
-56.25:10.56
(R. P2)3
-94.1610.68
D2 x E22 p2
-55.87+0.22
DD x E² p2 DD X E² (RN F2))
-97.48+0.18
E² x H x $
-55.9210.42
490.43+0.47
E x H2 X (vMF 8²)
-55.82+00433
DP X E² X (RN 1P_
96. 96.18-00.21
E x 1 (vMF 8i)
-55.77+0.51
EE X 1HI S
40.80
(UR)3 U
-55.56+0.13
E² X H x S? E² X H² x (vMF 8?)
99..6611.00
96.56=10.27
-0..8440.38
(U2)3 U6
-97.12
-97.26=+0.16
of dimension 2048, and then three consecutive transposed convolutional layers with 256, 64, and 3 channels. All layers are followed by a ReLU activation function, except for the last one. All convolutions have x 4 kernels with stride 2, and padding of size
The first 10 epochs for all models are trained with a fixed curvature starting at and increasing in absolute value each epoch. This corresponds (217)). For learnable curvature approaches we then use Stochastic Gradient Descent with learning rate 10 and let the optimizers adjust the value freely, for fixed curvature approaches stays the last burn-in value. All our models use the Wrapped Normal distribution, or equivalently Gaussian in Euclidean components, unless specified otherwise. All fixed curvature components are denoted with Mr or M-1 subscript, learnable curvature components do not have subscript. The observation model for the reconstruction loss term were Bernoulli distributions for MNIST and Omniglot, and standard Gaussian distributions for BDP and CIFAR.
burn-in period similarly Nickel & Kiela
As baselines, we train VAEs with spaces that have a fixed constant curvature, i.e. assume a single Riemannian manifold (potentially product of them) their latent space. It is apparent that models with a single component, like ST correspond to Davidson et al. (2018) and Xu & Durrett 2018), H is equivalent to the Hyperbolic VAE of Nagano al. (2019). p corresponds to the V.- of Mathieu et al. (2019), and E" is equivalent to the Euclidean VAE. the following, we present selection of all the obtained results. For more information see Appendix [E] Bold numbers represent values that are particularly interesting. Since the Riemannian Normal and the von Mises- Fischer distribution only have spherical covariance matrix, i.e. single scalar variance parameter component, we evaluate all our approaches with a spherical covariance parametrization as well.
Binary diffusion process For the BDP dataset and latent dimension 6 (Table 3). we observe that all VAEs that only use the von Mises-Fischer distribution perform worse than Wrapped Normal. However, when a VMF spherical component was paired with other component types, it performed better than if Wrapped Normal spherical component was used instead. Riemannian Normal VAEs did very well on their own the fixed Poincaré VAE (R. p2,)" obtains the best score. did not fare well when tried learn curvature with it, however.
8

---

Published as a conference paper at ICLR 2020
Table 4: Summary of selected models (mean and standard deviation), latent space dimension (left)
and 12 (right), diagonal covariance, on the MNIST dataset.
Model
LL
Model
LL
99.51
(Sg) )
09.992
De
-07.89=0.10
( 6
-0.153
E
-96.8820.16
B12
-79.51+0.00
H6
-0.73
(H² ,)
-0.,5440.23
p6
-0.13
H
-79.37 #0.14
S6
1..24
(P22)6
-8..39
D66
20.22
g12
-0..99=0.27
(HI2)3
-96.86=0.31
D
-80.37
H
-0..90=0.26
H
-99.77
p6
197.26=0016
(12)6
-80.31+0.08
D& x E x P2
-97.37
(DR)2 ( (P21)²
-80.1410.11
D x L x p2
-07.29=0.16
DJ x E x IPd
-0..14
B x H x 8
-9..71
(Ea)² x (H2))² x (83)²
79.59
E² x H2
-96.66-0.27
E1 x H x S1
-79. 79.6940.11
(U2)3
-9..0620.13
(U2)6
-79.6 6..6 10.06
T
-9.990.0.10
TJ12
-80.01 10.30
An interesting observation is that all singlc-component VAEs M performed worse than product VAEs (M2)s when curvature was learned, across all component types. Our universal curvature VAE (U2)3 managed to get better results than all other approaches except for the Riemannian Normal baseline, but within the margin of error of some other models. It also outperformed its single- component variant US. However, we did not find that it converged to specific curvature values, only that they were the approximate range of (-0.1, +0.1).
Dynamically-binarzzz MNIST reconstruction On MNIST (Table with spherical covariance, we noticed that VMF again under-performed Wrapped Normal, except when it was part of a product like E x H2 x (vMF ) When paired with another Euclidean and a Riemannian Normal Poincaré disk component, it performed well, but that might be because the RN P-1 component achieved best results across the board on MNIST. It achieved good results even compared diagonal covariance VAEs on 6-dimensional MNIST. Several approaches are better than the Euclidean baseline. That applies mainly the above mentioned Riemannian Normal Poincaré ball components, but also ş6 both with Wrapped Normal and VMF, as well as most product space VAEs with different curvatures (third section of the table). Our (U2)3 performed similarly to the Euclidean baseline VAE.
With diagonal covariance parametrization (Table 4. we observe similar trends. With latent di- mension 6, the Riemannian Normal Poincaré ball VAE still the best performer. The Euclidean baseline VAE achieved better results than its spherical covariance counterpart. Overall, the best result is achieved by the single-componed ingly, all single-component VAES performed better than their (M2)3 counterparts, except for the hyperboloid, but only by tiny margin. Products of different component types also achieve good results. Noteworthy that their fixed curvature variants seem perform marginally better than learnable curvature ones. Our universal VAES perform at around the Euclidean baseline VAE performance. Interestingly, all of them end up with negative curvatures -0.3 K 0.
spherical model, with learnable curvature Sg. Interest-
Secondly, we run our models with a latent space dimension of 12. We immediately notice, that not many models can beat the Euclidean VAE baseline 12 consistently, but several are within the margin of error. Notably, the product VAEs of H., S, and IE, fixed and learnable 12 and our universal VAE (12)). Interestingly, products of small components perform better when curvature fixed, whereas single big component VAEs are better when curvature is learned.
Dynamically siaamiclly-binarizedd Omniglot reconstruction For latent space of dimension 6 (Table 5) the best of the baseline models the Poincaré VAE of (aahhieu et al. 2019). Our models that come

---

Model Published as a conference paper at ICLR 2020
Table 5: Summary of selected models (mean and standard deviation), latent space dimension 6, diagonal covariance, on the Omniglot (left) and CIFAR-10 dataset (right), respectively.
LL
Model
LL
S
136.69
E
1896.1942..00. 19422.51
1137.42=1.20
H
1888 22
-136.03=20.0
p-
1899.2740..0
H
-137.09
D6
Pe
-135.86=0.20
g6
1889. 76.1
(S²)3
136.1410.27
-1891. 2591.4012.14
36
-136.20=0.11
DD2 X EE x 1p2
1899.90++1.60
(ID2)3
-136.13=0.17
E H x x
-1895.46200.
ID:
-136.30=0008
(U²)3
-1895. 4492.
(12)3
11.6.77=0.09
116
136.24=0.32
(IP2)3
136.09=0.07
Ip6
- -3...05:0.44
2 E22 p2
-1.5.89=0040
2 x H2 x S²
- -135.93=0.48
(T2)3
133.21 10.07
U
136.0410.1.
very close to the average estimated marginal log-likelihood, and are definitely within the margin of error, are mainly (82) 2 E² x 12 and U However, with the variance of performance across different runs, we cannot draw a clear conclusion. In general, hyperbolic VAEs seem to be doing a bit better on this dataset than spherical VAEs, which is also confirmed by the fact that almost all universal curvature models finished with negative curvature components.
CIFAR-10 reconstruction For latent space dimension 6, we can observe that almost all non- Euclidean models perform better than the euclidean baseline ES, Especially well-performing is the fixed hyperboloid H 1, and the learnable hypersphere $6 Curvatures for all learnable models on this dataset converge to values in the approximate range of (-..15,00.15..
Summary conclusion, a very good model seems to be the Riemannian Normal Poincaré ball VAE RN Ip.. However. it has practical limitations due to rejection sampling algorithm and an unstable implementation. On the contrary, von Mises-Fischer spherical VAES have almost consis- tently performed worse than their Wrapped Normal equivalents. Overall, Wrapped Normal VAEs in all constant curvature manifolds seem to perform well at modeling the latent space.
key takeaway is that our universal curvature models U and (U3)L//2 seem to generally out- perform their corresponding Euclidean VAE baselines in divee--imesiioall latent spaces and, with minor losses, manage to keep most the competitive performance as the dimensionality goes up, contrary to VAES with other Fuclliden components.
CONCLUSION
By transforming the latent space and associated prior distributions onto Riemannian manifolds of constant curvature, has previously been shown that we can learn representations on curved space. Generalizing on the above ideas, we have extended the theory of learning VAEs to products of con- stant curvature spaces. To do that, we have derived the necessary operations in several models of constant curvature spaces, extended existing probability distribution families to these manifolds, and generalized VAEs to latent spaces that are products of smaller "component" spaces, with learnable curvature. On various datasets, show that approach is competitive and additionally has the property that it generalizes the Euclidean variational autoencoder if the curvatures of all compo- nents go recover the VAE of Kingma & Welling (2014).
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We would like thank Andreas Bloch for help in verifying some of the formulas for constant curvature spaces and for many insightful discussions; Prof. Thomas Hofmann and the Data Analytics Lab, the Leonhard cluster, and ETH Zürich for GPU access.
Work was done while all authors were at ETH Zürich.
Ondrej Skopek (eskopexégooglee com) now Google.
Octavian-Bugen Ganea (oct @mit edu) is now at the Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology.
Gary Bécigneul gaazybbecigneule1ngg ethz ch) is funded by the Max Planck ETH Center for Learning Systems.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4):433-459 2010. doi: 10.100/wwss.101.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
Gregor Bachmann, Gary Bécigneul, and Octavian Eugen Ganea. Constant Curvature Graph Convo- lutional Networks, 2020. URL
Hx
Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, and Sam Gershman. Nonparametric Spherical Topic Modeling with Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Shori Papers), pp. 537-542. Associa- tion for Computational Linguistics, 2016. doi: 10.18653/v1/P16-2088
Gary Bécigneul and Octavian-Eugen Ganea. Riemannian Adaptive Optimization Methods. In ternational Conference on Leurning Representations, 2019.
Marcel Berger. panoramic view of Riemannian geometry. Springer Science & Business Media, 2012.
Janos Bolyai. Appendix, Scientiam Spatii absolute veram exhibens: a veritate aud falsitate Axioma- tis XI. Euclidei (a priori haud unquam decidenda) independentem; adjecta ad casum falsitatis, quadratura circuli geometrica. Auctore Johanne Bolyai de eadem, Geometrarum in Exercitu Cae- sareo Regio Austriaco Castrensium Capitaneo. Coll. Ref., 1832.
Silvere Bonnabel. Stochastic Gradient Descent on Riemannian Manifolds. IEEE Transactions on Automatic Control, 58:2217-2229, 2013.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating Sentences from a Continuous Space. In Proceedings of The 20th SIGNLL Confer- ence on Computational Natural Language Learning, pp. 10-21. Association for Computational Linguisties, 2016. doi: 10.18653/11/K11--1000
M. M. Bronstein, Bruna, LeCun, A. Szlam, and Vandergheynst. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, July 2017. ISSN 1053-5888. doi: 10.1 10.1100//SP2221.226992
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. In Proceedings of the 4th International Conference on Learming Representations, ICLR 2016, 2016.
James W Cannon, William J Floyd, Richard Kenyon, Walter Parry, al. Hyperbolic geometry. Flavors of geometry, 31:59-115, 1997.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. In Proceedings of the 2nd Interna- fional Conference Learning Representations, ICLR 2014, 2014.

---

Published as a conference paper at ICLR 2020
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspher- ical Variational Auto-Encoders. In LAI, 2018.
Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew M Dai, and George Dahl. Embedding Text in Hyperbolic Spaces. arXiv preprint arXiv: 043100000043,,3 2018.
Robert L. Foote. A Unified Pythagorean Theorem in Euclidean, Spherical, and Hyperbolic Geome- tries. Mathematics Magazine, 90(1):59-69, 2017. ISSN 0025570X, 19300980.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Neural Networks. S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. $345-5355,
Curran Associates, Inc., 2018a.
Sraviann-uugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. arXiv preprint 180xv:880400888, 2018b.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing Flows on Riemannian Manifolds. arXiv preprint arXiv:1611.022 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning Mixed-Curvature Representa-
tions in Product Spaces. In International Conference on Learning Reprexentations, 2019.
Søren Hauberg. Directional Statistics with the Spherical Normal Distribution. In 2018 2 Interna- tional Conference on Information Fusion (FUSION), pp. 704-711, IEEE, 2018.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. arXiv preprint arXiv: 1804.00779, 2018.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, et al. Adversarially regularized autoen- coders. arXiv preprint arXiv: /70ivv a76.v21706.04223, 04223, 2017.
Diederik Kingma and Jimmy Lei Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Max Kochurov, Sergey Kozlukov, Rasul Karimov, and Viktor Yanush. Geoopt: Adaptive Rieman- nian optimization in PyTorch. htttps:/ com wtt /ttp:///ggooottgggtgtttoo 2019.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Uni- versity of Toronto. 2009.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266);133--33388 2015. ISSN 0036-8075. doi: 10. 2000000.nce.uab3550.
Johann Lambert. Observations trigonométriques. Mem, Acad. Sci. Berlin, 24:327-354, 1770.
Mare T Law, Jake Snell, and Richard S Zemel. Lorentzian distance learning, 2019.
Yann LeCun. The MNIST database of handwritten digits, 1998. URL http::tttyypaaia com/ com/ exdb/mistmms
J.M. Lee. Riemannian Manifolds: An Introduction to Curvature. Graduate Texts in Mathematics. Springer New York, 1997. ISBN 9780387983226.
Hongbo Li, David Hestenes, and Alyn Rockwood. A Universal Model for Conformal Geometries of Euclidean, Spherical and Rouble-HHpertiee Spaces, pp. 77-104. Springer Berlin Heidelberg, Berlin, Heidelherg, 2001. ISBN 978-3-662-04621-0. doi: 10. 0007/978-3-662--4663 4.
12

---

Published as a conference paper at ICLR 2020
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv:1811.12355 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprini arXiv:151 1.05644, 2015.
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Hierarchi- Representations with Poincaré Variational Auto-Encoders arXiv preprint arXiv:1901.0603: 2019.
Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-VAE: Learning Basic Visual Concepts with a Constrained Varia- tional Framework. In Proceedings of the 5th International Conference Learning Representa- tions, ICLR 2017,2017
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A differen- tiable gaussian-like distribution on hyperbolic space for gradient-based learning, arXiv preprint arXiv:1902.02992, 2019.
Maximilian Nickel and Douwe Kiela. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry. In Proceedings of the 35th International Conference on International Conference on Machine Learning- Volume 50, ICML'18. 18, 2018.
Maximillian Nickel and Douwe Kicla. Poincaré Embeddings for Learning Hicrarchical Represen- tations. In Advances in Neural Information Processing Systems 30, pp. 6338-6347. Curran Asso- ciates, Inc., 2017.
Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. arXiv preprint arXiv: arXiv:/802.04440 180200402744 2018.
Xavier Pennec. Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measure- ments. Journal of Mathematical Imaging and Vision, 25(1):127, Jul 2006. ISSN 1573-7683. doi: 10.10077/10851-0006 -6228-4
Peter Petersen, S Axler, and KA Ribet. Riemannian Geometry, volume 171. Springer, 2006.
John Ratcliffe. Foundations of Hyperbolic Manifolds, volume 149. Springer Science & Business Media, 2006.
Danilo Jimenez. Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. arXiv preprint arXiv:1505.05770, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. arXiv preprint 4rXiv:1401.40822 2014.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Re. Representation tradeoff's for hyper- bolic embeddings. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4460 4469, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Ruslan Salakhutdinov and lain Murray. On the Quantitative Analysis Deep Belief Networks. In Proceedings of the 25th International Conference on Machine Learning, ICML'08, pp. 872-879, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-22 doi: 10. 145/1390156 1..11590..66
Martin Simonovsky and Nikos Komodakis. Graph Towards generation of small graphs using variational autoencoders. International Conference on Artificial Neural Networks, pp. 412- 422. Springer, 2018.
John Parr Snyder. Map projections-A working manual, volume 1395. US Government Printing Office, 1987.
Alexandru Tifrea, Gary Bécigneul, and Octavian- Eugen Ganea. Poincaré Glove: Hyperbolic Word Embeddings. In International Conference on Learning Representations, 2019.
13

---

Published as a conference paper at ICLR 2020
Abraham Albert Ungar. A Gyrovector Space Approach to Hyperbolic Geometry. Synthesis Lectures on Mathematics and Statistics, 1(1):1-194 2008.
Benjamin Wilson and Matthias Leimeister. Gradient descent in hyperbolic space. arXiv preprint arXiv: 08207, 2018.
Richard C. Wilson and Edwin R. Hancock. Spherical embedding and classification. Edwin R. Hancock, Richard C. Wilson, Terry Windeatt, Ilkay Ulusoy, and Francisco Escolano (eds.), Struc- tural, Syntactic, and Statistical Pattern Recognition, pp. 589-599, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-14980-11
Jiacheng Xu and Greg Durrett. Spherical Latent Spaces for Stable Variational Autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 4503 4513. Association for Computational Linguistics, 2018.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A. Efros. Generative Visual Manip- ulation on the Natural Image Manifold. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
14

---

Published as a conference paper at ICLR 2020
A GEOMETRICAL DETAILS
A. RIEMANNIAN GEOMETRY
An elementary notion in Riemannian geometry that of real, smooth manifold M E"", which is collection of real vectors æ that is locally similar to a linear space, and lives in the ambient space R At each point of the manifold æ E M a real vector space of the same dimensionality as M is defined, called the langent space al point æ: MM.M. Intuitively, the tangent space contains all the directions and speeds at which one can pass through æ. Given a matrix representation G( E Rux= of the Riemannian metric tensor g(z), we can define scalar product on the tangent space: ). MTM x Ta.M + M, where (a, b)x = g(x) (a b) = a a G(z)b for any a, b E M. A Riemannian manifold is then the tuple (M, The scalar product induces norm the tangent space M: ||a|| V(a, a), Ya E M (Petersen et al. 2006).
Although seems like the manifold only defines a local geometry, it induces global quantities by integrating the local contributions. The metric tensor induces local infinitesimal volume element on each tangent space M and hence measure induced as well d.M(x) vGG(x) le where dæ is the Lebesgue measure. The length of curve t + r(t) E M, E [0, 1] is given by
L(>) = So VIl II r())
Straight lines are generalized constant speed curves giving the shortest path between pairs of points E M, so called geodesics, for which it holds y = arg min, L(7), such that y(0)
2, 711 = y. and > = 1. Global distances are thus induced on M by dm (x, = inf, L()). Using this metric, we can go on to define a metric space (M, dm). Moving from a point x E M in a given direction a E T with constant velocity is formalized by the exponential map: expx T + M. There exists a unique unit speed geodesic y such that 7(0) = a and dr(t) dt r=-
v, where M.
The corresponding exponential map then defined as expx(v) = (1). The logarithmic map the inverse log,, expx M - M. For geodesically complete manifolds, i.c. manifolds in which there exists a length-minimizing geodesic between every æ; y E M, such as the Lorentz model, hypersphere, and many others, expx is well-defined on the full tangent space T..M.
To connect vectors in tangent spaces, we use parallel transport PT T - Ty.M, which is an isomorphism between the two tangent spaces, so that the transported vectors stay parallel to the connection. It corresponds to moving tangent vectors along geodesics and defines a canonical way to connect tangent spaces.
A.2 BRIEF COMPARISON OF CONSTANT CURVATURE SPACE MODELS
We have seen five different models of constant curvature space, each of which has advantages and disadvantages when applied learning latent representations in them using VAEs.
A big advantage of the hyperboloid and hypersphere is that optimization in the spaces does not suffer from as many numerical instabilities as it does the respective projected spaces. On the other hand, we have seen that when K + O, the norms of points go to infinity. As we will see in experiments, this is not problem when optimizing curvature within these spaces in practice, except if we' re trying to cross the boundary at K = O and go from a hyperboloid to a sphere, or vice versa. Intuitively, the points are just positioned very differently the ambient space of HI., and S., for a small > 0.
Since points in the n-dimensional projected hypersphere and Poincaré ball models can be repre- sented using real vector of length n, it enables to visualize points in these manifolds directly for n = 2 or even n 3. On the other hand, optimizing a function over these models not very well-conditioned. In the case of the Poincaré ball, a significant amount of points lie close to the boundary of the ball (i.e. with a squared norm of almost 1/K), which causes numerical instabilities when using bt float precision in computations.
A similar problem occurs with the projected hypersphere with points that are far away from the origin 0 (i.e. points that are close to the "South pole" the backprojected sphere). Unintuitively,
15

---

Published as a conference paper at ICLR 2020
all points that are far away from the origin actually very close to each other with respect to the induced distance function and very far away from each other terms of Euclidean distance.
Both distance conversion theorems (A.5 and its projected hypersphere counterpart) rely the points fixn when changing curvature. they are somehow dependent on curvature, the convergence theorem does not hold. We conjecture that if points stay close to the boundary in P far away from in as - 0, this is exactly the reason for numerical instabilities (apart from the standard numerical problem of representing large numbers in floating-poin notation).
Because of the above reasons, we do some of our experiments with the projected spaces and others with the hyperboloid and hypersphere, and aim compare the performance of these empirically as well.
A.3 EUCLIDEAN GEOMETRY
A.3.1
EUCLIDEAN SPACE
Distance function
The distance function in E"
(1, y) = || yl12.
Due to the Pythagorean theorem, we can derive that
a
31 (x y. y) ||
2 (ax, 4) + 11
x 1eel 2 ||>0|| || cas
Ox
Exponential map
The exponential map in E is
(x) x + v.
The fact that the resulting points belong the space is trivial. Deriving the inverse function, i.e. the logarithmic map, also trivial:
log (v)
Parallel transport We do not need parallel transport the Euclidean space, as we can directly sample from Normal distribution. In other words, just define parallel transport to be an identity function.
A.4 HYPERBOLIC GEOMETRY
A.4.1 HYPERBOLOID
Do note, that all the theorems for the hypersphere are essentially trivial corollaries of their equiva- lents in the hypersphere (and vice-versa) (Section A.5.I). Notable differences include the fact that R2 = - K not H²2 1. and all the operations use the hyperbolic trigonometric functions sinh, cosh, and tanh, instead of their Euclidean counterparts. Also, often leverage the "hyperbolic" Pythagorean theorem, in the form cosh (a) sinh (a) = 1.
Projections Due to the definition the space as a retraction from the ambient space, we can project generic vector in the ambient space the hyperboloid using the shortest Euclidean distance by normalization:
projug (x) |||||c VK ||ع||ا
Secondly, the n + coordinates of point on the hyperboloid are co-dependent; they satisfy the relation (I. 2) 1/K. This implies, that if we are given vector with coordinates ă (I2...., F.4 we can compute the missing coordinate to place onto the hyperboloid:
|
|
K
This is useful for example in the case of orthogonally projecting points from T Hig onto the mani- fold.
16

---

Published as a conference paper at ICLR 2020
Distance function
The distance function in Hi is
(-
(x, y)
)
di (2, (2. R- Oz.y R cosh
-1
cosh -1 (-K (x, y) c).
It2
V-K
K
Remark A.1 (About the divergence of points in H()) are norm-constrrinee to
Since the points on the hyperboloid H
(a, x) K'
all the points on the hyperboloid go to infinity as goes to 0 from below:
lim (x, x) -00
This confirms the intuition that the hyperboloid grows "flatter", but to do that, it has to go away from the origin the coordinate space 0. A good example of point that diverges is the origin of , 0,,,0..... trying to learn sign-agnostic curvatures, similarly to the hypersphere.
the hyperboloid 1 (1 (1/K,0 1/K,,,..
0)* (R, . 0) That makes this model unsuitable for
Exponential map
The exponential map in H is
|
||م||c| R
Rv |ollee"
exp K (v) cosh
x + sinh
R
and in the case of x 140 (R, .., 0)?:
exp po (v) = cosh ||01|2
R; sinh ||8112
R
T
|
where e (0; 0²) T and ||m||c = ||0||2 = |:112-
Theorem A.2 (Logarithmic map in H For all x. y Hi. the logarithmic map in THTS maps y to langent vector at x:
cosh (a)
log (y)
y - yrx),
Va²
where a = K (xx (...))a
Proof. We show the detailed derivation of the logarithmic map as inverse function to the expo- nential map log, (y) = exp (z), adapted from (Nagano et al. 2019).
As mentioned previously,
||w||c R
||01|c
Rv
y exp (v) cosh
sinh
R
|
Solving for u, we obtain
||0||c
|
cosh
[|0]]
Rsinh
R
However, still need to rewrite ||llic in evaluatable terms:
||u|lc
|
0 = 0((x, -2,
(or, v) cosh
(x, a)
R Rsinh
[|0||]] [
R
R2
hence
|
cosh
(x, v) c
R
R2
and therefore
|| = R Roosh-
(x, ()
cash (K cosh-t(K
v)) = =d (xr, y)
R
= V-K
17

---

Published as a conference paper at ICLR 2020
Plugging the result back into the first equation, obtain
|ellcc
| e
y cosh
x
R Rsinh
10
R
cosh (a)
sinh (
cosh Rosh"" (a)) y - cosh
R
Rcosh
(a) ((a)ss))
2
cosh (a)
sinh(cosh (a)) y cosh(cosh - (aa))x)
cosh (a)
Va² - (g/ ax))
where a - holds, since for all points æ, y E Hig it holds that (xx, y) S -R² and (æt, y) = - if and only if x y, due to Bouchy-Schwarz (Ratclifie 2006 Theorem 3. 3.1.6). Hence, the only case where this
गे (a. c K (xx, y) c and the last equality assumes |a| 1. This assumption
would be problem would be
y. but is clear that the result in that case
0.
Parallel transport Using the generic formula for parallel transport in manifolds for y E M and v E M
(log (y).
PTK 2- (v) = v dml x y)
(logæ (z) + log K (a:)),
(1)
and the logarithmic map formula from Theorem A.2
cosh - (a)) y - aar),
K (y) log
Va²
where a - RT (xe, v) c, we derive parallel transport in IH?::
(y, v)
PTK (1) - v Re (or, y) c
(x + y).
special form of parallel transport exists for when the source vector is po = (R.0....,0) 0)
1/1 π
(y. 2)2
y22
PTK Mo Ny (v) = 2 R Ry1
Un+1
A.4.2 POINCARÉ BALL
Do note, that all the theorems for the projected hypersphere essentially trivial corollaries of their equivalents in the Poincaré ball (and vice-versa) (ecction[A.5.)) Notable differences include the fact that R² A. not R² A and all the operations use the hyperbolic trigonometrie functions sinh, cosh, and tanh, instead of their Euclidean counterparts. Also, we often leverage the "hyperbolic" Pythagorean theorem, in the form cash?(a) (inh2(a) 1.
Stereographic projection
Theorem A.3 (Stercographic backprojected points of PP belong to H For all y e IP-
( K
Proof.
(VIx)
T
K
1
2
2y T
| (1)
K
1
2
K
1
22
+
18

---

Published as a conference paper at ICLR 2020
K 1 - -1)
4 11 2
+
K
K 11yl1 +
(K 11 112 + 1)a
(K
11 1)2 + 4 K |
(K 11 1)2
(K 11 - 1)2 4K 11
K
(K |1y122 1)2
(K 11 - 1)2 + 4K 112112
(K |1y| 2 + 1)2
K²
111l2 + 2K 11//1/2
K
(K 11y12 1)2
(K 11
+ +1) 1 K
K
(K 1
2 + 1)2
Distance function The distance function in PT is (derived from the hyperboloid distance function using the stereographic projection pk):
(2( y) da(Pk (x)), PK (y))
2K
U12
V-K cosh -1
(1 K [1(1 K 10)1
+1
21²2 | y112
R Reosh
(1+ (R2 || [
R2 -11y1 2
Theorem A.4 (Distance equivalence in PR... For all K < <0 and for all pairs of points æ, y E Pk the Poincaré distance between them equals the gyrospace distance
dp(x. y) = dr (xc y)-
Proof. Proven using Mathematica (File: distance 1 immtt. ws), proof involves heavy algebra.
Theorem A.S (Gyrospace distance converges to Fuclidean in PP)) For any fixed pair of points v. y E Pk, the Poincaré gyrospace distance between them converges to the Euclidean distance in the limit (up to a constant) as K + 0
lim dp (2r, 3) -21 -yl12- 1110-1l2-- K K-+0-
Proof.
tanh -1 (V- K | 0K 21122
lim de,, (x, y) = lim --
K K-
V-K
tanh -1 -1 K 11 - x (2)
= lim K-+0
K
2 y æ|1
where the second equality holds because of the theorem of limits of composed functions, where
f(a) tanh (av -K K
K
19

---

Published as a conference paper at ICLR 2020
g(K) 11 @K vl12.
We that
lim K K-0
g(K) 11 =
due to Theorem A.14 and
tanh (av K) f(a)
lim 2ll
V-K K
Additionally for the last equality, we need the fact that
tanh "(av/x=)) a
lim x-
Theorem A.6 (Distance converges to Euclidean as K o in Pkk) For any fixed pair points æ, y E K. the Poincaré distance between them converges to the Euclidean distance in the limit (up constant) as K 0
lim dz y) 2 x 1112
K K-+0
Proof. Theorem A.4 and A.5
Exponential map its inverse is
As derived and proven in Ganea et al. (0118)), the exponential map in Px and
K x
|0|1
K (v) +K expx
tanh
(V-K
2
V
|0012
log (y)
2 (anh (V- K
- 0 8112)
2 BK
V- K A
ED 811
In the case of a = M4 (0.
0)² they simplify to:
exp pd
K (v) tanh (V-K K (1,0112))
/- K |
log (y) = tanh 1
(V-K ||
y
Parallel transport Kochurov et al. 2 Ganea et al. (2018)) have also derived and imple-
mented the parallel transport operation for the Poincaré ball:
A A gyrly, - 1
P (v) 2
2
PTK no (1)) =
AR.
A
PTK x tpo (v) =
2 U,
where
gyr [x. y/v - -(x 0K y) 0x (x @x (y θx v))
is the gyration operation (Ungar 2008 Definition 1.11).
Unfortunately, on the Poincaré ball, ).. has a form that changes with respect to æ, unlike in the hyperboloid.
20

---

Published as a conference paper at ICLR 2020
A.5
SPHERICAL GEOMETRY
A.5.1 HYPERSPHIERE
All the theorems for the hypersphere are essentially trivial corollaries of their equivalents in the hyperboloid (Section A.4.1). Notable differences include the fact that R K. not R² K and all the operations use the Euclidean trigonometric functions sim, cos, and tan, instead of their
hyperbolic counterparts. Also, we often leverage the Pythagorean theorem, the form sin sin((a) + (a) 1.
Projections Due to the definition the space as a retraction from the ambient space, we can project generic vector in the ambient space the hypersphere using the shortest Euclidean distance by normalization:
projen- R
|
K ||01|
Secondly, the n + +1 coordinates of a point on the sphere are co-dependent; they satisfy the relation (x,x), x22 1/K. This implies, that if we given vector with n coordinates (22, .... In41); we can compute the missing coordinate to place it onto the sphere:
1 K
31
This is useful for example in the case orthogonally projecting points from Tuo onto the mani- fold.
Distance function
The distance function in S% is
(x. y)
de (x, y) = R. Oz.v R cos"
cos
(K (x, v)2).
R
VK
Remark A.7 (About the divergence of points in Sx). norm-constrrineed
Since the points on the hypersphere æ E Sj
1
(0,
K
all the points the sphere go infinity K goes to 0+ from above:
lim (z, x)2 cc. K-0+
This confirms the intuition that the sphere grows "flatter", but to do that, it has to go away from the origin the coordinate space 0. A good example of point that diverges the north pole of the sphere 11 (1/K, K, 0, ... 0)T (R, ..... 0)² That makes this model unsuitable for trying to learn sign-agnostic curvatures, similarly to the hyperboloid.
Exponential map
The exponential map in Sy
||0||2
[||||12
Rv
exp K (v) cos
+ sin
R
R
||0||2
Theorem A.8 (Logarithmic map in si)). For all x, y E Sk. the logarithmic map in Sjk maps y to a langent vector at æ:
cos ((a)
log (y)
vI a ( ax),
where a K (x, y)
Proof. Analogous to the proof of Theorem A.2 As mentioned previously,
||0|
Ro
= exp ((v) = cos
x
sin
R
R
21

---

Published as a conference paper at ICLR 2020
Solving for v, obtain
|
||r|| R.
v
|
cos
x
Riin
However, we still need to rewrite
|/0/22 in evaluatable terms:
||o||
||20
0 (a,, u)2
(x, v)2 cos
(2) 2
Rsin
[illla
R
R
hence
|
1
cos
(x,, 1122
R
17a
and therefore
||0|| R cos
(xe, 4)/
cos
-1 (K (x,, 3)2) - d5 y)
12
VK
Plugging the result back into the first equation, we obtain
||0|/2
||07
cos
x
|
R
Rsin
R.cos
(a)
{ [ (a)
1
Rsin 1 R cos (a))
y cos
R
cos (a)
( coss(coos coo(co)=((a))z) (a))xz)
sin(cos ((((ccosss(())))))) (a))
cos (a)
v1 /1-a² axx,
where 722h (ar, y) K (xe, y) and the last equality assumes la] > 1. This assumption holds, since for all points æ, y E S% it holds that (z. y) R², and (xr, y) R² if and only if x y, due to Secwar- (Ratcliffe, 2006, Theorem 3.1.6). Hence, the only case where this would be problem would be if y. but is clear that the result in that case is = 0.
Parallel transport Using the generic formula for parallel transport in manifolds (Equation A.4.1 for x; y S% and v e TeS and the spherical logarithmic map formula
cos (a)
log (y) v1 a (y az),
where K (x., y) derive parallel transport in S%:
(y, (,,
PTK (T (v)
(x: + y)
RR + (xx y)
K ( U)2
(x: y) K (x, y)
special form of parallel transport exists for when the source vector is μo = (R, .e.,0,......... 0)":
111 R
(y. v)2
y/2
PTk (v) =
12 + Ryi
Un+1
A.5.2 PROJECTED HYPERSPHERE
Do note, that all the theorems for the projected hypersphere essentially trivial corollaries of their equivalents in the Poincaré ball (and vere-versa) (Section A.4.2). Notable differences include the fact that R² K not R² sin, cos, and tan, instead of their hyperbolic counterparts. Also, we often leverage the Pythagorean theorem, in the form sin? (a) cos" cos2(a) 1.
j. and all the operations the Euclidean trigonometric functions
22

---

Published as a conference paper at ICLR 2020
Stereographic projection
Remark A.9 (Homeomorphism between SK and R.). We notice that pk is not homeomorphism
between the n-dimensional sphere and TR", as it is not defined at -140 (- R2 O²) If we addi- tionally changed compactified the plane by adding a point "at infinity" and set it equal to PK (po), PK would become homeomorphism.
Theorem A.10 (Stercographic backprojected points of Dj belong to 5k), ). For all D'k
l (v) K
Proof.
T12
K
|1
l (v)
(1)1
K
K
|B/
K
11
1
K 11
2
(K 11 11
4
2
V
K 11/112 +
+ 1)2
(K 11 - 1)2 + K|K 11//112
(K |1 + 1)2
1 (K 11 - 1)2 + 4K 11
K
(K 11u12 + 1)2
K2
|1//12 + 2K 1 1
K
(K 11y112 + 1)2
(K 11 + 1)²
K
(K | + K
Distance function The distance function in Dr is (derived from the spherical distance function using the stereographic projection pk):
dn( y) = da(px' (2r), PK (y))
1
2K x
y
cos -1
VK
(1 + K || [1 K 11))))
(
2R || - y/12
= RRoss -1
(R2 + x ( + 11
Theorem A.11 (Distance equivalence in DR).. For all K > and for all pairs of points x; y E DD the spherical projected distance between them equals the gyrospace distance
du Ax v) dD (x. y)
Proof. Proven using Mathematica (File: distance limits, ws), proof involves heavy algebra.
Theorem A.12 (Gyrospace distance converges to Euclidean in D*)). For any fixed pair of points x. y E Di the spherical projected gyrospace distance between them converges to the Euclidean distance in the limit (up to a constant) as K +
lim do (x. 3) ||æ 4 K
23

---

Published as a conference paper at ICLR 2020
Proof.
tan (VK || - Bx 21122
lim K-50+
augp (xc, y) = lim
-+
VK
tan 1(V K 11 2112)
lim
K K->01
K
ly 2212>>
where the second equality holds because of the theorem limits of composed functions, where
f(a) = tan (avK)
VK
g(K) 1 0x 2112-
We see that due to Theorem 14 and
lim g(K) = 11 - æl12
K K-+0
tan ((v K)
lim ((a
a- ||
2lla
VK
Additionally for the last equality, we need the fact that
tanh
(av [x]) a.
lim 2 R-0
Theorem A.13 (Distance converges to Euclidean as K - 0+ in DR). For any fixed pair of points a, y E Di the spherical projected distance between them converges to the Euclideun distance in the limit (up to a constant) as 0+:
lim dp (x. y) 2 [ 3112 +
Proof. Theorem A.ll and A.12
Exponential map Analogously to the derivation of the exponential map in Pk in Ganea et al. 2018a| Section 2.3-2.4), we can derive Möbius scalar multiplication DR:
ØK
lanh(r tanh -i [|x|||2)
ll)))
WK
tanh(ri tan ((V || l2)) [
WK
tan(r tttttannnVvA (V c (2))
VK
|x|||
where we use the fact that tanh (ix) = i tan (x) and tanh(ix) = ttan(z). We can easily see
that @ x æc
Hence, the geodesic has the form of
Ya (y x #x t 8x (-x Br y),
and therefore the exponential map in D7 is:
K |001/ tan VV
exp K (v) = x 0K
v
K
24

---

Published as a conference paper at ICLR 2020
The inverse formula can also be computed:
logz (y)
2
tan 1 (VK') - +K 21l2)
x #K y θx 211
VK K
In the case a = Mo = (O...., 0)" they simplify to:
exp K
uo (v) tan (VK [|0|12]
/K 1|01|2
log (y) tan (VK 11112))
y
K ggll2
Parallel transport Similarly to the Poincaré ball, we can derive the parallel transport operation for the projected sphere:
PTK x m- (v) = Aα gyr y,
2/00
X
2
PTK
po (y() =
1 U,
AK
PTK 2 (v) U,
where
gyrlæ: œ ylu (x #x y) 0K (o OK (y θ v))
the gyration operation (Ungar 2008
Definition 1.11).
Unfortunately, on the projected sphere, ( has a form that changes with respect to æ, similarly to the Poincaré ball and unlike in the hypersphere.
A.6 MISCELLANEOUS PROPERTIES
Theorem A.14 (Möbius addition converges to Eucl. vector addition).
lim (xr Ø
y.
Note: This theorem works from both sides, hence applies to the Poincaré ball as well as the projected spherical space. Observe that the Möbius addition has the same form for both spaces.
Proof.
(1 2K (x. y) K |1/l3)) a: + (1 +
K
lim (xc 0x y) -imm
K-+0
2K (x,, 2) + K2 | 2 1
x + y.
Theorem A.15 (px² is the inverse stereographic projection).
For all (E; MM & R
Pkl (p((::==="))) = x.
where M E H..
Proof.
Pr (7) = =Pr
x
25

---

Published as a conference paper at ICLR 2020
K
T
1
K
V-vVRR |RiE
K
K
K
+
KE
K
1
1/vlk
K
2V K oe
K
1;
1 v K
1
V KI
K
1
T
1/v K
K 1
1; 2/ TK x
K 2
V (K))
VIKTE)
(k(E)²
We observe that ||01|2 = K - _?,
because a E Mr. Therefore
PK (px ((;((({;T))T) (above)
2V K| T
K
52
K (1
1;
5 K VIRie)2
1 VIRTE
+1
1/V
VI+ Vi(Es)
2V K "x
T
1;
K 1 (K(())
(1- V[K(g)2
1 /
(1 /K|
T
1/VIK
1+ VIKIE
- 1; 2 2 OF
1+VIKK IKIE
ViKE
1 VIKE
+1
VIRIE
I/VIK
1+ VKKl - 1 + VKKEE
2/ [K o0
1+ L
1- 1-VIKTE
VIKTE
VINTE
1
(2VIK(6:2/1KTT))) = (5:
2
Lemma A.16 (A converges to 2 as K + 0). For all x in PR or D. it holds that
lim A 2. -00
Proof.
2
lim X lim
2
K-50
R <
1 + K
||00|/
Theorem A.17 (exp (v) converges to + as K - 0). For all in the Poincaré ball Pa or the projected sphere Di and M, holds that
lim expx K=0
K (v) exp, (v) = x +
hence the exponential map converges its Euclidean variant.
26

---

Published as a conference paper at ICLR 2020
Proof. For the positive case K >> 0
K 110112
lim
exp (v)
lim
0K
tank (viK)
K K
K K-0
2
K
110112
1
110112
lim
Lank
K-0+
|1:112
tan VV A 11
lim
K ||011
U,
due several applications of the theorem of limits of composed functions, Lemma A.16 and the fact that
tan (aa) a.
a-0
V/
The negative case K < 0 is analogous.
Theorem A.18 (log (y) converges y K 0). For all x, y in the Poincaré ball P or the projected sphere D' it holds that
lim log K-0
K (y) log (v) y x,
hence the logarithmic map converges its Euclidean variant.
Proof. Firstly,
K-0
BK y
y x.
due to Theorem A.14 For the positive case > 0
lim log (y) = lim K-0+
2
tank [V[K) |K| [z]12]
2
K-+0+
V
||
2
tan
K
(V[K)
[ [1=112)
lim
K-+0
AK
K ||
(VK 21212
tan
2
lim
lim
lim 2
K K-+0+
K
K-00
VK |
K -
1 1-111-1 (x
vy) x
due to several applications of the theorem of limits of composed functions, product rule for limits, Lemma A.16 and fact that
tan (Vaa) a.
lim a
Va
The negative case K < () is analogous.
Lemma A.19 (gyr(z, y converges to a as K + 0). For all in the Poincaré ball Pa or the M, holds that
projected sphere Dy and
lim gyr(æ, y|æ = V. K-0
hence gyration converges identity function.
27

---

Published as a conference paper at ICLR 2020
Proof.
lim gyrla, y/v = lim
(0x (ar #x y) 0x (a Ox (y 3x v)))
K-0
K-0
+ +
-- --z-y+zr
due to Theorem A. and the theorem of limits composed functions.
-- Theorem A.20 (PT ( (v) converges to as K + 0). For all x. in the Poincaré ball Pk or the
projected sphere ID's and Mm,, it holds that
lim PTK - (v)
Proof.
A
lim PTK K-0
x (w) = lim
gyr -2|
K-50
AR
lim
lim gyr 10
K-+
A
K-0
k K-0 ,
due to the product rule for limits, Lemma A.16 and Lemma A.19
B PROBABILITY DETAILS
B.1 WRAPPED NORMAL DISTRIBUTIONS
Theorem B.1 (Probability density function of WN(z;; E) in HH).
Rsinh |
log WN E) log log:N(;;00 (n 1) log
|
where u (f (z), PT (u), and R 1/V-K.
Proof. This was shown for the case K = by Nagano et al. 2019). The difference is that we do not assume unitary radius R = 1 = 1/V-K. Hence, our tranformation function has the form = exp PTK upo and f = PTK 4ー: o log
The derivative of parallel transport PTK-y(0)) for any I, E H& and v E Ta is map dPTK
x x-y (v) : ). Using the orthonormal basis (with respect to the Lorentz product) (E.....E.. {n), we can compute the determinant by computing the change with respect to each basis vector.
i
d dPTK
(-)
PTK
z
(v + +E)
de
0
(y. v + aE)
(v + (E)
(xr v) R² (z, 4)
de
-=0
(y, (4,E)
(x 4)
R2
(x, y)
e==
PTK x- (E).
Since parallel transport preserves norms and vectors in the orthonormal basis have norm 1, the (5)ll, (PTTK,,(5
change ||d PTK
28

---

Published as a conference paper at ICLR 2020
For computing the determinant of the exponential map Jacobian, choose the orthonormal basis { = u/ :
:......Su. S.), look at the change with respect each basis vector. For the basis vector E
where we just completed the basis based on the first vector. We again
d dexp x (E1)
0
u |/ullc/
= de
cxpx
u
+
==0
[|lell
a
[1llelllc tel)
R Rish
Pl
cosh
(1leell c + c)
ae
R
||ee|1e
1leel1c +
c-0
(1|22|1c
sinh
[1
cosh
[|aall
R ||22|1 +cl
x
2ell c
u
c=0
[Itallcc
|lullc
sinh
+
cosh
R
R
R
|rullc"
where the second equality is due to
u + ||elllc ||_ -
-1((
|(11
21 1c
|+
|eel
-11/el1e - cl.
|eell
|
For every other basis vector EN where 1: d exp K (E)
exp K (u + cE)
de
Rsinh [|utcell2
u + EElle
cosli
x +
(u + EE)
Oe
I/ + ebllc
c=0
Vilu R
V 11eel + +
Rsinh
cosh
(u + +E)
Oe
===
R
|uul + +
Viuu R
ccosh
(u +EE)
|ul12 +
(R22 11lll - Reeu + E(lluul1 + )))) sinh
v |ullite
R(I|ul 2 + 22)3/2
R² ||ul12 sinh [|ull
Rsinh ||ul|
E |//l/c E R(II 122 )3/2
where the third equality holds because Iu + cEl12 = ||u1|2 + 211111 - - (u, cE)
||ul|2 + + 2c (u, E)
= |/ul +
where the last equality relies on the fact that the basis orthogonal, and u is parallel to E u/ [|ull hence it is orthogonal all the other basis vectors.
29

---

Published as a conference paper at ICLR 2020
Because the basis is orthonormal the determinant is product of the norms of the computed change for each basis vector. Therefore,
( @ ( = 1" 1.
det
dv
Additionally, the following two properties hold:
dexp_ (1a/11 ) | =
2
sinh
|
c
+
cosh
Ilullc
u
R
R
R
c
|
|
|lul|c
l
sinh?
cosh?
R
R²
R
2
l|:
||eall
||2a||c
sinh
+ cosh
1.
R
R
and
| R sinh
2
|x (5)1
12
1/e1/c
Ra sinh?
[|alle
11E112
||u1122
[[ell Ra sinh?
R
||21122
Therefore, obtain
@eepp (w)
Rsinh
dot
du
|/2/|cc
Finally,
[bullc m-1
(af(v) 8v
[dexp x (u) du
@PPTK (o-+(()) =
Rsinh
det
det
det
dv
|/eella
Theorem B.2 (Probability density function of (N μ, E) in Si))
[ell22
R
sin
log WN(2; E) = log No; 0, Σ) (n 1) log
2 l
where u log (z), v PT (u), and R 1/VK.
Proof. The theorem is very similar to Theorem B. manifold changes from H to S hence K >
The difference is that in this one, Our tranformation function has the form
= ex PTK #a-1 and f-1 = PTK n-pp o llog
The derivative of parallel transport PTK
for any æ, S and E Ta is map dPTK ( : (T, Using the orthonormal basis (with respect the Lorentz product) (E.....E.) {,), we can compute the determinant by computing the change with respect to each ba- sis vector.
d KPTK m ( = de ==0
PTK xr
( + EE)
30

---

Published as a conference paper at ICLR 2020
(y. v + e
(0 + E
(o + y)
de
R²a + (x, y)2
==0
(2/, 6)2
6
(x: + y)
R2
+
(x,, yi
PTk
x (E)
Since parallel transport preserves norms and vectors in the orthonormal basis have norm 1, the change is Ild P (4)1, [ =.
For computing the determinant of the exponential map Jacobian, we choose the orthonormal basis {EI =u/ E2; ). where we just completed the basis based on the first vector. We again look at the change with respect to each basis vector. For the basis vector &1:
d dxpp K (E1)
0
u
expx
u
+
Ōe
|
[||
tell
2 2
Ris
008
x
+
(1/ee
+c)
2
Oe
R
22 l1
[/
+
e=0
(I|24 [u l2
+ E) sin
[|[xe|[a+e]]
R
cos
||
R
x +
R2] |ell22
+el
c=
||20|12
u
||:0|12
x
cos
sin
R
|e1|/2
R
R
where the second equality is due to u +
(
1
u
+
| 12
|||2/]|
+ cl-
2
2
1
For every other basis vector &n. where 1:
d dexp
(E)
0
expe (u + eg)
Ze
c=0
[Iutrellz 1t
u + ccll2
Rsin
+
(ze EE)
De
R
1|u + eEll2
c=0
Vllull3+
|ul1
Rsin
B
CoS
x +
(u + EE)
Oe
c=0
R
||
1/2 + c²
v ||ull +c²
é C08
(u + eE)
||2|| e
Vllul/++*
(R² || & RZeu e([lue|13 +)))z) ) sin
R
R([[u] 2 e2)3/2
6=0
R2a || sin ||uu|2
[x]]a
Rsin
& =
||21|/2 E.
R(IIu |[]1u|]))3/
31

---

Published as a conference paper at ICLR 2020
where the third equality holds because
a <Ell2
|/ul12 + c² 11E112 2 (u,, <E)
||ul2 + +e² 2e (u, 4)
= |/112 + +
where the last equality relies on the fact that the basis is orthogonal, and u is parallel to E u/ || hence orthogonal to all the other basis vectors.
Because the basis is orthonormal the determinant is product of the norms of the computed change for each basis vector. Therefore,
det ( (D
1
dv
Additionally, the following two properties hold:
cxp
2
cos
||el||2
u
sin
|
æ
||2/12
2
R
| l|
R
R
|u/l2
|
2
[u lg
|
sin? 2
+
cos?
2
R
R22
R
|
||ul|2
||aa|12
sin?
+ cos?
1.
R
R
and
[ell2 Rsin
dexp K (4)1
H
|lee
R² sin²
[|ullz
||E112
||/e|/
R2 sin?
[|ulla
|/u11/2
Therefore, we obtain
[|ull|
n-1
i @exp (w)
R sin
det
Or
|
Finally,
[ulla 1
m-1
(f((v) dv
/dexph t (u)
(PPTK
no - (v)
R sin
det
det
det
Ou
do
lz
Theorem B.3 (Probability density function of (N μt, Σ) in IPR))
log W.V pg (z: E log WW Eg (P (z): PT (n), E).
Proof. Follows from Theorem B. and A.3
Also proven by (Mathieu et al. (, I)). Given
2019 in a slightly different form for a scalar scale parameter
de(p,
log.N (2: 2I)
log (2To²)
202
2
32

---

Published as a conference paper at ICLR 2020
log (A(Z; = d# (H, 2)²
12
log (2TO²)
20
2
V kdp (1e, z) sinh V K d$ (Me, 2))
(n - 1)log
Theorem B.4 (Probability density function of WW E) in T7))
log WN (z; >) log WW 82 (Px (z); PK (11), E).
Proof. Follows from Theorem B.2 and A.3 adapted from IP to D.
C RELATED WORK
Universal models of geometry Duality between spaces of constant curvature was first noticed by Lambert (1770), and later gave rise to various theorems that have the same or similar forms in all three geometries, like the law of sines (Bolyai T832)
sin A sin sin Px(c)
Pk (a)
(b
where (r( 2n sing(r) denotes the circumference of a circle of radius r in space of constant curvature K, and
Kx.
K2x
(-1)
sing (x) = 3!
5!
(2i 1)!
E=
Other unified formulas for the law of cosines, or recently, a unified Pythagorean theorem has also been proposed (Foote 2017):
K
A(c) = A(a) + A(b)
A(a)
2x
where A(r) is the area of a circle of radius r in space of constant curvature K. Unfortunately, in this formulation A(r) still depends on the sign K w.r.t. the choice of trigonometric functions in its definition.
There also exist approaches defining universal geometry of constant curvature spaces. Li et al. (2001] Chapter 4) define unified model of all three geometries using the null cone (light cone) of a Minkowski space. The term "Minkowski space" comes from special relativity and is usually denoted as RI." similar to the ambient space of what we defined HH" with the Lorentz scalar product (- The hyperboloid H!! corresponds to the positive (upper, future) null cone of RI... All the other models can be defined in this space using the appropriate stereographic projections and pulling back the metric onto the specific sub-m manifold. Unfortunately, we found the formalism not useful for our application, apart from providing very interesting theoretical connection among the models.
Concurrent VAE approaches The variational autoencoder was originally proposed in Kingma & Welling 2014) and concurrently in Rezende et al.) (2014). One of the most common improve- ments on the VAE in practice is the choice of the encoder and decoder maps, ranging from linear parametrizations of the posterior to feed-forward neural networks, convolutional neural networks, etc. For different data domains, extensions like the Graph VAE (Simonovsky & Komodakis 2018) using graph convolutional neural networks for the encoder and decoder were proposed.
The basic VAE framework was mostly improved upon by using autoregressive flows (Chen et al, 2014) small changes to the ELBO loss function (Matthey et al., 2017: Burda et al. 2016). An important work in this area is 3-VAE, which adds simple scalar multiplicative constant to the KL divergence term in the ELBO, and has shown to improve both sample quality and (if 3 > 1) disentanglement of different dimensions in the latent representation. For more information on disentanglement, see Locatello et al. (2018).
33

---

Published as a conference paper at ICLR 2020
Geometric deep learning One of the emerging trends in deep learning has been to leverage non- Euclidean geometry to learn representations, originally emerging from knowledge-bait and graph representation learning (Bronstein et al. 2017).
Recently, several approaches to learning representations in Euclidean spaces have been generalized to non-Euclidean spaces (Dhingra et al., 2018: Ganea al. 2018b Nickel & Kiela 2017). Since
then, this research direction has grown immensely and accumulated ated more approaches, mostly for hyperbolic spaces, like Ganea et al. (2018a)) Nickel & Kiela (2018); Tiffrea al. (2019); Law al.
(2119). Similarly, spherical spaces have also been leveraged for learning con-Euclidenn representa- tions (Batmanghelich et al., 2016 Wilson & Hancock 2010).
To be able to learn representations in these spaces, new Riemannian optimization methods were required as well (Wilson & Leimeister
2018 Bonnabel 2013 Bécigneul & Ganea 2019).
The generalization to products of constant curvature Riemannian manifolds is only natural and has been proposed by Gu al. 2019). They evaluated their approach by directly optimizing distance- based loss function using Riemannian optimization in products of spaces on graph reconstruction and word analogy tasks, both cases reaping the benefits of non-Euclidean geometry, especially when learning lower-dimensional representations. Further use of product spaces with constant cur- vature components to train Graph Convolutional Networks was concurrently with this work done by Bachmann et al. (2020).
Geometry in VAEs
One of the first attempts at leveraging geometry in VAEs was Arvanitidis et al. (2018). They examine how a Euclidean VAE benefits both in sample quality and latent represen- tation distribution quality when employing non-Euclidean Riemannian metric in the latent space using kernel transformations.
Hence, a potential improvement area of VAEs could be the choice of the posterior family and prior distribution. However, the Gaussian (Normal) distribution works very well in practice, as is the maximum entropy probability distribution with a known variance, and imposes constraints on higher-order moments (skewness, kurtosis, etc.) of the distribution. Recently, non-Euclidean geom- etry has been used in learning variational autoencoders as well. Generalizing Normal distributions to these spaces is in general non-trivial..
Two similar approaches, Davidson et al. 2018 and Xu & Durrett (2018), used the von Mises- Fischer distribution on the unit hypersphere to generalize VAHS to spherical spaces. The von Mises- Fischer distribution is again a maximum entropy probability distribution on the unit hypersphere, but only has a spherical covariance parameter, which makes it less general than a Gaussian distribution.
Conversely, two approaches, Mathieu al. (2019) and Nagano et al. (2019), have generalized VAEs to hyperbolic spaces both the Poincaré ball and the hyperboloid, respectively. They both adopt a non- non-maximum entropy probability distribution called the Wrapped Normal. Additionally, [Math-] ieu et al. (2019) also derive the Riemannian Normal, which is a maximum entropy distribution on the Poincaré disk, but in practice performs similar to the Wrapped Normal, especially in higher dimensions.
Our approach generalizes on the mentionet geometrical VAE work, by employing "products of spaces" approach similar to Gu et al. (2019) and unifying the different approaches into a single framework for all spaces constant curvature.
D EXTENDED FUTURE WORK
Even though we have shown that one can approximate the true posterior very well with Normal- like distributions in Riemannian manifolds of constant curvature, there remain several promising directions of explorations.
First of all, interesting extension of this work would be to try mixed-curvature VAEs graph data, e.g. link prediction on social networks, as some of our models might be well suited for sparse and structured data. Another very beneficial extension would be to investigate why the obtained results have a relatively big variance across runs and try to reduce it. However, this is problem that affects the Fuclidean VAE well, even if not flagrantly.
34

---

Published as a conference paper at ICLR 2020
Secondly, have empirically noticed that it seems to be significantly harder optimize models in spherical spaces - they seem more prone to divergence. In discussions, other researchers have also observed similar behavior, but more thorough investigation is not available at the moment. We have side-stepped some optimization problems by introducing products of spaces previously, it has been reported that both spherical and hyperbolic VAEs generally do not scale well to dimensions greater than 20 or 40. For those cases, we could successfully optimize a subdivided space (82)56 instead of one big manifold S72, However, that also does not seem be conclusive rule. Especially in higher dimensions, we have noticed that our VAEs (S?) with learnable curvature and DD² with fixed curvature seem consistently diverge. In few cases g with fixed curvature and even the product (IE2)²2 x (1H2)12 x (S²)12 with learnable curvature seemed to diverge quite often as well.
The most promising future direction seems to be the use of "Normalizing Flows" for variational inference presented by Rezende & Mohamed (2015) and Gemici et al. (2016). More recently, it was also combined with "autoregressive flows" in Huang et al. (2018). Using normalizing flows, one should be able achieve the desired level of complexity of the latent distribution in VAE, which should, similarly to our work, help to approximate the true posterior the data better. The advantage of normalizing flows is the flexibility of the modeled distributions, the expense of being more computationally expensive.
Finally, another interesting extension would be to extend the defined geometrical models to allow for training generative adversarial networks (GANs) (Goodfellow et al., 2014) in products constant curvature spaces and benefit from the better sharpness and quality of samples that GANs provide. Finally, one could synthesize the above to achieve adversarially trained autoencoders in Riemannian manifolds similarly Pan et al. (2018); Kim al. (2017) 7): Makhzani al. (2015) and aim achieve good sample quality and well-formed latent space at the same time.
E EXTENDED RESULTS
3d2-comp_000 002 d2
0.6
3h2-comp 000
3h2 comp. 001 h:2
0.4
3h2 comp 002,002,11
3p2-comp 000 p2
3p2 comp 001 p2
3pp-ccmp 002 p2
362-comp 000
362-comp 001
Curvature
362-comp
3u2-comp 000 000_020000
3u2-comp 001
0.2
3u2 3u2-comp 332-compp_00__ju 002_02000______00
d2, d2,e22.2-coomp
d2,e22.p2ccomp 00pp__01_pp_____101
00.4
d2.e2.p2 comp 002
d6-comp 000 d6
e2.h2,s2-comp 000 c2
e2,h2,s2-comp 001 h2
e2.h2.s2-comp. 002_s2
h6-comp 000 h6
00.8
p6-comp 000 p6
s6-cemp de-cemp_000___
u6-comp_000 000 u6
Epoch
Figure 1: Learned curvature across epochs (with standard deviation) with latent space dimension of
diagonal covariance parametrization, on the MNIST dataset.
35

---

Published as a conference paper at ICLR 2020
Table 6: Summary of results (mean and standard-deviation) with latent space dimension of 6, spher- ical covariance parametrization, on the BDP dataset.
Model
LL
ELBO
BCE
KL
($f)3
55.89+0.36
56.72=0.40
51.01=0.31
5.7220.09
Si
+0.35
+0.35
=0.47
51.1620.78
5.41=:0.4
(vMF $
-57.87+1.52
-58.64=1.68
53.96+22.10
4.68++0.5
MMF Si
-58.78+0.88.
-60.74=2.22
56.03+2.66
4.71
(D))3
-56.01+0.0
-56.67=0.31
51.02+0.44
5.65##0.10
D
-55.78+0.07
-56.38=0.06
50.85+10.20
5.53:10.24
(E²)3
-5.334+0.455
-56.94=0.50
51.32:+0.55
5.62:+0.19
136
- -5..28+0.85
-06.99=0.59
51.58+00.6
5.41+0.29
(H22 1)3
56.08 10.52
56.80
50.94.10.38
5.86
H6
-56.1810.32
-57.10
51.48.10.47
5.62
(P2 1)
- -55 55.98 20.62
-56,49=0.e6
50.96
5.52+0.31
P6
56. 0.555
-57.61 20.74
52.07 =10.71
5.60
(RA T2
1..
-5..90
52.42.1.71
3.48
(82)3
-6.005+0.2
-6..9900330
51.07=+0.21
5.61+0.22
(vMF 8 86
-57.56
-57.80
52.68.11.6
5.12.10.84
-56.0620.55
-56.65=0.61
50.93
5.72+0.40
MMF S6
-58. -58.21+0.92
-1..87=1.51
54.99++.77
4.88++0.39
(D2)
56.06-20.36
-6..6900554
50.95-10.40
5.74
D
56.1010.22
-5..69
50.90
5.7910..0
(H2)3
55.80110.2
-66.72
51.14
5.58
HI
56.03+0.22
-56.82=0.20
50.994+0.16
5.83
(P2)3
666.2990.05
-57.11 =0.22
51 51.41:10119
5.6940.30
p6
56.40.1.31
-57.13=1.225
51.17
5.96-10.27
(RA p2)3
56.25+0.56
-57.26=0.15
53.16+1.0r
4.11+0.64
D2 x E x P2
- 55.87 10.22
-56.35
50.67:10.57
5.69
D x E² p2
-5..06+0.41
-56.86=0.65
51.23-10.0
5.64:t0.11
m x E x (RN p2)
- 56.35+4..2
-57.06
51 80.90..7
5.17:0.12
D2 X x (RA 122)
- -5..17 17.1171-0.44
-56.75=0.56
51 .0.1..00
4.95-10.32
E x $2
-55.921..44 10.42
56.55
51.13
5.4110..0
Ea x HJ X 3
-5..04+0.57
-6..710.77
51.09#+..80
5.62+0.12
E x HP x (vMF 82)
- 55.82++043
56.3220047
51.10
40.20
E X H2 (vMF )F
55..740.51
-56.3420.65
51.33-10.57
5.01 10.17
(U2)3 U
55.56110.15
-6.005.0.32
10..8-10.23
5.37 10.10
10.38
-56.46.0.41
50.66.10.38
5.81 181110.88
36

---

Published as a conference paper at ICLR 2020
Table Summary of results (mean and standard-deviation) with latent space dimension of 6, spher- ical covariance parametrization, on the MNIST dataset.
Model
LL
ELBO
BCE
KL
06.777
101.66=0.32
87.0410.49
14.62++0.1
(vMF spys
-97.7240.22
-102.98=0.m 15
87.77=0.18
10.01
S
-9..71+0.17
-101.55=0.w
86.90+0.30
14.65+00.10
vMF S
-97.03+0.11
-102.12=0.20
$7.42+0.22
14.69+0.03
(ID))s
-97.84+0.10
-102.75=0.22
88. 08.43+0011
14.33++011
ID
-98.2110.23
-103.02=0.14
88 08.44
14.58+10.11
(E²)3
-9..04+0.14
-101.4400.18
86.77
14.67=+0.22
I66
-97.16+t0113
-101.67=0.11
87.17=t022
14.50=0.20
(H22 )3
-97.3111000
-102.20
87.81.101 1:0.23
10.19
H
-97.10110.4
-101.99 0.33
87. 87.32.10.22
14.5610.20
(IP2
-97.56+tv.04
- -102.33 0.222
87 $7.933t0.35
14.40+0.10
(R.V T22
-92.5440.18 19
-9..19-0221
88.4210.20
8.76+0.04
P6
-97.8010.05
102.60
88.1410.08
14.46+0000
(S2)3
96.444+0.12
101.30=0.17
86. 36.79.2.22
14.51+0.09
(vMF 8
-97.62.1030
102.72
87.48.10.37
15. 15.2410.03
S6
-96.72+..15
-
-0.1.39
86.69
14.70+0..13
YMF $ (TD2)3
-96.72+0.11
-101.55=0.21
86. 86.82.+0.28
14.73-10.02
-97.68.10244
102.51
88. 111111000 1:0.44
14.41
D
197.7210.15
-102.31 110.16
87.70.10.22
14.61
(1²))3
-97.3711011
-102.07
87.56:10.30
14.51:10111
1H
-97.47+0.16
-102.18=0.20
87.6440.23
14.53+0.07
(IP2)3
-97.62.10.08
-102.34-0.10
87.9210.16
14.43+0.06
(R.N T22yo
-94.16110.68
-9..65
89.27:10.79
9.38
166
-9.771+024
-102.55=0.21
88.24+0028
14.32+0.01
D E x p
99..4810.18
102.22
87.85
14.37-10.18
D2 E² x p2
97.55
1022.33=0115
87.75+10.15
14.49+0.15
m R2 x (R.V T22)
96.44310.47
-101.31
88.8 08..500.50
12.5040.03
2 X E x (R. T22 1
-9..1810.21
-100.91 10.31
88.58
12.33-10.19
E2 2 $2
96.80.10.20
-101.60.1.333
17.13
14.47
E2 X 1H X S
96.76+0.09
-101.48=0.13
86.99+0.11
14.49+0.05
E2 x H2 X (vMF 82)
96.56+4.27
-101.49=0.28
86.58+0.36
14.91+0.14
E2 TH (vMF 8?)
96.7611.39
-101.82-0.15
87.08.10.06
14.7440.13
(U2)3 U
99.11210.04
-101.68
87.1310.14
14.55410116
97.26.10.16
-102.05
87.54.0.21
14.51
37

---

Published as a conference paper at ICLR 2020
Table 8: Summary of results (mean and standard-deviation) with latent space dimension of 6, diag- onal covariance parametrization, on the MNIST dataset.
Model
LL
ELBO
BCE
KI.
-6.57=0.01
-101.34+0.
86.88+0.17
14.45=0.11 10
-0.09
-101.29+0.18
86.71
14.58+0.15
(D
-0.1
-102.5810.25
88.21
14.27:0.02
D4
-0..89=0.10
-102.65-t0.10
88.39
14.26+t0.08
(B²)3
96.994+031
101.34+0.11
86.89+0.36
14.44+0.11
E6
96.88110166
-101.3610.08
86.9010.14
14.46.10.07
(H21)3
00..1
-102.061..28
87.63
14.42:10.10
H
-0.338
102.22::.95
87.75
14.47:10.37
(P2
-97.57=012
102.22:0.18
87.83
14.3940.10
p6
97.33
102.02
87.71
14.31 10.04
(82)3
0...
101.43+0.21
86.93=0.28
14.50=0.05
86
1..24
101.1810.36
86.74
14.44
(D2)a
-0.161
-102.3710.28
87.96
14.41:0.06
D6
-0..53
-102.31+0.38
00.97
14.34+0.08
(H2)3
96.86-0.31
101.61+0.30
87.13
14.48110.08
H6
96.9010.26
-101.48.0.35
87.18
14.30
(P2)3
97.5220002
102.30
88.1110.07
14.1910.12
p6
97.26=2016
1022000+..1
87.58+0.16
14.42+0.08
D2 x E² p2
97.37=0.14
102.12
87.78+0.23
14.34+0.12
T
x x p2
97.29
101.864.0.16
87.54.0.17
14.3240.04
E
HP? $2
96.71
101.3410.16
86.91
14.43..0.06
E H2 S7
96.6610.27
-101.4610.44
87.0220.38
14.44
(U2)3
97.06=0.13
1101.6610.19
87.2210.12
14.44
96.90
-101.68+0.07
87.27
14.42+t.112
0
0
0
3
4
4 6
7
8
9
0
3
4
6
6
7
8
9
3
5
7
8
9
0
2
3
4
5
6
7
8
9
O
I
2
3
4
5
0
7
8
9
0
J
Z
3
4
5
4
7
8
9
(a) Original
(b)
(E
0
0
6
7
2
9
0
/
2
3
4
G
6
7
2
6
01
4
5
6
7
8
9
0
2
3
4
5
6
7
8
9
01
2
3
4
5
6
7
8
9
01
1
2
3
4
5
6
7
8
9
(0)
33
(d)
E
x H² S
0 2
4
5
6 6
7
8
9
0
/
2.
4 5
6
7
8
9
0
0
4
5
8
9
1
4
8
9
2
3
4
5
7
8
9
1
2 3 4
S
6
8
9
(e)
12
(B2
1
(2
(
(T
Figure 2: Qualitative comparison of reconstruction quality of randomly selected runs of selection of well-performing models on MNIST test set digits.
38

---

Published as a conference paper at ICLR 2020
Table 9: Summary of results (mean and standard-deviation) with latent space dimension of 12, diagonal covariance parametrization, on the MNIST dataset.
Model
LL
ELBO
BCE
KL
(
0..
-4.888
62.83+0.21
22.06=0.07
812
-80.72+0.44
-85.73=0.32
63.86+0.32
21.87+0.04
(ID
-80.53
-85.59=0.us
63.62:t0.12
21.97:1.16
ID:12
80.8110.12
-86.40=0.17
64. 64.42:t0.19
21.98=0.06
(1B2)6
79.51=0010
-83.91=0.12
61.84+0.06
22.07=0.13
12
79.51
83.95
61. 11.66.10
22.29
(H22 )
10.54
86.05
63.7810.26
22.27:10226
H
79.37=0.14
84.76=00.u
62.32+t.05
22.44+0.10
(IP2
80.39
-5..46-0.16
63.4840.22
187
P12
80.88
85.87
63.66.0..9
22.21
($2)6
79.95=0.14 14
84.99==022
62.83+0034
22.07+0.17
112
79.99
84.78 0.26
62.89
21.8910.18
(D2)6
-0.400
85.38
63.4940.12
21.89+0.18
12
80.37
85.26=00119
63.2440.15
22.02+0.15
(
80. $0.1310.06
85.22
63.32.10.44
21.9040.10
12
79. 79.77:10.10
84.58
62.4910.110
22.09
(
80.31
85.35=0.11
63.57
21.79:10.07
p12
80.66=0.09
-85.55=0.03
63.55+00117
22.00+0.14
(D2)² x (E2)² x (P2)2
80.30
85,22=0.du
63.52+00.6
21.70+0.11
(IDR)2 (E²)² (F2 (P2,)2
-80.14+0.1
-85.00=0.8
62.99+0.16
22.01+0.21
D x E x I
-80.1710.11
84.95
12.37
22.0810.118
Df E E x P4
-80.14:0.20
84.99
63.06:10.26
21.92.1.08
(E2)² 2 (52)2
-79.59
84.43=0.20
62.68+0.20
21.75++0.0
(B2)² 2
(8))
79.87
-8..82-0.61
62.66-1.42
22. 17+.20
x x S
79.69 0.
84.45
62.644-0.28
21.21 10021
E
H St
-79.77 1.009
84.75
62.6810.22 10.25
22.07 10.24
( 112
-79.61 10.06
84.13
61.92.10.22
22.21 10.23
-0.001
84.86=0.51
62.90+t0.6
21.96:10116
39

---

Published as a conference paper at ICLR 2020
Table 10: Summary results (mean and standard-deviationn with latent space dimension of 72, diagonal covariance parametrization, on the MNIST dataset.
Model
LL
ELBO
BCE
KL
36
78.43+0.44
84.99=0.49
56.88=0.28
28. 11:00.56
(DP) 36
- 76.033+.17
-83.04=0.25
54.35=0015
28. 28.69#0.11
(E2) 36
74.5310.06
-0..05
50.91
15.10.07
E
74..4210.06
-80.09
51.45=00.00
28.63.10.20
(H² )
-77.92+0.32
-84.76=0.55
56.85=0.60
27 27.91+t.42
H72
177.30+0.12
86.98_0.09
58.04 00.04
28.9441.225
(P2 336
76.1110.08
-2.663
53.89 10.36
28. 74..300
P?2
77.5010.05
84.53
55.80.10.20
28. 18.73.10.18
S?2
75.2410.01
81.39
53.03
28.36110.16
(DD))6
75.66+4006
-81.94=0.09
53.32
28.61#0.11
D
77.1142.21
-83.94=2.81
54.94-2.55
29.00+1.31
332
77.8710.02
-83.95=0.02
55.71
28. 24.244
H22
75.03.10.11
-81.2310.14
52.63.10.1
28.6110.11
(P2)36 36
75.7710.12
-8..07
53.65-00.8
28.43:10.39
p72
75.7140.08
-81.95=0.09
53.29=0.14
28.6740.05
(D²) x (22 x (122)12
77.40+0.35
-3.33==0.1
53.90=0.4u
29.45+t011
(DD))22 11 (IP2) 12
75.36+0.23
-81.53=042
53.02=0.39
28.51+0.45
D22 x E24 x p24
75.11
-0.099
52.48
28.5210.16
(E?) 12 x (H2,)) )12 x (87)
77.5310..4
-83.95-0.40
55.54
28.42 08.4240.08
2 x H 24
75.0440.16
-81.17=0.18
52.61
28.55440.38
(U2)30 36
74.64+u.0s
-80.52-0.10
52.04=0.10
28.48:2007
7
-75.46+0.08
-81.76-0.09
53.27=0.18
28.4940.3
40

---

Published as a conference paper at ICLR 2020
3
3
8
8
4
4
4
1
9
9
3
3
3
8
6
4
4
4
9
9
3
3
8
8
6
6
4
4
9
9
9
8
3
3
3
8
8
0
4
4
9
9
9
8
33
3
8
4
9
7
7
8
2
2
2
1
7
7
8
8
9
2
2
2
2
1
7
7
8
8
8
0
7
7
7
7
7
2
8
(a) First component of (E²)).
(b) Second component (B?)3.
b
7
7
4
b
b
7
7
9
4
5
9
9
6
4
8
3 7
5
5
6
9
9
0
4
8
3
3
7
5
5
5
8
8
2
2
2
6
7
7
7
5
5
8
8
8
2
2
2
1
7
7
3
3
3
8
2
2
2
2
7
3
3
3
3
2
2
2
2
(c) First (negative) component of (U*)².
(d) Second (negative) component of (0R)3
1
7
7
9
9
6
6
7
7
7
7
9
9
9
6
6
6
7
7
7
9
9
9
9
9
9
1
7
7
8
g
0
8
3
9
9
1
7
8
0
0
7 7
8
8
8
8
6
8
0
0
7
7
2
8
8
8
8
8
1
0
7
2
2
2
8
8
8
8
0
2 2
2
2
8
8
8
8
(e) 22 of
g2
1e ID
2 of E2 x 1
D=
Figure 3: Samples from various models of grid search around 0 of single component's latent space on MNIST test digits.
41

---

Published as a conference paper at ICLR 2020
(a) H E² X H x 82
(b) H
(c) E Rf H² 8
(e) S of E x H 8
(I) S6
Figure 4: Illustrative latent space visualization of a randomly selected run the models x S2 RO, HH and @ on MNIST with spherical covariance. E is visualized directly, 82 is visualized using a Lambert azimuthal equal-area projection (Snyder) 1987 Chapter 24), Hl is transformed to the Poincaré ball model using a stereographic conformal projection. All other latent space sizes were first projected using the respective transformation (to Poincaré ball, Lambert projection) if applicable, and then projected IR² using Principal Component Analysis (Abdi & Williams 2010 PCA) and visualized directly.
42

---

Published as a conference paper at ICLR 2020
Table 11: Summary of results (mean and standard-deviation with latent space dimension of 6, diagonal covariance parametrization, the Omniglot dataset.
Model
LL
ELBO
BCE
KL
(3
136.80#1.31
141.68
131.73:15.65
9.95:14.33
Si
136.69=0.91
-141.46+0.9
129.52:00.4
11.94+0.19
(ID2)3
136.21
- -140.44+0.17
128.93+00.1
11.51+0.01
ID
137. 1.2.4
-141.9511.94
130.7012.18
11.25.10.26
(E²))
-336.08
-10.44610.24
128.85:10.4
11.62.1.11
136.05
-140.50+..35
128.95:10.41
11.55::0.14
(THT2 )3
137.14.20.13
1
130.1810.21
11.694..10
137.09
114..22.0.19
130.37
11.85:10.12
(IP2 )
136.1610.22
140.63
129.2910.34
11.34.0..3
16
130.8610.20
140.361...99
128.92:10.28
11.44
(S²)3
140.68
188.9810
11.70+0.13
$6
136.20=0044
140.76+0.45
129. 1040.37
11.66:+0.13
(ID2)3
136.13
10.
129.1041..2
11.4910.12
136.3010.0s
140.7410.14
129.35.10.16
11.39
(1H2)3
136.17
140.65
129.26:10.07
11.39
1H16
233.2244.32
140.9220.33
129. 120.48
11.45+0.12
(122)3
136.09
-40.41
229.044..05
11.37
p6
13..05.0.44
140.-42...47
129. 229.04.1..5
11.38
2D x E x p2
13..8910.40
140.2810.42
128.75.10.40
11.53
D& x E2 x p2
136.0110.33
140.52
129.02.10.27
11.50
E H22 x 8
135.98=2.44
-4..511
128.85-10.48
11.66:10.4
E x 1 X 87
136.3410.41
-141.0210.46
129. 120.2440.47
11.7840.10
(U2)3
136.21=0.07
-140.65+0.30
129. 14+0.34
11.52+0.15
U6
1...0.4
-140.43+0.14
229.07=0.27
11.36+0.13
43

---

Published as a conference paper at ICLR 2020
Table 12: Summary results (mean and Standard-deviation with latent space dimension of 72, diagonal covariance parametrization, the Omniglot dataset.
Model
LL
ELBO
BCE
KL
($?) 36
-112.33+t..4
-118.94+o.4
91.04
27.90
[D0 36
1088.66+0.21
-116.06#0.1
85.95
30.11#0.01
[IE²2) )6
- -105.96+0.33
-112.41+0.35
79.80+0.72
32.61=0.11
E
-105.8910.16
-112.40.0.17
79.5210.19
12.89
(IH?2 ):36
-112:22:0.11
-119.0610.15
91.30110.47
27.7610.35
H
- -111.19+0.42
-120.49+0.33
91.11+0.73
29.38=00.0
(TP2 )36
109.05+0.09
-115.9910.1 10
85.81
30. 18.0.44
IP72
111.2410.28
-118.36.1.24
89.53
28.84
S72
109.39++0.3
-116.42+0.3
87.22+0.58
29.20 29.20=0.28
(ID2)36
108.8910.36
-
-115.6510.46
85.29
30.37
D
108.81
-115.71+0.00
85.681t..10
30.03#0.09
(1H2)36
112.21+0.28
-118.74+0.8
91.03++0.76
27.71=0.47
IHT22
108.62.1.400
-115.5410.30
85.1810.62
30.37
(32)
108.76 788..66
- -115.54.0.70
85.1611.35
30.38
p72
109.66:10.61
-116.50:t0.6
87.09411.49
20.42
(ID))12 x (E²)12 x (22)
107.02+1.56
-115.62+1.78
$8.5248.24
27.10
(D2) (E²) x (P2 12
108.06+0.47
-114.92+4..8
83.95+00.58
30.97
[ ( 12
12 12 12
114.85+0.38
-120.98+0.1
95. 12+0.49
20.86=00.0
(E²)) ) x
($7)
110.2810.37
- -116.90.1.42
87.7110.82
20.19
(U2)36
105.98+0.05
-112.7040.18
32.85+0.61 79.85+0.80
?
1 106.58+0.12
-113.68+0.11
81.5310.34
32.15=2.36
Table 13: Summary of results (mean and standard-devvation) with latent space dimension of 6, diagonal covariance parametrization, the CIFAR dataset. All nan standard deviation values below indicate the repeated experiment was not stable enough to produce a meaningful estimate of spread.
Model
LL
ELBO
BCE
KL
56
1893.16#men
-1901.32=oo
1885.97
15.35+mma
D26
189916992na
-1897.77
1884.12
13.64+man
g6
1896.1944 1942.54
1905.75+3.19
1889.97
2.888
15.78+00.2
H
1896.56#2.98
1882.05+22
14.51+0.34
p6
1893.27+0.61
1902.67
1887.44=0.
15.23+40.6
6
1893.85.1..96
1902.67 0.66
1887.37
15.30
86
1889.76+1..6
1897.31
1882.55=1.48
14.76+0.24
p6
1891.40+2.11
1899.68+2.74
1884.58+2.60
15.1040.1
DD2 x 2 p2
1899.9014.60
1904.63+ 1.46
1889.134 1.38
15.5040.08
E2 x $2
1895.4610.92
1897.57
1882.842..00
14.7340.24
(U2)3
1895.09++.27
1904.46=5.21
1888.89#4.71
15.57+0.51
44


Reviews:
[{"review_id": "rkeOhAPTYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper is about developing VAEs in non-Euclidean spaces. Fairly recently, ML researchers have developed non-Euclidean embeddings, initially in hyperbolic space (constant negative curvature), and then in product spaces that have varying curvatures. These ideas were developed for embeddings, and recent attempts have been made to build entire models that operate in non-Euclidean spaces. The authors develop VAEs for the product spaces case.\n\nThere's largely two aspects here: one is to be able to write down the equivalents for the operations in models (e.g., the equivalent of adding or multiplying matrices and vectors in Euclidean space have to be lifted to other spaces which no longer have a linear structure). The other are VAE-specific choices, particularly choosing a normal distribution on the manifolds. The authors consider several of these choices and then run a variety of experiments on small latent-dimension cases for VAEs. These reveal that sometimes non-Euclidean and in particular product spaces improve performance.\n\n\nStrengths, Weakness, Recommendation\nI like what the authors are trying to do here; embeddings and discriminative models on non-Euclidean spaces have been developed, offer credible benefits, and generative models are the next step. The authors push forward the machinery needed to do this, and the results seem like there's something there.\n\nOn the other hand, the entire work seems quite preliminary. It's hard to say what the takeaway is, or any suggestions for users. The paper is written in a pretty frustrating way. There's an enormous amount of stuff in a sprawling appendix (there are 43 results in the first appendix?!), and checking all of these details will take a great deal of time. \n\nOverall, I recommended weak accept, since a lot of these issues seem like they can be cleaned up.\n\nEDIT: I increased my score based on the authors' response.\n\nComments:\n- The approach taken here is quite similar to another ICLR submission this year, which basically does the same thing but applies these operations to GCNs instead of VAEs.\n\n- A better way to define curvature is just to talk about the sectional curvature, instead of the Gaussian curvature the authors mention at the beginning of section 2. Fortunately for the constant case all of these definitions will be the same.\n\n- It's not quite clear in Section 2.1 why we should care about the fact that you can't fully take K->0 there---why does this hurt anything? You can approximate flat curvature arbitrarily well even without K exactly 0.\n\n- On a similar theme, what's the point of doing the product of {E,S,D,H,P}, instead of just {E,S,H} or {E,D,P}? Seems a bit weird to consider all 5, given the equivalence between S-D and H-P.\n\n- In 2.3, the products of spaces section, the distance decomposition in the 2nd paragraph should have squares (it's an l2): d_M(x,y)^2 = \\sum_{i=1}^k d_{M_k_i^n_i)^2(x^i,y^i).\n\n- The discussion in 2.3 should be expanded and made more concrete (some of these you can write out the expressions for), and more pros and cons explained, e.g., which theoretical properties are lost for the wrapped distributions?\n\n- On page 6, I don't understand the first problem with the learnable curvature approach. Why is there no gradient w.r.t to K? Isn't the idea that you'll write this thing as a piecewise function (presumably it's continuous, since that's why the authors built those models that deform to flat), and differentiate the whole thing? Why wouldn't there be a gradient at ELBO(K)? Is it not differentiable at K=0? That doesn't follow directly from just saying the curvature is 0.\n\n-  What's the intuition for the component learning algorithm using 2 dimensions for each of the spaces?\n\n- The experiment section was written in a way where I couldn't understand why the choices being made were there. Why 6 and 12 dimensions here? More clarity here would be great. Also, are there any other models to compare against for these datasets? I'm not a VAE expert; what do other models typically obtain in the authors' regime?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lBH4iE9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper devised a framework towards modeling probability distributions in products of spaces with constant curvature and showed how to generalize the VAE to learn latent representations on such product spaces using Gaussian-like priors generalized for this case. Empirically the authors evaluate the VAEs on four different datasets (a synthetic tree dataset, binarized MNIST, Omniglot, and CIFAR-10) for various choices of product spaces (fixed curvature and learnable curvature) and choices of latent space dimensionality. \n\nEvaluation:\nOverall this seems to be a nice work, with balanced discussion of the empirical results, and is clearly written.\n--Past works have considered VAEs on single constant curvature spaces and hence it is well-motivated to consider a more flexible model that enables usage of products of such spaces.\n--Empirical evaluations seems fair as far as I can tell, but I am not familiar with benchmarks for VAEs. It was interesting to see the variability in best performing models, e.g. cases in which the mixed curvature models did well vs. the Euclidean one.\n--Paper is quite readable, though in a few parts seems to delve a bit unnecessarily into geometric formalism/definitions (e.g. I did not really follow or appreciate the relevance of gyrovector distances).\n--Main text is 10 pages long and I'm not sure the extra length is necessary.\n--I would have appreciated a more clearly delineated discussion on how the technical details of this work overlap with past papers, both those that have investigated product spaces (Gu et al 2019) and single curvature spaces in VAEs (spherical & hyperbolic)? How did the latter approaches deal with modified prior distributions and/or smoothly recovering the Euclidean K=0 limit? As a result, I'm a bit unsure as to the novelty or technical obstacles that are overcome in the proposed framework in comparison to these.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyxpFesnKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a general formulation of the notion of a VAE with a latent space composed by a curved manifold. It follows the current trend of learning representations on curved spaces by proposing a formulation of the latent distributions of the VAE in a variety of fixed-curvature spaces, and introduces an approach to learn the curvature of the space itself. Extensive mathematical derivations are provided, as well as experiments illustrating the impact of various choices of latent manifolds on the performance of the VAE.\n\nI believe this work should be accepted, as while the numerical results are not particularly impressive, it provides some clear foundational work for further exploration of the use of non-euclidean latent spaces in VAEs.\n\nThis paper provides extensive and detailed theoretical grounding for their work, ensuring that it is a well-founded extension the VAE formalism. It explores numerous alternatives and compares them, providing detailed experimental results on 4 datasets. The appendices provided a much welcome refreshing on non-euclidean geometry, as well as more details & experimental results.\n\nThe paper is already quite dense, especially with the appendices, however there are a few points that could still be detailed in my opinion:\n\nFirst of all, what were the observation models used for the reconstruction loss in the experiments? I suspect a bernouilli likelhood was used for the binarized dataset, but what about the other ones, and notably CIFAR? Was it a Gaussian observation, a discretized logistic, ...? Was its variance learned? This kind of information is in my opinion crucial for assessing a construction to the latent space of VAE model, as it can have a lot of influence on the kind of information the model will try to store in its latent space.\n\nSecondly, for the model using product of spaces, do you observe some preference of the VAE to store more information in some of the sub-component? This can be explored by comparing the values of the KL term in each of these subspaces.\n\nThird, the VAE with a factorized Gaussian euclidean latent space has a well-known tendency to sparcify its latent representations: unneeded dimensions of the latent space are ignored by the decoder and set to the prior by the encoder. This allows one to not worry too much about the size of the latent space as long as it is \"large enough\". Does this property remain in curved spaces? Especially in the case the VAE on MNIST with a 72-dimensional latent, as I suspect the 6 and 12 dimensional spaces are not \"large enough\" for this phenomenon to appear.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies generalizations of Variational Autoencoders to Non-Euclidean domains, modeled as products of constant curvature Riemannian manifolds. The framework allows to simultaneously learn the latent representations as well as the curvature of the latent domain. 

Reviewers were unanimous at highlighting the significance of this work at developing non-Euclidean tools for generative modeling. Despite the somewhat preliminary nature of the empirical evaluation, there was consensus that the paper puts forward interesting tools that might spark future research in this direction. Given those positive assessments, the AC recommends acceptance. 

Decision: Accept (Poster)