Poincaré Wasserstein Autoencoder
Authors: Ivan Ovinnikov
Venue:  (2020)
Keywords: Variational inference
hyperbolic geometry
hierarchical latent space
representation learning

Abstract: This work presents the Poincaré Wasserstein Autoencoder, a reformulation of
the recently proposed Wasserstein autoencoder framework on a non-Euclidean
manifold, the Poincaré ball model of the hyperbolic space H n . By assuming the
latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure
on the learned latent space representations. We show that for datasets with latent
hierarchies, we can recover the structure in a low-dimensional latent space. We
also demonstrate the model in the visual domain to analyze some of its properties
and show competitive results on a graph link prediction task.

Under review as a conference paper at ICLR 2020
POINCARÉ WASSERSTEIN AUTOENCODER
Anonymous authors
Paper under double-blind review
ABSTRACT
This work presents the Poincaré Wasserstein Autoencoder, a reformulation of the recently proposed Wasserstein autoencoder framework on a non- -uclldeid manifold, the Poincaré ball model of the hyperbolic space H". By assuming the latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure on the learned latent space representations. We show that for datasets with latent hierarchies, we can recover the structure in Now-dimensional latent space. We also demonstrate the model in the visual domain to analyze some of its properties and show competitive results graph link prediction task.
INTRODUCTION
Variational Autoencoders (VAE) 17 28 are established class of unsupervised machine learning models, which make use of amortized approximate inference to parametrize the otherwise intractable posterior distribution. They provide elegant, theoretically sound generative model used in various data domains.
Typically, the latent variables are assumed to follow Gaussian standard prior, a formulation which allows for a closed form evidence lower bound formula and is easy to sample from. However, this constraint on the generative process can be limiting. Real world datasets often possess a notion of structure such as object hierarchies within images or implicit graphs. This notion is often reflected in the interdependence of latent generative factors or multimodality of the latent code distribution. The standard VAE posterior parametrizes unimodal distribution which does not allow structural assumptions. Attempts at resolving this limitation have been made by either "upgrading" the posterior to be more expressive (27) or imposing structure by using various structured priors (34), (36). Furthermore, the explicit treatment of the latent space as a Riemannian manifold has been considered. For instance, the authors of (5) show that the standard VAE framework fails to model data with latent spherical structure and propose to use a hyperspherical latent space to alleviate this problem. Similarly, believe that for datasets with latent tree-like structure, using hyperbolic latent space, which imbues the latent codes with a notion of hierarchy, is beneficial.
There has recently been a number of works which explicitly make use of properties of non- non-Euulidean geometry in order to perform machine learning tasks. The use of hyperbolic spaces in particular has been shown yield improved results datasets which either present hierarchical tree-like structure such as word ontologies (24) or feature some form of partial ordering (4). However, most of these approaches have solely considered deterministic hyperbolic embeddings.
In this work, we propose the Poincaré Wasserstein Autoencoder (PWA), a Wasserstein autoencoder (33) model which parametrizes a Gaussian distribution in the Poincaré ball model of the hyperbolic space H By treating the latent space as a Riemannian manifold with constant negative curvature, we can use the norm ranking property of hyperbolic spaces to impose a notion of hierarchy on the latent space representation, which is better suited for applications where the dataset is hypothesized to possess a latent hierarchy. We demonstrate this aspect on synthetic dataset and evaluate it using distortion measure for Euclidean and hyperbolic spaces. We derive a closed form definition of Gaussian distribution in hyperbolic space H" and sampling procedures for the prior and posterior distributions, which are matched using the Maximum Mean Discrepancy (MMD) objective. We also compare the PWA to the Euclidean VAE visually on an MNIST digit generation task as well quantitatively semi-supervised link prediction task.
The rest of this paper is structured as follows: we review related work in Section 2, give an overview of the mathematical tools required to work with Riemannian manifolds well define the notion

---

Under review as a conference paper at ICLR 2020
of probability distributions on Riemannian manifolds in Section 3. Section describes the model architecture as well as the intuition behind the Wasserstein autoencoder approach. Furthermore, we derive method to obtain samples from prior and posterior distributions in order to estimate the PWA objective. We present the performed experiments and discuss the observed results in Section 5 and summary of our results in Section 6.
2 RELATED WORK
Amortized variational inference There has been a number of extensions to the original VAE framework (17). These extensions address various problematic aspects of the original model. The first type aims at improving the approximation of the posterior by selecting a richer family of distributions. Some prominent examples include the Normalizing Flow model (27) as well as its
derivates (2)), (16), 77. A second direction aims at imposing structure on the latent space by selecting structured priors such as the mixture prior (6),
(4), learned autoregressive priors (36) or imposing
informational constraints on the objective
(12),
(38 The use of discrete latent variables has been
explored number of works (3) (36). The approach conceptually most similar to ours but with a hyperspherical latent space and von-Mises variational distribution has been presented in (5).
Hyperbolic geometry The idea of graph generation in hyperbolic space and analysis of complex network properties has been studied in (19). The authors of (24) have recently used both the Poincaré model and the Lorentz model (25) of the hyperbolic space to develop word ontology embeddings which carry hierarchical information encoded by the embedding norm. The general idea of treating the latent space a Riemannian manifold has been explored in (2) A model for Bayesian inference for Riemannian manifolds relying on particle approximations has been proposed in (2I). Finally the natural gradient method is prime example for using the underlying information geometry imposed by the Fisher information metric to enhance learning performance ①..
Three concurrent works have explored an idea similar to ours. hyperbolic latent space using the traditional evidence lower bound (ELBO) formulation. They approximate the ELBO using MCMC samples as opposed to our approach, which uses a Wasserstein formulation of the problem. 23 propose to use wrapped Gaussian distribution to obtain samples on the Lorentz model of hyperbolic latent space. The samples are generated in Euclidean space using classical methods and then projected onto the manifold under concatenation of a parallel transport and the exponential map at the mean. The authors of (IO) also propose a similar approach but use an adversarial autoencoder model in their work instead.
propose train VAE with
HYPERBOLIC GEOMETRY
In this section, we briefly outline some of the concepts from differential geometry, which are necessary to formally define model.
3.1 RIEMANNIAN GEOMETRY: A SHORT OVERVIEW
Riemannian manifold is defined as a the tuple (M,g), where for every point x belonging to the manifold M, tangent space M is defined, which corresponds to a first order local linear approximation of M at point x. The Riemannian metric g is collection of inner products (-I-)x TxM x T - IR on the tangent spaces M. We denote by att) M to be smooth curves on the manifold. By computing the speed vector att) at every point of the curve, the Riemannian metric allows the computation of the curve length:
Laa,)((a) = - V (án(((t),
Given smooth curve a(a.b) - M, the distance is defined by the infimum over off): d inf (a)) The smooth curves of shortest distance between two points on manifold are called geodesics.
2

---

Under review as a conference paper at ICLR 2020
Given point E M, the exponential map (v) Tx M gives way map vector in the tangent space T at point x to the corresponding point on the manifold M. For the Poincaré ball model of the hyperbolic space, which is geodesically complete, this map is well defined the whole tangent space T.M. The logarithmic map logx (v) is the inverse mapping from the manifold to the tangent space. The parallel transport Px -, Mx TxM defines linear isometry between two tangent spaces of the manifold and allows to move tangent vectors along geodesics.
3.2 POINCARÉ BALL
of three existing types of isotropic spaces: the Euclidean spaces with zero curvature, the spherical spaces with constant positive curvature and the hyperbolic spaces which feature constant negative curvature. The Poincaré ball is one of the five isometric models of the hyperbolic space. The model is defined by the tuple (B,, 9H) where B, is the open ball of radius I. 9H is the hyperbolic metric and gø In is the Riemannian metric on the flat Euclidean manifold.
Hyperbolic spaces
2
2
B, (x E R" |||||
<1}
9н
9E
1
x
The geodesic distance on the Poincaré ball is given by
11x
11
d(x, (u) arccosh 1 + (1+2
(1)
[1 11)))
3.3 GYROVECTOR SPACES FRAMEWORK AND OPERATORS
In order to perform arithmetic operations on the Poincaré ball model, we rely on the concept of gyrovector spaces, which is generalization of Euclidean vector spaces to models of hyperbolic space based on Möbius transformations. First proposed by (35), they have been recently used to describe typical neural network operations in the Poincaré ball model of hyperbolic space (9). In order to perform the reparametrization in hyperbolic space, we use the gyrovector addition and Hadamard product defined diagonal matrix-gyrove natrix-gyrovector multiplication. Furthermore, we make use of the exponential exp,, and logarithm log, map operators in order to map points onto the manifold and perform the inverse mapping back to the tangent space. The Gaussian decoder network is symmetric to the encoder network.
MODEL
4.1 GAUSSIAN DISTRIBUTION IN H
The Gaussian distribution a common choice of prior for VAE style models. Similarly to the VAE, we can select generalization of the Gaussian distribution in the hyperbolic space as prior for our model. In particular, we choose the maximum entropy generalization the Gaussian distribution (26) on the Poincaré ball model. The Gaussian probability density function in hyperbolic space is defined via the Fréchet mean and dispersion parameter a > 0, analogously to the density in the Euclidean space.
(x
(x
20d
Z(o)
The main difference compared Euclidean space is the use of the geodesic distance d(x, 11) in the exponent and a different dispersion dependent normalization constant Z(a) which accounts for the underlying geometry. In order to compute the normalization constant, we use hyperbolic
this can be generalized radius / curvature e. Throughout this paper, we assume Poincaré ball radius he and omit it from the notation.
3

---

Under review as a conference paper at ICLR 2020
polar coordinates where d(x,, (4) is the geodesic distance between the and Ja. This allows the decomposition of (or) into radial and angular components.
(o) = Zz())Z(() = Vol(S" x
e 20 "erssinh (inh"-((r)dr
We derive the closed form of the normalization constant in appendix A. For two-dimensional space, the normalization constant is given by (30):
π
2
( = 2 oe erf
Dispersion representation The closed form of the hyperbolic Gaussian distribution (2) is only de- fined for scalar dispersion value. This can be limitation on the expressivity the learned represen- tations. However, the variational family which is implicitly given by the hyperbolic reparametrization allows for vectorial or even full covariance matrix representations, which can more expressive. Since the maximum mean discrepancy can be estimated via samples, we do not require a closed form definition of the posterior density as is the case with training using the evidence lower bound. This allows the model to learn richer latent space representations.
4.2 MODEL ARCHITECTURE
Our model mimics the general architecture of a variational autoencoder. The encoder parametrizes the posterior variational distribution go (z|x) ) and the decoder parametrizes the unit variance Gaussian likelihood pa px(2)) (2) In order to accomodate the change in the underlying geometry of the latent space, we introduce the maps into hyperbolic space and back to the tangent space. Both the encoder and decoder network consist of three eolly-conneettd layers with ReLU activations. We the recently proposed hyperbolic feedforward layer (9) for the encoding of the variational family parameters (k ). For the decoder fo fx(x),) use the logarithm map the origin logo (z) map the posterior sample z back into the tangent space.
Mean and variance parametrization In order to obtain posterior samples hyperbolic space, the parametrization of the mean uses hyperbolic feedforward layer (W, bHH) the last layer of the encoder network (proposed ()). The weight matrix parameters are Euclidean and are subject to standard Euclidean optimization procedures (we use Adam (15) while the bias parameters hyperbolic, requiring the use Riemannian stochastic gradient descent (RSGD) ()). The outputs of the underlying Euclidean network h are projected using the exponential map at the origin and transformed using the hyperbolic feedforward layer map where sh, is the hyperbolic nonlinearity? ?
fa (b) = (W (h) + bn) (P(()) = (xpooyslogg x))
4.3 HYPERBOLIC REPARAMETRIZATIUE TRICK
The reparametrization trick is a common method to make the sampling operation differentiable by using differentiable function g(e, ()) to obtain a reparametrization gradient for backpropagation through the stochastic layer of the network. For the location-scale family of distributions, the reparametrization function g(c, () can be written as z = fu + O in the Euclidean space where ~ (O, I). We adapt the reparametrization trick for the Gaussian distribution in the hyperbolic space by using the framework of gyrovector operators. We obtain the posterior samples for the parametrized mean u (x) and dispersion o (xx using the following relation:
z M (x) @ (i)g(o(x)))*
We can motivate the reparametrization (3) with the help of Fig. 1, which depicts the reparametrization graphical fashion. first step, we sample e from the hyperbolic standard prior NN 1) using a rejection sampling procedure we describe in Algorithm 1. The samples are projected to the
see Appendix C. for the operator definitions
4

---

Under review as a conference paper at ICLR 2020
tangent space using the logarithm map logo the origin, where they scaled using the dispersion parameter. The scaled samples are then projected back to the manifold using the exponential map expo and translated using na.
logo (c)
expo (c)
@ %
Figure 1: Hyperbolic Gaussian reparametrization
4.3.1 SAMPLING FROM PRIOR IN HYPERBOLIC SPACE
We choose the hyperbolic standard prior NH (0,1) as prior p(z). In order to generate samples from the standard prior, we use an approach based on the volume ratio of spheres in H" to obtain the quasi-uniform samples on the Poincaré disk (19) and subsequently use rejection sampling procedure to obtain radius samples. We use the quasi-uniforn distribution
g(r) ac (r- R)
as proposal distribution for the radius. Using the decomposition into radial and angular components, sample direction from the unit sphere uniformly and simply scale using the sampled radius to obtain the samples from the prior. An alternative choice of prior is the wrapped Gaussian distribution. (O, 1) in the tangent space and projecting them onto
The samples are obtained by sampling the latent space manifold. Empirically, we have found the prior obtained via the rejection sampling procedure and the exponential map prior to perform similarly in the context of the PWA objective. comparison of samples from both distributions is presented in Appendix D)
4.4 OPTIMIZATION
Evidence Lower Bound The variational autoencoder relics on the evidence lower bound (ELBO) reformulation in order to perform tractable optimization of the Kullback-Leible divergence (KLD) between the true and approximate posteriors. In the Fuclidean VAE formulation, the KLDD integral has closed-form expression, which simplifies the optimization procedure considerably.
The definition of the evidence lower bound can be extended to non-Euclidean spaces by using the following formulation with the volume element of the manifold dvol induced by the Riemannian metric gr.
log p(x: log / p(arl p(==)))(=)dvoll, log / .
p(xlz)p(=) (0(2)x) ao g=(=(x))(=)ddoolg.
log p(al))p(z) (0(=|r) go (2 (2(r)dvl Exras [logp( log log (=|=)]
By substituting the hyperbolic Gaussian (2) into (4) we obtain the following expressions for Exo(zr) log (2(xz):
dx p)) o2
E g (lola)/logge [2(x]] E Eg [log Z(o)
const Eg (log/ vx2 1)]
Due to the nonlinearity of the geodesic distance in the exponent, we cannot derive a closed form solution of the expectation expression (x)( [log (z)). One possibility is
Taylor expansion of the first two moments the expectation of the squared logarithm. This is however problematic from numerical standpoint due the small convergence radius of the Taylor expansion. The
5

---

Under review as a conference paper at ICLR 2020
ELBO can be approximated using Monte-Care samples, as is done in (22). We have considered this approach to be suboptimal due to large variance associated with one-sample MC approximations of the integral.
Wasserstein metric In order to circumvent the high variance associated with the MC approximation we propose use a Wasserstein Autoencoder (WAE) formulation of the variational inference problem. The authors of the WAE framework propose to solve the optimal transport problem for matching distributions in the latent space instead of the more difficult problem of matching the data distribution p(x) to the distribution generated by the model Py (z) as is done in the generative adversarial network (GAN) literature. Kantorovich's 's formulation of the optimal transport problem given by:
inf We(Pu.Po) = Tep(xxxx.y-pp
E ((xx))r(((xxx) y)]
(4)
where c(x. y) is the cost function, p(x ~ Px.y- Py) is the set of joint distributions of the variables Px and y ~ Py. Solving this problem requires a search over all possible couplings I' of the two distributions which is very difficult from an optimization perspective. The issue is circumvented in a WAF model follows. The generative model of variational autoencoder is defined by steps. First we sample a latent variable z from the latent space distribution p(z). In second step, we map it to the output space using a deterministic parametric decoder fe (x).). The resulting density given by:
p(x) / fo(xlz)p(2)
Under this model, the optimal transport cost (5) takes the following simpler form due to the fact that the transportation plan factors through the map fa.
inf We(Px:Py) DEp(x=pxxd) pupp)
E Fxxx))-r(c(xx))]-
inf Egg [c(x, fo((2))
g=a(x)==(x))
The optimization procedure is over encoders (x( instead of the couplings between px and Py-
The WAE objective is derived from the optimal tramsport cost (5) by relaxing the constraint the posterior q. The constraint is relaxed by using a Lagrangian multiplier and an appropriate divergence measure.
L inf Ep(x) Bycars (logg(x)))))) + 3D (x(
(5)
The Maximum Mean Discrepancy (MMD) metric with an appropriate positive definite RKHS kernel an example of such a divergence measure. MMD is known to perform well when matching high-dimensional standard normal distributions (T). MMD is metric on the space of probability distributions under the condition that the selected RKHS kernel is characteristic. Geodesic kernels generally not positive definite, however it has been shown that the Laplacian kernel 6(x,y' y) = exp(-X(dz) (x,y)) is positive definite if the metric of the underlying space is conditionally negative definite (8) In particular, this holds for hyperbolic spaces (1).. In practice, there is a high probability that geodesic RBF kernel is also positive definite depending on the dataset topology (8). We choose the Laplacian kernel as it also features heavier tails than the Gaussian RBF kernel, which has positive effect on outlier gradients (3).. The MMD loss function is defined over two probability measures and y in an RKHS unit ball as follows:
DMMD (p. g0) = -11/o k(z, )dp(z) - -000--1
(6)
There exists an unbiased estimator for DMMD (p, 90). finite sample estimate can be computed based on minibatch samples from the prior z ~ (zz) via the rejection sampling procedure described
RKHS Reproducing Kerne Hilbert Space

---

Under review as a conference paper at ICLR 2020
Table 1: Average distortion measure
Model
Dataset
Metric
T-SNE
N-VAE
PWA
Synthetic trees
Avg
0.73
0.82
0.49
Appendix and the approximate posterior samples 2 (6(z) obtained via the hyperbolic reparametrization:
D DANN (p(z).
A n(n 1)
Σ (
2();
A n(n 1)
Σ (
2A M(Zz 2g) n
k(z,, Rz)
Parameter updates The hyperbolic geometry of the latent space requires us to perform Riemannian stochastic gradient descent (RSGD) updates for subset of the model parameters, specifically the bias parameters of μ.. We perform full exponential map updates using gyrovector arithmetic for the gradients with respect the hyperbolic parameters similar to (9) instead of using retraction approximation as in (24). In order to avoid numerical problems at the origin and far away from the respectively. The
origin of the Poincaré ball, we perturb the operands the norm is close Euclidean parameters are updated in parallel using the Adam optimization procedure (15).
5
EXPERIMENTS
5.1 DISTORTION OF TREE-STRUCTUREN DATA
To determine the capability of the model to retrieve an underlying hierarchy, we have setup two experiments in which we measure the average distortion of the respective latent space embeddings. We measure the distortion between the input and latent spaces using the following distortion metric, where subscript U denotes the distances in the input space and V the distances in the latent space.
daog (dv(((()), ((b) du b) d b)
Noisy trees The first dataset is set of synthetically generated noisy binary trees. The vertices of the main tree are generated from a normal distribution where the mean the child nodes corresponds to the parent sample x, Nx(xtt), o.) and p(i) denotes the index of the parent node. In addition to the main tree, we add K noise samples x; = [x oj) for every vertex. The dataset D {[x;, [[x;,x)]b 8iJhi, a concatenation of x; and Xy. To encourage a good embedding a hyperbolic space, we enforce the norms of the tree vertices to grow monotonously with the depth of the tree by rejecting samples whose norms are smaller than the norm of the parent vertices. We have trained our model on 100 generated trees for 100 epochs. The tree vertex variance was set to o 1 and the noise variance to o, = 0.1. 0.1. We have also normalized the generated vertices to zero mean and unit variance. Table compares the distortion values of the test set latent space embeddings obtained by using the Euclidean VAE model compared to the PWA model. We can see that the PWA model shows less distortion when embedding trees into the latent space of dimension d 2. which confirms our hypothesis that hyperbolic latent space is better suited to data with latent hierarchies. As reference, we provide the distortion scores obtained by the classical T-SNE (37) dimensionality reduction technique.
5.2 MNIST
In this experiment, we apply model to the task of generating MNIST digits in order to get an intuition for the properties of the latent hyperbolic geometry. In particular, we are interested in the visual distibution of the latent codes in the Poincaré disk latent space. While the MNIST latent space not inherently hierarchically structured there is no obvious norm ranking that can be imposed we compare model to the Fuclidean VAE approach. We train the models dynamically
7

---

binarized MNIST digits and evaluate the generated samples qualitatively well quantitatively via the reconstruction error scores. We can observe in Appendix B that the samples present a deteriorating quality the dimensionality increases despite the lower reconstruction emror. This can be explained by the issue of dimension mismatch between the selected latent space dimensionality d: and the intrinsic latent space dimensionality dy documented in (29) and can be alleviated by additional p-norm penalty on the variance. We have not observed significant improvement by applying the L2-penalty for higher dimensions. We have also performed an experiment using two-dimensional latent space. We can observe that the structure imposed by the Poincaré disk pushes the samples towards the outside the disk. This observation can be explained by the fact that hyperbolic spaces grow exponentially. In order to generate quality samples using the prior, some overlap is required with the approximate posterior the latent space. The issue is somewhat alleviated in higher dimensions Under review as a conference paper at ICLR 2020
as the distribution shifts towards the ball surface.
7 9 / 5 |
9 a 2
166 1 8
2 5 4 1500301 6 07
6530468044
7 1 7 2622280 6
0674240300
42246744 8 42
5634600014
1 66
5654760026
7175577- 5
998045650a
5655831 9
0506463943
82 2 3 8
6628040066
6 6613
9647606607
/ 144760 8
Q0609
Figure 2: A comparison of the Euclidean VAE (left) and PWA samples (right), |
5.3 LINK PREDICTION ON CITATION NETWORKS
In this experiment, we aim at exploring the advantages of using a hyperbolic latent space on the task of predicting links a graph. We train our model on three different citation network datasets: Cora, Citeseer and Pubmed (32). We use the Variational Graph Auto-Encoder (VGAE) framework (18) and train the model in an unsupervised fashion using subset of the links. The performance is measured in terms of average precision (AP) and under curve (AUC) test set of links that were masked during training. Table shows comparison to the baseline with Euclidean latent space (N-VGAE), showing improvements the Cora and Citeseer datasets. We also compare our results to the results obtained using a hyperspherical autoencoder (S-V (S-VGAE) (5). It should be noted that we have used a smaller dimensionality for the hyperbolic latent space (16 vs 64 and 32 for the Euclidean and hyperspherical cases respectively). which could be attributed to the fact that a dataset with a hierarchical latent manifold requires latent space embeddings of smaller dimensionality to efficiently encode the information (analogously to the results of (24). We can observe that the PWA outperforms the Euclidean VAE on two of the three datasets. The hyperspherical graph autoencoder (S-VGAE) outperforms our model. One hypothesis which explains this is the fact that the structure of the citation networks has a tendency towards a positive curvature rather than a negative one. is worth noting that it is not entirely transparent whether the use of Graph Convolutional Networks (18), which present a very simple local approximation of the convolution operator on graphs, allows to preserve the curvature of the input data.
8

---

Under review as a conference paper at ICLR 2020
Table 2: Performance link prediction datasets
Model
Dataset
Metric
N-VGAE
S-VGAE
PWA
Cora
AUC
92.7
94.1
93.9 2.2
AP
93.21.4
94.1
93.2
Citeseer
AUC
90.3+.5
94.7+.2
92.2=.2
AP
91.5+.5
95.2
91.8=.2
Pubmed
AUC
97.1
96.01.1
95.9 1.2
AP
97.1
96.0+.2
96.3_.2
CONCLUSION
We have presented algorithm perform amortized variational inference the Poincaré ball model of the hyperbolic space. The underlying geometry of the hyperbolic space allows for an improved performance on tasks which exhibit a partially hierarchical structure. We have discovered certain issues related to the use of the MMD metric in hyperbolic space. Future work will aim to circumvent these issues as well as extend the current results. In particular, we hope demonstrate the capabilities of our model on more tasks hypothesized to have latent hyperbolic manifold and explore this technique for mixed curvature settings,
REFERENCES
[1] S.-I. Amari. Natural gradient works efficiently in learning. Neural computation, 10(2)251-276. 1998.
[2] G. Arvanitidis, L. K. Hansen, and Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11333 2017.
[3] S. Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, (889):2217-2223) 2013.
[4] B. P. Chamberlain, J. Clough, and M. P. Deisenroth. Neural embeddings of graphs in hyperbolic space. arXiv preprint arXiv: 10359, 2017.
[5] T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf. and J. M. Tomczak. Hyperspherical variational auto-encoders. arXiv preprint arXiv: 1804.008000 2018
[6] N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv: 1.02688, 2016.
I.. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 2016.
[8] A. Feragen, F. Lauze, and S. Hauberg. Geodesic exponential kernel:: When curvature and lincarity conflict. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3032-3042, 2015.
[9] O.-E. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic neural networks. arXiv preprint arXiv: arXiv:1805.00 1805.09/ 12, 2018.
[10] D. Grattarola, D. Zambon, C. Alippi, and Livi. Learning graph embeddings constant- curvature manifolds for change detection in graph streams. arXiv preprint 1rXiv:1805.06225 2018.
[11] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. kernel two-sample test. Journal of Machine Learning Research, 13(Mar);723-773, 2012.

---

Under review as a conference paper at ICLR 2020
[12] I. Higgins, L. Matthey, X. Glorot, A. Pal, B. Uria, C. Blundell. S. Mohamed, and A. Lerchner. Early visual concept learning with unsupervised deep learning, arXiv preprint XXXv:16000005779 2016.
[13] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv:1611.0114, 2016.
[14] Józiak. Conditionally strictly negative definite kernels. Linear and Multilineur Algebra, 63(12):2406-241 2015.
[15] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
[16] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Im- proving variational inference with inverse autoregressive fnow.,nips), 2016. URL http://arxiv. org/abs/1606.cm 14444444/60000.44444. 04934.
[17] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint preprin arXiv: 13Iivv1312.6111 1312.6114, 2013.
[18] T. N. Kipf and M. Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 1609.02207 2016.
[19] D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, and M. Boguná. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.
[20] H. Larochelle and Murray. The neural autoregressive distribution estimator. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 29-37, 2011.
[21] C. Liu and J. Zhu. Riemannian stein variational gradient descent for bayesian inference. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[22] E. Mathieu, C. L. Lan, C. J. Maddison, R. Tomioka, and Y. W. Teh. Hierarchical representations with poincar variational auto-encoders. arXiv preprint arXiv:1901.060:: 2019.
[23] Y. Nagano, S. Yamaguchi, Y. Fujita, and M. Koyama. A differentiable gaussian-like distribution on hyperbolic space for gradient-based learning. arXiv preprint arXiv:1902.029900 2019.
[24] M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations, In Advances neural information processing systems, pages 6338-6347, 2017.
[25] M. Nickel and D. Kicla. Learning continuous hicrarchics in the lorentz model of hyperbolic geometry. arXiv preprint /0X.vv/806.03317, 2010 2018.
[26] X. Pennec. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 25(1):127, 2006.
[27] D. Rezende and S. Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv: /.00..5557777 2015.
[28] D. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 4082,, 2014.
[29] K. Rubenstein, B. Schoelkopf, and I. Tolstikhin. On the latent space of wasserstein auto- encoders. arXiv preprint arXiv:1802.0376. 2018.
[30] S. Said, L. Bombrun, and Y. Berthoumieu. New riemannian priors the univariate normal model. Entropy, 16(7):40154433) 2014.
[31] S. Said, L. Bombrun, Y. Berthoumieu, and J. H. Manton. Riemannian gaussian distributions the space of symmetric positive definite matrices. IEEE Transactions on Information Theory, 63(4):2153-2170 2017.
10

---

Under review as a conference paper at ICLR 2020
[32] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliass--Rad. Collective classifica- tion network data. Al magazine, 29(3):93, 2008.
[33] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: 2017.
[34] M.. Tomezak and M. Welling. Vae with vampprior. arXiv preprint arXiv:1705.07/220 2017.
[35] A. A. Ungar. gyrovector space approach hyperbolic geometry. Synthesis Lectures on Mathematics and Statistics, 1(1):1-194, 2008.
[36] A. van den Oord, O. Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306-6315, 2017.
[37] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579 2609,,26655500 2008.
[38] S. Zhao, Song, and Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv: 17Xiv:1006.000 706.02262, 2017.

---

Under review as a conference paper at ICLR 2020
A PRIOR REJECTION SAMPLING
Algorithm 1: prior rejection sampling
Input:
maximum support radius Tmon, dimensionality d, quasi-uniform a parameter, hyperbolic
prior likelihood M( (r(0, 1)
Result: n samples from prior p(z)
while i < n do
sample ~~ (0, Ia);
compute direction on the unit sphere ep
|+l
sample u ~ (0, 1):
get uniform radius samples r; [0, Tmarl via ratio of hyperspheres:
= (u rd )max) )a;
evaluate p(x,) = fr();:
M = max(p:);
g(r) Truz);
sample u ~ 4(0.11 1);
if u Mag(r,) then accept sample x;; else end end
reject sample;
Output: prior samples rsp
B NORMALIZATION CONSTANT DERIVATION
We can derive the normalization constant Z(σ) for the Gaussian distribution in H for curvature
c -1 by using the hyperbolic polar coordinates. The integral form is given by (3I):
000 = Z.())aaa)) Vol(s"-1)> x /
c 20 sinh"-1(r)dd
The normalization constant can be factorized into the radius and the angle part. The volume of a unit hypersphere is given by:
(o Vol(Sn-1)
(" +1) 1)
We derive closed form for the normalization constant follows:
12

---

Under review as a conference paper at ICLR 2020
(o)
e.
sinh" (r))d
c" - -
26
dr
2n
1) ) (-1)
(clu-t-birel-mh (-rb)
2
dr
2
h=0
n-1
/
000
(-1)
20 cin 1-n--1ki1-- kir el dr
E
2
k
k=U
(-1)+ /0 ( dr
2n-1
k
k=0
We use the following identity for the solution of the definite integral:
π 2
(ax² +ha) do
2
-
erfe
2
Setting a 202: 2k + n.
O. we obtain
(2k: 1 n)o
Zz(a) 2n-1 h
1)
2 oe
erfe
v/2
where erfe is the complementary error function.
C LIST OF GYROVECTOR OPERATIONS
In this list of gyrovector operations and throughout this paper, assume the Poincaré ball radius to be c = and omit from the notation.
Gyrovector addition:
x x (1 2(x,)) + ||y 1 )x 1xx
1 x y) + |x |2
Matrix-gyrovector product:
M tanh
[1|MXl |||||
arctanh((| (|x(1)
Mx
||xx||
Exponential map:
tanh >x
>x
expx(v) = x x0
|
Logarithm map:
2 arctann([]] x
-
logx (v)
Ax
-
Parallel transport:
Axo
Px. Pxn-xx()) = looog (v (ve ① expxo (v))
Xxx
13

---

Under review as a conference paper at ICLR 2020
D HYPERBOLIC GAUSSIAN SAMPLES
This section presents a comparison of samples obtained from the hyperbolic Gaussian and the wrapped Gaussian distributions.
The means and variances are given as follows. (14 σi) = ((0.0,0.0),1..,,11) (1y 02) = (00.6, 0.0), (1.0,100), (13,003 ((0.6, 0.4), (1.0, 1.0)), (14,. (.0,6,04), (0.7, 0.3)). (My, 05) = (00.6, 0.4), (0.1,0.4)
Figure 3: Hyperbolic Gaussian samples
Figure 4: Wrapped Gaussian samples
14

---

Under review as a conference paper at ICLR 2020
E
MNIST VISUAL SAMPLES
7 6 f
6
2 254
06 / a H 6
1 9
8
8 2 7
25 42
0
07
66775118
9
G
f
7 4
2
2 8 6674400
S 8 8 0
4224 6
4
0 D 1100162557
1900409550
19001099
425 11117/00 66
384045211
4 7/750 5
5
6 1
4
0139960150
1 6 1 8 3
700
4
3 2 6 2
161
7550
9 803377
6
5 6
9 7 7 5
1
1
8
7
/
1
4
584
584
6
9
1
1
9
8
2
25
03
8 32
Figure 5: Euclidean VAE samples d (109.01, 94.58, 98.36)
{5, 10, 10,00,0;) 20). training reconstruction enror L
/ 6 164 6 0 1660283878
3 349 23 2992662206
9
37
6530468448
247 7 0
8
4 6 8 5
0674
57'70 (
4
8
5584600014
5BD715GE
34
5E54760026
Σ EIG
1 4
9990456500
1
0 0506462943
195722
6 662804046e
6 1.25533200
3
444444
8 964m60r60r
153575062
7
23
12344
Ao6o9
1073J2615
5
2
4
Figure 6: Poincaré WAE samples d 6
{5, 10, 20), training reconstruction L
(95.01, 69.70, 58.58)
15

---

Under review as a conference paper at ICLR 2020
01332222222
9
Figure 7: MNIST samples from two-dimensional hyperbolic latent space
16


Reviews:
[{"review_id": "r1x-hoU6KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work extend the variational autoencoder model to the hyperbolic space for exploring the hierarchical data structures. Experimental results on the synthetic and real data sets show the performance of the proposed model. In general, the paper is well organized.  The proposed algorithm is promising. My main concern is the motivation of the paper and the experiments. In particular, \n \n(1) It is unclear what is the major contribution of this paper over the existing work, e.g., [10,22,23]. The authors may want to provide more discussion regarding this in the literature review. \n(2) It is unclear where is the impact of the proposed PWA model in real applications. Also, it is unclear why the data like MNIST, CORA, CITESEER, PUBMED exhibit hierarchical data structure. \n(3) The experiments are questionable. In table 2, why  CORA, CITESEER, PUBMED are adopted as the evaluation datasets? why only VGAE is considered as the baseline methods in link predict? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx-F2-otr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper summary*\n\nThe authors endow a variational auto-encoder with a hyperbolic space for the latent space. This enables them to model hierarchically ranked relationships in the data in a natural way. They use a Wasserstein formulation instead of the standard ELBO, which they claim reduces the variance of the gradients during training.\n\n*Paper decision*\n\nThanks for an interesting paper, I enjoyed reading it. While there are parts of this paper I like very much and I think it is technically sound, I feel the experiments section is lacking somewhat (see comments below). I therefore am recommending a weak reject; although, upon a favourable rebuttal would happily upgrade this position.\n\n*Supporting arguments*\n\n- In the introduction is well written and the motivation for the introduction of a hyperbolic later space is well laid out.\n- The section on Hyperbolic geometry is very well written indeed and presents this non-trivial material in a very simple to understand manner. Maybe what it lacks a bit is motivation for a machine learning audience, for why these mathematical tools from differential geometry would be useful or needed by the community to tackle presenting machine learning problems.\n- The structure of the method is well laid out and every hyperbolic version of an auto-encoder operation is explained. I would prefer to see methods from cited works, which are deployed in this paper, written in the paper instead of just referred to, so that I don’t have to search through the cited works to find out how a key operation works.\n- I think the method itself is technically sound. \n- I am not sure about the novelty of the method, seeing that there is a very similar paper from earlier this year (Mathieu et al. 2019).\n- The experiments are lacking a little. I do not feel that they show a clear reason why a hyperbolic latent space should be used and compared to standard methods, they do not perform any. better. It would have been nice to have a more structured discussion on the merits of the method.\n\n*Questions/notes for the authors*\n\n- Section 3.2, please explain the significance of \\lambda_x^c\n- Please explain in the related work and/or introduce in more detail than you already have the differences and similarities between this work and your nearest neighbour “Hierarchical representations with poincaré variational auto-encoders” by Mathieu et al. (2019). I would like to know the advantages/disadvantages of using a Wasserstein formulation instead of a sampled ELBO.\n- In section 3.3, please point readers forward to the Appendix for a list of gyrovector operations.\n- Please use equation numbers on all equations. \n- In your section on “Dispersion representation” you state: “Since the maximum mean discrepancy can be estimated via samples, we do not require a closed form definition of the posterior density as is the case with training using the evidence lower bound. This allows the model to learn richer latent space representations.” Why exactly does not using a closed form posterior density lead to a richer latent space representation? Please back up this statement.\n- In the presentation of the ELBO, there is a typo after the second equality where p(z) is listed twice.\n- This may just be personal preference on my behalf, but I think a short treatment on VAEs and optimal transport would be useful in the related work and background sections.\n- Experimentally it would have been nice to have a direct comparison between this method and Mathieu et al. (2019), that paper being very similar to this one. At least a shared experiment would have been useful.\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1eOuBThKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a Poincare Wasserstein autoencoder for representing and generating data with latent hierarchical structures. \nThe proposed model extends the Wasserstein autoencoder to the hyperbolic space. It is a new and powerful member of the family of VAEs. \nA hyperbolic Gaussian reparametrization method is designed and a Wasserstein loss with MMD regularizer is applied as an objective function.\nThe paper is well-organized and easy to read. The notations are clear. \n\nOverall, I think this work is interesting. My main concern is about the experiments. \nThe results on MNIST is not sufficient to demonstrate the usefulness of the proposed method. \nAs the authors mentioned, a potential reason is that the MNIST is not a typical hierarchical dataset. \nI suggest adding experiments on real-world hierarchical datasets, e.g., representing/generating sentences (which can be viewed as trees of words) or articles (which can be viewed as trees of sections/subsections).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received 3, 3, 6. All reviewers agree that the method is technically interesting. The main concern shared by the reviewers are the experiments which are somewhat underwhelming. The AC believes that this is a solid technical paper that needs a little bit more work. The authors are encouraged to strengthen their evaluation and resubmit to a future conference.
Decision: Reject