A Rotated Hyperbolic Wrapped Normal Distribution for Hierarchical Representation Learning
Authors: Seunghyuk Cho
Juyong Lee
Jaesik Park
Dongwoo Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Hyperbolic space
Hierarchical representation learning
Distribution in Riemannian manifold

Abstract: We present a rotated hyperbolic wrapped normal distribution (RoWN), a simple yet effective alteration of a hyperbolic wrapped normal distribution (HWN). The HWN expands the domain of probabilistic modeling from Euclidean to hyperbolic space, where a tree can be embedded with arbitrary low distortion in theory. In this work, we analyze the geometric properties of the diagonal HWN, a standard choice of distribution in probabilistic modeling. The analysis shows that the distribution is inappropriate to represent the data points at the same hierarchy level through their angular distance with the same norm in the Poincar\'e disk model. We then empirically verify the presence of limitations of HWN, and show how RoWN, the proposed distribution, can alleviate the limitations on various hierarchical datasets, including noisy synthetic binary tree, WordNet, and Atari 2600 Breakout. The code is available at https://github.com/ml-postech/RoWN.

A Rotated Hyperbolic Wrapped Normal Distribution for Hierarchical Representation Learning
Seunghyuk Cho
Juyong Lee'
Jaesik Park
Dongwoo Kim
CSED POSTECH'
GSAI POSTECH
Abstract
We present rotated hyperbolic wrapped normal distribution (RoWN), simple yet effective alteration of a hyperbolic wrapped normal distribution (HWN). The HWN expands the domain of probabilistic modeling from Euclidean to hyperbolic space, where a tree can be embedded with arbitrary low distortion in theory. In this work, we analyze the geometric properties of the diagonal HWN, standard choice of distribution in probabilistic modeling. The analysis shows that the distribution is inappropriate to represent the data points the same hierarchy level through their angular distance with the same norm in the Poincaré disk model. We then empirically verify the presence of limitations of HWN, and show how RoWN, the proposed distribution, can alleviate the limitations on various hierarchical datasets, including noisy synthetic binary tree, WordNet, and Atari 2600 Breakout. The code is available at wttphttt: h/gi::://ttttiee tiighh...... com ml pospootcch/Rooi
Introduction
Hyperbolic space has served as an effective medium to learn parsimonious representations of hier- archical data, including vocabulary with relationships (Nickel and Kiela 2017 2018 Tifrea et al. 2019), knowledge graphs (Chami et al. 2020, Sun et al. 2020), and social networks (Zhao et al.] 2011 Shavitt and Tankel, 2008). Recent studies reveal that the underlying anatomy in much complex data is non Euclidaai,, supporting the success of representation learning in hyperbolic space (Bronstein jet al. 2017). However, due to the absence of well-defined distribution that is easy to sample and has an analytic density function in hyperbolic space, earlier studies have focused on the non-probabilistic learning framework.
Hyperbolic wrapped normal distribution (HWN) has been recently proposed as an alternative to the celebrated normal distribution in Euclidean space. With an analytic density function and easiness of sampling. the HWN expands the domain of probabilistic modeling from Euclidean to hyperbolic space. The HWN has been successfully applied in various probabilistic models, including VAE (Kingma and WWlling] (014)) and probabilistic word embeddings (Vilnis and McCallum, 2015). However, unlike the normal distribution in Euclidean space, the geometric characteristics of the HWN have been fully understood so far. Therefore, figuring out what can be done or not with the HWN is difficult.
In this work, we analyze the geometric properties of the diagonal HWN, standard choice of distribution in many probabilistic models (Mathieu et al. 2019 Nagano et al. (019).. Based on the observation that the principal axes of the diagonal normal distribution in Euclidean space have parallel structure with the standard bases, we also focus on the structure of the principal axes of HWN hyperbolic space delve into deeper understanding, Our analysis shows that the principal axes of the diagonal HWN are locally parallel to the standard bases in the Poincaré disk model.
Figure I (a) shows common understanding learned representation of hierarchical data in the Poincaré disk. The hierarchical structure spreads out like the spokes of a wheel. The local variation
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) The Poincaré disk model
(b) The Lorentz model
Figure 1: Visualization of hyperbolic space models. (a) The Poincaré disk model can embed a given tree-structured data with low distortions as shown in the illustration. The black segments refer to the shortest path between the points in the Poincaré disk model. The red dashed line denotes the continuous points in the same hierarchy level. (b) The Lorentz model is another model for hyperbolic space. The blue line is the shortest path between the points on the Lorentz model. The red line the projected geodesic via the diffeomorphism, which is still a geodesic of the Poincaré disk model.
in hierarchy then be represented angular difference between nodes the same level of the norm, i.c., the principal axis representing local variation is orthogonal to the radial axis. However, analysis of the geometric property reveals that the local variation can only be modeled along with the standard bases with the diagonal HWN.
To fix the structure of the principal axes in the diagonal HWN, we propose a simple yet effective alteration HWN, named rotated hyperbolic wrapped normal distribution (RoWN), By rotating diagonal covariance matrix before parallel transportation of HWN, we could resolve the limitations local variation structure while keeping the valuable properties, such as easy sampling and tractable density of the original HWN.
We verify the representation learned with RoWN agrees with the common characteristic of repre- sentation in hyperbolic space, which is barely observable with the diagonal and the full covariance HWN, by using synthetic noisy binary tree dataset. We demonstrate the usefulness RoWN on the benchmark datasets: WordNet and Atari 2600 breakout. We summarize our contributions as follows:
1. We provide an analysis of the geometric properties HWN and its potential limitations in representation learning.
2. We propose a novel and efficient method of using hyperbolic distribution, namely RoWN, and apply it to probabilistic models.
3. We demonstrate the performance of RoWN through the comparison with the Euclidean normal distribution, diagonal covariance HWN, and full covariance HWN on one synthetic dataset and two benchmark datasets.
Preliminaries
This section reviews the wrapped normal distribution defined on the Lorentz model. We first introduce the Lorentz model of hyperbolic space and necessary concepts to understand the wrapped normal distribution.
2.1 The Lorentz model
Hyperbolic space non-Euclidean space, having a constant negative Gaussian curvature. Figure illustrates two models for hyperbolic space, among four standard equivalent models: (1) Klein model, (2) the Poincaré disk model, (3) the Lorentz (hyperboloid) model, and (4) the Poincaré half-plane model. In particular, the Lorentz model is famous for its numerical stability in computing the distance and comes with a simpler closed form of geodesics (Nickel and Kiela, 2018). The Lorentz model R is the Riemannian manifold consisting of the set of points z E R satisfying (z, x) -1 and

---

20 O, where the Lorentizan product (,, 1 is defined as:
(x. y) c 20.0
Tithi
which also works the metric tensor on hyperbolic space, i.e, the metric tensor g of the Lorentz model is g(x) = diagl ,,,1
2.2 Tangent space of the Lorentz model
We denote the tangent space of IL" TaLL", which is set of points satisfying the orthogonality relation with a in terms of the Lorentzian product: TaL" := {u c) 03 The metric tensor g induces an inner product of two tangent vectors from a tangent space. Geodesic y [0, 1] + L generalizes straight lines in the Riemannian manifold which the shortest curve between two points. The exponential map expx TL + L" maps tangent vector E L=L" onto L" as:
expx ()) = cosh(]) (aah(|//llc))) + sinh([/ullc) |uulcc"
u
(I)
such that exp, (u) y. (0) x. (1) y. The log map, inverse of the exponential map, is defined as log, (3) == expx (u).
Parallel transport is an operation that transports tangent vector in the tangent space at to another vector the tangent space at y along the geodesic from to y without losing the parallel property. The parallel transport in the Lorentz model is given by:
aæ, v)C
PT PTn-w(U) = ++
(x +
a a+1
where (ax, y)
2.3 Hyperbolic wrapped normal distribution
One of the key challenges in adopting hyperbolic space probabilistic models is finding dis- tribution on hyperbolic space that is easy to sample and has a closed-form density function. The two most common distributions of hyperbolic space used previous work are Riemannian normal distribution (Mathieu et al., 2019] Said et al.] (2044) and hyperbolic wrapped normal distribution (HWN) (Nagano et al.) 2019). Our work is mainly based on the HWN because sampling in Rieman- nian normal distribution is limited with only a unit variance.
The sampling process with the HWN follows:
1. Sample R" from a Gaussian distribution (0, in Euclidean space.
2. Parallel transport the vector [0, v] To the tangent space.
3. Project the transported tangent vector to L using exponential mapping.
The density of sample can be measured via the change of variable method. For convenience, we denote the procedures 2 and 3 as a single operation:
fu(v) exp, (11 (())
(3)
with given μ E L and u E R and 0g = [1.....01 0] stands for the origin of L.
Rotated Hyperbolic Wrapped Normal Distribution
This section introduces several observations on the geometric properties of the HWN distribution transformation from the Euclidean space hyperbolic space. We show the limitations of the diagonal HWN for representation learning of hierarchical data. Then, we propose simple yet effective modification of the HWN, called rotated hyperbolic wrapped normal distribution.

---

Diagonal HWN
ROWN
(a)
(b)
(c)
Figure 2: Visualization of (a) the principal axes of the normal distribution in the Euclidean space, (b) the transported version of two axes in hyperbolic space, and (c) the probability density plot of the distributions. axis (red) and minor principal axis (blue). (b) The transformed principal axes become geodesics in hyperbolic space. The major principal axis of the RoWN passes the hyperbolic origin and crosses with the minor axis at the mean point, whereas the major and minor axes of the diagonal HWN is locally parallel to the standard axes. (c) The principal axes determine the shapes of the variational distributions.
(a) The contour line (green) can be represented as an ellipse with major principal
3.1 Observations on the hyperbolic wrapped normal distribution
First, we investigate the changes in the normal distribution during the transformation from Euclidean to hyperbolic space by Egquaiin As the principal axes characterize the covariance structure of the normal distribution in the Euclidean space, we investigate how the principal axes are transformed in hyperbolic space. Before deriving main proposition, first show that the straight lines that pass through the origin in the Euclidean space are transformed to the geodesics in hyperbolic space by Equation 3
Proposition 1. Suppose (1) = 18 E IR line passing through the origin, where 8 E IR is directional vector. Then the curve ((()a(t) in the Lorentz model IL." becomes geodesic.
The proof of the proposition is provided in Appendix A] The proposition indicates that every straight line that passes through the origin including the principal axes, is transformed into a geodesic in hyperbolic space.
Based on the first proposition, we provide our main proposition, which fully characterizes the structure of principal axes when projected to the Poincaré disk model:
Proposition 2. Define Proj(u) to be the projection function from the Lorentz, model the Poincaré f1:(0)
model, i.e., Proj(u) = zo(m)+T : Yu E L". If la a principal axis of the normal distribution defined in TRn and the mean of HWN in L,", then is the tangent vector of Proj(S (u (Ex)) on Proj(μ))
The proof of the proposition is provided in Appendix B The proposition reveals that the principal axes of the HWN are locally parallel to the standard bases the Poincaré disk model. To visualize the proposition, we plot the contour line and the principal axes of the two-dimensional diagonal normal distribution before and after the transformation in Figure 2] We observe that the tangent lines of the transformed principal axes are parallel to the standard bases in hyperbolic space.
When a popular diagonal normal distribution employed as variational distribution, the locally parallel principal axes might be problematic in learning hierarchical representations For example, suppose one tries to represent the variability along the radial direction or in angular differences. the case, both the major (red) and minor (blue) axes in Figure 2b cannot model the variability properly.

---

Algorithm Sampling process with the rotated hyperbolic wrapped normal distribution Input Mean μ IL" diagonal covariance matrix E IR
Output Sample _
æ [+ 0] E R" y pt: / x y) + (y" - x² v2²)>2((1 (x,
is determined by the sign of po
2: R + (y' x
3: Rotate Ε RER."
4: Sample (0 E)
5: return = fu(()
3.2 Rotated hyperbolic wrapped normal distribution
Based on the observation, we propose a simple yet effective alternative to the diagonal HWN, rotated hyperbolic wrapped normal distribution (RoWN). Rotating the covariance matrix to the direction of μ enables aligning the major axis the normal distribution in the Euclidean space to the radial axis in hyperbolic space visualized in the Figure 2
To construct the distribution, we start with a mean vector μ E LL and diagonal covariance matrix Σ as in the standard HWN. We change the covariance matrix of the normal distribution as follows:
1. Compute the rotation matrix R that rotates the x-axis ((+,,...,00 O] R JA:.-
2. Substitute the covariance matrix of Gaussian normal with RER".
Thus, the rotation matrix R. which rotates unit vector from y. be computed as:
R= I=I+ x² 4) 1+ (x, y)
(y x² y)
(4)
The pseudo-code of the sampling process of RoWN is in Algorithm Note that the construction straightforward but still keeps the following benefits of the HWN: 1) The sampling can be done efficiently, and 2) the computation of the probability density of the samples is tractable. As the HWN provides a tractable probability density function for any kind of covariance matrix, we can easily compute the probability density of given sample from RoWN. See Appendix C for more details.
samples
means
Diagonal HWN
(b) Full covariance HWN
(c) RoWN
Figure 3: Visualization of the variational distribution of hyperbolic VAE on synthetic binary tree dataset with different variational distributions. The red dots denote the variational means of the root and four representative children. The upper row shows the samples from the variational distributions where the color denotes the level of noise. The bottom row shows the means of the variational distributions. Overall, RoWN better aligns local variation in angular difference.
5

---

We empirically demonstrate the influence of different variational distributions in Figure We visualize the variational distributions of the synthetic binary tree dataset of depth two after training a hyperbolic VAE. As the result shows, the model with the diagonal HWN represents the variation in child parallel to the standard bases, whereas the model with RoWN represents the variation in angular difference. Please check the detailed description on the synthetic dataset in Secciinn 4.2
4 Experiments
In this section, we first explain the two applications of the distribution defined hyperbolic space: hyperbolic VAE and probabilistic hyperbolic word embedding model. We then conduct three different experiments to compare the performance of ROWN with four baselines, including the Gaussian distribution in the Euclidean space, the isotropic HWN (Nagano et al. 2201)), the diagonal HWN, and the full covariance HWN. We also provide an additional study on a variant of RoWN with learnable rotation direction in Algorithh and the results are in Table 9 The details of the experiments are described in Appendix
4.1 Applications of the hyperbolic distribution
The hyperbolic VAE, whose latent space is hyperbolic space, has been shown to be efficient for capturing the hierarchical structure of the data (Nagano et al. 2019 Mathieu et al. 2019). The evidence lower bound of the hyperbolic VAE can be written as:
Hyperbolic VAE.
CELBO (0, Φ) := EE (del))))
[o po
=)] - DKL (c
z) /det(g) ll p(*) /det(g))
where qs is the encoder, pe is the decoder, g is the metric tensor of the chosen model of hyperbolic space, and p(z) is a prior distribution. The distributions defined on hyperbolic spaces, such as HWN and RoWN, are used define encoder g æ=) /det(g) and prior p(z) /det(g). In hyperbolic VAFs, due to the absence of the flosed-form KI. divergence in the hyperbolic distributions, the KL divergence between the encoder and the prior is usually approximated with Monte-Carke Carlo sampling (Nagano et al.) 2019 Mathieu et al. 2019).
Probabilistic hyperbolic word embedding model. The probabilistic word embedding models aim to learn probabilistic representations of words (Vilnis and McCallum 2015 Nagano et al. 2019)
Tifrea et al., 2019). The embeddings learned on hyperbolic spaces have shown better performance than the Euclidean counterpart (Nagano et al.] 2019; Tifrea et al.] 2019). Given the hypernymy relationships between the words, the probabilistic hyperholic emhedding model learns the probabilistic representation of words by minimizing the following objective:
woord () := Ex(nt,ap(1) [max(0, m + DRI(g, ll a.) DKL (g> 4e))];
(5)
where is margin, z is distribution for word parameterized by 0, and
I and J denote the presence and absence of hypernymy relation between word pair and f, respectively.
4.2 Noisy synthetic binary tree
A synthetic binary tree dataset is first used to show the performance of representing hierarchy in Nagano et al. (2019), where each node in tree corresponds to a sequence of binary values. |Figure 4a shows example of the depth three binary tree, where parent and child only differ in one digit. We add spherical noises to the nodes in the same level of hierarchy as described in Figure 4a the noisy samples. With the noisy samples, we can create dataset containing local-level variation in the hierarchy. For the experiments, we uniformly sample the spherical noise from [0, π r/4].
We train hyperbolic VAE on noisy synthetic binary tree with varying depths. The detailed model description is available in Appendix D.I. We set the latent dimension the same as the depth. We report 1) the correlation between the hamming distance and the embedding distance and 2) the correlation between the depth and the Poincaré norm of the embeddings. The first correlation is computed over all possible pairs of test points. As Table I] shows, the full covariance HWN and RoWN improve the diagonal HWN except depth six, outperforming the Euclidean model in every setting. RoWN preserves the depth information better than the other distributions in general. We additionally visualize the variational mean obtained by training the tree of depth three FFiiuure 4b where the hierarchical structure is well preserved in the hyperbolic embedding space.

---

noisy samples
1000000
0000
0.4
cos sin 0000
0.4
1100000
10000
-04
1101000
110010
1010010
1010001
-04
85
(a) Noisy synthetic binary tree
(b) Learned representations
Figure 4: Illustration of noisy synthetic binary tree. (a) We construct noisy synthetic binary tree by adding spherical noises defined with 0. The continuous samples are generated at the same distance from the root. (b) We train the depth three noisy synthetic binary tree with hyperbolic VAE with RoWN as a variational distribution and visualize the means of the variational distributions. The black lines show the underlying hierarchical structure, and the color denotes the level of noise. The hierarchy and the local variations are well preserved the representations.
4.3 WordNet
We train probabilistic word embedding model with WordNet dataset (Fellbaum 1998). which consists of 82, nouns and 743,241 hypernymy relationships. We have initialized the embeddings from (O0, 0.012), which are then moved to the Lorentz model using the exponential map. We use the learning rate warm-up proposed in (Nagano et al. 2019). We evaluate the learned representations by computing the average rank of all the hypernymies. The rank of a given pair of words and t is computed among the distances between all possible pairs of the words s and t' without hypernymy. Table 2 shows the empirical performances of representing the word data. We report the performance with the mean rank (MR) and the mean average precision (mAP). RoWN preserves the hierarchical structure better than the other distributions, while the full covariance HWN often performs worse than RoWN.
Failure of the full covariance HWN. a similar performance to RoWN since the RoWN is the particular case of the full covariance. The
In theory, the full covariance HWN needs to have least
Table Results of noisy syuthetic binary tree. The results are averaged over 10 runs. The hyperbolic models outperform the Euclidean model in all settings. Overall, RoWN preserves the hierarchical information better than the other distributions.
Depth
5
6
7
Euclidean
0.748_..00
0.740
0.741
0.733
HWN (isotropic Σ)
0.809 1.16
0.7981..00s
0.735
Correlation w/ distance
HWN (diagonal )
0.814 .088
0.791 023
0.817 010
0.759+.020
HWN (full 2)
0.827 -077--012 .015
0.798+.028 026
0.798 010
0.794+.014
RoWN
0.820 0115
0.807
0.822+ 017
0.788+.016
Euclidean
0.762-.117
0.807
0.712+ 054
0.612+4.00
HWN (isotropic Σ)
0.902
0.867+.034
0.8114.029
0.602+.000
Correlation
HWN (diagonal Σ)
0.918 .228
0.808
0,862 035
0.697
w/ depth
HWN (full Σ)
0.956 015
0.878 051
0,870+ 033
0.815 055
RoWN
0.930
0.911+ 027
0.901 034
0.827

---

Table Results of WordNer. The results are an average runs. Based on the rank of hypernymy pairs among non non-hyyernymy pairs, we report the mean rank (MR) and mean average precision (mAP) for evaluation.
Latent dimension
10
20
Euclidean
13.968.10.00
3..62
1.955
HWN (isotropic Σ) HWN (diagonal 2)
14.568
2..60
3.0255
MR
16.590+1.146
3.891=0.117
2.062+0.088
HWN (full 2)
557.309
466.513=75.14 142
599.140=18.916
ROWN
16.271+2.985
2.888
1.783+0.090
Euclidean
0.565_0.004
0.801
0,902
HWN (isotropic ) HWN (diagonal Σ)
0.617=0.012
0.820=0.013
0..07
MAP
0.565
0.805
0.905
HWN (full Σ)
0.032=0.005
0.003=0.005
0.079
RoWN
0.593=0.024
0.844
0.921 +00005
performance the full covariance HWN often performs worse with the probabilistic hyperbolic word embedding models than the hyperbolic VAEs. We speculate that reason is because of the relatively simple prior in the hyperbolic VAE, whereas the probabilistic word embedding models need to compute the KL divergence between the full covariance HWNs. Our additional experiments reported in Table confirm our speculation the number of training samples for the KI divergence increases, the performance increases slightly.
Discussion of the root placement. Note that due to the isometry of the geometry, the root node can be placed anywhere hyperbolic space. In other words, infinitely many sets embeddings preserve the same pairwise distances between the nodes. To place the root near the origin, we initialize all embeddings with the zero mean distribution as done in (Nagano et al., 2019). We find that this initialization helps the root placed near the origin. A study the effects of the initialization methods shown in Table 3. Further details about the root placement can be found in Appendix
4.4
Atari 2600 Breakout
Trajectories of some Atari 2600 games can be structured as a tree-like hierarchy along the time horizon as Nagano et al.] (2019) points out. For example, given Atari 2600 Breakout, the root node can be the starting state of the game where blocks have been broken. As the game progresses the states of the blocks form a hierarchical structure depending on which blocks have been broken. To
Table 3: The effects of initializations. We test different initializations the deterministic hyperbolic word embedding models trained with the subset of the WordNet dataset. The near zero vector model initializes the embeddings with uniform distribution (0000001,, 0.001), while the near one vector model initializes the embeddings with uniform distribution 2((0.999, 1.001). While the Poincaré norm of the root node differs, the other metrics remain similar.
Latent dimension
2
10
20
Near vector
4.346
3.270
2.8284.00g
2.5084.049
MR
Near one vector
4.029
3.2094.094
2.8564.073
2.491
Near zero vector Near one vector
0.821+.016
0.891
0.891
0.895
mAP
0.829#.010
..894+.004
0.891+.00s
0.8964.003
The Poincaré norm of the root node
Near zero vector
0.122 .036
0.042 015
0.031
0.0244 001
Near one vector
0.505
0.650+.01
0.7324 003
0.801

---

Table 4: Results of Atari 2600 Breakout. The results are averaged over runs. We measure the correlation between the score of an mmgge and the Poincaré norm of the variational mean. The HWN with the isotropic covariance can be viewed as variant of RoWN
Latent dimension
15
20
Euclidean
0.3794.007
0.436
0.479
HWN (isotropic Σ)
0.513
0.598 L.01
0.607
Correlation btw. and norm
HWN (diagonal )))
0.4784.011
0.513 006
0.513+.008
HWN (full 2)
0.483
0.520 009
0.563+.010
ROWN
0.497+.014
0.556
0.561 029
learn the implicit hierarchy that can be observed from the trajectories of Breakout, we train the VAE models with the Atari 2600 Breakout images.
The images of Breakout are collected by using a pre-trained Deep Q-network (Mnih et al. 2015) and divided into training and test set with 90,000 and 10,000 images respectively. We label each image with the score obtained from the game environment. So the labels are correlated to the number DCGAN-based architecture, which originally used
of broken blocks. To train VAE, to evaluate HWN Nagano et al. (019). The detailed architecture provided in Appendix [D.4] To evaluate the models,
the correlation between the Poincaré norm of the test images and the labeled scores, The evaluation results and the generated images from the trained models are reported in Table 4 and Figure 5, respectively.
learning Breakout images, RoWN and the full covariance HWN outperform the diagonal HWN. The isotropic HWN, where the covariance matrix is invariant any rotation matrix, shows better correlation than the others in all the settings. However, as reported in Table 14] the test ELBO values relatively worse than the others. While the isotropic HWN shows high correlation but relatively lower test EL ELO RoWN shows competitive test ELBO to the others and well aligns the hierarchical structures with respect the norm in low latent dimensions.
Related work
Hyperbolic space for hierarchical representation learning. Earlier studies on the hierarchical
representation learning have focused modeling explicit hierarchical structures through the Bayesian
non-parametrics (Griffiths et al., 20033 Larsen et al. 2002 Salakhutdinov et al. [012 Teh et al. [007]]
Ghahramani et al. 2010 Heller and Ghahramani] 2005) or embedding the hierarchica structure into
Euclidean space (Nickel et al.) 2011 Grover and Leskoved 2016 Nguyen et al. 2017). Euclidean
The Poincaré disk model
Diagenal
RoWN
Figure The generated images from trained VAE endowed with RoWN by using Atari 2600 Breakout images. Every five images are generated from the randomly sampled latent vectors of dimensionality two having the Poincaré norm 0. 0.9, 0.95. The larger norm of the sample latent vectors, the more blocks are broken out in the generated images.

---

space is later shown to require an excessive number of dimensions to embed the original hierarchical structure without any distortion (Linial al. 1994).
Hyperbolic space has been proposed as an alternative medium to embed the hierarchical data. Theoretically, any tree-structured data can be embedded in hyperbolic space with arbitrarily low distortion (Sarkar 2011). Based on the theory, learning representation of hierarchical structure in hyperbolic space has been shown great success in various datasets including WordNet hierarchy,
graph-structured data, and social network data (Nickel and Kiela 2017 2018 Chami et al. 2020
Sun al. 2020; Zhao
2011 Shavitt and Tankel| 2008). However, these studies are only limited to modeling the explicit hierarchy, which means that the dataset contains an explicit relation between data points. Furthermore, the learning frameworks are limited in the proppobbaiilitti setting because of the absence of well-behaved distribution in hyperbolic space.
Distributions in hyperbolic space. The probabilistic model enables measuring the uncertainty and provides a principled way of learning. To extend the probabilistic learning framework from Euclidean to hyperbolic space or Riemannian manifold in general, one needs to define a well-behaved distribution that has tractable density and is easy to sample from. Recently, a few studies have proposed probabilistic learning schemes hyperbolic space (Mathieu et al., 2019: Nagano et al.. 2019). Mathieu et al. (2019) introduces parametrizable sampling schemes for the two canonical Gaussian generalizations defined on the Poincaré disk model. The scheme is used train hyperbolic VAE and show improved generalization performance with high interpretability. [Ngano al.) (0199) suggests a method of integrating the Bayesian framework with hyperbolic space in the Lorentz model where the simpler closed form of geodesics is defined. Normalizing flow (Rezende and Mohamed 2013) can be also used to define the hyperbolic distribution in the probabilistic learning framework. Bose et al. (2020) propose normalizing flows defined on hyperbolic space, which show improvements in learning hierarchical structures in graph data. Mathieu and Nickel| (2020) elevate the continuous normalizing flow (Chen al., 2018) defined on the Euclidean space arbitrary Riemannian manifold, including the hyperbolic space. Based on Nagano et al. | (2019), we analyze the geometric properties of the distribution lying on the Lorentz model and show the limitations of the existing method.
Limitations
We explore a better method of representing hierarchical data in hyperbolic space. To this end, we propose simple yet effective alternative of hyperbolic wrapped normal distribution. However, the proposed distribution is limited only to hyperbolic space, and no generalization method for Riemannian space is studied yet. To explore the usefulness of alternative Riemannian spaces, finding a common distribution that can work well in any Riemannian space will be necessary.
RoWN is a subset of the full covariance HWN. However, in many cases, RoWN outperforms the full covariance HWN in our experiments. general, optimizing the covariance matrix requires learning the quadratic number of parameters with respect to the dimensionality. We conjecture the hardness of optimization leads to the poor performance the full covariance HWN. To overcome this limitation, search for a better optimization algorithm in hyperbolic space needs to be explored.
7 Conclusions
In this work, we propose a novel method of using RoWN for representing the data with a hicrarchical structure. With in-depth analysis of the geometric properties of HWN, we demonstrate why the common choice of the diagonal covariance matrix for HWN may be inappropriate but the rotated covariance matrix. Our empirical results present that RoWN exhibits better representation ability, both qualitatively and quantitatively, compared to all the baselines: Euclidean normal distribution, diagonal HWN, and full covariance HWN. We hope that our method helps better understanding the anatomy of hyperbolic space and be a promising technique for efficient representation learning.
10

---

Acknowledgement
This work was partly supported by Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2019.0-0199) Artificial Intelligence Graduate School Program (POSTECH) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-202IRICIC111
References
Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. In International Conference on Machine Learning. 2020.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 2017.
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, July 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
Zoubin Ghahramani, Michael Jordan, and Ryan Adams. Tree-structured stick breaking for hierar-
chical data. Advances in neural information processing systems, 23, 2010.
Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16, 2003.
Aditya Grover and Jure Leskovee. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, 2016.
Katherine A Heller and Zoubin Ghahramani. Bayesian hierarchical clustering. In Proceedings of the 22nd international conference on Machine learning, pages 297-304, 2005.
Daniella Horan, Eitan Richardson, and Yair Weiss. When unsupervised disentanglement possible?
Advances in Neural Information Processing Systems, 34:51 1555155-1111, 2021.
Jaemin Jo and Jinwook Seo. Disentangled representation of data distributions in scatterplots. In 2019 IEEE Visualization Conference (VIS), pages 136-140. IEEE, 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
Jan Larsen, A Szymkowiak, and Lars Kai Hansen. Probabilistic hierarchical clustering with labeled and unlabeled data. International Journal of Knowledge Based Intelligent Engineering Systems, (1):56-63,, 2002.
N. Linial, E. London, and Y. Rabinovich. The geometry of graphs and some of its algorithmic applications. In Proceedings 35th Annual Symposium on Foundations of Computer Science, 1994.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 2020.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, Fox, and R. Garnett, editors, Advances Neural Information Processing Systems. Curran Associates, Inc., 2019.

---

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 2015.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and f Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient- based learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2019.
Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, and Ngoc Thang Vu. Hierarchical embeddings for hypernymy detection and directionality. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, September 2017.
Maximilian Nickel, Volker Tresp. and Hans-Peter Kriegel. A three- way model for collective learning on multi-relational data. In International Conference on Machine Learning. 2011.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
Maximillian Nickel and Douwe Kicla. Learning continuous hicrarchies in the Lorentz model of hyperbolic geometry. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. JMLR.org. 2015.
Joel W Robbin and Dietmar A Salamon. Introduction to differential geometry. Springer Nature, 2022.
Salem Said, Lionel Bombrun, and Yannick Berthoumieu. New riemannian priors the univariate normal model. Entropy, 2014.
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. One-shot learning with a hierar- chical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pages 195-206. JMLR Workshop and Conference Proceedings, 2012.
Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In Proceedings of the 19th International Conference on Graph Drawing. Springer-Verlee 2011.
Yuval Shavitt and Tomer Tankel. Hyperbolic embedding of internet graph for distance estimation and overlay construction. IEEE ACM Transactions on Networking, 2008
Zequn Sun, Muhao Chen, Wei Hu, Chengming Wang, Jian Dai, and Wei Zhang. Knowledge association with hyperbolic knowledge graph embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2020.
Yee Teh, Hal Daume III, and Daniel M Roy. Bayesian agglomerative clustering with coalescents. Advances in neural information processing systems, 20, 2007.
Alexandru Tifrea, Gary Becigneul, and Octavian Eugen Ganea. Poincare glove: Hyperbolic word
embeddings. In International Conference on Learning Representations, 2019.
Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Xiaohan Zhao, Alessandra Sala, Haitao Zheng, and Ben Y. Zhao. Efficient shortest paths on massive social graphs. In 7th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom) 2011.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is provided with the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material or URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "lHExB7pk6QW", "reviewer": "Reviewer_nB4o", "summary": "The authors build upon the Hyperbolic Wrapped Normal (HWN) distribution, which puts a distribution on hyperbolic space by transporting a distributed tangent vector from an origin to a mean point mu and taking the exponential map (a diffeomorphism in this case). The authors observe that a standard basis vector in the tangent plane at the origin, transported to point mu and then exp-mapped and projected to a Poincaré disk, is aligned with a standard basis vector in the disk. They posit that the resulting distribution is less desirable than a distribution that is aligned such that one of the tangent vectors points in the disk from mu to the origin. To fix this, they propose the Rotated HWN distribution that is constructed so that the first dimension of the tangent plane points from mu to the origin.\n\nEmpirically, the authors show that the RHWN outperforms HWN with diagonal and full covariance matrix on a synthetic task, a word embedding task and a task of embedding Atari rollouts. In the synthetic task, the goal is to have the hamming distance on a binary tree correlate with the (I presume geodesic) distance in embedding space or to correlate tree depth with the \"Poincaré norm\" (geodesic distance to origin, I presume). In the word embedding task, the goal is to have hypernymy correlate with distance. In the Atari task, the goal is to have distance to the origin correspond with progress in the game.", "questions": "Please see my point above on the isometry symmetries.", "limitations": "The authors fairly address the limitations of their work.\nI disagree though that rotating the tangent space to the root node is applicable only on hyperbolic space, see my comment above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The paper is very clearly and pleasantly written with many insightful illustrations - critical when dealing with geometry.\n- I like the construction of the paper: diagnosing an issue with a prior method, then proposing a simple fix\n- The proposed method is simple to describe and implement.\n- The proposed method outperforms prior work.\n\n## Weaknesses\nI have one major concern with the geometric analysis and some other points.\n### Symmetry of the geometry\nI am not sure that I agree with the diagnosis. As a geometric method, it shouldn't matter which model of the hyperbolic space we pick. Up to isometry there is only one space of n dimensions with constant -1 curvature. This space, $H_n$, is a homogenous (even symmetric) space meaning that there is an isometry mapping any tangent vector $v$ at point $p$ to any other tangent vector $v'$ at any point $q$ assuming that they have the same norm: $g_p(v, v)=g_q(v', v')$. Thus, I don't really see why in Fig 2 the unrotated image on the top and the rotated image on the bottom are different. I suspect there is an isometry that maps between them. The only reason the top image looks \"odd\" is the particular choice of coordinates. Similarly, the paper makes repeated reference to the \"origin\" and \"standard basis\" of the Poincaré disk, which are also contingent on a particular choice of coordinates. A geometric method shouldn't care about those. Hence, when constructing a single distribution in hyperbolic space, up to isometry, the added rotation in the RHWN effectively does nothing.\n\nThe main reason the isometry-symmetry is broken is the particular choice of prior $p(z)$ in the variational inference, taken to be centered around a particular point $\\mu_0$, which can then be called the origin. I suppose that's what the authors do. This would be the only reason why the root of a tree should be embedded close to the origin - geometry itself doesn't promote that. Yet the authors make no mention of the prior $p(z)$ in their analysis. I suppose the RHWN works well because first the prior encourages the root to be close to the origin and then it being preferable to have the basis such that one dimension points towards the root.\n\nI feel the analysis can be much improved if these considerations were taken into account. Concretely:\n- I'd like to see an argument why the root node should be place at the mean of the prior $p(z)$. For that, I'd also like to know what's special about the root node. If one interprets a (e.g. binary) tree as a graph, then it can be seen as a binary tree with any node as root node. I'd guess that this has to do with the balancing of the resulting tree - or with some notion of distance between that the nodes that's not encoded in an (unweighted) tree.\n- Then in the subsequent discussion, it'd be better if references to \"standard basis\" and \"origin\" (arbitrary coordinatization) would be made geometrically meaningful relative to the prior origin / placement of the root node.\n\n### Other points\n1. Due to my above points, it's not quite clear to me why the $x$ vector in the algorithm should be fixed, so that the first dimension of $v$ is rotated to point to the arbitrary origin. It would be interesting to have it be learned and given by the encoder network. This is somewhat similar to e.g. Householder flows [1]. I'd be great if the authors could compare to such a method. One motivation could be that one wants to align not to the root node, but to the parent node, which may not lie in the direction of the root node. It'd be very interesting if the learned point to rotate to would indeed be the parent.\n2. Why would the rotation be particular to a hyperbolic embedding of trees? In other manifolds with a prior with a well-defined mean and exp-mapped normal distributions, one could rotate one tangent vector towards the prior mean. For Euclidean spaces, this would be straightforward, but I could also imagine this working on Lie groups [2]. I'd be very interested to see if that improves the performance. The paper could make a much more general claim if they show improvement across different kinds of manifolds.\n3. The rotation matrix (4) is not the only rotation that maps $x$ to $y$. It'd be good if the authors discuss this and say why this particular one (which I believe is the one with the smallest SO(n) distance to the unit) is desirable.\n4. The authors use the word \"prior\" confusingly to not as the distribution $p(z)$ in variational inference, but as the variational family the posterior $q(z|x)$ belongs to. I see how this originates from the idea of \"prior\" as domain knowledge, but I'd prefer a different wording for that.\n5. Due to its simplicity, the proposed method is a quite incremental change over prior work, hence somewhat limited in originality.\n\n\n# Conclusion\nBecause I don't think the current analysis is correct without explicit mention of the role of the prior, I propose rejection of the paper. If the authors address those issues satisfactorily, I would recommend acceptance. If the authors could expand the paper by addressing my points (1) and (2), this could become a strong paper.\n\n-- Score updated\n-- Score updated again\n\n[1] Tomczak, Jakub M., and Max Welling. \"Improving variational auto-encoders using householder flow.\" (2016).\n\n[2] Falorsi, Luca, et al. \"Reparameterizing distributions on lie groups.\" (2019)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dgL6JOtAESU", "reviewer": "Reviewer_hLZa", "summary": "This paper studies the geometry of two sub-classes of the Hyperbolic Wrapped Normal distribution (HWN) with the goal deepen the understanding of which family is more appropriate for probabilistic machine learning when the data are in (or can be embedded into) hyperbolic spaces. The first subclass is the diagonal HWN that has been applied lately to different problems in hyperbolic probabilistic machine learning and the second the rotated HWN, that is proposed by the authors as a more appropriate alternative. \n\nThe diagonal HWN draws a sample in two steps. First samples from a diagonal Gaussian in the tangent space at the origin and secondly parallel transports the sample to the new mean and projects it to the manifold via the exponential map. The proposed rotated HWN interpolates an extra step. After sampling at the origin, it rotates the sample by the rotation matrix that brings the origin to the direction of the new mean and then performs last step of the diagonal HWN. The authors provide a proof (and an additional visual explanation) that for both the diagonal HWN and the rotated HWN the principal axes become geodesics that intersect orthogonally on the new mean. From that the authors infer that the rotated HWN is more appropriate (especially for the Poincare model) since the principal axes correspond to the radial and angular variations accordingly. \n\nThey perform 3 experiments on data that have hierarchical structure (and thus better embedded in hyperbolic space) to support their claims. The first is a toy dataset of a noisy binary synthetic tree. The goal is to show that the rotated HWN generrates posteriors that capture better the distances between the nodes as well as their level in the hierarchy (via their norm). In the second task a real language dataset is used and the goal is to embed the words and count their rank. From that task, the authors provide experimental proof that the full HWN is not appropriate for some hierarchical tasks and conjecture that this is due to the difficulty of the optimization problem. The last task is to embed the trees that can describe the evolution of an Atari game. The goal is to show that the angular direction in the rotated HWN is more informative than the one of the diagonal HWN, by describing \"sibling states\" of equivalent scores. ", "questions": "1) In Algorithm 1 why do we need to resort to the inner product of the ambient space i.e. a)the normalization of $\\mu$ is w.r.t. the Euclidean norm and not the norm induced by the Lorentz inner product, b) the rotation matrix. \n\n2) The inner product $\\langle \\cdot,\\cdot \\rangle_x$ is defined but not used. \n \n4) How are the correlation metrics in experiment 1 computed exactly and why are they more appropriate to present than the ELBO?", "limitations": "The authors discuss some limitations of the proposed distribution, i.e. that it only works for hyperbolic spaces (although it seems relevant to a lot of conformal maps) and that the full HWN might be better optimized if there were better optimization algorithms (although more samples might still be needed to distinguish the best distribution in the family)\n\n-I would advise the authors to extend their idea to more general families of distributions in hyperbolic space as well as more general conformal maps.\n\n-Moreover, to strengthen the intuition about how the angular and radial dependency decompose more appropriately the authors should provide the marginal distributions and an extensive discussion.  \n\n-(Minor) The authors should mention what exactly is the \"hyperbolic VAE\" and how it is modified by them in the main text.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \n\n(+) To my knowledge the observations and proofs in this paper are original. \n\n(-) The figures describing the spaces and the experimental evaluation are not novel but they seem standard.\n\nQuality:\n\n(+) High quality in writing and presentation.\n\nClarity:\n\n(+) Paper is clearly presented.\n\n(-) (Minor) Figure 2 (a): \"red\" and \"blue\" are interchanged, since red is the major principal axis. \n\n(-) In the preliminaries one more sentence about what is a geodesic would be appropriate. \n\nSignificance:\n\n(-) the paper proposes a simple modification to constraint the family of HWN. \n\n(+) However, this observation aligns well with the intuition gained from the conformality of the map from Euclidean to Poincare disk. It also disentangles better the angular and radial dimensions and the authors provide proofs, figures to build intuition and standard experimental evaluation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3rezcHPWOSf", "reviewer": "Reviewer_hPAy", "summary": "In this submission the authors introduce a parametric family of probability distribution on the hyperbolic space. They build on the hyperbolic wrapped normal distribution which rely on pushing Gaussian samples with parallel transport to the mean and then along the exponential map onto the manifold. Their main insight is based on the fact that when using a diagonal covariance matrix, the arbitrary choice of basis to sample this Gaussian (in the tangent space at the origin) is carried to the tangent space at the mean. Thus the principles axes of the distribution would not (apart if the mean lies on the x or y axes) be aligned with the \"hierarchical depth direction\" (which is given by the mean) and the \"hierarchical same-level direction) which is the hyperplane orthogonal to the \"depth\" direction. They highlight that these directions are the \"natural\" ones over which the uncertainty/variation should be expressed, and therefore proposed to rotate the diagonal covariance matrix so that these are aligned.\nThey empirically demonstrate that this parametric distribution is better able than the diagonal and full covariance variants in learning latent representation that are better correlated with variables of interest, hence leading to \"better\" learnt hierarchical representations in that sense.\n", "questions": "\nGlobal comments:\n- I would strongly suggest moving the figures/tables so that they are closer (in the same page) as the related paragraph. At the moment it's not the case, and I believe it would greatly improve the reader's experience.\n- As the proposed method is de facto a specific setting of the full covariance matrix, I believe that the investigation of the failure of full covariance model would be worthy of more space. I believe that Figure 3 is useful in that regard but I have a few remarks/questions on it later. It is claimed that this parametrisation lead to unstable optimisation, but it would be nice to dive a bit more, e.g. computing the condition number. Additionally, the isotropic variant where the covariance matrix is given by $\\Sigma = \\sigma I_d$ is not explored, whilst it would be easy to do so and interesting to see whether it is indeed not flexible enough.\n\n\nSpecific comments:\n- typo \"Hyperboilc\" in openreview title\n- Abstract: \"RoWN, the newly proposed distribution\" -> \"newly\" perhaps misleading as it's in some ways a specific setting of the full covariance matrix (which now depends on the mean).\n- Section 2\n    - Sec 2.3: I feel that the sampling process for the HWN would really benefit from having an illustrative figure on the side. This could also be used to illustrate Algorithm 1, which is the core contribution of the paper.\n- Section 3\n    - Sec 3.1: Proposition 1 is somewhat repeated before *and* after which seems unnecessary.\n    - Sec 3.2:\n        - Worth noting in Algorithm 1, that to sample the multivariate normal $v \\sim \\mathcal{N}(0, \\hat{\\Sigma})$, one can rely on computing $\\hat{\\Sigma}^{1/2} = R \\text{diag}(\\sqrt{\\sigma})$ (which exist and is unique since $\\Sigma$ is positive definite) which can be done easily.\n        Figure 3: \"different prior choices\" -> meant \"posterior\"? What are the dashed blue lines? Although I do find the upper row visually pretty, I don't understand from which posterior distributions the samples come from (from the ones whose means are represented in red?) nor the signification of the colour.\n- Section 4\n    - Sec 4.1:\n        - I would suggest giving a bit more of background on the model for the noisy synthetic tree, mainly on the loss and likelihood (based on the Hamming distance) and how this plays with the spherical noise since the nodes aren't binary anymore.\n        - Also I would be curious to see Table 1 for <4 and >7 depth, are the results similar?\n        - Additionally I was wondering whether \"better\" latent representations (measured via the reported correlations) lead to better likelihood (as estimated via IWAE [1])?\n    - Sec 4.2:\n        - The HWN (full covariance) dramatically fails, according to the author this due to \"unstable optimization\". What is the reason why this phenomenon did not occur in the VAE setting from Section 4.1?\n        - Table 2: Would suggest adding \"metric\" and \"model\" in the table's header\n        - Worth reminding what are the MR and mAP metrics for the readers unfamiliar with these? either via their mathematical formula or in an english sentence.\n    - Sec 4.3:\n        - What is the motivation for the Poincaré norm of the latent representation to be correlated with the score? I'm only partially familiar with breakout, is the score correlated with the number of broken blocks? Isn't the Poincaré norm simply correlated with time then?\n- Sec 5:\n    - The reader may benefit from starting the related work section with a paragraph on \"hierarchical representation learning\". These methods are most of the time relying on discrete hierarchical structures, there are quite a few in the Bayesian non-parametrics literature (e.g. [2, 3]).\n    - In the \"Distributions in the hyperbolic space\" paragraph, one could also mention normalizing flow related methods (e.g. [4, 5]).\n", "limitations": "I believe so", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe proposed idea is simple, pretty well presented and well executed, with nice figures to illustrate the model and pretty convincing empirical results.\nThe paper was mostly pleasant to read and easy to follow.\n\nI don't see any major weakness.\nIn the following section I make suggestions to improve the organisation of the paper, along with some questions which I hope will lead to some clarification.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept