Hyperbolic Neural Networks++
Authors: Ryohei Shimizu
YUSUKE Mukuta
Tatsuya Harada
Venue: ICLR 2021 Poster (2021)
Keywords: Hyperbolic Geometry
Poincaré Ball Model
Parameter-Reduced MLR
Geodesic-Aware FC Layer
Convolutional Layer
Attention Mechanism

Abstract: Hyperbolic spaces, which have the capacity to embed tree structures without distortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data. In this study, we generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincaré ball model. This novel methodology constructs a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under a unified mathematical interpretation, without increasing the parameters. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.

Published as a conference paper at ICLR 2021
HYPERBOLIC NEURAL NETWORKS+
Ryohei Shimizu', Yusuke Mukuta 12, Tatsuya Harada 1.2
'Tee University of Tokyo
PRIKEN AIP
(shimizu, mukuta, harada emi.t.u-tokko.ace jp
ABSTRACT
Hyperbolic spaces, which have the capacity to embed tree structures without dis- tortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data. In this study, we generalize the fundamental components of neural networks in a single hyper- bolic geometry model, namely, the Poincaré ball model. This novel methodology constructs a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under unified mathematical interpretation, with- out increasing the parameters. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
INTRODUCTION
Shifting the arithmetic stage of a neural network to non-Fuclidean geometry such hyperbolic space is a promising way to find more suitable geometric structures for representing or processing data. Owing to its exponential growth in volume with respect its radius (Krioukov et al., 2009; 2010), hyperbolic space has the capacity to continuously embed tree structures with arbitrarily low distortion (Krioukov al., 2010; Sala al., 2018). has been directly utilized, for instance, to visualize large taxonomic graphs (Lamping et al., 1995), to embed scale-free graphs (Blasius et al., 2018), or to learn hierarchical lexical entailments (Nickel Kiela, 2017). Compared to the Euclidean space, a hyperbolic space shows a higher embedding accuracy under fewer dimensions in such cases.
Because a wide variety of real-world data encompasses some type of latent hierarchical structures (Katayama & Maina, 2015; Newman, 2005; Lin & Tegmark, 2017: Krioukov et al., 2010), it has been empirically proven that a hyperbolic space is able to capture such intrinsic features through representation learning (Krioukov et al., 2010: Ganea et al., 2018b; Nickel & Kiela, 2018; Tifrea et al., 2019; Law et al., 2019; Balazevic et al., 2019; Gu et al., 2019). 9).. Motivated by such expressive characteristics, various machine learning methods, including support vector machines (Cho et al., 2019) and neural networks (Ganea et al., 201 8a; Gulcehre et al., 2018; Micic & Chu, 2018; Chami al., 2019) have derived the analogous benefits from the introduction of hyperbolic space, aiming to improve the performance on advanced tasks beyond just representing data.
One of the pioneers in this area is Hyperbolic Neural Networks (HNNs), which introduced an casy-to-interpret and highly analytical coordinate system of hyperbolic spaces, namely, the Poincaré ball model, with corresponding gyrovector space smoothly connect the fundamental functions common to neural networks into valid functions in a hyperbolic geometry (Ganea et al., 201 8a). Built upon the solid foundation HNNs, the essential components for neural networks covering the multinomial logistic regression (MLR), fully-connected (FC) layers, and Recurrent Neural Networks have been realized. In addition the formalism, the methods for graphs (Liu et al., 2019), sequential classification (Micic & Chu, 2018), Variational Autoencoders (Nagano et al., 2019; Mathieu et al., 2019; O Ovinnikov, 2019; Skopek et al., 2020) are further constructed. Such studies have applied the Poincaré ball model natural and viable option in the area of deep learning.
Despite such progress, however, there still remain some unsolved problems and uncovered regions. In terms of the network architectures, the current formulation of hyperbolic MLR (Ganea al., 201 8a) requires almost twice the number of parameters compared to its Euclidean counterpart. This makes both the training and inference costly in cases in which numerous emhedded entities should be

---

Published as a conference paper at ICLR 2021
classified or where large hidden dimensions employed, such as in natural language processing. The lack of convolutional layers must also be mentioned, because their application is now ubiquitous and is no longer limited to the field of computer vision.
For the individual functions that are commonly used in machine learning, the split and concatenation of vectors have yet be realized in hyperbolic space in manner that can fully exploit such space and allow sub-vectors to achieve a commutative property. Additionally, although several types of closed-form centroids in a hyperbolic space have been proposed, their geometric relationships have not yet been analyzed enough. Because a centroid operation has been utilized in many recent attention- based architectures, the theoretical background for which type of hyperbolic centroid should be used would be required in order to properly convert such operations into the hyperbolic geometry.
Based on the previous analysis, we reconsider the flow of several extensions to bridge Euclidean operations into hyperbolic operations and construct alternative or novel methods on the Poincaré ball model. Specifically, the main contributions of this paper are summarized as follows:
1. We reformulate hyperbolic MLR reduce the number of parameters to the same level as a Euclidean version while maintaining the same range of representational properties.
2. We further exploit the knowledge of as a replacement of an affine transformation and propose novel generalization of the FC layers that can more properly make use of the hyperbolic nature compared with a previous research (Ganea et al., 2018a).
3. We generalize the split and concatenation of coordinates to the Poincaré ball model by setting the invariance of the expected value of the vector norm as a criterion.
4. By combining 2 and 3, we further define novel generalization scheme of arbitrary dimen- sional convolutional layers the Poincaré ball model.
5. We prove the equivalence of the hyperbolic centroids defined in three different hyperbolic geometry models, and expand the condition of non- non-negative weights to entire real values. Moreover, integrating this finding and previous contributions 1, 2, and 3, we give a theoretical insight into hyperbolic attention mechanisms realized in the Poincaré ball model.
We experimentally demonstrate the effectiveness of our methods over existing HNNs and Euclidean cquivalents based on a performance test of MLR functions and experiments with Set Transformer (Lee et al., 2019) and convolutional sequence to sequence modeling (Gehring et al., 2017).'
2 HYPERBOLIC GEOMETRY
Riemannian geometry. An n-dimensional manifold M is an n-dimensional topological space that can be linearly approximated to an n-dimensional real space at any point æ E M, and each local linear space is called a tangent space M. A Riemannian manifold is pairing of differentiable manifold and a metric tensor field g as function of each point æ. which is expressed as (M, g). Here, g defines inner product each tangent space such that Yu, E M, (u, v) = u g v, where 92 is a positive definite symmetric matrix defined on T. The norm of a tangent vector derived from the inner product is defined as ||D||,, = Vk, Te A metric tensor g.r provides local information regarding the angle and length of the tangent vectors in T,M, which induces the global length of the curves on M through an integration. The shortest path connecting two arbitrary points on M constant speed is called geodesic, the length of which becomes the distance. Along geodesic where one of the endpoints is æ. the function projecting a tangent vector e Ta as an initial velocity vector onto M is denoted as an exponential map exp.. and its inverse function is called logarithmic map logg In addition, the concept of parallel transport P Pæ- TM - T generalized to the specially conditioned unique linear isometry between two tangent spaces. For more details, please refer to Spivak (1979); Petersen et al. (2006); Andrews & Hopper (2010).
Note that, in this study, we equate g with a.. if g. is constant, and denote the Euclidean inner product, norm, and unit vector for any real vector u, v E IR as (u, v). ||w/], and [o] w/|/v/ respectively.
Hyperbolic space. A hyperbolic space a Riemannian manifold with a constant negative curvature, the coordinates of which can be represented in several isometric models. The most basic model is an
The code is available at httpt::////gg tww.itthbb........ com/mi 1--tooooooyyprrboooo i pl_nn_pl
2

---

Published as a conference paper at ICLR 2021
n-dimennsioal hyperboloid model, which is a hypersurface H. in an (n dimensional Minkowski space 17+1 composed of one time-like axis and space-like axes. The manifolds of Poincaré ball model _ and Beltrami-Kii Klein model K" are the projections of the hyperboloid model onto the different n-dimensional space-like hyperplanes, depicted in Figure 1. For their mathematical definitions and the isometric isomorphism between their coordinates, see Appendix A.
Poincaré ball model. The n-dimensional Poincaré ball model of a
constant negative curvature -c is defined by , g"). where B {ar RD c||ar||2 1} and 0h (Xş) ) In. Here, B is an open ball of radius e 2, and X 2(1 c||a 12)-1 a conformal factor, which induces the inner product (u, v) (2, v) and norm I = ||| for u, v E T, The exponential, logarithmic maps and parallel transport are denoted as expe. logg and 2-+y' respectively, as shown in Appendix C.
EGC
To operate the coordinates vector-like mathematical objects, the Möbius gyrovector space provides an algebra that treats them as gyrovectors, equipped with various operations including the general- ized vector addition, that is, a noncommutative and non-associative binary operation called the Möbius addition @. (Ungar, 2009). lim @. converges to + in connection with a Euclidean geometry, the curvature of which is zero. For more details, see Appendix B.
Figure I: Geometric relation- ship between H. B and K depicted in IR I=++1
Poincaré hyperplane. As specific generalization of hyperplane into Riemannian geometry, Ganea et al. (201 8a) derived a Poincaré hyperplane ftāpp. which is the set of all geodesics containing an arbitrary point p IB and orthogonal to arbitrary tangent vector Ty based the Möbius gyrovector space. As shown in Appendix C.2, they also extended the distance de between two points B into the distance from a point in Ba a Poincaré hyperplane in a closed form expression.
HYPERBOLIC NEURAL NETWORKS++
Aiming to overcome the difficulties discussed in Section 1, we build novel scheme of hy- perbolic neural networks in the Poincaré ball model. The core concept is re-generalization of (a, x)- -b b type equations with no increase in the number of parameters, which has the potential to replace any affine transformation based on the same mathematical principle. Specifically, this section starts from the reformulation of the hyperbolic MLR, from which the variants the FC, convolutional, and multi-head attention lay- ers are derived. Several other modifications are also proposed to support neural networks with broad architectures.
Hab
1l.
(a) Reformulation in R
(b) Generalization to R.!
Figure 2: Whichever pair of a and p is chosen, it determines the same discriminative hyperplane. Considering one bias point qa, per one discrimina- tive hyperplane solves this over-parameetriizaton.
3.1 UNIDIRECTIONAL REPARAMETERRIZATIO OF HYPERBOLIC MLR LAYER
Given input E R", MLR is an operation used to predict the probabilities of all target outcomes E 2,., K} for the objective variable y as a log-linear model and is described as follows:
ply k: a)) xx exp (vx (x)). where Un (x: (ax, .)) by, as Rd be R.
(1)
Circumvention of the double vectorization. To generalize the linear function ts to the Poincaré ball model, Ganea et al. (2018a) first re-parameterized the scalar term by as a vector ps E _ in the form (ax, x) by (ax, PP: + x). where by (ax: Pa). and then discussed the properties which must be satisfied when such vectors become Möbius gyrovectors. However, this causes an undesirable increase in the parameters from 1 to 2n in each class k. As illustrated in Figure (a), this reformulation is redundant from the viewpoint that there exist countless choices of pæ to determine the same discriminative hyperplane Hap.by {ar R (az; Φ) by 0). Because the
3

---

Published as a conference paper at ICLR 2021
|bll
(a) expo (A logo (a)) @c
(b) F F> Z, r) (ours)
Figure 3: Comparison of FC layers in input spaces B The values certain dimension of output spaces are illustrated as contour plots. Black arrows depict the orientation parameters, and they are fixed for the comparison. Their orthogonal curves show discriminative hyperplanes where the values are zeros. As a bias parameter b or Th changes, the outline of the contour landscape in (a) remains unchanged, whereas in (b) the focused regions are dynamically squeezed according the geodesics.
key of this step is to replace all variables with vectors attributed to the same manifold, we introduce another scalar parameter rı E R instead, which makes the bias vector Ja parallel to ag:
(ak: x) by (ak: qac."= z), where 9ax.r= rx[aa] s.t. by rs lasll
(2)
One possible realization of pæ: is adopted to reduce the previously mentioned redundancies without loss of generality or representational properties compared to the original affine transformation, and induces another notation: Ha,.,, = (xr IR" (ak: Qar, + x)) = 0} Has rx/los
I|- Based on distance d from a point to a hyperplane, Equation 2 can be rewritten as with Lebanon & Lafferty (2004) in the following form: (ass (as--a:: qar,rr + æ) sigu((as, Qaa, r + xx) d(æ;, Il Har. (aa..))aaa I)|aal I| which decomposes the inner product into the product of the norm of an orientation vector and the signed distance between an imput vector x E R and the hyperplane Har.re.
Unidirectional Poincaré MLR. Based on the observation that qaz.r starts from the origin and the concept Poincaré hyperplanes, we can now generalize dy for æ. Qa,,ra E BB and a E Taagira B
(x) sign((ax. Oe qax.ru De x)) de (xx, Hay.rr) ||a)///alll ||aallg...**
where Jar.r, erp((ri(az)), H == (ar E IB" (as, Oe Par., B. x.) = 0}, (4)
which are shown in Figure 2 (b). Importantly, the circular reference between ag Ta B and qag.ru can be unraveled by considering the tangent vector at the origin, E ToB from which is parallel transported by pe w-y TT + Ty described in Appendix C.3 as follows:
0g Po (zx) soch? (Verk) zk. Gan.ra = (xx)) = az
(5)
Combining Equations 3, 5, and 23, conclude the derivation of the unidirectional re-generalization of MLR, the parameters of which are гы E IR and N₁ E To = Rn for each class k:
ey (x) = 2 2 2 || sinh (vex, [zx]] cosh (2verk) (X 1) sinh (2Vorz)). (6)
For more detailed deformation, see Appendix D.1. Note that we recover the form of the standard Euclidean MLR in lime-so (x(xx) 4 4( æ² bk), which is proven in Appendix D.2.
3.2 REFORMUI FC LAYERS TO PROPERLY EXPLOIT THE HYPERBOLIC PROPERTIES
We next discuss the FC layers, described as a simple affine transformation = Ax - b. in an element-wise manner with respect to the output space as yi: (a,, x) by where σ, ag E R" and by IR. This can be interpreted as an operation that linearly transforms the input æ and treats the output score Y/ as the coordinate value at, or the signed distance from the hyperplane containing the origin and orthogonal to, the k-th axis of the output space R"., Therefore, combining them with a generalized linear transformation, as described Section 3.1, we can now generalize the FC layers:
Poincaré FC layer. Given an input E Bgg with the generalized linear transformation in Equation and the parameters composed of Z = {=* E ToB Rn m jk= which is generalization of and {T& E R2 mm representing the bias terms, the Poincaré FC layer outputs the following:
y F Z,r) w01 + V1 c|| | where == (c à sinh (VCu(()))
It can be proven that the signed distance from y each Poincaré hyperplane containing the origin, and orthogonal to the k-th axis, equals (g(x), (z), as shown in Appendix D.3, satisfying the aforementioned
properties. We also recover FC layer in lim yk 4 ((ar, 2²) rk ||ax]))-
4

---

Published as a conference paper at ICLR 2021
Comparison with previous method. Ganea et al. (2018a) proposed hyperbolic FC layer op- erating a matrix-vector multiplication in a tangent space and adding a bias through the following Möbius addition: y exp9 (A (g)()) @e b, which indicates that the discriminative hyperplane determined in To is projected back to BB by the exponential map at the origin. However, such surface is no longer Poincaré hyperplane, except for b = O. Moreover, the basic shape of the contour surfaces in the output space 1 is determined only by the orientation of each row vector in A, whereas their norms and a bias term b contribute to the total scale and shift. Conversely, the parameters in our method cooperate to realize more various contour surfaces. Notably, discriminative hyperplanes become Poincaré hyperplanes, i.e., the set of all geodesics orthogonal to the orientation zi and containing point expó [x)]). As shown in Figure 3, the input space B is separated in more meaningful manner as a hyperbolie space for each dimension of the output space B
3.3 REGULARIZING SPLIT AND CONCATENATION
Split and concatenation are essential operations for realizing small process branches in parallel or combining feature vectors. However, the Poincaré ball model, merely splitting the coordinates lowers the norms of the output gyrovectors and limits the representational power, and concatenating them is invalid because the norm of the output can easily exceed the domain of the ball. One simple solution is to conduct an operation in the tangent space. The aforementioned problem regarding split operation, however, remains. Moreover, as the number of inputs to be concatenated increases, the output gyrovector approaches the boundary of the ball even if the norm of each input is adequately small. The norm of the gyrovector crucial in the Poincaré ball model owing to its metric. Therefore, reflecting the orientation of inputs while preserving the scale of the norm is considered to be desirable.
Generalization criterion. In Euclidean neural networks, keeping the variance of feature vectors constant is an essential criterion (He et al., 2015). As analogy, keeping the expected values of the norms constant is a worthy criterion in the Poincaré ball because the norm of any Möbius gyrovector upper- bounded by the ball radius and the variance of the coordinates cannot necessarily remain intact when the dimensions in each layer vary. Such replacement of the statistic invariance target from each coordinate the norm also suggested by Becigneul & Ganea (2019). To satisfy this criterion, we propose the following generalization scheme with a scalar coefficient Bn = B(2 2) where B indicates a beta function.
Poincaré 3-split. First, the input E IB is split in the tangent space with integers 8.1. Σ ?4 + v logE (x) = (oT E R uN E RON)T. Each split tangent vector is then properly scaled and projected back to the Poincaré ball as follows: + = cxp6 (3n,,,''er)) v.).
Poincaré A-concatenation. Likewise, the inputs {x, B Ji=1 first properly scaled and concatenated in the tangent space, and then projected back to the Poincaré ball in the following manner: x; vz logo (z;) E Bo = (3,,8,,) uu] ...., f
y expi (v).
We prove the previously mentioned properties under a certain assumption in Appendix D.4. One can also confirm that the Poincaré 3-concatenation is the inverse function of the Poincaré 8-split.
Discussion about the concatenation. Ganea et al. (2018a) generalized a vector concatenation under the premise that the output must be followed by an FC layer, but such an assumption possibly limits its usage. Furthermore, it requires Möbius additions N times sequentially due to the noncommutative and non-associ properties of the Möbius addition, which incurs heavy computational cost and an unbalanced priority in each input gyrovector. Alternatively, our method with a pair of exponential and logarithmic maps has a lower computational cost regardless of N and treats every input fairly.
3.4 ARBITRARY DIMENSIONAL CONVOLUTIONAL LAYER
The activation of D-dimensional convolutional layers with kernel sizes of [K is generally described as an affine transformation y/ = (ag. x) - by for each channel k where x E R= K is an input vector per pixel, and is concatenation of K = II, K; feature vectors contained in receptive field of the kernel. This notation also includes a dilated operation. It is now natural to generalize the convolutional layers with Poincaré f-concatenation and Poincaré FC layer.
5

---

s=1 Published as a conference paper at ICLR 2021
Poincaré convolutional layer. At each pixel in the given feature map, the gyrovectors {x, B K contained in receptive field of the kernel are concatenated into single gyrovector æ E Bak in the manner proposed in Section 3.3, which is then operated in the same way as a Poincaré FC layer.
3.5 ANALYSIS OF HYPERBOLIC ATTENTION MECHANISMS IN THE POINCARÉ BALL MODEL
As preparation for constructing hyperbolic attention mechanisms, is necessary to theoretically consider the midpoint operation of multiple coordinates in hyperbolic space. For the Poincaré ball model and Beltrami- Klein model, Ungar (2009) proposed the Möbius gyromidpoint and Einstein gyromidpoint built upon the framework of gyrovector spaces, respectively, which represented in different coordinate systems but are geometrically the same, as shown in Appendix D.5. On the other hand, Law et al. (2019) proposed another type of hyperholic centroid based the minimization problem of the squared Lorentzian distance defined in the hyperboloid model. Based on the above situation, for the major concern of which formulation to utilize, we proved the following theorem.
Theorem 1. (The equivalence of three hyperbolic midpoints) The Möhius gyromidpoint, Finstein gyromidpoint, and the centroid of the squared Lorentzian distance exactly match each other, which indicates they are the same midpoint operations projected on each manifold.
The proofs are given in Appendix D.5 and D.6. Furthermore, based on this equivalence, we can characterize the Möbius gyromidpoint gyrometrics, which we proved in Appendix D.7.
minimizer of the weighted sum of calibrated squared
With Theorem I, we can now exploit the Möbius gyromidpoint as unified option to realize hyperbolic attention mechanisms. Moreover, we further generalized the Möbius gyromidpoint by extending the condition of non-negative weights to entire real values by regarding negative weight as an additive inverse operation: The midpoint E B of Möbius gyrovectors {b, E B B with the real scalar weights {V/ E J i==1
is given by
N Di=l Vi
Ng.b;
b, vil. = 8o 2
(8)
E |us/ (A6.
which is shown in Appendix D.8. Note that the sum of weights does not need to be normalized to one because any scalar scale is cancelled between the numerator and denominator. On the basis of this insight, in the following, we describe the construction of multi-hean attention specific example, aiming at a general approach that can be applied to other arbitrary attention schemes.
Multi-head attention. Given a source sequence S - of length Lg and target sequence T E Re of length T.,, the module first projects the target onto query Q E Le and the source onto key K E RR,, and value E TR. LR2 with the corresponding FC layers. These are split into d-dimensional vectors of h heads, which is followed by similarity function between Q' and K* producing weight. IT' = {softmax 4 qf K for 1 <i The weights are utilized to aggregate V into a centroid, giving X = Vi Finally, the features in all heads are concatenated.
Poincaré multi-head attention. Given the source and target as sequences of gyrovectors, they are projected with three Poincaré FC layers, followed by Poincaré B-splits to produce Q' {g] e Bdy K [k] Bd)
I. (2) and V = {ví Bd) Applying similarity function f and activation g, each weight Ty.. = (g) k))) is obtained and the values are aggregated as follows: x Iises, [v], o.. mf.s). Finally, the features in all heads are Poincaré @-concatenated.
For the similarity function fc there are mainly two choices to exploit. One is the inner product in the tangent space indicated by Micic & Chu (2018), which the naive generalization the Euclidean version. Another choice is based on the distance of two points: { k) -rd (q, k) 7, where an inverse temperature and 7 is a bias parameter, which was proposed by Gulcehre et al. (2018). As for the activation 9, g(x)
exp (x:) is the most basic option because it turns to be softmax operation due to the property of gyromidpoint. Gulcehre et al. (2018) also suggested g(x) = (xx). In light of the property of the generalized gyromidpoint, g as identity function is also exploitable.

---

Published as a conference paper at ICLR 2021
Table Test Fl scores for four sub-trees of the WordNet noun hierarchy. The first column indicates the number of nodes in each sub-tree for the training and test times. For each setting, we report the 95% confidence intervals for three different trials. Note that the number of parameters of the Euclidean MLR and our approach is D 1, whereas for the MLR layer of HNNs, it is 2D.
RootNode
Model
D=2
D=3
D=5
D=10
Unidirectional (ours)
60.69.14.05
67.88
86.26:1466
99.1510.16
animal.n.01 3218/ 798
HNNs
19.25:11688
1.35
85.89
99.34:10.9
Euclidean
39.96=t0.9
60.20
66.20
98.33+1.12
Unidirectional (ours)
7..27=1..0
63.90
84,36
85.60+273
group.n.01 6649 / 1727
HNNs
4..6
66.79 :-112
84.44:
86.87+122
Euclidean
47.65
55.15
71.21 1:81
81.01
Unidirectional (ours)
63.48
94.98 1:387
99.30.10.30
99.17
mammal.n 953 / 228
HNNs
16.96.113.86
95.18
98 1.229
98.751051
Euclidean
15.78:to.66
36.88+3383
60.53
65.63+2298
Unidirectional (ours)
42.60
66.70:2267
78.18
92.34:1184
location.n.0f 2689 / 673
HNNs
42.57+
62.21 +:2644
77.26
85.1441226
Euclidean
34.50+0.34
31.44
63.86
82.99433.5
4 EXPERIMENTS
In this section, evaluate methods in comparisons with HNNs and Euclidean counterparts. The implementation of hyperbolic architectures is based on the Geoopt (Kochurov et al., 2020).
4.1 VERIFICATION OF THE MLR CLASSIFICATION CAPACITY
We first evaluated the performance of our unidirectional Poincaré MLR on the same conditioned experiment designed for the MLR of HNNs, that is, a sub-tree classification on the Poincaré ball model. In this task, the Poincaré embeddings of the WordNet noun hierarchy (Nickel & Kiela, 2017) are utilized as the data set, which contains 82,115 nodes and 743,241 hypernymy relations. We pre-trained the Poincaré embeddings of the same dimensions as the experimental settings in HNNs, ie., two, three, five, and ten dimensions, using the open-source implementation" to extract several sub-trees whose root nodes certain abstract hypemymies, e.g., animal. For each sub-tree, MLR layers learn the binary classification to predict whether each given node is included. All nodes are divided into 80% training nodes and 20% testing nodes. We trained each model for 30 epochs using Riemannian Adam (Becigneul & Ganea, 2019) with a learning rate of 0.001 and a batch size of 16.
The F1 scores for the test the hyperbolic MLRs to outperform the Euclidean version in all settings, which illustrates that MLR considering the hyperbolic geometry better suited the hyperbolic embeddings. In particular, our parameter-reducee approach obtains the same level of performance as a conventional hyperbolic MLR in more stable training, as can be seen from the relatively narrower confidence intervals.
shown in Table From the results,
confirm the tendency of
4.2 AMORTIZED CLUSTERING OF MIXTURE OF GAUSSIANS WITH SET TRANSFORMERS
For the evaluation of Poincaré multi-heas attention, we utilize Set Transformer, which we consider is proper test case to eliminate the implicit influence of unessential operations, e.g., positional encoding. The task is an amortized clustering of a mixture of Gaussians (MoG). In each sample in mini-batch, models take hundreds of two-dimensional points randomly generated by the same K <<-oompneett MoG, and directly estimate all the parameters, i.e., the ground truth probabilities, means, and standard deviations, in single forward step. We basically follow the model architectures and experimental settings of the official implementation' except that we employed the hyperbolic Gaussian distribution (Ovinnikov, 2019) as well as the Euclidean distribution aiming to verify the performance of the hyperbolic architectures both for the ordinary settings and for their desirable data
7

---

Published as a conference paper at ICLR 2021
distributions. When the hyperbolic models need to deal with Euclidean coordinates, the inputs or outputs are projected by an exponential map or logarithmic map, respectively, with a scalar parameter for scaling the values to the fixed-radius Poincaré ball BT. Note that we omit ReLU activations for our models because the hyperbolic operations are inherently non-linear. We also remove normalization layers because there does not exist enough research the normalizing criterion in the hyperbolic space and existing methods possibly reduce the representational power of gyrovectors by neutralizing their norms. For the hyperbolic attentions, based on a preliminary experiment, we choose to utilize the distance based similarity function and exponential activation.
Table 2: Negative log-likelihood on the test set. For each setting, we report the 95% confidence intervals for five trials. The numbers in brackets indicate the diverged trials, the final scores of which were higher than 10.0, and those trials are not accounted into the reported scores.
Model
K=4
K=S
K=6
K=7
K=8
Gaussian distribution on Euclidean space
Set Transformer w/o LN
1.556112114))
1.912.2.:01 (2)
2.03220.19300
5.06615229
2.608
Set Transformer
1.558
1.776_:0.300 (0)
2.04610000 (0)
2.297:1007 (0)
2.5191002000
Ours
1.558
1.833=000600)
2.081 (0)
2.370 (0)
2.682200:6400
Generalized Gaussian distribution on the Poincaré ball model (Ovinnikov, 2019)
Set Transformer Ours
3.084
3.298 (0)
3.327 (0)
3.923 (0)
3.51920125(0)
2.920=a..0200 (0)
3.087=2a.014 (0)
3.252 (0)
3.375 (8)
3.46211001300
The results are shown in Table 2. For the Euclidean distribution, our models achieved almost the same performance as Set Transformers, while the training of those without Layer Normalization for the same conditioned comparison often failed under all settings. This result suggests the intrinsic normalization properties of our methods, which we attribute to the computation using vector norms. For the hyperbolic distribution, our models outperformed the Euclidean counterparts with an order of magnitude smaller confidence intervals, which indicates that our hyperbolic architectures are indeed suited to their assumed data distribution.
4.3 CONVOLUTIONAL SEQUENCE TO SEQUENCE MODELING
Finally, we experimented with the convolutional souueece-to-sequene modeling task for machine translation of WMT 17 English-German (Bojar et al., 2017). Because the architecture is composed of convolutional layers and attention layers, the hyperbolic version which has already been verified in Section 4.2, it would provide comparison focusing convolutional operations. It also has a practical aspect as a task of natural language processing, in which lexical entities are known to form latent hierarchical structures. We follow the open-source implementation of Fairseq (Ott et al., 2019), where preprocessed training data contains 3.96M sentence pairs with 40K sub-word tokenization in each language. our hyperbolic models, feature vectors are completely treated as Möbius gyrovectors because token embeddings can be learned directly on the Poincaré ball model. Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmically mapped just like hyperbolic Gated Recurrent Units proposed by Ganea et al. (2018a). We train various scaled-dows models to verify the representational capacity of our hyperbolic architectures, with Riemannian Adam for 100K iterations. For more implementation details, please check Appendix E.
Table 3: BLEU-4 scores (Papineni et al., 2002) on the test sets newstest2013. The target sentences were decoded using beam search with a beam size of five. D indicates the dimensions of token embeddings and the final MLR layer.
Model
D=16
D=32
D=64
D=128
D=256
ConvSeq2Sey
2.68
8.43
14.92
20.02
21.84
Ours
9.81
14.11
16.95
19.40
21.76
The results are shown in Table 3. Our model demonstrates the significant improvements compared to the usual Fuclidean models in the fewer dimensions, which reflects the immense embedding
8

---

Published as a conference paper at ICLR 2021
capacity of hyperbolic spaces. On the other hand, there is salient differences observed in higher dimensions, which implies that the Euclidean models with higher dimensions than a certain level can obtain a sufficient computational complexity through the optimization. This would fill the gap with the representational properties of hyperbolic spaces. also implies that the proper construction of neural networks with the product space of multiple small hyperbolic spaces using methods has the potential for the further improvements even in higher dimensional architectures.
CONCLUSION
We showed novel generalization and construction scheme of the wide range of hyperbolic neural network architectures in the Poincaré ball model, including parameter-reduced MLR, geodesic- aware FC layers, convolutional layers, and attention mechanisms. These were achieved under unified mathematical backbone based on the concepts of Riemannian geometry and the Möhius gyrovector space, which endow our hyperbolic architectures with theoretical consistency. Through the experiments, we verified the effectiveness of our approaches from diversified tasks and perspectives, such as an embedded sub-tree classification, amortized clustering of distributed points both on the Euclidean space and the Poincaré ball model, and neural machine translation. We hope that this study will pave the way for future research the field of geometric deep learning.
ACKNOWLEDGMENTS
We would like to thank Naoyuki Gunji and Yuki Kawana from the University of Tokyo for their helpful discussions and constructive advice. We would also like to show our gratitude to Dr. Lin Gu from RIKEN AIP, Kenzo Lobos-Tsuneken proofreading the manuscript for English language.
from the University of Tokyo, and Editage" for
This work was partially supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, IST CREST Grant Number JPMJCR2015, JSPS KAKENHI Grant Number JP19H01115, and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
REFERENCES
Ben Andrews and Christopher Hopper. The Ricci flow in Riemannian geometry: a complete proof of the differentiable 1/4-pinching sphere theorem. Springer, 2010.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. Multi-relational Poincaré Graph Embeddings.
In Advances in Neural Information Processing Systems 32, pp. 4463 4473, Curran Associates, Inc., 2019.
Gary Becigneul and Octavian -ugenn Ganea. Riemannian Adaptive Optimization Methods. In International Conference Learning Representations, 2019.
Thomas Blasius, Tobias Friedrich, Anton Krohmer, Soren Laue, Anton Krohmer, Soren Laue, Tobias Friedrich, and Thomas Blasius. Efficient Embedding of Scale-Free Graphs the Hyperbolic Plane. IEEF/ACM Trans. Netw., 26(2):920 April 2018. ISSN 1063-6692. doi: 10. 10.11099TN 2810186.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang. Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pp. (19-214, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic Graph Convolutional Neural Networks. In Advances in Neural Information Processing Systems 32, pp. 4868 4879. Curran Associates, Inc., 2019.
htttp:/xww.ed

---

Published as a conference paper at ICLR 2021
Hyunghoon Cho. Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large-Margh Classification in Hyperbolic Space. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp. 1832-1840. PMLR, 16-18 Apr 2019.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic Neural Networks. In Advances in Neural Information Processing Systems 31, pp. 5345-5355. Curran Associates, Inc., 201 8a.
Irtavia-EEugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. In ICML, 2018b.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional Sequence Sequence Learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 Proceedings of Machine Learning Research, pp. 1252- International Convention Centre, Sydney, Australia, 06-11 1-11 Aug 2017. PMLR.
Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. Learning tions in Product Spaces. In International Conference on Learning Representations, 2019.
Representa-
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic Attention Networks. arXiv preprint arXiv:/805.09786. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance ImageNet Classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaoru Katayama and Ernest Weke Maina. Indexing Method for Hierarchical Graphs based on Relation among Interlacing Sequences of Eigenvalues. Journal of Information Processing, 23(2): 210-220, 2015. doi: 10.2197/ipsjjip.23.21
Max Kochurov, Rasul Karimov, and Sergei Kozlukov. Geoopt: Riemannian Optimization in Py Torch. arXiv preprint arXiv:2005.02819 2020.
Dmitri Krioukov, Fragkiskos Papadopoulos, Amin Vahdat, and Marián Boguná. Curvature and temperature of complex networks. Physical Review E, 80(3):03100, 2009.
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.
John Lamping, Ramana Rao, and Peter Pirolli. A Focus+Context Technique Based on Hyperbolic Geometry for Visualizing Large Hierarchies. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, CHI '95, pp. 401-408, New York, NY, USA, 1995. ACM Press/Addison- -Weeessssnnnnnyly Publishing Co. ISBN 0-201-84705-1. doi: 10. 10.1145/223044222999
Mare Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian Distance Learning for Hyperbolic Representations. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3672-3681, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Guy Lebanon and John Lafferty. Hyperplane Margin Classifiers on the Multinomial Manifold. In Proceedings of the Twenty- First International Conference Machine Learning, ICML '04, pp. 66. Association for Computing Machinery, 2004.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set Transformer: A Framework for Attention based CermutationIIvariint Invariant Neural Networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Henry W Lin and Max Tegmark. Critical Behavior in Physics and Probabilistic Formal Languages. Entropy, 2017..
10

---

Published as a conference paper at ICLR 2021
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic Graph Neural Networks. In Advances in Neural Information Processing Systems 32, pp. -220-.241. Curran Associates, Inc., 2019.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders, In Advances in Neural Information Processing Systems 32, pp. 12565-12576. Curran Associates, Inc., 2019.
Marko Valentin Micic and Hugo Chu. Hyperbolic Deep Learning for Chinese Natural Language Understanding. arXiv preprint arXiv: 18/2./0082 2018.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A Wrapped Normal Distribution on Hyperbolic Space for Gradient- Gradient-Based Learning. In ICML, 2019.
Mark EJ Newman. Power laws, Pareto distributions and Zipf's law. Contemporary physics, 46(5): 323-351, 2005.
Maximilian Nickel and Douwe Kiela. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry. In ICMI., 2018.
Maximillian Nickel and Douwe Kiela. Poincaré Embeddings for Learning Hierarchical Repre- sentations. In Advances in Neural Information Processing Systems 30, pp. 6338-6347. Curran Associates, Inc., 2017.
Myle Ott, Sergey Edunov. Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. FAIRSEQ: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019 Demonstrations, 2019.
Ivan Ovinnikov. Poincaré Wasserstein Autoencoder. arXiv preprint wrXiv:1901.014222 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318 Association for Computational Linguistics, 2002.
Peter Petersen, S Axler, and KA Ribet. Riemannian Geometry, volume 171. Springer, 2006.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Re. Representation Tradeoffss for Hyperbolic Embeddings. In Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4460-4469, Stockholmsmässaan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Ondrej Skopek, Octavian-Eugen Ganea, and Gary Bécigneul. Mixed-curvature Variational Autoen- coders. In International Conference on Learning Representations, 2020.
Michael Spivak. A Comprehensive Introduction to Differential Geometry. Publish or Perish, 1979.
A. Tifrea, G. Becigneul, and O.-E. Ganea, Poincaré Glo Hyperbolic Word Embeddings. In 7th International Conference Learning Representations (ICLR), May 2019.
Abraham Ungar. Gyrovector Spaces And Their Differential Geometry, Nonlinear Functional Analysi and Applications, 10, 2005.
Abraham Ungar. Einstein's Special Relativity: The Hyperbolic Geometric Viewpoint. PIRT Confer- ence Proceedings, 02 2013.
Abraham A Ungar. Hyperbolic Trigonometry and its Application in the Poincaré Ball Model of
Hyperbolic Geometry. Computers & Mathematics with Applications, 41(1-2):155147, 2001.
Abraham A Ungar. Analytic Hyperbolic Geometry and Albert Einstein's Special Theory of Relativity,
World Scientific, 2008
Abraham Ungar. Beyond the Einstein Addition Law and its Gyroscopic Thomas precession: The Theory of Gyrogroups and Gyrovector Spaces, volume 117. Springer Science & Business Media, 2012.
Abraham Albert Ungar. A Gyrovector Space Approach to Hyperbolic Geometry. Synthesis Lectures Mathematics and Statisties, 1(1):1-194, 2009.

---

Published as a conference paper at ICLR 2021
A HYPERBOLIC GEOMETRY
In this section, we review the definition of the hyperbolic geometry models other than the Poincaré ball model and the relationships between their coordinates.
Hyperboloid model. The ri-dimensional hyperboloid model a hypersurface in an (n + 1)- dimensional Minkowski space RT++1 which is equipped with an inner product (ae, y) = a g for "z E [R where O = diag(- 1, 1 Given a constant negative curvature -c, the manifold of the hyperboloid model is defined by H = {x = (Zg. .n)" E [R c(2, = - To > 0}.
Note that, in this standard (n 1) -iimmensionall coordinate system, the metric tensor positive definite matrix for the n-dimensional hyperboloid manifold cannot be defined. Instead, when the hyperboloid model is represented in specific n-dimensional coordinates, e-g., hyperbolic polar coordinates, then its metric tensor has the corresponding representation as the n-dimensional positive definite matrix.
Isometric isomorphism with the Poincaré ball model. The bijection between an arbitrary point h = (z, k kT) E H& and its unique corresponding point B. depicted in Figure 1, is given by the following:
H
B b = (h)
1 Vez
H h h (b) - (2 (b), Is (b)) = ( 1 1+ c|bb2
26
B
(10)
Ve c|b|] 1 1 b|||
Beltrami-Klein model. The n-dimensionaal Beltrami-Klein model of constant negative curvature -c is defined by (IKg g*), where K2 f R" 1|2 1] and if (1 <||=)1²) 2xx T Here, K an open ball of radius 1/ L/
I, + (1
Isometric isomorphism with the Poincaré ball model. The bijection between an arbitrary point n E K. and its unique corresponding point b E B depicted in Figure 1, is given by the following:
K - B = (n))
n
(11)
1+ V cl
2b
B
K n (b)
(12)
c||bb12"
B MOBIUS GYROVECTOR SPACE
In this section, we briefly introduce the concept of the Möbius gyrovector space, which is a specific type of gyrovector spaces. For rigorous theoretical and detailed mathematical background this system, please refer to Ungar (2005; 2009; 2001; 2012).
gyrovector space is an algebraic structure that endows the points in a hyperbolic space with vector-like properties based ordinary vector spaces that provides a Euclidean space with the well-known vector operations based on the notion of groups. As particular example in physics, this helps to understand the mathematical structure of the Einstein's 's theory of special relativity where no possible velocity vectors including the sum of velocities in an arbitrary additive order can exceed the speed of light (Ungar, 2008; 2013). Because hyperbolic geometry has several isometric models, a gyrovector space also has some variants where the Möbius gyrovector space is a variant for the Poincaré ball model.
special concept called gyrogroup. This gyrogroup is similar to
As an abstract mathematical system, a gyrovector space is constructed through the following steps: (1) Start from a set G. (2) With a certain binary operation @, create a tuple called groupoid, or gyrogroup. These axioms include several important properties of gyrovector spaces, such as the left gyroassociative law and an operator called a gyrator gyr G x G - (ut @), which generates automorphism Aut(Gr G, @) 9 gyr x, yl G + G given by + gyr(æ [ y' x, called a gyration, from two arbitrary points æ and y G. The notion of the gyrocommutative law and gyrogroup cooperation given in this step. (4) Adding ten more axioms related to the statements about a real inner product space and a scalar multiplication 60, the gyrovector space (G, @, (0) is thus defined.
magma (G. (). (3) Based on five axioms, define a specific type of magma
12

---

Published as a conference paper at ICLR 2021
Some of the important properties of a gyrovector space are listed below. Here, æ. y G.
Gyroassociative laws. Although the binary operation @ is not necessarily associative in general. obeys the left gyroassociative law + (3y @ x) (æ y) @ gyr(æ, y and right gyroassociative law (ar @ u) @ 2 = x @ (y @ gyrly, x|z). These equations also provide general closed- expression of the gyrations: gyr a. y 2 O
(x @ y 2)).
Cases in which gyrations become identity maps. If at least one element for gyr is E G. the gyrations become identity map I: gyr x. gyr 00 æi| = I. With the loop properties of the gyrations given by gyr(æ, y] = gyr( + y: y) gyt a @ a:), many other cases can be also derived.
Gyrocommutative law, Although binary operation obeys the equation x: @ y = gyr(z y (y @ 2;), the gyrogroup is called gyrocommutative.
necessarily commutative in general, if
Gyrogroup cooperation. Regarding @ the primal binary addition, the second binary addition in G is defined as the gyrogroup cooperation W, which is given by æ M has duality symmetries with the first binary operation @ such that x + y a M gyrlæ, yly. In addition, corresponding to the left cancellation law @ @ (x + y) = y inherent in 4D, the gyrogroup cooperation induces two types of the right cancellation laws: (x + y) 3 y
æ @ gyrl syr|e, Gyly. This
y) a y
In this formalism, the Möbius gyrovector space is then defined as (I BBc 8c), where B is as previously introduced in Section 2, and @. and 8e are shown in the following subsections.
B. MÓBIUS ADDITION
In the Möbius gyrovector space, the primary binary operation is denoted as the Möbius addition De B B + B. which is noncommutaive and nonassociative addition, given by the following:
(1 + 20(x, y) + +cbll)) (1 -0]==/=>>))>) O y 3c (-y)-
Be y
(13)
+ 2c(x y)
B.2 MOBIUS GYRATOR
The expression of gyrations in the Möbius gyrovector space can be expanded using the equation of the Möbius addition &e. which is described by Ungar (2009) as follows:
(c(x, z) | (y. 2) (1 + 20 y))) x +
(cly, z) |0e|2 + (x, z)) y
gyrlær, y) 2 2c
20 (r, y)
(14)
By writing down all the special operators @. for the gyrovectors in By into the normal vector operations, the expression of the gyrations can be now seen as general function for the any real vector z E IR". Indeed, gyrations are extended to invertible linear maps R (Ungar, 2009).
The Möbius gyrator endows the Möbius gyrovector space with gyrocommutative nature.
B.3 MOBIUS COADDITION
The gyrogroup cooperation in the Möbius gyrovector space called the Möbius coaddition, and given by the following:
x E.e = x Be gyr[2, S.yly (1 clly/ +
[2)
With the gamma factor % = -||==|/ for E By. this is also described in the following manner:
-2 + 75 72 1 - -1'
z - y
(15)
Note that the Möbius coaddition is not associative but commutative.
13

---

Published as a conference paper at ICLR 2021
B.4 MÓBIUS SCALAR MULTIPLICATION
The Möbius scalar multiplication for æ E BE and r E I is given by the following:
tanh (r tanh (vc (x)))) [x] expo (r logö (æ)).
(16)
In terms of the Riemannian geometry, the Möbius scalar multiplication adjusts the distance of æ from the origin by the scalar multiplier The expressions of the logarithmic map logg and distance in the Möbius gyrovector space are described in the following subsections.
C POINCARÉ BALL MODEL
Owing to the algebraic structure provided by the Möbius gyrovector space, many properties related to the geometry of the Poincaré ball model can be described in implemeentaionnffiendllyy closed-form expressions.
C.I EXPONENTIAL AND LOGARITHMIC MAPS
The exponential map expe T By B described in (Ganea et al., 2018a, Lemma 2) as follows:
exp (v) +e tanh [vcgg
1
[v].
Væ E B T.
(17)
Vc
The logarithmic map log (cxp5) -1
B - To By is also given by the following:
log (z) = 2
tanh -1 (Vell E. G. vll) |9 t. y). Yx, B.
(18)
C.2 DISTANCE
C.2.1 POINCARÉ DISTANCE BETWEEN TWO ARBITRARY POINTS
The distance function de is originally and preliminary defined as binary operation for indicating the distance between two arbitrary points æ, y E B Based the notion of the Möbius addition, the distance de B. x B + R is succinetly described as follows:
d (x. v) tanh (vol O O vll) = log (y)
(19)
Vc
Despite the noncommutative aspect of the Möbius addition B.. this distance function in Equation 19 becomes commutative thanks the commutative aspect of the Euclidean norm of the Möbius addition, which is expressed as follows:
| + 2(xx y) + |/
|| O. yll
-V
1 + 20(x, y) + c [ a B
(20)
C.2.2 DISTANCE FROM A POINT TO POINCARÉ HYPERPLANE
the Euclidean geometry, the generalized concept of two-dimensional plane to higher dimensional space RM is hyperplane containing arbitrary point p IR" and is the set of all straight lines orthogonal to an arbitrary orientation vector a E R". Because straight lines Euclidean spaces geodesics in terms of the Riemannian geometry, hyperplane Riemannian manifold M such that the hyperplane contains an arbitrary point E A and the set of all geodesics orthogonal M" This concept in the Poincaré ball model has been rigorously defined in (Ganea et al., 2018a, Definition 3.1) for p E B"
be generalized as another
arbitrary orientation vector p, namely, the tangent vector
TB as follows:
Ha {a E B (log) (z), a) 0) expt (fa)*)
(21)
x E TB (O.p B. x, a) 0).
(22)
14

---

Published as a conference paper at ICLR 2021
Note that {a} is the set of all tangent vectors p and orthogonal a.
Ganea et al. (2018a) have proven the closed-form description of the distance from a point æ E ! to an arbitrary Poincaré hyperplane a.p by considering the minimum distance between a and any point in Happ'
de (ax, Hap))
( ( B. a. a) (1-cl O C x 3) |3|||||||
inf de (x) sinh weHa,
(23)
Vc
C.3 PARALLEL TRANSPORT
The concept of parallel transport is traditionally derived from differential geometry. In the hyperbolic geometry, the gyrovector space provides the algebra to formulate the parallel transport of gyrovector (Ungar, 2012). When gyrovector Gex BB w B rooted at a point E Ba is transported parallel to another gyrovector O.y @e z E BB rooted at point y E By along a geodesic connecting a and y, the equation below is satisfied:
Oey == = O.a) (O.z G. w).
(24)
Because the exponential map in the Poincaré ball model is bijective function, the parallel transported gyrovectors u and z can regarded as the exponential mapped tangent vectors v E T rooted at and u E TyBa rooted y, respectively, that is,
Yey @e expy (u) = gyrly, eca] (Eox Ge expx (v)).
(25)
With Equations 17 and 18 and the properties of the Möbius gyration described in Appendix B, succinct expression of the tangent parallel transport Pe-ty" TT + TT can be obtained as follows:
x X
P (o) (v) = u gyrly G,alv.
(26)
Note that, in a special case in which æ = 0 and v E To this equation is simplified as follows:
A x
Po Po (v)
= (1 c|ly|l3)
(27)
One can confirm that Equation 26 deserves to be called parallel transport in terms of the differential or Riemannian geometry by checking the covariant derivative associated with the Levi-Civita con- nection of Po along tangent vector field (t) smooth curve (t)) from to y vanishes to 0.
D SUPPLEMENTAL PROOFS FOR PROPOSED METHODS
D. FINAL DEFORMATION OF THE PROPOSED UNIDIRECTIONAL POINCARÉ MLR
Proof. First, clarify the relation between the Poincaré hyperplane Ha described in Appendix C.2.2, and the variants Hü,r introduced in Section 3.1:
H = A a.qag.rk
(28)
We then start the derivation of Equation 6 from the variables and qazire described in Section 3.1. Following Equation 28 and the concept of the distance from a point to a Poincaré hyperplane described in Equation 23, the generalized MIR score function eg in Equation 3 can be written as follows:
||aull
2/cleo O. x, ak)
Up (x)
sinh
x E B.
(29)
Vc
(1 cl Oe aaxiri O x112) |as
With Equation 20, we obtain
| 2- 2(x, Aas.re) + qaz. 1 20(x, rax.re ) + e² |x 9a
ee aax.r 22 2012=
(30)
15

---

Published as a conference paper at ICLR 2021
Therefore, expand the term inside the sinh -1 function in Equation 29 in the following manner.
2 2 De H, ax)
(1 cll Oe 9arrri Te x|12) |ax1
2/0 - (11 20(e,, las .r.) + cl 2|)3) (a ax) + (1 cl Aax. (13) (ae, ax²
(31)
|a
1 - -20(2, axn, + c2 || 112 c |/ [- - 2(x, Jar,rx) + 1ax 113)
2/ (1 (1-20(2) nz) + 2/|=011) (m [ax)) + (1 cll rxl19) (xx, [au])
20/2x,
(32)
- c 90 rx 2|2 cl x 1 c² |1121|1|| 12
2
Vē (1 20/2 ao + c/|== (qa lan] Velar [ax]}
(33)
1 - c||=|
9an.ra 1|
With Equations 5 and 17, the term in the outer brackets in Equation 33 can be further expanded into the form using r% and 2ą described in Section 3.1:
vc (1 2c(2, (an.rr) + cl|c=|1²) (qauork (axl) vitar, (as])
cllqar 11
(1 - 1-22ettanle (Vors) (x, [=x]) + c||2)113) tanh (verx) vela, [=x]]
(34)
tanh 2 (Verk)
(1 + c||xe112) sinh (Vers) cosh (Verk) + Velar, +1+22 2sinh (Verk))
(35)
1 + c||== sinh (2Vork) + velæ, [zx]) cosh (2Vork).
(36)
2
In addition, we also expand the term outside the sinh function in Equation 29 using Equations 5 and 17 as follows:
N qorir
|awll
2 ||(a
2 |scchh (Vers) 2x 2x||__ 2 |||
(37)
Vē (1 cl Jau.n [22
Vē(1 tanh? 2 (Verw))
Ve
Combining Equations 29, 33, 36, and 37, we finally conclude the proof through the following:
|x| ||
sinh -1 ( (1- x
(2ve(z, [zx]) cosh (2Verk)
1 + el| x|| sinh (2Ver))
ux(x)=
(38)
Ve
1 - c|
2| ||| Ve
sinh -1 (Xg (Vox, [=x]) cosh (2Verk) - (Xg sinh (2Vers)).
(39)
D.2 CONVERGENCE PROOF UNIDIRECTIONAL POINCARÉ MLR EUCLIDEAN MLR
Proof.
For the intended proof, we first introduce the following proposition:
Proposition For x of 0, sinh(r)) over x converges to in the limil x - 0:
lim sinh(x)
1
(40)
x
Proof. The result can be obtained based the definition of the differentiation of scalar function:
sinh(x)
e
E
e x
lim
lim x-
lim
(41)
2x
x
8" 0
de
lim
=
1.
(42)
dr
From Proposition 1, we derive the following two propositions.
16

---

Published as a conference paper at ICLR 2021
Proposition 2. For t R, + 0, sinh(tx) over converges to in the limit - 0:
sinh(tx))
lim
(43)
x
Proof. We divide this proof into two cases:
sinh(tx)
(1 0)
lim
sinh(tx)
(44)
lim
(t o O, Proposition 1)
to
Proposition 3. For t E IR + O, sinh sinh- (tx) over x converges to t the limit x + 0:
sinh (tx) I.
lim 2-+0
(45)
x
Proof. We can directly utilize Proposition as follows:
sinh
sinh-l(ta
ts
lim x-0
lim
(s = sinh (tz))
(46)
+0 sinh(s)
sinn((s)
-1
lim
(Proposition 1).
(47)
With Propositions 2 and 3, we can now take the limit of Equation 6 as follows:
lim (x) c+0
2/ zA|
(2ve(ar, [ze]) cosh (2Verz)
+ c||a=|1 sinh (2vorz)
lim c-0
sinh
(48)
Vc
1 cl |
c||>|
2||2
2| ||
2(xx [zx] cosh (2Verk)
1 cl d||=|| sinh (2 (2vCrt)
lim c00
sinh
Ve
(49)
Vc
c||
1 c |
vc
= 2 -| (2x (zx] 2rx) = ((x, mか} rk ||
(50)
Moreover, with Equation 5, we can confirm that N₁ matches ag in the limit c ->> 0:
1
lim a lim sech? (Verk) 2 0-
lim c
cosh (Vers) Zk Zk.
(51)
c+0
Combining with Equations and 50, we finally conclude the proof as follows:
lim un (x) ((x, an) - rh |ax = 4 ((ax, x)) - bx) where by == rk ||ax||.
(52)
c
Here, the factor is derived from the squared conformal factor (22)² degenerating into constant
value. This corresponds to the fact that the Poincaré ball model B converges to the Euclidean space Rn in the limit
except for the same multiplier lim d (Xe)² "4 owing to its metric tensor.
D.3 PROOF OF THE PROPERTIES OF OUTPUT COORDINATES OF POINCARÉ FC LAYER
Proof. To check the properties of the Poincaré FC layer described in Section 3.2, first clarify the Poincaré hyperplane containing the origin and orthogonal to the k-th axis in 1 The k-th axis is geodesic passing through the origin and any point it except the origin has non-zero element in only the k-th coordinates. Therefore, an arbitrary point x E I along the k-th axis can be written as follows:
x = (R)) where e (k) = ( E (- 1 CR.
2
1
(53)
which is as intuitive as in a Euclidean space. Specifically, - 0 represents the origin.
We then easily describe the intended Poincaré hyperplane follows:
17

---

Published as a conference paper at ICLR 2021
Definition 1. (Poincaré hyperplane containing the origin and orthogonal the k-th axis)
2 {xr (21, 22, ................................ Im) E B (e(t), x) I/k =].
(54)
which is also intuitively obtained.
With Definition the preparation for constructing y in Equation 7 is complete.
Derivation of y. Let B and = (V1 .2...... 3m)) B be the input and output of the Poincaré FC layer, respectively. Below, we start the proof with the score functions (x(x) for Vk (1,2..... m) already obtained in the same way as in Equation 6.
To endow y the properties described in Section 3.2, i.e, the signed distance from to each Poincaré hyperplane containing the origin and orthogonal to the k-th axis is equal (x(2), we generate simultaneous equation for VI as follows:
de (v. Hr ( v1 (xx).
(U.
(55)
With Equations 54 and 28 and the notion of the distance from point to Poincaré hyperplane described in Equation 23, these equations are expanded as follows:
( 2/ yk
sinh-
clull rk (x))
(56)
Vc
Therefore, obtain the following notation of the coordinates:
1 - clyll sinh (v)r) Vk.
Vi 2/e0
(57)
When considering the Euclidean norm of using Equation 57, the equation for ||y|| can be derived follows:
1 c|
m
|
Σ k=1
sinh? (V)))(
(58)
2/cC
This can be succinetly rewritten
|v = 1 - c|ly|?
||0||, where w ( sinh (Verx(z))
(59)
2
k=1
By solving this quadratic equation, the closed form of | is obtained through the following:
|
+
(60)
|
c
2
Substituting Equations 59 and 60 for Equation 57 leads to Equation in the notation of the coordinates:
Wi
yk
Wk
(61)
cl
1 / c
Confirmation of the existence of Finally, we conclude the proof by checking that is always within the domain of the Poincaré ball B (y- E Rm c|//|| 1}:
2
/
c
1
c
>
(62)
w
18

---

Published as a conference paper at ICLR 2021
D.4 PROOF OF THE PROPERTIES OF POINCARÉ 3-SPLIT AND B- B-CONCATENATION
In this section, we prove the properties of the Poincaré 8-split and the Poincaré 3-concatenation described in Section 3.3. The Poincaré ball model is different from Euclidean neural networks, on the simple calculation of the expected value and the variance of a particular value related to feature vector or weight matrix owing to the linearity in their operations. the Poincaré ball model, calculating such values without any postulate for the probabilistic distribution that the feature gyrovectors or tangent vectors follow is difficult owing the nonlinear transformations in the exponential and logarithmic maps. Thus, we first make the following naive assumption:
Assumption 1. Each coordinate of an n-dimensional tangent vector in TolB" follows a normal distribution centered at zero with certain variance
The reasons why we assume the distribution on the tangent space rather than on the Poincaré ball model itself are as follows:
1. It is improper to assume continuous and smooth distribution onto the space with upper- bounded radius because there must be no probability density on or outside the boundary. The rough idea of discontinuing such probabilities outside the domain of the Poincaré ball and discretely taking only the inside into account seems to lack rationality.
2. One simple way to avoid the above issue is to apply a uniform distribution from zero the ball radius based on the norm of the gyrovector. However, there is no guarantee that such constancy in the distribution can be realized on a complexly curved geometric structure of the Poincaré ball model.
3. Conversely, a tangent space is linear space that is attached to the manifold and can be treated as ordinary vector space.
4. The Poincaré ball model is conformal to the Euclidean space, i.e., preserving the same angles, and at the origin, the gyrovectors having the same norms are projected onto the tangent vectors which also have the same norms with their angles unchanged.
5. In Euclidean neural networks, the normal distribution is one of the most popularly considered priors. The multivariate normal distribution is occasionally approximated as an independent and identically distributed distribution for easier calculation.
Because the Poincaré 3-split and the Poincaré 3-concatenation are inverse functions to each other, is sufficient prove the properties of either one of these operations. Here, we show proof for the Poincaré B-concatenation. Recalling that 3,, = B( 2 22) and considering the following:
Poincaré @-concatenation. The input gyrovectors fa; E ) are first scaled by certain co- efficients and concatenated in the tangent space, and then projected back to the Poincaré ball as follows:
Bn BR
Bn Bax
T
34 U. (x;) E To
UN
y expo (v) B.. (63)
Proof. At first, consider the expected value of the norm of each tangent vector v, which is the target of the Poincaré 3-concatenation. Because the value 4 == <lle,]1 follows a x2 distribution based on Assumption 1, the expected value of || can be obtained as follows:
o
1
|
1 dt.
(64)
2
On;
xt -
C
dt;
(65)
2
00
2 2
#++1
2
(66)
2 R4
I
Vc
2x
One
(67)
1 2
c
19

---

Published as a conference paper at ICLR 2021
2T ons c Br.
(68)
Therefore, when the norm of each input tangent vector v; is kept the same by the former part of neural networks before applying this operation, the standard deviation Fn. must be expressed follows:
On; CBnn where C = const.
(69)
In addition, using Equation 63, the squared norm of the Poincaré 3-concatenated tangent vector v is obtained as follows:
Bn 2 Bry
82 cl C²
C c
N
|| 2
[uil |vi12 =
Σ
4
(70)
n,
This leads the value === <||a|12
where On = Cpn, which is expressed as follows:
4
(71)
Here, f. also follows a x² distribution, and the expected value of the norm of v is obtained as follows:
F|[ ( /
||0|| - 2
2t On VcB. 1-=Vol
2
(72)
Bn
which is the same as the norms of the input tangent vectors. This indicates that each coordinate of follows normal distribution centered zero with a variance L satisfying the Assumption 1.
Based on the results above, the expected value of the norm of each input gyrovector x; is expressed by the following:
1
E =|| =
|<:|||
|l
e
2 1
dt;
(73)
2T
(4
1
tanh (Vcll e- ,,,,,= -1
dt;
(74)
2d l
Vc
m -1 dli
tanh
(on, vti))
=,,2"
(75)
2 r
2
2 (22) -1) Boj (on, vli) 23-1
(1))20-1
e -
4 dli
(76)
Th 1
(2j)!
22 (22) - 1) Bajaz21 21-1
/
000
4 _ +j
c
2t;
dt;
(77)
2 a
l
11)) VC
(2j))
2 (22)- - 1) B2j02
2j-1
2 m[-]
N24
l
j+
(78)
2
r
11 Ve j=1
(2))
2
20
225 (22)
1) B2d
(22TC)
2/-1
[(#) 2j-2
T
2j T
j
14: -
(79)
(2j)
(
2
Note that, for the calculation between Equations 75 and 76, we utilize the Taylor series expansion of tanh for real value. Furthermore, considering the Laurent series expansion at infinity, we can obtain the following expressions:
4 3+
23-1
1
T
j
+
(2e)
ni
0
(80)
n
n?
r
2 4
1
(
= (2c)
+
O
(81)
r
n: 2
2 V
m
2
20

---

Published as a conference paper at ICLR 2021
Therefore, in the general cases in which n4
> 1, we obtain the following approximation:
r
25- 74 1
(
R4
r
2/
2j-1 T
=T
(82)
P
m 2
2
2
T
T
((=21
2j
2 1 V
"
+2-2
(2e)
11
(83)
23 V
n1 2
2j
r R2
2
(84)
r ni+1
R1
1--1
2j
VT (2e)
2 n2
~ 2 mm
(85)
Vi (2e) 7
(n4 + 1) 7
n4 nigl
= -2-
(20) (n4 + 1) 2
(86)
n1; (((
= nn
(87)
(n; + 1)ins
nn
ed
14
(88)
n2 + 1
(89)
(90)
Note that, for the calculation between Equations 84 and 85, we utilize Stirling's 's approximation, i.e. r(=)
In addition, we utilize the definition of Napier's constant for the approximation between Equations 88 and 89, i.e., lim + 1 =e
2x
Combining Equations 79 and 90, the expected value of ||a:|| || can be approximately expressed by the following:
00 220 (220
1) Baj (V2TC)
25-1
E||
(91)
(2j)1
j=1
(V2*C)
tanh
(92)
Ve
tanh
(93)
Vc
In the same way, the expected value of the Poincaré B-concatenated gyrovector x is obtained by the following:
El |
~ tanh (V27C)
(94)
Vc
vc tanh (VcEllloll).
(95)
which concludes the proof.
21

---

Published as a conference paper at ICLR 2021
D.5 THE MÓBIUS GYROMIDPOINT AND THE EINSTEIN GYROMIDPOINT
Einstein gyromidpoint. In the Beltrami-Klein model, the midpoint n E K among {n: E Ky and the non-negative scalar weights {v/ E J+ given as follows:
1
where vi
(96)
N
v - cl n 2
This operation is called the Einstein gyromidpoint (Ungar, 2009).
Based on the above, we prove the equivalence of the Möbius gyromidpoint and Einstein gyromidpoint.
Proof. Let the points (b; E ] correspond to {n, E K Ji=1 respectively, i.e., b, is a projection of n; to the Poincaré ball model using Equation 11. From Equations 12 and 96, obtain the following:
cl bb 1
(97)
n
1
6
Substituting Equations 12 and 97 for Equation 96 leads to the representation of the Einstein midpoint using the coordinates in the Poincaré ball model:
2b:
v;28,b0 b;
b
72
(98)
1 + c||b:112
Σ
Σ u
-1)
1 c||b;|12
Therefore, the point b E B which is projection of n the Poincaré ball model using Equation 11, is expressed in the following manner:
N
b:
b
b
Lob.
where b n
i=1
(99)
1
/ c c/bl2
N
Σ
(Xe. - -1)
This concludes the proof.
D.6 MÖBIUS GYROMIDPOINT AND CENTROID OF SQUARED LORENTZIAN DISTANCE
Weighted centroid in the hyperboloid model (Law al., 2019). With Lorentzian norm ||
Vkz, æ / = for R the center of mass h Hद among (h, (2), k
H 1==1 and the non-negative scalar weights (v, I given follows:
N
h
where h = Σ v.h.
(100)
Vcl blll'
1=1
This is based on the minimization problem of the weighted sum of squared Lorentzian distances expressed as follows:
N
h
arg mii
Us h. 12
(101)
In the following, we prove the equivalence of the Möbius gyromidpoint and the weighted centroid in the hyperboloid model.
22

---

Published as a conference paper at ICLR 2021
Proof. Expanding Equation 100 with the coordinates, obtain the following:
Σ
Vizi
h
(102)
ve
N
2
N
Vi
Σ
Viki
The point b E By, which a projection of h the Poincaré ball model using Equation 9, is expressed in the following manner:
Σ
(103)
Ve
N
Σ
Vi24
ki
Dividing both the numerator and denominator by E, M/2s, this can be rewritten follows:
N
kgki
b 1 +
0c b. where vo
i=1
(104)
1
b
N
1/21
i=1
Next, considering the points (b, E B which also correspond to (hr \ respectively, we can transform the expression of b into expression with only the coordinates in the Poincaré ball model:
Σ
ug bi Σ
U1
c
b = 2 1=1
(105)
Vi + 1b:1
Vi (Ag, 1))
c b
This concludes the proof.
D.7 MÖBIUS GYROMIDPOINT AS A SOLUTION OF THE MINIMIZATION PROBLEM
The discovery of the equivalence between the weighted centroid in the hyperboloid model and the Möbius gyromidpoint enables discuss what the Möbius gyromidpoint is a minimizer of. In the following, we prove that the Möbius gyromidpoint can be regarded as minimizer of the weighted sum of calibrated squared gyrometrics.
Theorem 2. The Möhius gyromidpoint is solution of the minimization problem of the weighted sum of calibrated squared gyrometrics, which is expressed as follows:
b = arg min
A B.ba.. Oe b @. b. b.
(106)
Each @e b @r bill indicates the norm of the respective gyrovector b, viewed from the Möbius gyromidpoint which equals the gyrodistance of b to b, and is also called gyrometric (Ungar, 2009). In addition, each Ae B. is a conformal factor of the metric tensor of the Poincaré ball model for such gyrovector. Therefore, the minimization objective in Equation 106 can be interpreted as the weighted sum squared gyrometrics, each of which is calibrated by a scaling factor at the respective point.
23

---

Published as a conference paper at ICLR 2021
Proof. Let the point b 2 be projection of the weighted centroid h Hy. With Equation 101 and the notation of Equation 10, we obtain the following straightforward expression:
b = arg min Σ Il, h(b)ll
(107)
Expanding Equation 107 with the coordinates, we obtaín the following:
N
b arg min
2
Vi
zz=(b) + (le,, h(b))
(108)
i=1
arg min
Σ
1/4
zzz(b) (he,, (b,))(()))
(109)
Ji=i which correspond to {h+ respectively, we can transform Equation 109 into an expression with only the coordinates in the Poincaré ball model:
Considering the points {b, 1
11 + b, 1 +
c||B||
2b:
2b
b arg min
1/4
cl c b, 1 6|1
(1
b
1 1 b
(110)
20² b b1|2
arg min
(111)
b
(1
b. [3) c b23
2v: e b 0. 1
arg min
(112)
cl Oe b te 1
b
arg min
A
b.b
O b 0. 6.1l?
(113)
This concludes the proof.
D.8 WEIGHT GENERALIZATION OF THE MÖBIUS GYROMIDPOINT
As mentioned in Section 3.5, we extend the condition of the weights of the Möbius gyromidpoint to all real values {u; R by regarding negative weight additive inverse operation, that is, regarding any pair (v, by) as sign
pel Aainn(os) sign(u;)
(114)
usl (Nagententos -1)
|us| (A6, -1)
E IMPLEMENTATION DETAILS
E.I PARAMETER INITIALIZATION
Unidirectional Poincaré MIR.. When the dimensions of the input gyrovector is n, each element of the weight parameter Z is initialized by normal distribution centered at zero with standard deviation n 1. The bias parameter r is initialized as a zero vector.
Poincaré FC layer. When the dimensions of the input gyrovector and the output gyrovector are and m, respectively, each element the weight parameter initialized by a normal distribution centered at zero with a standard deviation (2nm)-+ The bias parameter r is initialized as a zero vector.
24

---

Published as a conference paper at ICLR 2021
Poincaré convolutional layer. When the dimensions of the input gyrovector and the output gy- rovector are nt and m, respectively, and the total kernel size K, each element of the weight parameter Z is initialized by a normal distribution centered at zero with a standard deviation (2nKm) 2. The bias parameter r is initialized zero vector.
Embedding on the Poincaré ball model. As mentioned by Ganea et al. (2018a), we confirmed the tendency of the parameters the Poincaré ball model to adjust their angles at the first phase of the training before increasing their norms. In addition, we consider that, due to the exponentially growing distance metric of the hyperbolic space, the farther gyrovector parameter is placed from the origin, the more costly it moves such point to another point through the optimization. Therefore, the embedding parameters on the Poincaré ball model should be initialized with a particular small gain cg, given as hyperparameter, aiming to accelerate such adjustment and make the later optimization smooth. We set the value ey; to be 10 2 in the experiment in Section 4.3.
E.2 HYPERPARAMETERS OF THE EXPERIMENT IN SECTION 4.2
Optimization. We used the Riemannian Adam optimizer with 31 0.9, 32 0.999 and c 10-8 for both of the Euclidean and our hyperbolic architectures. The learning rate was set to 10 3.
E.3 HYPERPARAMETERS OF THE EXPERIMENT IN SECTION 4.3
Model architectures. Let D be the dimension of the source and target token embeddings. Each model for the experiment in Section 4.3 has the encoder and decoder, both of which are composed of five convolutional layers with kernel size of three and a channel size of D, five convolutional layers with kernel size of three and channel size of 2D, and two convolutional layers with kernel size of one and channel size of 4D. The output feature maps of the last convolutional layer in the encoder are projected into D-ddimensional feature maps. They are utilized as the key for the encoder-decoder attentions. Likewise, the output feature maps of the last convolutional layer in the decoder are projected into D-dimensional feature maps for the final token classification.
Training. In each iteration of the training phase, fed each model mini-batch containing approximately 10,000 tokens at most. In this setting, the batch size, or the number of the sentence pairs in mini-batch, dynamically changes.
As a loss function, we utilized the cross entropy function with a label smoothing of 0.1.
Optimization. We used the Riemannian Adam optimizer with 31 0.9, 32 0.98 and 10-9 for both of the Euclidean and our hyperbolic architectures. For the scheduling of the learning rate n, we linearly increased the learning rate for the first 4000 iterations warm-up. and utilized the inverse square root decay with respect the number iterations thereafter as r) (Dt) é.
25


Reviews:
[{"review_id": "t_scbfdbcl0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe paper provides a reformulation of the fundamental operations in Euclidean space that are used in neural networks for the Poincaré ball model of hyperbolic space (and thus hyperbolic space generally). The paper’s reformulation differs from previous reformulations (Ganea et al. 2018) in several ways. For multinomial logistic regression, the excess n-1 degrees of freedom available in previous formulations when defining a Poincaré hyperplane via a point on the hyperplane and a normal vector are eliminated by using a canonical choice of normal vector along with a scalar quantity corresponding to the distance to the hyperplane from the origin. Fully connected (FC) neural network layers are also reformulated in a way that keeps with the interpretation of an affine transformation as returning a point whose individual coordinates are distances to a set of different hyperplanes. On the other hand, the previous reformulation directly used Möbius matrix-vector multiplication, which does not this same interpretation. The paper then gives hyperbolic reformulations of further types of neural network operations in Euclidean space, namely split/concatenation (less computationally intensive than that in previous work), convolution (not present in previous work). Turning its focus to attention models, the paper proves a theorem regarding the equivalence of various hyperbolic midpoints proposed in previous work. Finally, the paper carries out experiments testing each part of its reformulation on appropriate datasets.\n\n------------------------------------------\nStrengths and weaknesses:\n\nThe paper gave clear motivations for each of part of its hyperbolic reformulation and explained how it differed from previous reformulations. The paper was well-written, and the authors performed regular sanity checks, such as re-deriving the Euclidean case for c -> 0. The experiments were appropriate and demonstrated that the reformulation works as well as previous reformulations (possibly better, with regard to stability). I liked the paper a lot, and I think it will definitely be of interest to people working on non-Euclidean deep learning. I’m assigning a score of 8 (a very good conference paper), and I think that the paper is more or less ready for publication as is. I’ve included a few questions below (to help my own understanding), as well as some typos I spotted whilst reading the paper.\n\n------------------------------------------\nQuestions and clarification requests:\n\n1) Which of Spivak (1979), Petersen et al. (2006), and Andrews & Hopper (2010) were you working from when writing the brief summary of Riemannian Geometry on page 2? Some of the definitions given like “An n-dimensional manifold M is an n-dimensional topological space that can be linearly approximated to an n-dimensional real space at any point x ∈ M” and “g_{x} is a positive definite symmetric matrix defined on T_{x}M” struck me as unusual (and possibly imprecise).\n2) In equation 4, you write q as the exponential of r_{k}[a_{k}], even though a_{k} ∈ T_{q}M, not T_{0}M. Is this correct?\n3) At the beginning of section 3, you write “The core concept is re-generalization of ⟨a,x⟩−b type equations with no increase in the number of parameters, which has the potential to replace any affine transformation in a shared manner.” What did you mean by “in a shared manner”?\n4) At the end of section 4.1, you write “In particular, our parameter-reduced approach obtains the same level of performance as a conventional hyperbolic MLR in a more stable training, as can be seen from the relatively narrower confidence intervals.” Do you think your approach is genuinely more stable? If so, do you have any intuition as to why performance was more stable?\n\n------------------------------------------\nTypos and minor edits:\n- Section 1, paragraph 4, sentence 1 – “Despite such progresses” -> “Despite such progress”\n- Section 2, Poincaré hyperplane – Ganea et al. 2018 certainly discusses Poincaré hyperplanes, but hyperplanes in Riemannian geometry have been studied a very long time before this.\n- Section 3.3, paragraph 1, sentence 5 – “approaches to the” -> “approaches the”\n- Section 3.3, last paragraph, last sentence – “treats every inputs fairly” -> “treats every input fairly”\n- Section 4.2, paragraph 1, sentence 2 – better to rephrase this\n- Section 4.2, paragraph 2, sentence 2 – “our models achieved the almost the same performance as Set Transformers” -> “our models achieved almost the same performance as Set Transformers”\n- Section 4.3, paragraph 1, sentence 1 – “we experimented the convolutional sequence to sequence modelling task” -> “we experimented with the convolutional sequence-to-sequence modelling task”\n- Table 2 – “on the Euclidean space” -> “on Euclidean space”\n- Section 4.3, paragraph 1, sentence 2 – “the hyperbolic version of which have already been verified” -> “the hyperbolic version of which has already been verified”\n- Section 4.3, paragraph 1, sentence 6 – “Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmic mapped just as hyperbolic Gated Recurrent Units proposed by” -> “Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmically mapped just like hyperbolic Gated Recurrent Units proposed by”\n- Conclusion, final sentence – “for the future researches” -> “for future research” / “for future researchers”", "extra_scores": {}, "extra_text": {}}, {"review_id": "sVO8F6S-MSh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "edit after rebuttal: The idea of the paper was interesting to me but some motivations or choices were unclear. After reading the rebuttal and other reviews, the authors have addressed most of my concerns. Therefore, I am ready to increase my score.\n\n======\n\nThis paper introduces an improvement of the \"Hyperbolic Neural Networks\" (HNN) proposed by Ganea et al. and published at Neurips 2018.\n\nThe authors consider in particular the multinomial logistic regression problem. The probability score of a category is determined by a linear separator and a bias term. \nIn the vanilla HNN, two sets of parameters, each of dimensionality n (i.e. 2n in total), are learned to adapt the problem to hyperbolic geometry. \nIn the submission, the authors propose to learn only n+1 parameters per category. Thanks to this reformulation, the authors propose a new generalization of the fully connected layers of neural networks to hyperbolic geometry. This allows them to define arbitrary dimensional convolutional layers in the Poincare ball model. In Ganea et al., the authors do not exploit fully the hyperbolic geometry when they introduce their bias term, their discriminative hyperplane is instead determined in some tangent space (which is not hyperbolic), and then projected back to the hyperbolic space via the logarithm map. HNN++ proposes instead the exploit the hyperbolic geometry of their discriminator.\n\nAlthough HNN and HNN++ are both hyperbolic neural networks, I think that the proposed approach should not be called HNN++ as it seems to suggest that this is an incremental extension of HNN.\nIn general, I think that the paper is hard to read. This is probably due to the 8 page format that makes some parts of the paper very dense. In particular:\n- The split operation is not motivated until the multi-head attention part (i.e. 2 subsections after it was defined).\n- What does the notation B(n/2,1/2) mean at the end of the \"Generalization criterion\"?\n- The paper contains a lot of equations and formulae, but it would benefit from having a general pipeline explaining how to code exactly the approach. It currently seems hard to reimplement the approach, even with Section E in the Appendix. Will the code be available?\n\nConcerning contributions, I do not think that the reduction of numbers of parameters from 2n to n+1 is a novel contribution, I also do not think it causes problems if the training data is large enough. However, I understand how proposing an approach that better exploits hyperbolic geometry to implement fully connected layers can help. My main concern is that the experiments do not seem to show how the proposed approach improves performance compared to HNN in practice. \n\n- Section 4.1:\nI do not understand the point of the experiments in Section 4.1. The scores obtained by HNN and HNN++ look very similar (are the scores on the right standard errors or standard deviations btw?). \nIt is mentioned in the last sentence of Section 4.1 that the \"parameter-reduced approach obtains the same level of performance as a conventional hyperbolicMLR in a more stable training.\" What do the authors mean by more stable? Can they provide more details about the stability? Do they use the same optimizer between HNNs and the proposed approach?\n\n- Why can't HNN be used as a baseline in Sections 4.2 and 4.3? Why is ConvSeq2Seq the only baseline in Table 3?\n\n- Can the authors provide an experimental analysis on the contribution 4 (arbitrary dimensional convolutional layers) in page 2? How can tuning the \"arbitrary dimensions\" of convolutional layers have impact on performance?\n\n- I have a general question about the gyromidpoints. I understand that the centroid of the squared Lorentzian distance is a minimizer over a sum of squared Lorentzian distances, just like the Euclidean centroid (average vector) would minimize a sum over squared Euclidean distance. However, is this also the case for the other types of gyromidpoints? What is the optimization problem(s) that those gyromidpoints minimize?\n\n\nIn conclusion, I would like to see an application where HNN++ is much better than HNN.\n\n\nMinor comment: There is a minor mistake in the \"Riemannian geometry\" paragraph of Section 2. It is mentioned that g_x is a symmetric positive definite matrix. \nThis statement is correct for the Beltrami-Klein model because it is the sum of 2 symmetric positive definite matrices. However, this is not correct for the hyperboloid model because g_x is a diagonal matrix (see Section A of Appendix) and it would then be positive definite if and only if all its diagonal elements were positive. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "oWtLidOJxwf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "_Summary_:\nHyperbolic Neural Networks++ extends the existing work of applying hyperbolic manifolds to neural networks. It proposes new ways to reparametrize hyperbolic multinomial logistic regression (MLR) layers to reduce the number of parameters, to generalise fully connected layers  as well as split and concat operations to be more flexible and less computationally expensive. Further they expand hyperbolic neural networks to convolutional layer and attention mechanisms. The evaluation include direct comparison to HNN, and clustering with Transformers and seq-2-seq modeling.\n\n_Strengths_: \n- Paper is well-written and easy to understand. I also appreciate a good overview over hyperbolic geometry and consistent notation.\n- The work proposed extends existing architectures with convolutional layers and attention mechanism which is prevalent in many deep architectures nowadays and thus an important contribution.\n- The evaluation included a range of different tasks and architectures.\n\n_Weaknesses_:\n- The evaluation did not convince me completely of its superiority to HNNs. This is also not really discussed.\n\n_Overall assessment_: I do believe that the extensions and generalization to Hyperbolic Neural Networks are important contributions. Therefore, I am recommending an accept.\n\n_Detailed comments and questions_:\n- Reduction of number of parameters needed for hyperbolic MLR: You correctly point out that the current formulation of hyperbolic MLR requires double the number of parameters in comparison to the Euclidean counterpart. This is reduced in this work. However, w.r.t. space complexity both are linear (=O(N)). Has the reduction in number of parameters other implications?\n- Concatenation (\"heavy computational cost\"): Can you quantify that?\n- Comparisons to HNN (Table 1): In 1/3 of the results HNN is superior to your approach? Do you have an explanation for that?\n\n_Post-rebuttal_: I do appreciate the authors addressing my comments and updating their paper. Furthermore, the authors also addresses a lot of concerns of my fellow reviewers. As this was a good paper to begin with, I am keeping my score of recommending an accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BSb-filc10P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper bolsters the library of hyperbolic neural network building blocks by defining a variety of layers in a natural way. These building blocks include a tighter parameterization of the basic fully connected layer, and the concatenate/split operations which are then used to define a version of hyperbolic multi-head attention.\n\nThe experimental section is adequate, covering standard hyperbolic applications such as embedding word hierarchies, as well as interesting tasks to validate the attention and convolutional capabilities of hyperbolic neural networks. Some details could be improved; for example, the BLEU score is difficult to interpret, the architecture and task details in Section 4.2 are sparse, and the first two experiments report only in very low dimensions which has been a toy testbed for hyperbolic models but can be unrealistic. Overall, the model shows general improvement over the baseline hyperbolic neural network models.\n\nOverall, this paper does not present fundamentally new ideas in the development of non-Euclidean geometry for deep learning, but offers solid improvements in various aspects of constructing architecture components for hyperbolic neural networks and is a useful contribution.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces new methods and building blocks to improve hyperbolic neural networks, including a tighter parameterization of fully connected layers, convolution, and concatenate/split operations to define a version of hyperbolic multi-head attention. The paper is well written and relevant to the ICLR community. The proposed methods offer solid improvements over previous approaches in various aspects of constructing hyperbolic neural networks and also extends their applicability. As such, the paper provides valuable contributions to advance research in learning non-Euclidean representations and HNNs. All reviewers and the AC support acceptance for the paper's contributions. Please consider revising your paper to take feedback from reviewers after reubttal into account.
Decision: Accept (Poster)