Laplacian Eigenspaces, Horocycles and Neuron Models on Hyperbolic Spaces
Authors: Ming-Xi Wang
Venue:  (2021)
Keywords: hyperbolic learning
hyperbolic neural network
Poincare embedding

Abstract: We use hyperbolic Poisson kernel to construct the horocycle neuron model on hyperbolic spaces, which is a spectral generalization of the classical neuron model. We prove a universal approximation theorem for horocycle neurons. As a corollary, this theorem leads to a state-of-the-art result on the expressivity of neurons of the hyperbolic MLR. Our experiments get state-of-the-art results on the Poincare-embedding tree classification task and the two-dimensional visualization of images.

Under review as a conference paper at ICLR 2021
LAPLACIAN EIGENSPACES, HOROCYCLES AND NEU- RON MODELS ON HYPERBOLIC SPACES
Anonymous authors Paper under double-blind review
ABSTRACT
We use hyperbolic Poisson kernel to construct the horocycle neuron model on hyperbolic spaces, which is spectral generalization of the classical neuron model. We prove universal approximation theorem for horocycle neurons. As a corollary, obtain of-the-r-tee-tt result on the expressivity of fa. which is used in the hyperbolic multiple linear regression. Our experiments get state-of-the-arr results the Poincare-embedding subtree classification task and the classification accuracy of the two-dimensional visualization images.
INTRODUCTION
Conventional deep network techniques attempt to use architecture based on compositions of simple
functions to learn representations of Euclidean data (LeCun al. (0155. They have achieved
remarkable successes in a wide range of applications (Hinton et al. 2012 He et al. 2016). Geometric deep learning, a niche field that has caught the attention of many authors, attempts to generalize conventional learning techniques to non-Euclidean spaces (Bronstein et al. 2017 Monti et al. 2017).
There has been growing interest in using hyperbolic spaces in machine learning tasks because they are well-suited for tree-like data representation (Ontrup & Ritter
2005
Llanis-Lobato et al. [0166]
Nickel Kiela| 2017 Chamberlain et al., 20188] Nickel & Kiela [018|SSal et al. [018 Ganea
al. 2018b] Tifrea al. 2019 Chami al. 2019 Liu al. 2019
Balazevic al. 2019: Yu & Sa
2019;
Gulcehre et al. 2019 Law et al. 2019). Many authors have introduced hyperbolic analogs of
classical learning tools (Ganea et al.) 2018a] Cho et al. 2019 Nagano et al. 2019 Grattarola et al.)
2019
Mathieu et al. 2019 Ovinnikov [2020] Khrulkov et al. 2020 Shimizu et al. 2020).
Spectral methods successful in machine learning, from nonlinear dimensionality reduction (Belkin & Partha 2002) to clustering (Shi & Malik 2000) Ng et al. 2002) to hashing (Weiss et al. 2009) to graph CNNs (Bruna al., 2014) to spherical CNNs (Cohen et al. 2018) and inference networks (Paau et al. 2019). Spectral methods have been applied to learning tasks on spheres (Cohen et al.) 2018) and graphs (Bruna et al., 22014), but not yet on hyperbolic spaces. This paper studies a spectral generalization of the FC (afline) layer on hyperbolic spaces.
Before presenting the spectral generalization of the affine layer, introduce some notations. Let (g be the inner product, |- the Euclidean norm, and activation function. The Poincaré ball model of the hyperbolic space H" H" (n22) a manifold {a=R" [<]<<]) equipped with a Riemannian metric = <(1- dxx" The boundary of H under its canonical embedding in R is the unit sphere 5" The classical neuron y=p((z: (x., +-b) is of input zER", output yER, with trainable parameters weR", beR. An affine layer R - R is concatenation of m neurons. An alternative representation of the neuron x=+p((Z, w) + is given by
r PER" ((Axx,w)e
wES--1 b.R.
This neuron is constant over any hyperplane that is perpendieular fixed direction w. In H horocycle is n-1 dimensional sphere (one point deleted) that is tangential to 5n-1, Horocycles are hyperbolic counterparts of hyperplanes (Bonola 2012 Horocyclic waves (x,w) = 1 log x= f--up
are constant over any horocycle that is tangential to S at w. Therefore,
r p(\(x, wI +b), WEST-1 1,,,,,,,bee, beR.
(2)
lf o (0. .0.....0, 0), one take ==/ /e/l|wl,
|ao|; else, one take À and any 5

---

Under review as a conference paper at ICLR 2021
Figure 1: (Left) o((:,w))) (middle) p((:,w) ()):: (right) p(aap()) In this figure, w=(1,0). (1, 0), and p is tanh. The colorbar represents function values.
generalizes the classical neuron model (T) and concatenation finitely many generalizes the FC (affine) layer. We call 2 horocycle neuron. Figure | (middle) is an example on H
The neuron models in building blocks of the Euclidean (respectively hyperbolic) Laplacian eigenspace. Moreover, many L spaces have a basis given by Laplacian eigenfunctions (Einsiedler Ward 2017). On one side, all Euclidean (respectively hyperbolic) eigenfunctions are some kind of "superposition" of (-,w))
are related to spectral theory because (+, w) (respectively (,w))) are
(respectively (,,W) H). On the other side, neural networks based on (1) (respectively P represent functions that another kind of "superposition" of w) (respectively (-. They heuristically explain why the universal approximation property is likely to hold for networks constructed by (T) and 2 By using the Hahn Banach theorem, an injectivity theorem of Helgason, and integral formula, we prove that finite sums of horocycle neurons (2) are universal approximators (Theorem (
Let E H" (H" be the tangent space of H" at E (H""), @ be the Möbius addition (Ungar 2008 We remind the reader that the following functions
(x)(2)
2 a T- 1p
sinh
x| "1111-p001
2( 2 Dz, ( -1-1-p
are building blocks of many hyperbolic learning tools (Ganea et al. 2018a Mathieu et 2019
Shimizu et al. 2020). Figure] illlstrrates examples of different neuron models 2 on H
In Lemma we shall present close relationship between (2) and (3). Using this relationship and Theorem we obtain a novel result on the expressivity of fa.p (Corollary
This article contributes hyperbolic learning. We first apply spectral methods, such as the horocycle, to hyperbolic deep learning. We prove results on the expressivity of horocycle neurons (2) and Sup (3) With horocycle neurons, we obtain state-of-the-ard results on the Poincaré-embedding subtrec classification task and the classification accuracy of the 2-D visualization of images in in the experiment.
RELATED WORK
Universal approximation There a vast literature on universal approximation (Cybenko) 1989
Horiik et al. 1989 Funahashi 1989; Leshno et al. 1993). Cybenko (1989)'s existential approach uses the Hahn Banach theorem and Fourier transform Radon measures. To prove Theorem we also use the Hahn Banach theorem, and additionally integral formula 77) and injectivity Theorem of Helgason. Generalizing integral formulas and injectivity theorems is easier than generalizing Fourier transform of Radon measures most non-Euclidean spaces. (Carroll & Dickinson 1989) uses the inverse Radon transform to prove universal approximation theorems. This method relates to ours, injectivity theorems akin to inverse Radon transforms. However, using the injectivity theorem is an existential approach while using the inverse Radon transform is a constructive one.
Spectral methods Spectral methods Bronstein al. (017):
7): Bruna al. (2014); Cohen al. (2018) use a basis of (x) given by eigenfunctions, where X is a fimite graph or the sphere. Because (.* has no eigenfunctions as a basis, our approach different from theirs.
Hyperbolic deep learning One part of hyperbolic learning concerns embedding data into the hy- perbolic space (Nickel & Kiela 2017 Sala et al. 2018). Another part concerns learning architectures with hyperbolic data as the imput (Ganea et al. (2018a); Cho et al. (2019)). Ganea et al. (2018a) proposes two ways to generalize the affine layer on hyperbolic spaces: one by replacing the linear and bias part of an affine map with (25, 26) of their paper; another by using concatenation of fi in
2

---

Under review as a conference paper at ICLR 2021
Comnspanding
Figure 2: (Left) A; are pairwise parallel lines in the sense of Gauss (Bonola 2012): (Middle) A family of horoeycles tangential to O; (Right) Each Horoeycle locus of corresponding points (Bonola 2012) [3333 of parallel lines. This fact justifies that horocycles are analogs of hyperplanes.
relevant ours. A level set of Sa.p is hypercycle that has the same distance to a chosen geodesic hypersurface, while level set of horocycle neuron is horocycle that has the same "spectral" distance to ideal point at infinity, Mathicu et al. (2019); Shimizu et al. al. (2020) build the gyroplane layer and Poincaré FC layer. Ganea et al. (2018a); Cho et al. (2019) take geodesics as decision hyperplanes, while we (initially) take horocycles. We shall construct the horocycle multiple linear regression (MLR), where decision hypersurfaces are geodesics. Geodesics decision hyperplanes (Ganea al. Ba, Cho et al. (0199) and geodesic decision hypersurfaces here arise from different methods. Khrulkov al. (2020) investigates hyperbolic image embedding, where prototypes (or models) of each class center-based. We study different one, and we shall call prototypes end-based.
their hyperbolic multiple linear regression (MLR). The latter
Based on functions similar to fa. Mathieu
HYPERBOLIC SPACES
This section reviews facts from hyperbolic geometry that used in the proof of Theorem 2 For the reader who is not interested in the proof, is enough for the implementation.
Hyperbolic metric We use the Poincaré model. The hyperbolic space H" is the manifold (xER" |x|| <1} equipped with a Riemannian metric ds² Ei= 4(1 [-]--=|²>>> - da?. Let o be the
origin of H The distance function -hp satisfies dh (o, x = arctanh |z]]_
Geodesics, horocycles and corresponding points Geodesics in H are precisely circular ares that are orthogonal 5T-1 Horocycles in H" are precisely (n- -d--e--mimnsiioall spheres that are tangential 5m-1 (Helgason 1970). Horocycles are hyperbolic analogs of hyperplanes. Figure illustrates geodesics and horocycles H²
Hyperbolic Poisson kernel
The Poisson kernel for H is P(x,w)= 1
where
rEH" WES"-I Helgason
1088). The function (,,w), defined by (x, w)
2(n - 1) log P(.,w) 2 log 1 |x
(4)
constant over any horocycle that is tangential 5 at w (Figure (middle),
Riemannian volume The Riemannian volume induced by the metric ds² on H" is
dVol = 21 "el
(5)
Horocycles Let be the set of horocycles of H and let E., be the set of all horocycles that tangential 5m-1 at w. Given λER, we let ε be the unique horocycle that connects w and tanh (A/2) w. We have E.,- = UX and =at UsES" 1 I The length of any geodesic (that ends w) line segment cut by εx, and &xa. equals (\ A2l (A.2). Therefore [Λ Ag is natural distance function defined on E.., and the map X + Exw is an isometry between R and E... This isometry is closely related (,,w) ( for any
Ex...
(x,, (),W) = A/2.
(6)
The annoying /2 in 6 is tradeoff that the metric here is different from that in Helgason (2000).

---

Under review as a conference paper at ICLR 2021
Integral formula For fixed w E S = Let d dool be the measure induced by ds² on Exar Let be family of geodesics that end at w. ō > and U=Ln (UsgaSAtd(a.w) For I E L, d Exas n Ex+6,)=0 A.2). hence d Vol(U) = Vo (U n (xww and therefore
J (())d = (1
f())dVolex. dA.
(7)
The above proof (for H is essentially the same that in (Helgason 2000 [p.37] (for H²). To further convince the reader that 7 holds for all n, we give another simple proof in A.4
Injectivity theorem With respect to the canonical measure on E. Helgason 19700 [p.13 proved
Theorem (Helgason), f L' (H)) and Jg Vo)d = for a.e & E E, then f = a.e..
Theorem demonstrates that if the integral of f (H") over almost every horocycle is zero then is also zero. This theorem and the integral formula are essential for the proof of Theorem 2
LEARNING ARCHITECTURES AND EIGENFUNCTIONS OF THE LAPLACIAN
In this section, we discuss a heuristic connection between the representation properties of eigenfunc- tions and classical neurons, and then we define some horocycle-related learning tools.
4.1 EIGENSPACES AND NEURON MODELS
On Riemannian manifold X, the Laplace-Beltrami Beltrami Lx is the divergence of the gradient, and it has a well-known representation property (Einsiedler Ward 2017): is a compact Riemannian manifold or bounded domain in R then (x*(X) has basis given by eigenfunctions. This statement false is R or H (Hislop. 1994).
Eigenspaces of on R" and H Our work is motivated by the theory of eigenspaces, in which Euclidean (respectively hyperbolic) eigenfunctions are obtained from (x,w)) (respectively (r, w) #) by some kind of superposition. For example, all smooth eigenfunctions LR" are precisely the functions (M. Hashizume & Okamoto [p.543
[(x)
e Manw) dT)),
(8)
and eigenfunctions of LH" are precisely the functions (Helgason
1.7, p.199]
f(x)
Maxww) a (T((),
gn-1
where 7' in and 9 are some technical linear forms of suitable functional spaces on Sn-1
Neuron models By 8 and Fuclidean eigenfunctions (respectively classical neurons)
superpositions of (:ww) and exp (respectively p). with homogeneity and additivity. By 9 and 2 hyperbolic cigenfunctions (respectively horocycle neurons) are superpositions of w) и and exp (respectively p). The representation property of cigenfunctions on compact manifolds and bounded
domains suggests that the universal approximation property likely hold for networks constructed
by (.,))g or (,,w).. However, this heuristic is not proof (A.5)
4.2 HOROCYCLE BASED LEARNING ARCHITECTURES
Horocycle neuron In the implementation of the horocycle neuron 2). we take 1 log
(1==0)/=> +
for (z, (). H. where is small constant to ensure numerical stability. For updating w, use the sphere optimization algorithm (Absil et al. 2008 Bonnabel] 2013) (A.6).
Horocycle feature and horocycle decision hypersurface Given non-origin point E H", for y E H we define (y) = (y. ///==///)) and call the horocycle feature attached to x. This feature useful in the Poincaré embedding subtree classification task (see the experiment and lefft). The horocycle the hyperbolic analog of the Euclidean hyperplane, and therefore could be possible choice of decision hypersurface, which may arise from a level set of horocycle feature.
4

---

Under review as a conference paper at ICLR 2021
Figure 3: (Left) Horocycle decision hypersurface. Colored points form Poincaré embedding of WordNet nouns H². The yellow point is group.n.01. Points from the subtree rooted at group.n.01 are in red, and the rest are in blue. The green horocycle separates the blue and the red. (Right) End-based clustering and geodesic decision hypersurfaces. It is an embedding of CIFAR- 10 in H². Different classes are in different colors. Thin diamonds on S prototypes. Decision regions are separated by geodesic decision hypersurfaces.
End-based clustering and end prototype Natural clustering is topic in representation learning (Bengio et al. 2013). and the common prototype-based clusters are center-based (Tan et al. (005).. We propose type of clustering that embeds high-dimensional data in H and places prototypes
in S 5
Figure right] an example for n = 2. For w E 5 and any E R. the function
E H log
+ measures the relative distance of H from w in Gromov's bordification
---
theory (Bridson & Haefliger
2009 (I18). A.18 Moreover, we define Dist H" x 5 x R R by
|z| + b 2(20 b.
(i, (,, = log
(10)
1 w|2
is a relative distance function, and this is why Dist may assume negative values and why there is a bias term in (10p. Consider classes Cls = {C1, Co...., CM j and labeled training YM)), where X E RD are D-dimensional input features and
examples {(X1, Y1), ..... (X
Y' E 21, 2.... M} Each example X' belongs to the class Cy,, In light of (10), our goal is find neural network NN, R + H that is parameterized by 0, prototypes wi,.... WM E S and real numbers .i,...,bu bM such that
IsiSN: Y arrgiin (Dist(NN 15J5M
(11)
N
maximized. We call (N و S N) the end-hased clustering and w; end prototypes (in hyperbolic geometry, the end is equivalence class of parallel lines in Figure left)). In experiments, we take NNg = Exp o NN,, where NN₃, RD + R" is a standard neural network parameterized by 0 and Exp: RD H" is the exponential map of the hyperbolic space.
Horocycle layer, horocyele multiple linear regression (MLR) and geodesic decision hypersur- faces We call a concatenation of (2) horocycle layer, and we shall carefully describe a prototypical learning framework for and-based clusterings. Using the same notions as in the previous paragraph, the classification task has M classes, and NN, Exp o NN: : R R + H is a deep network. For prototypes w1.....
........ ...,.e War E 5m-1 real numbers b1....,byy .b. R. and any example X, our feedforward for prediction will be
r NNo(X), (X,(X) = -istt( Wj, by),
(Feature descriptor)
(Scores; Similarity) (Classifier)
X E Cang (SC ISISM
The goal is to maximize the accuracy (I]), and then we need a loss function for the backpropagation. Following the convention of prototypical networks (Snell et al. 2017 Yang et al. |l.] 2018).. we choose an increasing function p (in our experiments, p(x) = x or p = tanh. and let the distribution over classes for an input (with label Y' be
PY C2lX) (X))
pC (x))),(X))
"nee often takes p(x) 2 in metric learning, which improper here because Dist(x) could he negative.
5

---

3.1]. We shall use this figure to explain an Under review as a conference paper at ICLR 2021
13
66
-4
-10
-10
Figure 4: Prediction probabilities of classifiers. Suppose the classification task has 10 classes. Let NNo R R be the feature descriptor, X the input, and x=NNx(X).. Let w;=( cos ((i-1)=/5), sin ((i-1)=/5)((1333
(Left) po(Y C1 (z) on R when score functions are Poisson neurons SC, (X) ) = BatchNNorm(P -1, (Right) po C1 on R when score functions are (X) = (z, wy)a.
Therefore, given batch of training examples, the loss function is
L -Ex log Cys X3)
(12)
#Batch
The training proceeds by minimizing I., and we call this framework a horocycle MLR. The set of parameters of the framework is {0} U [w),... war) U {b..... by J. It is worth mentioning that decision boundaries of the horocycle MLR geodesies, which follows from
1- 20
1 1- 1z
pw-wil
= 4-
(X) = log
b; log
by
w wi
o 1
x w--wyl
and the theorem of Apollonian circles
Poisson neuron and Poisson multiple linear regression (MLR) Although (x. w)H (4) is well- motivated by the theory of eigenspaces (9) and fits naturally into metric learning (see TO or also Corollary []. it is only defined on H Some readers might not be convinced that the neuron has to be defined on hyperbolic spaces. Therefore, we try to remove the log in ) and define the Poisson
neuron model by Pe (,.),((r) = p
1 --- + b) for E R R,, E R. which is well-defined on
R" Notice that if [ar| < |w] then |=P-1z 2/=//////////x In A.8 Figure illustrates
|--w|
example of Poisson neuron R In the implementation,
take
for z/? where
mo) a small constant for numerical stability. We call concatenation of Poisson neurons Poisson layer, and we use it with deep neural network NN, RDD + R to construct the Poisson MLR, which similar the horocycle MLR. Let w WM R and bi,...,bur bM R. the feedforward for prediction of our framework is
x = (N) (X) x0= Batc Py.. 1. (x)), E
(13)
1835
We let the Po PY = X) x (x) and take 12) as the loss. This framework is called Poisson MLR. We use the usual optimization algorithms to update parameters in the Poisson neuron. The BatchNorm(Ioffe Szegedy 2015) seems crucial for (13) in the experiment. Figure illustratte that high-confidence prediction regions (deep red areas) of the Poisson MLR are compact sets, in contrast to classical classifiers Hein et al. experiment in Section 6.4
5 REPRESENTATIONAL POWER
In this section, 0 is a continuous sigmoidal function (Cybenko) 1989), ReLU(Nair & Hinton, 2010),
ELU et al. 2016), Softplus(Dugaa et al., 2001). We remind the reader that p is sigmoidal lim p(l) = and lim p(t) = 0. The following theorem justifies the representational power of ト- -o
t-
horocycle neurons.
Theorem 2. Let K be a compact set in H and 1 Sppoo. Then finite sums of the form
F(x)= Σ asp wi) u w busss-l,as,.ae Ac, D;ER
(14)
dense in TP(K H)), where μ is either d Vol 5) the induced Euclidean volume.

---

Under review as a conference paper at ICLR 2021
We provide sketch of the proof here and go through the details in A.9 It suffices to prove the theorem for a sigmoidal function p and μ = d Vol, as other cases follow from this one. Assume that these finite sums not dense in LP(K,, d Vol). By the Baatn-Banach theorem, there exists some nonzero heT d Vol), where q=p/(p
1) if p>>1 and g=oc if p=1, such that fg (()))((r)d Vol(z) = 0 for all finite sums of the form (4). Extend h be a function H that is defined on H by assigning (x))=((a) if zEK and H (z)=0 if zEH" Using the property of sigmoidal functions, the bounded convergence theorem, and the integral formula 7 we prove that the integration of H on almost every horocycle is zero. By the injectivity Theorem 1, H is almost everywhere zero, which contradicts our assumption and completes the proof.
In A.10 we shall prove the same result for Poisson neurons. In A.II we prove the following lemma, which demonstrates close relationship between horocycle neurons and the widely used fá.p 3
Lemma 1. Let K be a compact set in H", 5n-1, and a E T. (HH) such that the function D(x) = (T) + d (a,, w) satisfies (K), <
There are c, c,d R. P H and
This lemma suggests that H is a boundary point of some "compactification" of the space of fa.p' The above lemma together with Theorem 2 implies
Corollary 1. Let K be compact set in H" and 15pcx0. Finite sums of the form
N Σ
F(x)
auplessa di), ps E E Tp, (H").. ai, a, E R.
are dense in (K, 10), where d Vol or is the induced Euclidean volume.
This result provides novel insights into the hyperbolic neural network (Ganea et al. 2018a), gyroplane layer (Mathieu et al. ) and Poincaré FC layer (Shimizu et al. 2020p. Although level sets of fa.p are hypercycles, our proof of Lemma rreies on the theory of horocycles. It would be interesting to have more natural approaches to treat the expressivity of fa.p
EXPERIMENTS
In this section, we first play with the MNIST toy. Next, we apply a horocycle feature to the Poincaré embedding subtree classification task. After that, we construct 2-D clusterings of image datasets by using the horocycle MLR. Finally, we provide evidence for further possible applications of the Poisson MLR. We use the framework or some functions of Tensorflow, Keras, and scikit-learn (Abadi et al.
2015
Chollet et al. 2015
Pedregosa et al. 2011
6.1
MNIST
The MNIST (LeCun et al.
1998) task is popular for testing hyperbolic learning tools (Ontrup &
Ritter 2005 Nagano et al. 2019; Mathieu et al. 2019 Grattarola et al. 2019: Ovinnikov 2020
Khrulkov et al. 2020). We train two different classifiers. A.12 A. and code contain details. The first one a single horocycle layer followed by the softmax classifier. The average test error rate after 600 epochs is 1.96%, and Theorem proovides the rationale for this experiment 13.1 The second one is a Poisson MLR. It is the best hyperbolic geometry related MNIST classifier (Table I In this table, Ontrup & Ritter (2005) uses the hyperbolic SOM, Grattarola et al. (2099) uses the adversarial autoencoder, and Khrulkov et al. (2020) uses the hyperbolic MLR. Our experiment performs well on MNIST suggests that horocycle and Poisson neurons are computationally efficient and easily coordinate with classical learning tools (such as the convolutional layer and the softmax).
Table 1: Test error rates of hyperbolic geometry related MNIST classifiers
Ontrup & Ritter 20055)
Grattarola et al. (2019)
Khrulkov et al. 2020
This paper
5.4%
4.2%
1%
0.35%
7

---

Under review as a conference paper at ICLR 2021
6.2 POINCARÉ EMBEDDING SUBTREE CLASSIFICATION
Given a Poincaré embedding (Nickel & Kiela nouns and given a node a E [WordNet noun the task is to classify all other nodes as being part of the subtree rooted
2017) PE (WordNet noun) - HD of 82114
(Ganea et al. 2018ap. Our model is logistic regression, where the horocycle feature p E (WordNet noun + hppea)) (PE())5s) (s is a hyperparameter lying in [1, 1.5) is the only predictor, and the dependent variable is whether p is in the subtree rooted at The decision hypersurface of this model is horocycle, as illustrated in Figure 3 (left). In the experiment, we pre-train three different Poincaré embedding
Blin in each of H², H² H. H For each (animal, group, location, mammal, worker) and D f2. 10), J. randomly select one of three pre-trained Poincaré embedding PE: {WordNet noun) - H and then test the model.
Table the Fl classification scores and two standard deviations of 100 trials for each {x, D}. Different Poincaré embeddings account for the most variance of the performance. Our model is different from the existing ones. Firstly, we take the horocycle as the decision hypersurface, while others take the geodesic. Secondly, we train a logistic regression on top of the horocycle feature attached to (x)) which is efficiently calculated, while others train the hyperbolic MLR with different parametrizations, On the number of parameters, we have three (independent of D), Ganea ctt al. 2a) has 2D, and (Shimizu et al. (2020) has D + 1. The number of parameters explains why our model is prominent in low dimensions.
Table 2: Average test Fl classification scores (%) for five subtrees of WordNet noun tree
RootNode
Model
H²
H
H²
H
unimal.
Ganea et
2018a
47.43:+1.07
91.92: 0.61
90.07
99. 0..29
4016 modes
Smmzzu et
2020
60.69:14.05
1..8811.18
16.26
99.15.10.46
This paper
5..3225.33
86.9842.005
98.57114.98
98.76:42.43
group.n.D1
Ganea el al
2018a
81.72:10.17
89.87
87.8910.880
91.91 1.3.07
8376 nodes
Shimi7u
74.27+1.50
63.90-1:..4
84.3611 1.79
85.60:2275
paper
90.08+12.0 .1
90.91=114 .2
97.52+1.11
97. 17.88-2 17..661..M 1.92
location.a.01
Shimizu et al
2020
12.60=2.69 69
66, 66.70:22.6
78. 966
92.31121.88
nodes
This puper
93.19:11.70
4.65
45.23:14.11
97.37:1.75
mamal.n.0
Ganca et
2018a
32.01=17.14
17.51+1.55
87.73+3.22
91.3746.00
1181 nodes
Sammzz cl al
2020
63. 63.48:4:3.6
94.98:+3.87
99.30 00:-0--0330
99. 1.55
puper
1.93
96.37 2.42
93.3418.86
93.8011.62
worker.n001
Ganea et al
2018a
12.68-10.82
24.09-
55.46115.49
66.83 11.38
1115 nodes
This paper
94.47=4.30
92. 6..533--..
95. 95.47-2271
94. 04.2.11.18
6.3 BND-BASED CLUSTERING FOR 2D DIMENSION REDUCTION
In this experiment, we use the horocycle MLR (Section to construct end-based clusterings NN R + H2 for MNIST, Fashion-MNIST Xiao et al. 2017 7). and CIFAR- 10/AA---//rizhevskk 2012). We take NN, Exp NN where Exp the exponential map of H and NN, R R is a
network with four convolutional blocks for MNIST/Fashion-MNNS or a ResNet-32 structure for CIFAR- 10. A. and code contain details.
Faucu
Figure 5: End-based clusters of MNIST, Fashion-MNIST, and CIFAR-10. Test error rates from the left to the right: 0.63%, 9.98% 8.42%. The performance for CIFAR- is better than FaSHIOn-MNIST because NN, is of ResNet-32 structure for CIFAR-10 but of only four convolutional layers for Fashion-MNiST. Thin diamonds are prototypes, and shallow areas are decision regions.
8

---

Under review as a conference paper at ICLR 2021
Figure iillustrates end-based clusterings for MNIST, Fashion-MNIST, and CIFAR-IO, 10, with perfor- 8% higher than all numbers presented in McInnes al. (2020). Moreover, Table 3 compares the numbers of Yang et al. (2018);
mance reported in the caption. Our accuracy for
Ghosh & Kirby (020), and ours for MNIST, and our methods are similar. We all use convolutional
networks the (Feature descriptor) and prototype-based functions the loss. However, Yang et al.
(2018): Ghosh & Kirby (2020) use the center-based prototype loss, while we use the end-based (12).
Yang et al. (Figur 1] points out that the traditional CNN is good linearly separating feature representations, but the learned features are of large intra-class variations. The horocycle MLR leads the inter-class separability in the same way (angle accounts for label difference) traditional CNN
does. At the same time, it also obtains intra-class compactness (Figure
Table 3: Test error rates 2D embedded MNIST by dimensionality reduction techniques
GerDA
Centroidencoder
dG-MCML
dt-MCML
CPL
GCPL
This paper
3.2%
2.61%
2.13%
2.03%
0.72%
0.67%
0.63%
6.4 POISSON MLR
Using a Poisson MLR whose feature descriptor is ResNet-32 structure, we obtain a classifier with a test error rate of 6.46% on CIFAR-10. on par with other methods with similar network structures (Yagg et al.] 2018). Moreover, we apply Poisson MLR to the classification task of flowers (Tensorflow), which is a typical example of overfitting. Replacing the MLR part of the Keras model (Tensorflow) with a Poisson MLR, the new Poisson model shows better generalization performance (Figure A.17 and code contain the details. This subsection provides evidence for further applications of horocycles.
42-
Keras model Accuracy
Keras model Validitton on Accuracy
Training Accuracy Poisson madel Validation Accuracy
Figure 6: Keras' MLR and our Poisson MLR on the task of flowers. Figure Fexplans this experiment. In the earlier epochs of the training, feature vectors NN,(X) of the Poisson model are not even close to its compact high-confidence prediction regions (deep red areas), and therefore on the test data, it is not much better than random guess. In the end, feature vectors lying in these compact regions are of small intra-class variations, which is good for generalization (Yang et al. 2018).
7 CONCLUSION
Based on the spectral theory of hyperbolic spaces, we introduce several horocycle-related learning tools. They find applications in the hyperbolic neural networks, the Poincaré embedding subtree classification task, and the visualization and classification of image datasets. We give an existential proof of universal approximation theorem for shallow networks constructed by horocycle neurons or fl.p. Hopefully, it will trigger further research the expressivity problems, such constructive approaches, quantitative results, and benefit of depth (Mhaskar & Poggio 2016). on horocycle general manifolds.
neurons, ft and similar functions

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Abadi, A. Agarwal, Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp. G. Irving, M. Isard, Y. Jia, Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, Shlens, B. Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large- scale machine learning on heterogeneous systems, 2015. URL WeKLLtttt///tesoriiiio Software available from tensorflow.org.
P.-. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.
G. Alanis-Lobato, Mier, and M.A. Andrade-Navarro. Efficient embedding of complex networks to hyperbolic space via their Laplacian. Scientific Reports, 6, 2016.
V. I. Amold. On functions of three variables. In Collected Works. Vladimir LArnold-Collected Works. Springer, 2009.
I. Balazevic, Allen, and T. Hospedales. Multi-relational Poincaré graph embeddings. In Advances in Neural Information Processing Systems 32, pp. 4463-4473, Curran Associates, Inc., 2019.
M. Belkin and N. Partha. Laplacian eigenmaps and spectral techniques for embedding and cluster- ing. In T. G. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances Neural Information Processing Systems 14, pp. 585-591. MIT Press, 2002.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEFE Trans. Pattern Anal. Mach. Intell., 1788 August 2013. ISSN 0162-8828.
Bonnabel. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 580)):227722299 2013.
R. Bonola. Non-Euclidean Geometry. Dover Books on Mathematics. Dover Publications, 2012. ISBN 9780486155036.
M. Bridson and Haefliger. Metric Spaces of Non-Positive Curvature, volume 319. 01 2009. ISBN 978-3-642-08399-00 doi: 10. 11..07/978-3-662-2244444
M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34-4):18-42, 2017.
Bruna, Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks graphs. In International Conference Learning Representations (ICLR2014), CBLS, April 2014, 2014.
S. Carroll and B. Dickinson. Construction neural nets using the Radon transform. International 1989 Joint Conference on Neural Networks, pp. 607-611 vol. 1989.
B.P. Chamberlain, J.R. Clough, and M.P. Deisenroth. Hybed: Hyperbolic neural graph embedding,
2018.
I. Chami, Z. Ying, C. Ré, and J. Leskovec. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems 32, pp. 4868-4879, Curran Associates, Inc., 2019.
H. Cho, B. DeMeo, Peng, and B. Berger. Large-margin classification in hyperbolic space. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp. 1832-184h PMLR, 16-18 Apr 2019.
F. Chollet et al. Keras. https: /ttpos://kera..oo 2015.
D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In Proceedings of the International Conference on Learning Representatioss(CLIR 20/6), 2016.
10

---

Under review as a conference paper at ICLR 2021
T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical CNNs. In International Conference Learning Representations, 2018.
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989.
C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second-order functional knowledge for better option pricing. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 472-478. MIT Press, 2001.
M. Einsiedler and T. Ward. Functional Analysis, Spectral Theory, and Applications, volume 276 of Graduate Texts in Mathematics. Springer International Publishing, Cham, 2017. ISBN 978-3-319- 58539-0, doi: 10. 1001007978---339-55555
K.-I Funahashi. On the approximate realization of continuous mappings by neural networks, Neural Networks, 193 1989.
O.-E. Ganea, G. Becigneul, and Hofmann. Hyperbolic neural networks. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems 31, pp. 5345-5355. Curran Associates, Inc., 2018a
O.-E. Ganea, G. Becigneul, and T. Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. volume 80 of Proceedings of Machine Learning Research, pp. 1646-1655. PMLR. 10-15 Jul 2018b.
T. Ghosh and M. Kirby. Supervised dimensionality reduction and visualization using centroid-encoder, 2020.
F. Girosi and T. Poggio. Representation properties of networks: Kolmogorov's 's theorem is irrelevant. Neural Computation NECO, 1:465-469, 12 1989.
D. Grattarola, L. Livi, and C. Alippi. Adversarial autoencoders with constant-curvatu latent manifolds. Appl. Soft Comput., 81, 2019.
C. Gulcehre, M. Denil, M. Malinowski, A. Razavi, R. Pascanu, K. M. Hermann, P. Battaglia, V. Bapst, D. Raposo, A. Santoro, and N. de Freitas. Hyperbolic attention networks. In International Conference Learning Representations, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
M. Hein, M. Andriushchenko, and J. Bitterwolf. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. 2019 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 41-50, 2019.
S. Helgason. A duality for symmetric spaces with applications group representations. Adv. Math., 5:1-154, 1970.
S. Helgason. Groups and Geometric Analysis: Integral Geometry, Invariant Differential Operators, and Spherical Functions. Mathematical surveys and monographs. American Mathematical Society, 2000. ISBN 9780821826737.
D. Hilbert. Mathematische probleme. In Dritter Band: Analysis Grundlagen der Mathematik Physik Verschiedenes. Springer, 1935.
G.E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6): 2012.
P. Hislop. The geometry and spectra of hyperholic manifolds. Proceedings of the Indian Academy of Sciences Mathematical Sciences, 104:715-776, 1994.
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approxi- mators. Neural Networks, 2(5):359-3666 1989.
11

---

Under review as a conference paper at ICLR 2021
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, (C,L'15, pp- 448-456. JMLR.org, 2015.
V. Khrulkov, L. Mirvakhabova, E. Ustinova, Oseledets, and V. Lempitsky. Hyperbolic image embeddings. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6417-6427, 2020.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. Y. Bengio and Y. LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings. 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
M. Law, R. Liao, J. Snell, and R. Zemel. Lorentzian distance learning for hyperbolic representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3672-3681, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521:436-444, 2015. doi: 10.1038/ nature14539,
M. Leshno, V. Ya. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
Q Liu, M. Nickel, and D. Kiela. Hyperbolic graph neural networks. In Advances in Neural Information Processing Systems 32, pp. 8230 8241. Curran Associates, Inc., 2019.
K. Minemura M. Hashizume, A. Kowata and K. Okamoto. An integral representation of an eigen- function of the Laplacian on the Euclidean space. Hiroshima Math. J., 2:535-545, 1972.
E. Mathicu, C. Le Lan, C. J. Maddison, R. Tomioka, and Y.W. Teh. Continuous hicrarchical representations with Poincaré variational auto-encoders. In Advances in Neural Information Processing Systems 32, pp. 12565-12576 Curran Associates, Inc., 2019.
L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifold approximation and projection for dimension reduction, 2020.
Mettes, E. van der Pol, and C. Snoek. Hyperspherical prototype networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 1487-1497. Curran Associates, Inc., 2019.
H. Mhaskar and T. A. Poggio. Deep vs. shallow networks An approximation theory perspective. ArXiv, hbss/6608.0387,, 2016.
F. Monti, D. Boscaini, J. Masci, E. Rodolå, J. Svoboda, and M. M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5425-5434, 2017.
Y. Nagano, S. Yamaguchi, Y. Fujita, and M. Koyama. wrapped normal distribution on hyperbolic space for gradient-based learning. In ICML, 2019.
V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Pro- ceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 807-814, 2010.
A. Y. Ng. M. I. Jordan, and Y. Weiss. On spectral clustering: Analysis and algorithm. In T. G. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances in Neural Information Processing Systems 14, pp. 846-856. MIT Press, 2002.
12

---

Under review as a conference paper at ICLR 2021
M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameett (eds.), Advances in Neural Information Processing Systems 30, pp.
Curran Associates, Inc., 2017.
M. Nickel and Kiela. Learning continuous hierarchies the Lorentz model of hyperbolic geometry.
In ICML, 2018.
R.H. Nielsen. Kolmogorov's mapping neural network existence theorem. In Proceedings of the IEEE First International Conference on Neural Networks (San Diego, CA), volume III. pp. 11-13, 1987.
Ontrup and H. Ritter. A hierarchically growing hyperbolic self-organizing map for rapid structuring of large data sets. 2005.
I. Ovinnikov. Poincaré wasserstein autoencoder, 2020.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-2836 2011.
D. Pfau, S. Petersen, A. Agarwal, D. G. T. Barrett, and K. L. Stachenfeld. Spectral inference networks: Unifying deep and spectral learning. In International Conference on Learning Representations, 2019.
Sala, C. De Sa, A. Gu, and Re. Representation tradeoff's for hyperbolic embeddings. volume 80 of Proceedings of Machine Learning Research, pp. 4460-4469. PMLR, 10-15 Jul 2018.
J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 22(8):888-905, 2000.
R. Shimizu, Y. Mukuta, and T. Harada. Hyperbolic neural networks++, 2020.
J. Snell, K. Swersky, and R.S. Zemel. Prototypical networks for few-shot learning. CoRR, abs/1703.05175, 2017. URL httthhttp///aaet
hwapr//////////// abs 1703. 170110117575755 05175
P.-N. Tan, M. Steinbach, and Kumar. Introduction Data Mining. Addison Wesley, May 2005.
Tensorflow. Image
classification
A. Tifrea, G. Becigneul, and O.-E. Ganea. Poincare glove: Hyperbolic word embeddings. In International Conference on Learning Representations, 2019.
A. A. Ungar. A gyrovector space approach to hyperbolic geometry. Synthesis Lectures on Mathematics and Statistics, 1(1):1-194, 2008
Y. Weiss, Torralba, and R. Fergus. Spectral hashing. In Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1753-1760. Curran Associates, Inc., 2009.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms, 2017.
H. Yang, X.-Y. Zhang, Yin, and C. Liu. Robust classification with convolutional prototype learning. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, 3474-3482, 2018.
T. Yu and C. De Sa. Numerically accurate hyperbolic embeddings using tiling-based models. In Advances in Neural Information Processing Systems 32, pp. 2023-2033. Curran Associates, Inc., 2019.
13

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
NOTATIONS AND SYMBOLS
Default Notations
Notation
Description
Related formula
R
The set of real numbers
R
TI dimensional Euclidean space
I E R" = (FI,...,II.) In)
(v))
Euclidean inner product
x E R y E R², RR, (x,
= -i= Tiyl
Hyperbolic analogue of (::)) Euclidean norm
x y E S S" (, = log
1-1=
1
I+]
# R" v (I,I))
H"
dimensional hyperbolic space
H fo R" [ac|
Tx(X)
Tangent space of X p
T(X)
Tangent space of X
T(X) Up) do - 1(1-|z/) -2 do
dsit"
The canonical metric on HD with curva-
ture 11
Vol
Riemannian volume on H space
dVol = 21 - |m|²) - dx dxn
dP(K,
TP(K, = {fl Pfl < oo)
II. -IIxp(K,vv))
norm
f measurable on K, || (1P = (Jxc ISPavo))
s--1
1 dimensional sphere
a set, -n--
R |>|=
P(..)
Hyperbolic Poisson kernel
I E H H E Sn1 P(a. wi = ( (1
fa.
Model in the hyperbolic ML.
(x = T-
2/el sinh -1
2(-p5r.))
dhm
The hyperbolic distance function
E
The space of horocycles
E
The set of horocycles that are tangential to g at w
Lx
Laplace-Reltrami Beltrami operator on X
hz
The horocycle feature function
hz(y) = (g,, x/ #//a))
Ex,"
The unique horocycle connecting w and tanh X/2 w.
Multiple linear regression
dim
dimension
Ix
the indicator function of K
Dist
Relative distance function Set of classes
Dist(r, -((xr, W)H Cls = (C,,,2,..... Ca,..,,,,CAl
Cls
NN;
A network parameterized by 0 network parameterized by
NN;
Exp
Exponential map of the hyperbolic space
(X+
Labeled sample
SC,
Score function
Po = C2(X)
Prediction probability
Loss function
P2 (x)) = (11mp-1ag) l=- +
Pa P...
Poisson neuron
PE
Poincaré embedding
14

---

Under review as a conference paper at ICLR 2021
Conventional symbols
Symbol
In most cases it refers
m,
integers
2, y, w
points in R td or H or real numbers
the origin of R or H real numbers
b, d, ax
real or complex number
real number, represent the timestamp in optimization
point in Sn-1
p
activation function functions
f.9
K
a compact set manifold a point in H or on a manifold an element in (H") a horocycle a measure a family of geodesics lines a geodesics line a set in H functions number of classes dimension
X
P
a
E
p
I
U
F,
M
D
A.2 PROOF OF THE ISOMETRY
Given WES and λER, we let &x,w the unique horocycle that connects and tanh (A/2) w. The length of any geodesic (that ends at w) line segment cut by Ex and Exa equals [1 Agl. This fact is obvious in the half-space model.
There is a Riemannian isometry F : {2 E R |2| < 1} + { ... .I,) : 31 > 0} (the latter such that F(w) oo and F(o) 01, Using dH (o. tanh(A, (anh(A,/2)w) = |dal. di(an) and Jm. ((1, .,...,,,, 0), (edt:: ...... 0)) F(w) oo and F(o) = (1, .........) 0), we have (tanni(X;/2)a = (e^e, ,0,, o,.......
with the metric ds² do? do o
........) 0). Therefore, maps ÊA, to {(11, 222 ... ,,, 51 e²,), Any geodesic (that ends at w) line segment cut by Ex, and Exg mapped by to {tt, 02 ......... an) an) e x²2) < 0} for some fixed aj. It is easy to check the length this segment with respect tod+ on this segment) is [} Az/-
(as a; are constants, the metric reduces du /x?
xf
A.3 PROOF OF 66
Because is on εx which is sphere with center 14tanh A/2 w and radius 1-tanh A/2 2
we have
1+tanh A/2
1-tanh A/2
which leads to |æ|| (1+ (1nt A/2)(z, w) F + tanh A/2|w/2=
2 w
2
15

---

Under review as a conference paper at ICLR 2021
and then
1+lanh A/2
x w/2
1-tanh A/2 (1w2) - |a/²)), and finally (r,w) = 1/2 log |w|²-|a|?
[z-w|=
log
1+tanh A/2
2/2.
I-tanh X/2
A.4 ANOTHERR PROOF OF THE INTEGRAL FORMULA 7
We use 4 for the upper half space model {(T),-.. Fn) : æ1 > 0} with the Riemannian volume day dannd Let w = (oo, 0.),),..., and be (1,0,...,00) 0) as (.2)) then Êxw
{111, 22,,,,, d Volss is de (respectively d of variable I1 =
σ1 ed}. The induced Riemannian metric on Exw (respectively volume 1)x For any integral function f H1 using change
Jm ((I1...., dr ......da dxn
J. (o /. fle,, I2, ,2,...,,
dx .. dxa ed end
JER /
1. I J. Vo)e (z)dA.
drg dr d.n c(u-1J2
x x2, ,n)
dd
The above identity is equivalent to the integral formula SH f(z)d R Iex f()dd Volex. (=) dd. presented in 7 according to the Riemannian isometry in (A.2).
A.5 THE HEURISTIC NOT A PROOF
The spectral theory does not directly lead to universal approximation theorems because of the following: 1, superpositions in T 2 and
are different (similarly, although another kind of superposition in Hilbert's s 13th problem (Hilbert 1935 Amoold 2009) was driving force for universal approximation theorems (Nielsen 1987). the former is hardly relevant for networks (Girosi) & Poggio 1989)) ) 2, desired representation properties of hyperbolic eigenfunctions are unknown, partially because H non-compact; 3, results in spectral theory favor Hilbert spaces, while universal approximation theorems embrace more than 2 space.
A.6 OPTIMIZATION
The parameters update for the horocycle unit ( involves the optimization problem the sphere (for w) and the hyperbolic space (for z). We use standard algorithm of sphere optimization (Absil et al.) 2008) to update w, and in the supplement we present an optimization approach based on the geodesic polar-coordinates to update z.
In the implementation of horocycle layer, the forward propagation is trivial, while the backpropaga- tion involves optimization on the sphere and hyperbolic space. In the following. 17 is the learning rate, or the value of (a may be 7), 8, 2, .....) the 1-th step, X is the tangent fiber at p. the gradient, and VH is the hyperbolic gradient. It suffices to consider the layer (=,ww))
Optimization on the sphere the sphere. The projection of aCa Vs(wr) aCe سيييي Tw, R onto Tw, S 1 is given by Absil
The parameter update of w in 8= w) involves the optimization on
de
al. 2008 (p.48)
OCe 2t W+
OCo
2t Wt
dLg 2t
(at, we Wr)
Vt Øs
2, wi Is
22
12t
w/
We
wi
Is
12t w/ 2
Two well-known update algorithms of we Absil et al.
2008 [p.76] are:
W++1 = cos (ytel U/ Jois - sin (nel
[ell -
W++1 (w) m)ttt//lus \/wus merel
16

---

Under review as a conference paper at ICLR 2021
1.30
0.90
0.8
14.75
Figure 7: Poisson neuron (1.o)
(1.0),1/3,0* The level sets of a Poisson neuron P w, are horocycles of |w]} that are tangential to far |ac| [w]] at and their inverses with respect to
the ball {x: |æ:] < {x : x [
A.7 A PROOF OF APOLLONIUS THEOREM
Theorem (Apollonius). Given distinct W1, W2 Sn and positive number X, the locus fx
J Wi| A/. w2l) is a sphere orthogonal to 5n-1
Proof. If A is one then is trivial. We assume now is not one. By Jac wil = A/e w2/. we can have
W1 Xw2 2
W1
Xwze
x
1.
1
Al2
The locus is a sphere with center O د - and radius Apollonius (in all dimension) claims that this sphere is orthogonal to 5n1 To prove this, it suffices to prove 100 = 1 + R2 (recall o is the origin of H"), which follows from
Awal 1-M XIp
The theorem of
1->
W1 Aw2 12
W1
Awz
+
1
2
A.8 INVERSION
On R" U ((oo), given the sphere {x : 2 - wel = r}, the corresponding inversion is given by
r² Wp) wo 2
I0(x) wo +
For Rd U (oo), Tu(x)) is called the inverse with respect to fx |x wol = rr.
A.9 PROOF OF THROREM 2
Theorem Let K be compact set in H", and 15p< Then finite sums of the form
N
F(x) Σ wx)()
+b),, w.ES"-1 ,0 a.E8s-l,a,,,.0 A,
are dense in LP(K, H), where is either d Vol 5 or the induced Euclidean volume.
Proof. We first treat the case is sigmoidal and μr dd Vol. Assume that these finite sums are not dense in LP(K, d. Vol). By the Hahn-Banach theorem, there exists some nonzero he d Vol), where qpp/(p- - 1) if p>1 and 9= if p=1, such that fx F(z)h(z)d Vol(z) = 0 for all fi- nite sums of the form (14. As K is compact set, by Hölder's inequality, fx [h(z)) d Vol < (Ex d (Vol))//ll//llxxxccaaa which leads hET, Vol). Extend to be function H
17

---

= Under review as a conference paper at ICLR 2021
that is defined H by assigning H(x)=h(a) if xek and H(r)=0 if z Then
HEL' (eo, (H Vol) and
J H
(()))()) =0
(15)
for all finite sums of the form (4 For any WES and X, b,R we set F.,A,())) p({((z,)) These functions are uniformly bounded, (FLA) Moreover,
lim ( = {1 is (x,w))>b, A-00
(16)
if (x,w)
According (5). for w, A, b, have SH (x))))))(()dVVll)) 0. Functions (F.. AER converge pointwise as X-x00, and they are uniformly bounded by HHEL'(HT (H",, d bounded convergence theorem, for all WES "ires",,,bRR, we have
By the
(oz)d 0.
(17)
By the integral formula 7 (with notations defined there), (6) and (17, for all b,
Jom (1em. ((2)) (z) dt 0.
(18)
Taking the derivative of L (Se... (1):) (z) dt with respect to b, we deduce from (18) that Sa... (2) 0 for a.e. bER. In other words, the integration of H a.e. E E is zero. This fact is valid for all w Therefore, the integration of H on a.c. ç E E is zero. By the a.e., which contradicts our assumption. Therefore, finite sums of the form (14) are dense in L"(K, d Vol). The case is ReLU, ELU or Softplus and μ = d Vol follows from the above case and the fact that + p(x- - 1) p(x) sigmoidal. The case μ is the Euclidean volume follows from previous cases and the fact that the Euclidean volume on compact K is bounded from above by Ad Vol for some constant
SE (2))dVols......
injectivity Theorem 1, H
A. UNIVERSAL APPROXIMATION THEOREM FOR POISSON NEURONS.
In this section, p continuous sigmoidal function (Cybenko 1989, Reiu & Hinton 2010). (leverl et al. 2016), or Softplus(Dugas et al. 2001). We also recall the Poisson neuron:
Pd (=)))())= = =p(A (x |w| |x|2
w/2 +b) WER", A.bER.
Theorem 4. Let K be a compact set in H", and 1 Spp000. Then finite sums of the form
N
a, F(x)= = Σ a WHA. (r), "yESs-l,a,,,,, A,E,RR
(19)
are dense in IP(K, R), where is either d Vol or the induced Euclidean volume.
Proof. We first treat the case is sigmoidal and μ dd Vol. Assume that these finite sums are not dense in JP(K d Vol). By the Hahn-Banach theorem, there exists some nonzero heT K, d Vol), where pp - 1) if p>1 and 9=00 if p=1, such that fx F(z))((z)dd Vol(x) = 0 for all fi- nite sums of the form 19 As K is compact set, by Hölder's inequality, fx [4(x)) d Vol < (Sx d Vol))///l/lll>ck that is defined on H by assigning H(x)=h(x) if zeK and H(x)=0 if ze Then HEL (H", (H",, Vol) and
which leads to heL' Vol). Extend h to be function H
[ (())
(20)
18

---

Under review as a conference paper at ICLR 2021
for all finite sums of the form 19 For any WES 1 E R, and >> O, set
(z) = P W.A.- -2b (x) (z)= = (* |xi|
b))
I w/2
These functions are uniformly bounded, as [u.A,,tT)]E1 Moreover,
1 if
1- Ia
lim Fa)) A
(21)
0
if
1-|=
m--
According to 20p. for w, A, b, have SH- (()))()((()) 0. Functions (F b) AER converge pointwise as X-x00, and they are uniformly bounded by (H", By the bounded convergence theorem, for all wES"=1, beR, we have
(Lm:aan)))>0000)000 (z))dVol(z)= 1
H (zz)doll(z) 0.
(22)
1 s0]
By the integral formula
(with notations defined there), 66) and 22 for all beR,
J.ee (1 ((=)d (x) dt 0.
(23)
Taking the derivative of Hogg (Se... (ol)dVolgg. (2) dt with respect to b, deduce from 23 that Jewor H (2)) Volging (2) for a.c. b>0. In other words, the integration of H on a.c. E E is zero. This fact is valid for all WE Sm- Therefore. the integration of H on a.c. ç E E is zero. By the injectivity Theorem 1,
0 a.e., which contradicts our assumption. Therefore, finite sums of the form (19) are dense in I.P(K, d Vol). The case is ReLU, ELU or Softplus and μ d Vol follows from the above case and the fact that + p(x + 1) p(x) sigmoidal. The case μu the Euclidean volume follows from previous cases and the fact that the Euclidean volume on compact K is bounded from above by Ad Vol for some constant A.
We refere the reader to the difference of (6)) and 21 and 22p. and 18 and However, basically the proofs the same. The points are the integral formula 77. the injectivity Theorem 1 and the fact that level sets of horocycle/Poisson neurons are horocycles. Moreover, as a corollary of Theorem 4 we have
Corollary 2. Let K be a compact set in R.", and 1 Spp000. Then finite sums of the form
F(z)= = Σ P (.), W a4 b,E
dense in LP(K, n). where p. is the Euclidean volume.
Proof. Because K is compact, there exists positive number R such that K fxr E R |ol < RJ. By the above theorem, finite sums of the form
N
F(x) = Σ ag Pe
(n w;E55-1 ,n, Ai, "oi
DER BER
are dense (KKR/,,)) Then the corollary follows from
po pe (...b(z)) = pp
a. (a/ R).
19

---

Under review as a conference paper at ICLR 2021
A.11
PROOF OF THE LEMMA
Recall
2/a| ( 2( - xr a) } 1- lpl
(a)
sinh
p (
(24)
(1 -
The proof of Lemma I follows from the following direct computation.
Proof. Let (0. Take pt to and a₁ -w, then we have
- ((1 2M F) + |ar)) ) + (1 )r
Pt x=
1 - 2f(w F)S + t///2
Let F((x) = 2(-p. then
[1-1-p:zzP](aa]
Fi(z)
2(-pt O F)
(w.x) o/
(1 - Pt + 22)
|-
(1- (1-211u,0) -= (1-2211
2t(1 2t(w x) + (z) 21( M) + |x
2(1 P) 21 2) +11x1 (xx, w)
(1 21( F) + 12 13) (11 -1( D) + (x0 + (1 2))z12
A,(o)/B((r),
where A,, B, are defined as the corresponding numerator and denominator. We have
A(())[==1 2/.r w
B,(2)=== = 0
(B) = 2/x (1=)2 1).
Let Gx(x) sinh (F)())) + log 1+1
1-1 then
Ax(x)) (x)
A7(x)
1 1
(1 (1-t)Ac
(1++t) + (1-1)2A(())
(x(z) = log
+ 1
+ llog
= log
B (z) (z)
1
(1 + t)
(1+t)²B((z)
By L'Hôpital's rule,
(1 (1-1)A((p) (<1,0+1 (1)
-A,(x) (1-1))(((c)
|x - w/
lim
B((x) + (1+
1=1
2
Therefore,
w22
lim Gi(z) log (41,111
1 |
For < 1, we take p: = tw,, 0g = - 04 = 1-1 dj = log + then for all x E K.
lim fa (I) + dg = lim -1 G, = log (1- |r| (41111
1
= (H.
2
2 [x - w|2
If there exists 6t C2 c2
such that (t, + d= (x.(2))/2) c2 for all / (c1, 1),x
K, then by the dominated convergence theorem, there exists t such that ||ct.a...... (x) + dg (x, w) H Hllze(K.m) < c, which proves the lemma. Note that
(1 (x)
- (1-1-
- + U(,,)
(1)))
-2 - - [| - -))(22-111---11 - -) Lig () +t)
2/r- w/t U,
t)²
2/x - -w/ -/ 1x1)) 4 Md (1+))
20

---

Under review as a conference paper at ICLR 2021
where Un and L, continuous functions defined on K x (w). There exist positive numbere C3, C4 and C1 E (0, 1) such that for all E K and t E (C1, 1),
c3 x 2 w/* - c4:
C3 _ 2/2 - w1 (1 + +) S cs.
W] -1/%,
2 ==1
63
L(x, w) (1 +t]-
2
2
Therefore, for x E and I E (c1, 1), we have
C3
I (1 ()-())((x)
204 + C3. (c(z)
2c1
C3
This implies that for E (c,, 1), G.lx and therefore Ica + dellx are uniformly bounded, which finishes the proof of the lemma.
A.12 THE FIRST MNIST CLASSIFIER
At the preprocessing stage, we compute the projection of the 28 x 28 input pattern the 40 principal components and then scale them so that the scaled 40-dimensional PCA features are within the unit ball. In network,
1. Input layer: scaled 40-dimensional PCA features;
2. First layer: 40 inputs/1000 outputs horocycle layer (tanh activation);
3. Last layer: 1000 inputs/10 outputs affine layer;
4. Loss: cross entroy loss.
Take learning rate 1, learning rate decay 0.999, and batch size 128, and run it three times. The average test error rates after 600 epochs is 1.96%.
PCA follows LeCun et al. (1998 (1998) (C.3), where 40 PCA is used for the quadratic network. Quadratic network has a similar structure to ours, because our neuron are contructed by quotient of quadratic functions followed by log.
A.13 HOROCYCLE LAYER FOLLOWED BY MLR CAN APPROXIMATE THE CLASSFICATION FUNCTION
Suppose the MNIST classification function M is defined on _g_ C H0 where K; are relatively compact and M|K, = j. By Theorem 2 for 05j59, there ex- ist Fạ(x) - a3ap(Aj, (x,,.)) +bb) such that F2 approximates TK,, where T is the indicator function. Therefore, a network with the first (horocycle) layer given by p()z, (2, w -b, by ( followed by classical MLR with parameters given by IsiSN, (with arg max for prediction) approximates. M.
A. 14 THE SECOND MNIST CLASSIFIER IN 6.1
At the preprocessing stage, we do data augmentation by letting each image step toward each of corners, so that our traning set has 300000 examples. In our network.
1. Input layer: (28,28, 1);
2. First block: 32-filters 3 x convolution, ReLU, 2 x 2 max-pooling, BatchNorm;
3. Second block: 64-filters
convolution, ReLU, BatchNorm;
4. Thrid block: 64-filters 3 x 3 convolution,Rel.UU max-pooling, BatchNorm;
21

---

Under review as a conference paper at ICLR 2021
5. Fourth block: 128-filters 3 x 3 convolution, ReLU, 2 x 2 max-pooling, BatchNorm;
6. Fifth block: FC 1000, ReLU, BatchNorm;
7. Last block: 1000 input/10 output Poisson layer, sigmoid, BatchNorm;
8. Loss: cross entroy loss.
In optimization, we take Adam Kingma & Ba 2013). The batch size is 128 the first 5 epochs, and 1024 in the next 15 epochs. After 5 epochs, we w, in the Poisson layer to be non-trainable. We train our network five times, the average test error rate after 20 epochs 0.35%.
The in | =|" is an important hyperparameter for the numerical stability. We train this MNIST [z-w|=+e model with E {10-1, 10-2,0220000 100 10-6, 10- 10-10, 10 10-2"). They all show robust performance.
A.15 EXPERIMENT OF POINCARE TREE CLASSIFICATION TASK
Given a Poincaré embedding (Nickel & Kiela 2017) PE: : (WordNet noun) + HD of the 821 14 WordNet noun nodes and given node F, the task is classify all other nodes being part of the subtree rooted at x (Ganea et al. 2018a) Our model is logistic regression, where the horocycle feature p [WordNet noun) + (x) (PE(p)/s) (s is a hyperparameter lying in [1, 1.5)) is the only predictor, and the dependent variable is whether is in the subtree rooted at x. Let be the set of all nodes in the Poincare embedding, and let p range from P.
1. Input: (PE(p)))))))) (s is hyperparameter.)
2. Only layer: input/1 output affine layer. (two parameters: one for input, one for bias.)
3. Loss:
Logistic. (with respect to 1 if p in the tree rooted at 2;; 0 else.)
In each training, a is one of (animal, group, location, mammal, worker), ), dim is one of 12.3,5,10) and Poincaré embeddings are from the animatioontrain..py of Ganea et al. (0118)) D (with tree=wordnet_ f_ulll model=poincare, dim=dim, seed randomly E 8, 9)), All nodes in the subtree rooted are divided into training nodes (80%) and test nodes (20%). The same splitting procedure applies for the rest nodes. We choose that has the best training F1, and then record the corresponding test F1. For each x and dim, we do the training 100 times. The average test F1 classification scores are recorded in 2
The horocycle feature performs well here because it is compatible with the Poincaré embedding algorithm. Let x be node that is not at the origin. It seems that the Poincaré embedding algorithm tends to pull all nodes that are from the subtree rooted at x towards the direction of æ therefore [
y
y
is a suitable feature for this task.
A. END-BASED CLUSTERING IN H
For MNIST, at the preprocessing stage, we do data augmentation by letting each image step toward each of its 4 corners, so that our traning set has 300000 examples. Our network for H2 embedding of MNIST dataset is
1. Input layer: (28,28, 1);
2. First block: 32-filters 3 x convolution, ReLU, 2 x 2 max-pooling, BatchNorm;
convolution, ReLU, BatchNorm; 4. Thrid block: 64-filters 3 x 3 convolution,,eLU.22 max-pooling, BatchNorm;
3. Second block: 64-filters
5. Fourth block: 128-filters 3 x convolution, ReLU, x 2 max-pooling, BatchNorm;
6. Fifth block: FC 1000, ReLU, BatchNorm;
7. Sixth block: FC 2, ReLU, BatchNorm, Exp;
8. Last block: 2 input/10 output horocycle layer, sigmoid;
22

---

Under review as a conference paper at ICLR 2021
9. Loss: cross entroy loss,
where Exp is the exponential map T. (= R²) - H². We apply the data augmentation as in A.14 In optimization, learning rate is 0.1, learning rate decay is 0.99, batch size 128, epochs 50.
Our network, data augmentation and optimization for H² embedding of Fashion-MNIST dataset is completely the same as that for MNIST.
For MNIST and Fashion-MNISTT we use sphere optimization. We would like to remark that there are interesting new features in sphere optimization. Because the S' compact, for any continuous function f. there exists = argmazs:: f. The derivative of f at vanish, so the usual optimization algorithm find the minimum will fail in the general case. In our experiments, we solve this problem by adding the following tricks:
1. Observation: the class C. are all close to w E S and the end prototype wa for the class C. is around -w, then wa maximum point of the loss function and therefore can not be improved through normal SGD. We solve this problem by adopting an idea (speervidd variation) of k-means clustering. In each early epochs, optimization consists of two parts. In the first part, the normal SGD applies. In the second part, we move end prototypes (w;) to the average direction of the class (using training data).
2. Observation: if the class C. and class C3 are all close to w S!, and the end prototype Wa Wy are also both around w, then all points in class C. and class C3, end prototypes Wa we will all be pulling w by the SGD, and finally the network can not distinguish class Co and class Cg. We solve this problem by adding a loss if two prototypes are close.
With these small tricks, our 2D end-based clustering algorithm is very stable for MNIST and Fashion- MNIST. We run it MNIST 10 times, and they all get test acc around 99% within 20 epochs.
Suppose the classification task has M classes and the prototype of the i-th class is wr. We write down the additional loss function for the second observation follows
i (hancooChoice((,..... M})
RandomChoice/ ((l. M} {i})
(wi, wi))
LObservation2 arctanh (10 ReLU 0.9 - c)),
where 6 is small constant for numerical stability.
For CIFAR-10, our network for 2² embedding of CIFAR-10 dataset
1. Input layer: (32,32, 3);
2. First block: ResNet-32/128 output;
3. Second block: FC 2, ReI.U, BatchNorm, Exp;
4. Last block: 2 input/10 output horocycle layer;
5. Loss: cross entroy loss.
In the data augmentation, we apply horizontal/vertical shifts and horizontal flip. We use Adam. The batch size is 32 in the first 100 epochs, or 1024 in the next 50 epochs. The weights of the horocycle layer are fixed at the beginning of the training and are non-trainable, which follows an idea of Mettes et al. (019))
A.17 POISSON MLR
For CIFAR-IO, we use a ResNet-32 structure as the feature descriptor, and we apply corizonnallvertice shifts and horizontal flip. In our network,
1. Input layer: (32,32, 3);
2. First block: ResNer-32/128 output;
3. Second block: FC 128, ReL.U, BatchNorm;
23

---

Under review as a conference paper at ICLR 2021
4. Last block: 128 input/10 output Poisson layer, BatchNorm;
5. Loss: cross entroy loss.
use Adam. The batch size 32 in the first epochs, or 1024 in the next 20 epochs. Test acc greater than 93.5%.
For the classification task flowers (Tensorlow). The dataset of 3670 photos of flowers contains classes: daisy, dandelion, roses, sunflowers and tulips. The keras model is
1. Input layer: (180,180, 3);
2. First block: 16-filters 3 x convolution, ReLU, x max-pooling;
3. Second block: 32-filters
convolution, ReLU, max- max-poolings
4. Thrid block: 64-filters 3 x convolution,,elL122 2 max- max-pooling;
5. Fourth block: FC 128, ReLU;
6. Last block: 128 input/10 output FC layer;
7. Loss: cross entroy loss.
Our Poisson model is
1. Input layer: (180,180, 3);
2. First block: 16-filters 3 x convolution, ReLU, 2 x 2 max-poolle max-pooling;
3. Second block: 32-filters 3 convolution, ReLU, 2 max- max-pooling:
4. Thrid block: 64-filters 3 x 3 Cenvolution.,eLL122 2 max- max-pooling;
5. Fourth block: FC 128, ReLU;
6. Last block: BatchNorm, 128 input/10 output Poisson layer, sigmoid, BatchNorm;
7. Loss: cross entroy loss.
We 2936 photos for training and the rest 734 for testing. We train two models 5 times, and Figure 6 records the average of training and test accuracies in 20 epochs. Although the test accuracy of the Poisson model bad in the beginning, it fairly higher (around 4%) than the test accuracy of the keras model in the end.
A.18 Dist IS A RELATIVE DISTANCE FUNCTION
As H is complete metric space, the absolute distance between any pair pf points (r,ww H" x 5n-1 is always +00. This absolute distance is not useful, hence we look for relative one. Moreover, if a function reasonably measures the relative distance of H from w 5n-1 then so does D + for any constant c. This section explains why Dist (10) is reasonable relative distance function. For (I, H 5 R, we recall
|z// J w/2
Dist log
+ 2((2,)) + b.
A.18.1 A NAIVE VIEWPOINT
For fixed Sn-1 and R. Dist(s, defined on H and its level sets are the of horocycles that are tangential to S at w: E = (. For X E R, using (6), we have for all εx,w,
Dist (,,, - +
The above identity implics the following: æ1,22 are on same horocycle εx.w then Dist(æ1, b) = Dist( w, the horocycle Exw moves to w), then for E Ex, the function value Dist(r, w. b) goes to -o;; À moves to oo (equivalently speaking, the Êx,s moves away from w), then for Ex, the function value Dist(a, w. b) goes to Dc. The above observation heuristically explains that Dist(,,w,b measures the relative distance of H from wt
- + b. Moreover, as À moves to (equivalently speaking,
24

---

Under review as a conference paper at ICLR 2021
A.18.2
THE BUSEMANN FUNCTION VIEWPOINT.
Fix Sm-1, and then let e [0, 00) H be the unique geodesic ray (with unit speed) that satisfies
c(0 = (o......). d(oo) = w.
For the purpose of this section, we do need the definition of Busemann functions (we refer the interested reader to Bridson & Haefliger| (200ggeer((00)))) (110)) Instead, it suffices to know the following result of the theory: dh- be the hyperbolic distance functiin, and for any x H
lim (d (I, c(t)) dи (c(0), c(t)) ((,, w)H.
(25)
We read the above 25 in the following way: for fixed
H - dH (z)) dh (c(0),,(())
is function that measures the relative distance of (x. c(0)} from c(t), and therefore the left hand side of (25)
H lim (dir e()) dh (c(O),
(-000
a function that measures the relative distance of (I,, c(0) from lim c(t) = w. and finally the right hand side of (25)
EEH H" + -2x
a function that measures the relative distance of {z, c(0) from
Moreover, if the geodesic ray e starts from a different E H and then there will be a corresponding bias term added to 255). Therefore, for any b R and E S
H Dist w. b) -2x w) + b
is a function that measures the relative distance of H from
25


Reviews:
[{"review_id": "uCxedy8fCxh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes new neural models for hyperbolic space, which unlike previous hyperbolic NN works, relies on the notion of horocycle in the Poincare disk. This novel framework has connections to spectral learnig in hyperbolic space. Representation theorems alla Cybenko for layers constructed from these neurons are presented. Finally, various experiments on clustering and classifying datasets using these neurons to generate hyperbolic embeddings are presented. \n\nWith the caveat that this paper is outside my main area of expertise, I must say that I have mixed feelings about it. On the one hand, I want to like it - the topic is quite interesting and timely, the theoretical connections are intriguing, the representation results seem quite remarkable, and the experiments seem to suggest (modulo some questions I have, see below) that this is a promising approach. On the other hand, the writing, dry exposition, utter lack of discussion or intuition for most results, and the confusing setup of the experiments make it hard to produce a confident assessment. In addition, these drawbacks probably imply that the paper might not be accessible but to a few niche in the community, and might have a very limited impact. \n\nFor the reasons above, I'm leaning towards rejection, but I think that this could be a very solid paper if: (i) the results hold, (ii) the writing and exposition is improved, and (iii) the results are better discussed and motivated .\n\nStrengths:\n* Interesting problem in a flourishing but not-yet-too-crowded corner of the representation learning literature\n* Seemingly very strong theoretical results (representation theorems for neural nets in hyperbolic space)\n* Seemingly very convincing experimental results, outperforming alternative methods by wide margins\n\nWeaknesses:\n* The paper needs thorough rewriting. There's various typos, confusing grammatical choices, and overall, confusing writing. \n* Besides grammar, etc, the paper needs to be written with an ICLR audience in mind, most of which might not be experts in hyperbolic geometry, so more hand holding is needed. \n* The paper needs restructuring. Too much space is devoted to listing prior results without further explanation or discussion (e.g. Theorem 1 - what's the importance, implication of this result?). In turn, the contribution of this paper, mostly contained in Section 4.2, could benefit from more detailed discussion and motivation. In particular, I find sentences like \"Suppose this Poisson neuron is non-trainable ... \" very confusing. I have no idea what this whole sentence is trying to convey.\n* The results in Section 5 need more discussion. Theorem 2 at least is reminiscent of other representation theorems in the NN literature, but what is reader supposed to take away from Lemma 1 and Corollary 1? Instead of provding a full proof of Theorem 2, I would suggest deferring that to the appendix, and using the additional space to discuss the importance of all these results.\n* The experiments seem quite impressive, but then again, I'm not sure whether I can gauge their soundness with confidence. There are many details about the experimental setting that are either missing or not well explained. For example:\n    * Are the G/S/H models in Table 1 all directly comparable? Do they have a similar number of parameters? Similar training?\n    * The reported advantage of H over G/S seems to be mostly prominent in low dimensions of the Poincare ball. I would like to see a discussion on why this is the case.\n    * Given how much variance the results in Table 1 seem to have, standard deviation or error bars should be reported along with the means\n    * It is not clear what exactly is meant by test error on a clustering task in Section 6.2. How are the train / test samples used? \n    * Many experimental/design choices are not well justified - e.g., why is the input layer scaled down with PCA in 6.3?\n\nOther issues:\n* The notion of end prototypes seems quite interesting, but I feel like it could be better explained / elaborated on, e.g., at the end of Section 4.\n* In Theorem 2, it isn't clear where K is coming into play in the definition of F (given that the density argument is on $L_p(K,\\mu)$, I suppose F is only defined for $x in K$?). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "sNWhihCXMDo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Review summary**\n\nThe proposed models are theoretically sound, as it is as expressive as satisfying the universality theorem. Also, the proposed methods are empirically superior to existing methods. However, I think there is much room for improvement in the presentation of the paper. Besides, it is unclear to me what the research question is and how the proposed methods solve the problem. I would recommend reconsidering the organization of the paper. \n\n**Summary of the paper**\n\nThis paper proposed a horocycle neuron that acts on the hyperbolic space. It uses the Hyperbolic Poisson kernel in place of the standard Euclid norm on the Euclidean space. This paper proposed an architecture called horocycle MLR, which used a horocycle neuron as a building block, and Poisson neural MLR. They showed the universality of a model with a single hidden layer of horocycle neurons or $f^1_{a, p}$, which has been used in the existing literature. They applied the horocycle feature to a subtree classification task of Poincare embedding, a horocycle MLR to a clustering task of 2D embedding, and horocycle and Poisson MLRs to classification tasks on image datasets.\n\n**Claim**\n\nIf I understand correctly, this paper claims that the horocycle and Possison neurons are theoretically sound and empirically effective. However, it is not clear to me the research question that this paper addressed and how the theoretical and empirical properties of the proposed methods answer the question. It is true that they discussed the heuristic connection between the universal approximation property and the integral representation of the form (8) of a horocycle neuron. However, I think it is not a research question but supporting evidence that the universal approximation property is likely to hold.\n\n**Soundness of the claims**\n\nCan theory support the claim?\n- The authors proved the universal approximation theorem for horocycle and $f^1_{a, p}$. Although it is not a constructive proof due to the Hahn-Banach theorem's nature, as the paper pointed out, it gives an affirmative answer for the theoretical justification and is a good first step to study the expressive power. \n- If I do not miss any information, the Poisson neuron model (Section 4.2, Paragraph 4) is introduced without its motivation nor justification. In addition, this paper does not provide the theoretical superiority of the model. For example, Theorem 1 and Theorem 2 does not apply to the Poisson neuron model. I want to know if there are theoretical justifications for the Poisson model.\n\nCan empirical evaluation support the claim?\n- Section 6.1: I confirm that the horocycle model's overall performance is better than Ganea et al. (2018a) and Shimizu et al. (2020). Especially, the proposed method significantly outperforms them when the embedding dimension is two, or the subtree is \"worker.n.01\". \n- Section 6.2--6.4: I confirm that the proposed method's error rate is smaller than the existing methods.\n- Section 6.5: I could not understand the motivation for the experiments of the CIFAR-10 and Fashion-MNIST datasets in this section. Figure 8 claimed that Poisson MLR shows good generalization in the Flowers dataset. However, this paper does not provide such a comparison in the CIFAR-10 and Fashion-MNIST datasets. Also, the performance in these datasets is not as good as the SOTA models (I referenced [1] for CIFAR-10 and [2] for Fashion-MNIST]). Therefore, I think these results do not support the empirical superiority of Poisson MLR.\n[1] https://paperswithcode.com/sota/image-classification-on-cifar-10\n[2] https://paperswithcode.com/sota/image-classification-on-fashion-mnist\n\n\n**Significance and novelty**\n\nNovelty\n- To the best of our knowledge, this is the first study that proves the universal approximation theorem for a single-hidden layer model on a hyperbolic space.\n\nRelation to previous work\n- Although this paper mentioned Ganea et al. (2018a) and Shimizu et al. (2020), with which this paper compare the proposed method in the experiment, it did not compare the methodological difference (especially novelty and superiority) of the proposed method from the two. Same is true of the baseline methods in Table 2 and the method by Ontrup & Ritter (2005) and Grattarola et al. (2019) in Table 3. I would like to recommend to make it clear what is the drawback of the existing model.\n\n\n**Correctness**\nIs the theory correct?\n- Yes, So far as I check the proof, Theorem 1 and Corollary 1 (universality of horocycle neurons and the function $f^1_{a, p}$) are correct.\n\nIs the experimental evaluation correct?\n- Yes, I did not find any methodologically incorrect point in the experimental procedures. In Table 1, ideally, we should compare three methods with the same train/test partitions because the class label is highly imbalanced (e.g., 1115/82114 is positive in the case of worker.n.01 ); I am wondering if the performance variance caused by the randomness of data partition could be high.\n\nReproducibility of the experiments\n- Yes. It explains experimental settings in detail in the appendix. Also, it has a runnable code with trained parameters.\n\n**Clarity**\n\nI would say that there is much improvement in the clarity of the paper.\n     First, I took some time to understand how sections are related and how paragraphs in a section are related. I think adding discourse markers and organizing sentences so that readers can do paragraph reading could make the paper more understandable. Take the introduction section as an example. I feel there is a large gap between the third and fourth paragraphs. In addition, I could not understand that the fourth paragraph intends to explain the horocycle neuron until I reached the end of the paragraph. Also, although the function $f^1_{a, p}$ is introduced in the fifth paragraph, the introduction does not mention it in the remaining part and goes back to the explanation of horocycle neurons. \n     Another problem is that the tables and figures are not prepared appropriately. For example, Table 3 is inserted within a paragraph. Also, captions and legends of figures are tiny and hard to read.\n\n**Additional feedback**\n- Abstract: The acronym MLR is used without what it stands for. So, I recommend writing the meaning of MLR without abbreviation.\n- Section 1, Paragraph 3: This paper study → studies\n- Section 1, Second bullet: Although this sentence mentioned the Poisson neuron and the horocycle MLR, they were not mentioned before this sentence. Similarly, the term \"end-based\" is used in the introduction but is explained in Section 4.2 for the first time. I would recommend writing their explanation before they are used.\n- Section 2 Paragraph 3 (Hyperbolic deep learning): I could not see what this paper intended to mean by the term \"prototype\" at first reading. This wording may need some definition.\n- Section 4.1, Paragraph 3 (Neuron models): What does the following sentence mean?: We accept the representation properties of eigenfunctions on compact manifolds. \n- Section 4.2, Paragraph 5 (End-based clusters, end prototypes): It was hard for me to understand the relationship between sentences in the paragraph. For example, it is not clear at first sight how RBF is related to the discussion of clustering algorithms. I would recommend reconsidering the organization of the paragraph.\n- Section 5 (9): $\\langle x, \\omega_i \\rangle$ → $\\langle x, \\omega_i\\rangle_H$\n- Section 6.1, Table 1: Could you explain what $H^2$, $H^3$, etc. means?\n- Section 6.1: The task (Ganea et al., 2018a) is to classify all other nodes as [...]. → It is not clear what \"other\" nodes mean solely from the main text. I understand it after I read the first sentence of Section A.11.\n- Appendix A.15: This paper says that it adds a loss to distinguish the prototypes of 4 and 9. However, looking at the code, it seems the algorithm randomly selects two classes and adds the loss from prototypes of these classes. I want to confirm if my understanding is correct and recommend explaining the procedure if it is correct.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fLHwEvoWNVH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a new hyperbolic neuron based on horocycles (hyperbolic counterparts of hyperplanes). The authors proved that these neurons in H^n are as useful as traditional neurons in R^n through theoretical arguments and demonstrated they can significantly improve learning in hyperbolic embeddings of tree datasets and MNIST/CIFAR datasets.\n\nQuality:\nThis contribution has both theoretical and practical strengths. Theoretically, they proved that the proposed hyperbolic neurons are universal approximators (Theorem 2). Practically, they introduced a new kind of hyperbolic neuron, with its difference with existing literature clearly demonstrated through formulations and density plots. It shows supervisor performance improvements in several examples.\n\nClarity:\nThe language is well polished. The formulations and statements are clear and consistent. The presentation has high clarity with good intuitions through illustrations.\n\nOriginality:\nThe proposed method is mostly related to hyperbolic neural networks constructed using Mobius arithmetic operators. Their difference is demonstrated both intuitively and empirically through experiments. The relationship with previous works is clearly stated in section 2. The references are proper with page numbers mentioned.\n\nSignificance:\nThis paper establishes a new connection between hyperbolic geometry and deep learning. Therefore it should be interesting to the large group of audiences in those areas.\n\nMy main concern and questions are listed as follows:\n\nMost importantly, the introduction and the theorem are based on equation (2); while the experiments are based on the Poisson layer introduced in section 4.2. I can see some inconsistency here: clearly they are different functions. Please fill this gap in the rebuttal and next version.\n\nAre there any explanations and technical arguments of the good empirical performance?\n\nClearly, the hyper-parameter epsilon is important to maintain numerical stability. There should be some demonstrations in the main text or the  supplementary material to show the robustness to instability (e.g. by setting epsilon=0)\n\nFinally, I summarize the pros and cons as follows.\n\nPro:\n- new hyperbolic deep learning\n- proof of representation power on H^n\n- strong empirical results\n\nCon:\n- missing connection between equation(2) and Poisson layer\n\nOverall, based on the above assessment measures I recommend strong acceptance.\n\nHere are more comments for the authors' revision:\n\nabstract: \"MLR\" is the abbrev of?\n\nIntroduction: introduce notation T_p(H)\n\nAfter theorem 1 there have to be some remarks to explain the statement.\n\nSame for Lemma/Corollary 1.\n\nTheorem 2 is referred to before the statement.\n\nThe volume element \"dm\" is a bit hard to read\n\nWhere are the notation h_x(y) used?\n\nFigure 8 x-axis and y-axis are not clear\n\n-----\nAfter rebuttal:\n\nThank you for the revision and the clarifications.\n\nIt is now clear that this work actually proposed two different neurons: the horocycle neuron defined on H^n and the Position neuron defined on R^n (removing one point). After the revision, they are proved to satisfy the universal approximation property. They share similar level sets (although the density of the level sets is different). It would be interesting to see their relationships through formal arguments and a more careful empirical comparison.\n\nThis work needs background knowledge in hyperbolic geometry and may not be easy to read at the beginning. That could explain the criticism regarding clarity. Overall, I believe this paper developed important tools along the line of hyperbolic deep learning and still recommends strong acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V8GxZEqsUqJ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a MLR based on hyperbolic geometry. The idea is based on well-known concept of horocycle and horospheres which are known to be hyperbolic counterpart of line and plane in Euclidean geometry (see Coxter). Then the authors show the universal approximation which kind of follows similarly from the Euclidean counterpart. In fact we can probably conject that this universal approximation holds for any manifolds with constant sectional curvature.\n\nStrength: To the best of my knowledge, this is the first paper to deal with linear models on hyperbolic spaces by borrowing geometric tools like horocycles.\n\n\n\nMajor weakness:\n\nThe ideas are borrowed from well-known geometric tools, although this is not a weakness but the theorems closely follow Euclidean counterpart. This essentially reduces the ``````\"novelty\" of the paper. Moreover, the experiments are ``\"synthetic\", there is no motivation to use such a construction in real experiment. It will be good to see the authors discuss in which real cases we need to use such a hyperbolic MLR.\n\n1) The work should be better motivated, for example what is the motivation of using Horocycle layer and Poisson neuron layer?\n2) In section 6.2, the 2D output after 4 convolutional layers seems very less expressive, why not increase the dimension? Also what is the motivation to map it to H^2?\n3) In Theorem 2, eq. 9, why the inner product is Euclidean instead of hyperbolic? \n4) The universal approximation theorem in Theorem 2 almost follows from the Euclidean counterpart, e.g., see https://cbmm.mit.edu/sites/default/files/publications/CBMM-Memo-054.pdf\n5) What is the additional consequence of Corollary 1 other than showing we can approximate any function, similar as Theorem 2?\n6) The statement in section 6.3 stating \"t is the best hyperbolic geometry related MNIST classifier\" does not carry much weight, e.g., what is the motivation of using MNIST images for MLR using hyperbolic geometry? \n7) There is not much point for section 6.4. In most practical cases, the 1-dimensional reduction is not meaningful as it can not carry much information.\n8) Section 6.5 seems very rushed including the Fig. 8 and experiment of Flowers. This section seems more like placeholder.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers generally appreciate the contributions of the paper, namely the horocycle neuron, Poisson neuron, and the universal approximation properties. However, there are concerns, especially by R4 and R5, that the presentation is confusing, lacks clarity, and should be substantially improved.

Note: Theorem 1.7 in (Helgason, 1970) is proved explicitly for the case n=2, not for general n as claimed in (9).  Thus the Laplacian eigenspace motivation needs to be re-written/re-examined.
Decision: Reject