Laplacian Eigenspaces, Horocycles and Neuron Models on Hyperbolic Spaces

1 INTRODUCTION . Conventional deep network techniques attempt to use architecture based on compositions of simple functions to learn representations of Euclidean data ( LeCun et al. , 2015 ) . They have achieved remarkable successes in a wide range of applications ( Hinton et al. , 2012 ; He et al. , 2016 ) . Geometric deep learning , a niche field that has caught the attention of many authors , attempts to generalize conventional learning techniques to non-Euclidean spaces ( Bronstein et al. , 2017 ; Monti et al. , 2017 ) . There has been growing interest in using hyperbolic spaces in machine learning tasks because they are well-suited for tree-like data representation ( Ontrup & Ritter , 2005 ; Alanis-Lobato et al. , 2016 ; Nickel & Kiela , 2017 ; Chamberlain et al. , 2018 ; Nickel & Kiela , 2018 ; Sala et al. , 2018 ; Ganea et al. , 2018b ; Tifrea et al. , 2019 ; Chami et al. , 2019 ; Liu et al. , 2019 ; Balazevic et al. , 2019 ; Yu & Sa , 2019 ; Gulcehre et al. , 2019 ; Law et al. , 2019 ) . Many authors have introduced hyperbolic analogs of classical learning tools ( Ganea et al. , 2018a ; Cho et al. , 2019 ; Nagano et al. , 2019 ; Grattarola et al. , 2019 ; Mathieu et al. , 2019 ; Ovinnikov , 2020 ; Khrulkov et al. , 2020 ; Shimizu et al. , 2020 ) . Spectral methods are successful in machine learning , from nonlinear dimensionality reduction ( Belkin & Partha , 2002 ) to clustering ( Shi & Malik , 2000 ; Ng et al. , 2002 ) to hashing ( Weiss et al. , 2009 ) to graph CNNs ( Bruna et al. , 2014 ) to spherical CNNs ( Cohen et al. , 2018 ) and to inference networks ( Pfau et al. , 2019 ) . Spectral methods have been applied to learning tasks on spheres ( Cohen et al. , 2018 ) and graphs ( Bruna et al. , 2014 ) , but not yet on hyperbolic spaces . This paper studies a spectral generalization of the FC ( affine ) layer on hyperbolic spaces . Before presenting the spectral generalization of the affine layer , we introduce some notations . Let ( · , · ) E be the inner product , | · | the Euclidean norm , and ρ an activation function . The Poincaré ball model of the hyperbolic space Hn ( n≥2 ) is a manifold { x∈Rn : |x| < 1 } equipped with a Riemannian metric ds2Hn= ∑n i=1 4 ( 1−|x| 2 ) −2dx2i . The boundary of Hn under its canonical embedding in Rn is the unit sphere Sn−1 . The classical neuron y=ρ ( ( x , w ) E+b ) is of input x∈Rn , output y∈R , with trainable parameters w∈Rn , b∈R . An affine layer Rn → Rm is a concatenation of m neurons . An alternative representation of the neuron x 7→ρ ( ( x , w ) E+b ) is given by 1 x∈Rn 7→ ρ ( λ ( x , ω ) E+b ) , ω∈Sn−1 , λ , b∈R . ( 1 ) This neuron is constant over any hyperplane that is perpendicular to a fixed direction ω . In Hn , a horocycle is a n−1 dimensional sphere ( one point deleted ) that is tangential to Sn−1 . Horocycles are hyperbolic counterparts of hyperplanes ( Bonola , 2012 ) . Horocyclic waves 〈x , ω〉H : = 12 log 1−|x|2 |x−ω|2 are constant over any horocycle that is tangential to Sn−1 at ω . Therefore , x∈Hn 7→ ρ ( λ〈x , ω〉H+b ) , ω∈Sn−1 , λ , b∈R ( 2 ) 1if w 6= ( 0 , . . . , 0 ) , one can take ω = w/|w| , λ = |w| ; else , one can take λ = 0 and any ω ∈ Sn−1 . generalizes the classical neuron model ( 1 ) , and a concatenation of finitely many ( 2 ) generalizes the FC ( affine ) layer . We call ( 2 ) a horocycle neuron . Figure 1 ( middle ) is an example on H2 . The neuron models in ( 1 , 2 ) are related to spectral theory because ( · , ω ) E ( respectively 〈· , ω〉H ) are building blocks of the Euclidean ( respectively hyperbolic ) Laplacian eigenspace . Moreover , many L2 spaces have a basis given by Laplacian eigenfunctions ( Einsiedler & Ward , 2017 ) . On one side , all Euclidean ( respectively hyperbolic ) eigenfunctions are some kind of “ superposition ” of ( · , ω ) E ( respectively 〈· , ω〉H ) . On the other side , neural networks based on ( 1 ) ( respectively ( 2 ) ) represent functions that are another kind of “ superposition ” of ( · , ω ) E ( respectively 〈· , ω〉H ) . They heuristically explain why the universal approximation property is likely to hold for networks constructed by ( 1 ) and ( 2 ) . By using the Hahn Banach theorem , an injectivity theorem of Helgason , and integral formula , we prove that finite sums of horocycle neurons ( 2 ) are universal approximators ( Theorem 2 ) . Let p ∈ Hn , Tp ( Hn ) be the tangent space of Hn at p , a ∈ Tp ( Hn ) , ⊕ be the Möbius addition ( Ungar , 2008 ) . We remind the reader that the following functions f1a , p ( x ) = 2|a| 1− |p|2 sinh −1 ( 2 ( −p⊕ x , a ) E ( 1− | − p⊕ x|2 ) |a| ) ( 3 ) are building blocks of many hyperbolic learning tools ( Ganea et al. , 2018a ; Mathieu et al. , 2019 ; Shimizu et al. , 2020 ) . Figure 1 illustrates examples of different neuron models ( 1 , 2 , 3 ) on H2 . In Lemma 1 , we shall present a close relationship between ( 2 ) and ( 3 ) . Using this relationship and Theorem 2 , we obtain a novel result on the expressivity of f1a , p ( Corollary 1 ) . This article contributes to hyperbolic learning . We first apply spectral methods , such as the horocycle , to hyperbolic deep learning . We prove results on the expressivity of horocycle neurons ( 2 ) and f1a , p ( 3 ) . With horocycle neurons , we obtain state-of-the-art results on the Poincaré-embedding subtree classification task and the classification accuracy of the 2-D visualization of images in in the experiment . 2 RELATED WORK . Universal approximation There is a vast literature on universal approximation ( Cybenko , 1989 ; Hornik et al. , 1989 ; Funahashi , 1989 ; Leshno et al. , 1993 ) . Cybenko ( 1989 ) ’ s existential approach uses the Hahn Banach theorem and Fourier transform of Radon measures . To prove Theorem 2 , we also use the Hahn Banach theorem , and additionally an integral formula ( 7 ) and an injectivity Theorem 1 of Helgason . Generalizing integral formulas and injectivity theorems is easier than generalizing Fourier transform of Radon measures on most non-Euclidean spaces . ( Carroll & Dickinson , 1989 ) uses the inverse Radon transform to prove universal approximation theorems . This method relates to ours , as injectivity theorems are akin to inverse Radon transforms . However , using the injectivity theorem is an existential approach while using the inverse Radon transform is a constructive one . Spectral methods Spectral methods in Bronstein et al . ( 2017 ) ; Bruna et al . ( 2014 ) ; Cohen et al . ( 2018 ) use a basis of L2 ( X ) given by eigenfunctions , whereX is a finite graph or the sphere . Because L2 ( Hn ) has no eigenfunctions as a basis , our approach is different from theirs . Hyperbolic deep learning One part of hyperbolic learning concerns embedding data into the hyperbolic space ( Nickel & Kiela , 2017 ; Sala et al. , 2018 ) . Another part concerns learning architectures with hyperbolic data as the input ( Ganea et al . ( 2018a ) ; Cho et al . ( 2019 ) ) . Ganea et al . ( 2018a ) proposes two ways to generalize the affine layer on hyperbolic spaces : one by replacing the linear and bias part of an affine map with ( 25 , 26 ) of their paper ; another one by using a concatenation of f1a , p in their hyperbolic multiple linear regression ( MLR ) . The latter seems more relevant to ours . A level set of f1a , p is a hypercycle that has the same distance to a chosen geodesic hypersurface , while a level set of a horocycle neuron is a horocycle that has the same “ spectral ” distance to an ideal point at infinity . Based on functions similar to f1a , p , Mathieu et al . ( 2019 ) ; Shimizu et al . ( 2020 ) build the gyroplane layer and Poincaré FC layer . Ganea et al . ( 2018a ) ; Cho et al . ( 2019 ) take geodesics as decision hyperplanes , while we ( initially ) take horocycles . We shall construct the horocycle multiple linear regression ( MLR ) , where decision hypersurfaces are geodesics . Geodesics decision hyperplanes ( Ganea et al. , 2018a ; Cho et al. , 2019 ) and geodesic decision hypersurfaces here arise from different methods . Khrulkov et al . ( 2020 ) investigates hyperbolic image embedding , where prototypes ( or models ) of each class are center-based . We study a different one , and we shall call our prototypes end-based . 3 HYPERBOLIC SPACES . This section reviews facts from hyperbolic geometry that are used in the proof of Theorem 2 . For the reader who is not interested in the proof , ( 4 ) is enough for the implementation . Hyperbolic metric We use the Poincaré model . The hyperbolic space Hn ( n≥2 ) is the manifold { x∈Rn : |x| < 1 } equipped with a Riemannian metric ds2 = ∑n i=1 4 ( 1−|x|2 ) −2dx2i . Let o be the origin of Hn . The distance function dHn satisfies dHn ( o , x ) =2 arctanh |x| . Geodesics , horocycles and corresponding points Geodesics in Hn are precisely circular arcs that are orthogonal to Sn−1 . Horocycles in Hn are precisely ( n−1 ) -dimensional spheres that are tangential to Sn−1 ( Helgason , 1970 ) . Horocycles are hyperbolic analogs of hyperplanes . Figure 2 illustrates geodesics and horocycles on H2 . Hyperbolic Poisson kernel The Poisson kernel for Hn is P ( x , ω ) = ( 1−|x|2 |x−ω|2 ) n−1 , where x∈Hn , ω∈Sn−1 ( Helgason ( 1970 ) [ p.108 ] ) . The function 〈· , ω〉H defined by 〈x , ω〉H = 1 2 ( n− 1 ) logP ( · , ω ) = 1 2 log 1− |x|2 |x− ω|2 ( 4 ) is constant over any horocycle that is tangential to Sn−1 at ω ( Figure 1 ( middle ) , ( 6 ) ) . Riemannian volume The Riemannian volume induced by the metric ds2 on Hn is dVol = 2n ( 1− |x|2 ) −ndx1 . . . dxn . ( 5 ) Horocycles Let Ξ be the set of horocycles of Hn , and let Ξω be the set of all horocycles that are tangential to Sn−1 at ω . Given λ∈R , we let ξλ , ω be the unique horocycle that connects ω and tanh ( λ/2 ) · ω . We have Ξω = ∪λ∈R { ξλ , ω } and Ξ = ∪ω∈Sn−1Ξω . The length of any geodesic ( that ends at ω ) line segment cut by ξλ1 , ω and ξλ2 , ω equals |λ1 − λ2| ( A.2 ) . Therefore |λ1 − λ2| is a natural distance function defined on Ξω , and the map λ→ ξλ , ω is an isometry between R and Ξω . This isometry is closely related to 〈· , ω〉H ( A.3 ) : for any x ∈ ξλ , ω , 〈x , ω〉H = λ/2 . ( 6 ) The annoying /2 in ( 6 ) is a tradeoff that the metric here is different from that in Helgason ( 2000 ) . Integral formula For fixed ω ∈ Sn−1 , Hn=∪λ∈Rξλ , ω . Let dVolξλ , ω be the measure induced by ds2 on ξλ , ω . Let L be a family of geodesics that end at ω , δ > 0 , and U=L ∩ ( ∪λ≤α≤λ+δξα , ω ) . For l ∈ L , dH ( l ∩ ξλ , ω , l ∩ ξλ+δ , ω ) =δ ( A.2 ) , hence dVol ( U ) = δ · dVolξλ , ω ( U ∩ ξλ , ω ) and therefore∫ Hn f ( x ) dVol ( x ) = ∫ R ( ∫ ξλ , ω f ( z ) dVolξλ , ω ( z ) ) dλ . ( 7 ) The above proof ( for Hn ) is essentially the same as that in ( Helgason , 2000 ) [ p.37 ] ( for H2 ) . To further convince the reader that ( 7 ) holds for all n , we give another simple proof in A.4 . Injectivity theorem With respect to the canonical measure on Ξ , Helgason ( 1970 ) [ p.13 ] proved Theorem 1 ( Helgason ) . If f ∈ L1 ( Hn ) and ∫ ξ f ( z ) dVolξ ( z ) = 0 for a.e ξ ∈ Ξ , then f = 0 a.e .. Theorem 1 demonstrates that if the integral of f ∈ L1 ( Hn ) over almost every horocycle is zero then f is also zero . This theorem and the integral formula ( 7 ) are essential for the proof of Theorem 2 .

Summary: This paper proposes new neural models for hyperbolic space, which unlike previous hyperbolic NN works, relies on the notion of horocycle in the Poincare disk. This novel framework has connections to spectral learnig in hyperbolic space. Representation theorems alla Cybenko for layers constructed from these neurons are presented. Finally, various experiments on clustering and classifying datasets using these neurons to generate hyperbolic embeddings are presented.