Hyperbolic Neural Networks++

1 INTRODUCTION . Shifting the arithmetic stage of a neural network to a non-Euclidean geometry such as a hyperbolic space is a promising way to find more suitable geometric structures for representing or processing data . Owing to its exponential growth in volume with respect to its radius ( Krioukov et al. , 2009 ; 2010 ) , a hyperbolic space has the capacity to continuously embed tree structures with arbitrarily low distortion ( Krioukov et al. , 2010 ; Sala et al. , 2018 ) . It has been directly utilized , for instance , to visualize large taxonomic graphs ( Lamping et al. , 1995 ) , to embed scale-free graphs ( Blasius et al. , 2018 ) , or to learn hierarchical lexical entailments ( Nickel & Kiela , 2017 ) . Compared to the Euclidean space , a hyperbolic space shows a higher embedding accuracy under fewer dimensions in such cases . Because a wide variety of real-world data encompasses some type of latent hierarchical structures ( Katayama & Maina , 2015 ; Newman , 2005 ; Lin & Tegmark , 2017 ; Krioukov et al. , 2010 ) , it has been empirically proven that a hyperbolic space is able to capture such intrinsic features through representation learning ( Krioukov et al. , 2010 ; Ganea et al. , 2018b ; Nickel & Kiela , 2018 ; Tifrea et al. , 2019 ; Law et al. , 2019 ; Balazevic et al. , 2019 ; Gu et al. , 2019 ) . Motivated by such expressive characteristics , various machine learning methods , including support vector machines ( Cho et al. , 2019 ) and neural networks ( Ganea et al. , 2018a ; Gulcehre et al. , 2018 ; Micic & Chu , 2018 ; Chami et al. , 2019 ) have derived the analogous benefits from the introduction of a hyperbolic space , aiming to improve the performance on advanced tasks beyond just representing data . One of the pioneers in this area is Hyperbolic Neural Networks ( HNNs ) , which introduced an easy-to-interpret and highly analytical coordinate system of hyperbolic spaces , namely , the Poincaré ball model , with a corresponding gyrovector space to smoothly connect the fundamental functions common to neural networks into valid functions in a hyperbolic geometry ( Ganea et al. , 2018a ) . Built upon the solid foundation of HNNs , the essential components for neural networks covering the multinomial logistic regression ( MLR ) , fully-connected ( FC ) layers , and Recurrent Neural Networks have been realized . In addition to the formalism , the methods for graphs ( Liu et al. , 2019 ) , sequential classification ( Micic & Chu , 2018 ) , or Variational Autoencoders ( Nagano et al. , 2019 ; Mathieu et al. , 2019 ; Ovinnikov , 2019 ; Skopek et al. , 2020 ) are further constructed . Such studies have applied the Poincaré ball model as a natural and viable option in the area of deep learning . Despite such progress , however , there still remain some unsolved problems and uncovered regions . In terms of the network architectures , the current formulation of hyperbolic MLR ( Ganea et al. , 2018a ) requires almost twice the number of parameters compared to its Euclidean counterpart . This makes both the training and inference costly in cases in which numerous embedded entities should be classified or where large hidden dimensions are employed , such as in natural language processing . The lack of convolutional layers must also be mentioned , because their application is now ubiquitous and is no longer limited to the field of computer vision . For the individual functions that are commonly used in machine learning , the split and concatenation of vectors have yet to be realized in a hyperbolic space in a manner that can fully exploit such space and allow sub-vectors to achieve a commutative property . Additionally , although several types of closed-form centroids in a hyperbolic space have been proposed , their geometric relationships have not yet been analyzed enough . Because a centroid operation has been utilized in many recent attention-based architectures , the theoretical background for which type of hyperbolic centroid should be used would be required in order to properly convert such operations into the hyperbolic geometry . Based on the previous analysis , we reconsider the flow of several extensions to bridge Euclidean operations into hyperbolic operations and construct alternative or novel methods on the Poincaré ball model . Specifically , the main contributions of this paper are summarized as follows : 1 . We reformulate a hyperbolic MLR to reduce the number of parameters to the same level as a Euclidean version while maintaining the same range of representational properties . 2 . We further exploit the knowledge of 1 as a replacement of an affine transformation and propose a novel generalization of the FC layers that can more properly make use of the hyperbolic nature compared with a previous research ( Ganea et al. , 2018a ) . 3 . We generalize the split and concatenation of coordinates to the Poincaré ball model by setting the invariance of the expected value of the vector norm as a criterion . 4 . By combining 2 and 3 , we further define a novel generalization scheme of arbitrary dimensional convolutional layers in the Poincaré ball model . 5 . We prove the equivalence of the hyperbolic centroids defined in three different hyperbolic geometry models , and expand the condition of non-negative weights to entire real values . Moreover , integrating this finding and previous contributions 1 , 2 , and 3 , we give a theoretical insight into hyperbolic attention mechanisms realized in the Poincaré ball model . We experimentally demonstrate the effectiveness of our methods over existing HNNs and Euclidean equivalents based on a performance test of MLR functions and experiments with Set Transformer ( Lee et al. , 2019 ) and convolutional sequence to sequence modeling ( Gehring et al. , 2017 ) .1 2 HYPERBOLIC GEOMETRY . Riemannian geometry . An n-dimensional manifoldM is an n-dimensional topological space that can be linearly approximated to an n-dimensional real space at any point x ∈ M , and each local linear space is called a tangent space TxM . A Riemannian manifold is a pairing of a differentiable manifold and a metric tensor field g as a function of each point x , which is expressed as ( M , g ) . Here , g defines an inner product on each tangent space such that ∀u , v ∈ TxM , 〈u , v〉x = u > gxv , where gx is a positive definite symmetric matrix defined on TxM . The norm of a tangent vector derived from the inner product is defined as ‖v‖x = √ |〈v , v〉x| . A metric tensor gx provides local information regarding the angle and length of the tangent vectors in TxM , which induces the global length of the curves onM through an integration . The shortest path connecting two arbitrary points onM at a constant speed is called a geodesic , the length of which becomes the distance . Along a geodesic where one of the endpoints is x , the function projecting a tangent vector v ∈ TxM as an initial velocity vector ontoM is denoted as an exponential map expx , and its inverse function is called a logarithmic map logx . In addition , the concept of parallel transport Px→y : TxM→ TyM is generalized to the specially conditioned unique linear isometry between two tangent spaces . For more details , please refer to Spivak ( 1979 ) ; Petersen et al . ( 2006 ) ; Andrews & Hopper ( 2010 ) . Note that , in this study , we equate g with gx if gx is constant , and denote the Euclidean inner product , norm , and unit vector for any real vector u , v ∈ Rn as 〈u , v〉 , ‖v‖ , and [ v ] = v/‖v‖ , respectively . Hyperbolic space . A hyperbolic space is a Riemannian manifold with a constant negative curvature , the coordinates of which can be represented in several isometric models . The most basic model is an 1The code is available at https : //github.com/mil-tokyo/hyperbolic_nn_plusplus . n-dimensional hyperboloid model , which is a hypersurface Hnc in an ( n+ 1 ) -dimensional Minkowski space Rn+11 composed of one time-like axis and n space-like axes . The manifolds of Poincaré ball model Bnc and Beltrami-Klein model Knc are the projections of the hyperboloid model onto the different n-dimensional space-like hyperplanes , as depicted in Figure 1 . For their mathematical definitions and the isometric isomorphism between their coordinates , see Appendix A. Poincaré ball model . The n-dimensional Poincaré ball model of a constant negative curvature −c is defined by ( Bnc , gc ) , where Bnc = { x ∈ Rn | c‖x‖2 < 1 } and gcx = ( λcx ) 2In . Here , Bnc is an open ball of radius c− 1 2 , and λcx = 2 ( 1− c‖x‖2 ) −1 is a conformal factor , which induces the inner product 〈u , v〉cx = ( λcx ) 2〈u , v〉 and norm ‖v‖cx = λcx‖v‖ for u , v ∈ TxBnc . The exponential , logarithmic maps and parallel transport are denoted as expcx , log c x and P c x→y , respectively , as shown in Appendix C. To operate the coordinates as vector-like mathematical objects , the Möbius gyrovector space provides an algebra that treats them as gyrovectors , equipped with various operations including the generalized vector addition , that is , a noncommutative and non-associative binary operation called the Möbius addition ⊕c ( Ungar , 2009 ) . limc→0⊕c converges to + in connection with a Euclidean geometry , the curvature of which is zero . For more details , see Appendix B. Poincaré hyperplane . As a specific generalization of a hyperplane into Riemannian geometry , Ganea et al . ( 2018a ) derived a Poincaré hyperplane H̃ca , p , which is the set of all geodesics containing an arbitrary point p ∈ Bnc and orthogonal to an arbitrary tangent vector a ∈ TpBnc , based on the Möbius gyrovector space . As shown in Appendix C.2 , they also extended the distance dc between two points in Bnc into the distance from a point in Bnc to a Poincaré hyperplane in a closed form expression . 3 HYPERBOLIC NEURAL NETWORKS++ Aiming to overcome the difficulties discussed in Section 1 , we build a novel scheme of hyperbolic neural networks in the Poincaré ball model . The core concept is re-generalization of 〈a , x〉−b type equations with no increase in the number of parameters , which has the potential to replace any affine transformation based on the same mathematical principle . Specifically , this section starts from the reformulation of the hyperbolic MLR , from which the variants to the FC , convolutional , and multi-head attention layers are derived . Several other modifications are also proposed to support neural networks with broad architectures . 3.1 UNIDIRECTIONAL REPARAMETERIZATION OF HYPERBOLIC MLR LAYER . Given an input x ∈ Rn , MLR is an operation used to predict the probabilities of all target outcomes k ∈ { 1 , 2 , ... , K } for the objective variable y as a log-linear model and is described as follows : p ( y = k | x ) ∝ exp ( vk ( x ) ) , where vk ( x ) = 〈ak , x〉 − bk , ak ∈ Rn , bk ∈ R. ( 1 ) Circumvention of the double vectorization . To generalize the linear function vk to the Poincaré ball model , Ganea et al . ( 2018a ) first re-parameterized the scalar term bk as a vector pk ∈ Rn in the form 〈ak , x〉 − bk = 〈ak , −pk + x〉 , where bk = 〈ak , pk〉 , and then discussed the properties which must be satisfied when such vectors become Möbius gyrovectors . However , this causes an undesirable increase in the parameters from n+ 1 to 2n in each class k. As illustrated in Figure 2 ( a ) , this reformulation is redundant from the viewpoint that there exist countless choices of pk to determine the same discriminative hyperplane Hak , bk = { x ∈ Rn | 〈ak , x〉 − bk = 0 } . Because the key of this step is to replace all variables with vectors attributed to the same manifold , we introduce another scalar parameter rk ∈ R instead , which makes the bias vector qak , rk parallel to ak : 〈ak , x〉 − bk = 〈ak , −qak , rk + x〉 , where qak , rk = rk [ ak ] s.t . bk = rk‖ak‖ . ( 2 ) One possible realization of pk is adopted to reduce the previously mentioned redundancies without a loss of generality or representational properties compared to the original affine transformation , and induces another notation : H̄ak , rk : = { x ∈ Rn | 〈ak , −qak , rk + x〉 = 0 } = Hak , rk‖ak‖ . Based on distance d from a point to a hyperplane , Equation 2 can be rewritten as with Lebanon & Lafferty ( 2004 ) in the following form : 〈ak , −qak , rk + x〉 = sign ( 〈ak , −qak , rk + x〉 ) d ( x , H̄ak , rk ) ‖ak‖ , which decomposes the inner product into the product of the norm of an orientation vector ak and the signed distance between an input vector x ∈ Rn and the hyperplane H̄ak , rk . Unidirectional Poincaré MLR . Based on the observation that qak , rk starts from the origin and the concept of Poincaré hyperplanes , we can now generalize vk for x , qak , rk ∈ Bnc and ak ∈ Tqak , rkB n c : vk ( x ) = sign ( 〈ak , c qak , rk ⊕c x〉 ) dc ( x , H̄cak , rk ) ‖ak‖cqak , rk , ( 3 ) where qak , rk = exp c 0 ( rk [ ak ] ) , H̄ c ak , rk : = { x ∈ Bnc | 〈ak , c qak , rk ⊕c x〉 = 0 } , ( 4 ) which are shown in Figure 2 ( b ) . Importantly , the circular reference between ak ∈ Tqak , rkB n c and qak , rk can be unraveled by considering the tangent vector at the origin , zk ∈ T0Bnc , from which ak is parallel transported by P cx→y : TxBnc → TyBnc described in Appendix C.3 as follows : ak = P c 0→qak , rk ( zk ) = sech 2 ( √ c rk ) zk , qak , rk = exp c 0 ( rk [ zk ] ) = qzk , rk . ( 5 ) Combining Equations 3 , 5 , and 23 , we conclude the derivation of the unidirectional re-generalization of MLR , the parameters of which are rk ∈ R and zk ∈ T0Bnc = Rn for each class k : vk ( x ) = 2 c − 12 ‖zk‖ sinh−1 ( λcx〈 √ cx , [ zk ] 〉 cosh ( 2 √ c rk ) − ( λcx − 1 ) sinh ( 2 √ c rk ) ) . ( 6 ) For more detailed deformation , see Appendix D.1 . Note that we recover the form of the standard Euclidean MLR in limc→0 vk ( x ) = 4 ( 〈ak , x〉 − bk ) , which is proven in Appendix D.2 .

Summary: Hyperbolic Neural Networks++ extends the existing work of applying hyperbolic manifolds to neural networks. It proposes new ways to reparametrize hyperbolic multinomial logistic regression (MLR) layers to reduce the number of parameters, to generalise fully connected layers  as well as split and concat operations to be more flexible and less computationally expensive. Further they expand hyperbolic neural networks to convolutional layer and attention mechanisms. The evaluation include direct comparison to HNN, and clustering with Transformers and seq-2-seq modeling.