Ultrahyperbolic Neural Networks

1 Introduction . Riemannian manifolds of constant curvature are the most common representation spaces in machine learning . They include the Euclidean space ( of constant zero curvature ) , the d-sphere ( of constant positive curvature ) and the hyperbolic space ( of constant negative curvature ) . The choice of a geometry to represent data mainly depends on the kind of relationship that needs to be described . For instance , Gromov [ 10 ] showed the relevance of hyperbolic geometry to represent trees ( i.e. , graphs without cycles ) . Since many hierarchies can be described as trees , hyperbolic representations have been used to represent hierarchical relationships ( e.g. , hypernymy between words [ 19 ] ) . Nonetheless , in many domains ( e.g. , social networks or protein structures ) , hierarchical graphs contain cycles . In hyperbolic geometry , the considered manifold is not a vector space and is not equipped with the standard dot product . Therefore , most hyperbolic neural networks [ 5 , 8 , 18 , 27 ] represent the weights of their last layer in the tangent space of some reference point . That tangent space is equipped with a positive definite metric tensor and the learned model can then be optimized with Riemannian gradient descent [ 1 , 4 ] . In particular , since there exists a geodesic between any pair of points , the parameters are often optimized by using parallel transport ( also called parallel translation ) or the logarithm map . The Riemannian gradients are then parallel translated to the reference tangent space in which the model parameters lie . We refer the reader to [ 23 ] for a recent survey on hyperbolic neural networks . Recently , Law & Stam [ 15 ] proposed ultrahyperbolic embeddings . They are a type of embedding that lies on a pseudo-Riemannian manifold of constant nonzero curvature [ 2 , 21 , 30 ] . Pseudo-Riemannian manifolds ( also called semi-Riemannian manifolds ) are generalizations of Riemannian manifolds where the nondegenerate metric tensor is not constrained to be positive definite [ 16 ] . In particular , when the metric tensor is not positive definite ( e.g. , when it is indefinite ) , the negative of the ( pseudoRiemannian ) gradient is not a descent direction [ 9 ] . Law & Stam [ 15 ] proposed an efficient method to calculate a descent direction and learn ultrahyperbolic ( nonparametric ) embeddings . The main motivation of representing data on an ultrahyperbolic manifold is that it contains hyperbolic and spherical parts ( see Fig . 1 and supp . material for details ) . It can then describe relationships specific to hyperbolic and spherical geometries ( e.g. , to represent parts of a graph that are trees or cycles ) and is more flexible . Ultrahyperbolic embeddings were experimentally shown to be more appropriate than hyperbolic embeddings to represent hierarchical graphs with cycles on several datasets [ 15 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , since there exist pairs of points on the ultrahyperbolic manifold considered in [ 15 ] that can not be joined by an ( unbroken ) geodesic , gradients might not be parallel translated via a geodesic and the logarithm map joining two given points might not be defined . Directly extending hyperbolic neural networks [ 5 , 8 , 18 , 27 ] to ultrahyperbolic space is then problematic . In this paper , we propose a method to learn ultrahyperbolic representations with neural networks . Unlike [ 15 ] , we consider the pseudo-Riemannian quotient manifold defined such that every point x = ( x0 , . . . , xd ) > is equivalent to its antipodal point −x = ( −x0 , . . . , −xd ) > . In this way , for any other point y , there always exists at least one geodesic joining ( x , y ) or ( −x , y ) . We provide sufficient conditions to minimize a function defined on our quotient manifold . Since tangent vectors ( hence gradients ) of quotient manifolds are abstract objects , we explain how the function can be optimized with the horizontal lift operator . Our optimization framework is general , so we also introduce an extension to Graph Neural Networks ( GNNs ) [ 18 ] such that the activation representations at each layer of our GNN lie in ultrahyperbolic space . We then obtain a deep ultrahyperbolic model to represent graphs given as input . We evaluate our approach in different graph classification tasks . 2 Pseudo-sphere and Quotient Manifold . We extend the ultrahyperbolic manifold described in [ 15 ] ( denoted by Sp , qr ) to a quotient manifold denoted by Pp , qr where ( p , q ) is the metric signature ( see page 343 of [ 16 ] ) of the pseudo-Riemannian manifold and 1/r2 is its curvature . The motivation is that any pair of points of Pp , qr can be joined by at least one geodesic , which allows us to optimize parametric models . We consider three pseudoRiemannian manifoldsPp , qr ⊂ Sp , qr ⊂ Rp+1 , q that we define below . We explain howPp , qr generalizes elliptic and hyperbolic geometries in the special cases where q = 0 and p = 0 , respectively . Notation . We denote points on a smooth manifoldM [ 16 ] by boldface Roman characters x ∈M . [ x ] : = { x , −x } is a pair of antipodal points . TxM is the tangent space ofM at x and we write tangent vectors ξ ∈ TxM in boldface Greek fonts . Rd is the d-dimensional Euclidean space equipped with the ( positive definite ) dot product 〈· , ·〉 defined as 〈x , y〉 : = x > y . I is the identity matrix . The inverse function of the cosine ( resp . hyperbolic cosine ) is denoted by cos−1 ( resp . cosh−1 ) . Ambient space Rp+1 , q . Our ambient space Rp+1 , q is a vector space of dimensionality d + 1 = p+ q + 1 ∈ N called pseudo-Euclidean space [ 21 ] . It is equipped with the following scalar product ( i.e. , nondegenerate symmetric bilinear form ) of signature ( p+ 1 , q ) : ∀x = ( x0 , . . . , xd ) > , y = ( y0 , . . . , yd ) > , 〈x , y〉q : = p∑ i=0 xiyi − d∑ j=p+1 xjyj = x > Gy , ( 1 ) where the signature matrix G = G−1 = Ip+1 , q is the ( d+ 1 ) × ( d+ 1 ) diagonal matrix with the first p + 1 diagonal elements equal to 1 and the remaining q equal to −1 . Following general relativity literature and spacetime terminology [ 7 ] , Rp+1 , q has p+ 1 space dimensions and q time dimensions . Since it is a vector space , we can identify its tangent space to the space itself by means of the natural isomorphism TxRp+1 , q ≈ Rp+1 , q . Finally , the Euclidean space Rd+1 is the special case of Rd+1,0 which contains zero time dimension , and where G = Id+1,0 = I . Total space Sp , qr . Our total space Sp , qr is a pseudo-sphere of radius r > 0 embedded in Rp+1 , q . It is the following hypersurface : Sp , qr : = { x ∈ Rp+1 , q : 〈x , x〉q = r2 } , ( 2 ) It is equivalent to work with the pseudo-hyperboloid Qq , pr : = { x ∈ Rq , p+1 : 〈x , x〉p+1 = −r2 } and the pseudo-sphere Sp , qr as they are anti-isometric to each other ( see supp . material ) . Moreover , the radius r > 0 plays a role of scaling factor so we consider it to be 1 although it can be learned [ 5 , 14 ] . Finally , both x ∈ Sp , qr and its antipodal point −x lie on Sp , qr since 〈x , x〉q = 〈−x , −x〉q . Quotient manifold Pp , qr . We consider as equivalence relation the two-element group { ±1 } consisting of the identity map x 7→ x and the antipodal map x 7→ −x . This means that two points x ∈ Sp , qr and y ∈ Sp , qr are equivalent iff y = x or y = −x . We define the following projective space : Pp , qr : = Sp , qr /± 1 = Sp , qr /± I = { { x , −x } : x ∈ Sp , qr } . ( 3 ) Every point of Pp , qr is an unordered pair that we denote by [ x ] : = { x , −x } . Since Pp , qr is a projective space , every point [ x ] ∈ Pp , qr can be interpreted as the intersection of the pseudo-sphere Sp , qr with a line passing through the origin of Rp+1 , q . In some cases , it might be easier to interpret points of Pp , qr as lines through the origin , and to study their properties when they intersect the pseudo-sphere . Each point [ x ] ∈ Pp , qr is also a submanifold of Sp , qr and a discrete space . In the following , we explain how Pp , qr extends spherical geometry to elliptic geometry ( i.e. , when q = 0 ) , or naturally describes the hyperboloid model of hyperbolic geometry ( i.e. , when p = 0 ) . Elliptic geometry ( q = 0 ) . In spherical geometry , points lie on the unit d-sphere Sd : = Sd,01 = { x ∈ Rd+1 : 〈x , x〉 = 1 } . The geometry of the projective d-space Pd : = Sd/± 1 is called elliptic geometry [ 24 , 30 ] . Geodesic distances of Pd naturally account for the fact that they compare sets . Let dγ : Sd × Sd → R be the geodesic distance of Sd ( i.e. , spherical distance ) . The geodesic distance between [ x ] ∈ Pd and [ y ] ∈ Pd is dγ ( [ x ] , [ y ] ) = mina∈ [ x ] , b∈ [ y ] dγ ( a , b ) . We then have : dγ ( [ x ] , [ y ] ) : = min { dγ ( x , y ) , dγ ( −x , y ) } = cos−1 ( |〈x , y〉| ) = cos−1 ( |〈x , y〉q| ) , ( 4 ) which is a distance metric . The fact that the spherical geometry is antipodally symmetric ( i.e. , every point can be inverted w.r.t . the origin ) leads to a duplication of geometric information [ 24 ] . Identifying each pair of antipodal points to one point eliminates the antipodal duplication in spherical geometry . The hyperboloid model of hyperbolic geometry is similar to the geometry of P0 , q1 ( p = 0 ) . The q-dimensional manifold S0 , q1 ⊂ R1 , q contains two separate sheets ( i.e. , two connected components ) and is anti-isometric to the hyperboloid of two sheetsQq,01 . Pairs of antipodal points lying on different sheets of S0 , q1 are considered as a single point of P 0 , q 1 . Let x ∈ S 0 , q 1 and z ∈ S 0 , q 1 be two points lying on the same sheet of S0 , q1 , there exists no geodesic joining x and −z . Their geodesic distance with respect to S0 , q1 can then be considered to be dγ ( x , −z ) = +∞ , and we have : dγ ( [ x ] , [ z ] ) : = min { dγ ( x , z ) , +∞ } = dγ ( x , z ) = cosh−1 ( 〈x , z〉q ) = cosh−1 ( |〈x , z〉q| ) , ( 5 ) which is similar to the hyperbolic distance metric of the hyperboloid model studied in [ 20 ] . Ultrahyperbolic geometry ( or indefinite elliptic geometry ) . In this paper , we propose a parametric model that learns representations lying on the quotient manifold Pp , qr . When both p and q are positive , the metric tensor of Pp , qr is nondegenerate ( see page 343 of [ 16 ] ) and indefinite . This means that the manifold is pseudo-Riemannian but not Riemannian due to the lack of positive definiteness of the metric tensor . Pp , qr is also called an indefinite elliptic space [ 30 ] in the literature . We refer the reader to Chapters 11 and 12 of [ 30 ] or Chapter 7 of [ 21 ] for details . As an example , Fig . 1 illustrates the manifold P1,11 . Our main motivation for considering Pp , qr is that it is more flexible than hyperbolic and elliptic geometries since it contains hyperbolic and elliptic parts ( i.e. , time-like and space-like geodesics in Fig . 1 ) . This flexibility allows us to better represent graphs that are not entirely trees or cycles , but that contain tree-like or cycle subgraphs . We experimentally verify our intuition .

Summary: This paper presents a (graph) neural network in ultrahyperbolic (semi-Riemannian) space that can be used to model hierarchical graphs with cycles. The motivation is that ultrahyperbolic manifold generalizes hyperbolic and spherical manifolds, thus providing inductive bias to those geometries. In order to avoid broken cases, the paper considers ultrahyperbolic quotient manifold such that every two points in the manifold can be connected by a geodesic. The method is evaluated in graph and node classification tasks.