Constant Curvature Graph Convolutional Networks

1 INTRODUCTION . Graph Convolutional Networks . The success of convolutional networks and deep learning for image data has inspired generalizations for graphs for which sharing parameters is consistent with the graph geometry . Bruna et al . ( 2014 ) ; Henaff et al . ( 2015 ) are the pioneers of spectral graph convolutional neural networks in the graph Fourier space using localized spectral filters on graphs . However , in order to reduce the graph-dependency on the Laplacian eigenmodes , Defferrard et al . ( 2016 ) approximate the convolutional filters using Chebyshev polynomials leveraging a result of Hammond et al . ( 2011 ) . The resulting method ( discussed in appendix A ) is computationally efficient and superior in terms of accuracy and complexity . Further , Kipf & Welling ( 2017 ) simplify this approach by considering first-order polynomials approximations obtaining high scalability . The proposed graph convolutional networks ( GCN ) is interpolating node embeddings via a symmetrically normalized adjacency matrix , while this weight sharing can be understood as an efficient diffusion-like regularizer . Recent works extend GCNs to achieve state of the art results for link prediction ( Zhang & Chen , 2018 ) , graph classification ( Hamilton et al. , 2017 ; Xu et al. , 2018 ) and node classification ( Klicpera et al. , 2019 ; Veličković et al. , 2018 ) . Euclidean geometry in ML . In machine learning ( ML ) , data is most often represented in a Euclidean space for various reasons . First , some data is intrinsically Euclidean , such as positions in 3D space in classical mechanics . Second , intuition is easier in such spaces , as they possess an appealing vectorial structure allowing basic arithmetic and a rich theory of linear algebra . Finally , a lot of quantities of interest such as distances and inner-products are known in closed-form formulae and can be computed very efficiently on the existing hardware . These operations are the basic building blocks for most of today ’ s popular machine learning models . Thus , the powerful simplicity and efficiency of Euclidean geometry has led to numerous methods achieving state-of-the-art on tasks as diverse as machine translation ( Bahdanau et al. , 2014 ; Vaswani et al. , 2017 ) , speech recognition ( Graves et al. , 2013 ) , image classification ( He et al. , 2016 ) or recommender systems ( He et al. , 2017 ) . Riemannian ML . In spite of this success , certain types of data ( e.g . hierarchical , scale-free or spherical data ) have been shown to be better represented by non-Euclidean geometries ( Defferrard et al. , 2019 ; Bronstein et al. , 2017 ; Nickel & Kiela , 2017 ; Gu et al. , 2019 ) , leading in particular to the rich theories of manifold learning ( Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ) and information geometry ( Amari & Nagaoka , 2007 ) . The mathematical framework in vigor to manipulate nonEuclidean geometries is known as Riemannian geometry ( Spivak , 1979 ) . Although its theory leads to many strong and elegant results , some of its basic quantities such as the distance function d ( · , · ) are in general not available in closed-form , which can be prohibitive to many computational methods . Representational Advantages of Geometries of Constant Curvature . An interesting trade-off between general Riemannian manifolds and the Euclidean space is given by manifolds of constant sectional curvature . They define together what are called hyperbolic ( negative curvature ) , elliptic ( positive curvature ) and Euclidean ( zero curvature ) geometries . As discussed below and in appendix B , Euclidean spaces have limitations and suffer from large distortion when embedding certain types of data such as trees , e.g . fig . 1 . In these cases , the hyperbolic and spherical spaces have representational advantages providing a better inductive bias for the respective data . The hyperbolic space can be intuitively understood as a continuous tree : the volume of a ball grows exponentially with its radius , similarly as how the number of nodes in a binary tree grows exponentially with its depth . Its tree-likeness properties have long been studied mathematically ( Gromov , 1987 ; Hamann , 2017 ; Ungar , 2008 ) and it was proven to better embed complex networks ( Krioukov et al. , 2010 ) , scale-free graphs and hierarchical data compared to the Euclidean geometry ( Cho et al. , 2019 ; Sala et al. , 2018 ; Ganea et al. , 2018b ; Gu et al. , 2019 ; Nickel & Kiela , 2018 ; 2017 ; Tifrea et al. , 2019 ) . Several important tools or methods found their hyperbolic counterparts , such as variational autoencoders ( Mathieu et al. , 2019 ; Ovinnikov , 2019 ) , attention mechanisms ( Gulcehre et al. , 2018 ) , matrix multiplications , recurrent units and multinomial logistic regression ( Ganea et al. , 2018a ) . Similarly , spherical geometry provides benefits for modeling spherical or cyclical data ( Defferrard et al. , 2019 ; Matousek , 2013 ; Davidson et al. , 2018 ; Xu & Durrett , 2018 ; Gu et al. , 2019 ; Grattarola et al. , 2018 ; Wilson et al. , 2014 ) . Computational Efficiency of Constant Curvature Spaces ( CCS ) . CCS are some of the few Riemannian manifolds to possess closed-form formulae for geometric quantities of interest in computational methods , i.e . distance , geodesics , exponential map , parallel transport and their gradients . We also leverage here the closed expressions for weighted centroids . “ Linear Algebra ” of CCS : Gyrovector Spaces . In order to study the geometry of constant negative curvature in analogy with the Euclidean geometry , Ungar ( 1999 ; 2005 ; 2008 ; 2016 ) proposed the elegant non-associative algebraic formalism of gyrovector spaces . Recently , Ganea et al . ( 2018a ) have linked this framework to the Riemannian geometry of the space , also generalizing the building blocks for non-Euclidean deep learning models operating with hyperbolic data representations . However , it remains unclear how to extend in a principled manner the connection between Riemannian geometry and gyrovector space operations for spaces of constant positive curvature ( spherical ) . By leveraging Euler ’ s formula and complex analysis , we present to our knowledge the first unified gyro framework that smoothly interpolates between geometries of constant curvatures irrespective of their signs . This is possible when working with the Poincaré ball and stereographic spherical projection models of respectively hyperbolic and spherical spaces . How should one adapt graph neural networks to non-flat geometries of constant curvature ? In this work , we propose constant curvature GCNs to model non-Euclidean data . Node embeddings lie in spaces of constant curvature or product of those instead of a Euclidean space , thus leveraging both the representational power of these geometries and the effectiveness of GCNs . Concurrent to our work , Chami et al . ( 2019 ) ; Liu et al . ( 2019 ) propose hyperbolic graph neural networks using tangent space aggregation . 2 THE GEOMETRY OF CONSTANT CURVATURE SPACES . Riemannian Geometry . A manifoldM of dimension d is a generalization to higher dimensions of the notion of surface , and is a space that locally looks like Rd . At each point x ∈M , M can be associated a tangent space TxM , which is a vector space of dimension d that can be understood as a first order approximation ofM around x . A riemannian metric g is given by an inner-product gx ( · , · ) at each tangent space TxM , gx varying smoothly with x . A given g defines the geometry ofM , because it can be used to define the distance between x and y as the infimum of the lengths of smooth paths γ : [ 0 , 1 ] →M from x to y , where the length is defined as ` ( γ ) : = ∫ 1 0 √ gγ ( t ) ( γ̇ ( t ) , γ̇ ( t ) ) dt . Under certain assumptions , a given g also defines a curvature at each point . Unifying all curvatures κ . There exist several models of respectively constant positive and negative curvatures . For positive curvature , we choose the stereographic projection of the sphere , while for negative curvature we choose the Poincaré model which is the stereographic projection of the Lorentz model . As explained below , this choice allows us to generalize the gyrovector space framework and unify spaces of both positive and negative curvature κ into a single model which we call the κ-stereographic model . The κ-stereographic model . For a curvature κ ∈ R and a dimension d ≥ 2 , it is defined as stdκ = { x ∈ Rd | −κ‖x‖22 < 1 } equipped with its Riemannian metric gκx = 4 ( 1+κ||x||2 ) 2 I = : ( λ κ x ) 2I . Note in particular that when κ ≥ 0 , stdκ is Rd , while when κ < 0 it is the open ball of radius 1/ √ −κ . Gyrovector spaces & Riemannian geometry . As discussed in section 1 , the gyrovector space formalism is used to generalize vector spaces to the Poincaré model of hyperbolic geometry ( Ungar , 2005 ; 2008 ) . In addition , important quantities from Riemannian geometry can be rewritten in terms of the Möbius vector addition and scalar-vector multiplication ( Ganea et al. , 2018a ) . We here extend gyrovector spaces to the κ-stereographic model , i.e . allowing positive curvature . For κ > 0 and any point x ∈ stdκ , we will denote by x̃ the unique point of the sphere of radius κ− 1 2 in Rd+1 whose stereographic projection is x . As detailed in appendix C.2.2 , it is given by x̃ : = ( λκxx , κ − 12 ( λκx − 1 ) ) . ( 1 ) For x , y ∈ stdκ , we define the κ-addition , in the κ-stereographic model by : x⊕κ y = ( 1− 2κxTy − κ||y||2 ) x + ( 1 + κ||x||2 ) y 1− 2κxTy + κ2||x||2||y||2 ∈ stdκ . ( 2 ) The κ-addition is defined in all the cases except for spherical geometry and x = y/ ( κ‖y‖2 ) as stated by the following theorem proved in Appendix C.2.1 . Theorem 1 ( Definiteness of κ-addition ) . We have 1 − 2κxTy + κ2||x||2||y||2 = 0 if and only if κ > 0 and x = y/ ( κ‖y‖2 ) . For s ∈ R and x ∈ stdκ ( and |s tan−1κ ‖x‖| < κ 1 2π/2 if κ > 0 ) , the κ-scaling in the κ-stereographic model is given by : s⊗κ x = tanκ ( s · tan−1κ ||x|| ) x ||x|| ∈ stdκ , ( 3 ) where tanκ equals κ−1/2 tan if κ > 0 and ( −κ ) −1/2 tanh if κ < 0 . This formalism yields simple closed-forms for various quantities including the distance function inherited from the Riemannian manifold ( stdκ , g κ ) , the exp and log maps , and geodesics , as shown by the following theorem . Theorem 2 ( Extending gyrovector spaces to positive curvature ) . For x , y ∈ stdκ , x 6= y , v 6= 0 , ( and x 6= −y/ ( κ‖y‖2 ) if κ > 0 ) , the distance function is given bya : dκ ( x , y ) = 2|κ|−1/2 tan−1κ ‖ − x⊕κ y‖ , ( 4 ) the unit-speed geodesic from x to y is unique and given by γx→y ( t ) = x⊕κ ( t⊗κ ( −x⊕κ y ) ) , ( 5 ) and finally the exponential and logarithmic maps are described as : expκx ( v ) = x⊕κ ( tanκ ( |κ| 12 λ κ x||v|| 2 ) v ||v|| ) ; logκx ( y ) = 2|κ|− 12 λκx tan−1κ ||−x⊕κy|| −x⊕κ y || − x⊕k y|| ( 6 ) aWe write −x⊕ y for ( −x ) ⊕ y and not − ( x⊕ y ) .

Summary: The authors propose using non-Euclidean spaces for GCNs. This is inspired by the recent work into non-Euclidean, and especially hyperbolic, embeddings. A few papers have recently tried to go past embeddings into building non-Euclidean models, requiring the lifting of standard operations in Euclidean space to non-Euclidean settings. This has been done in particular in hyperbolic space, but some datasets benefit from more complex spaces. The authors combine the mixed-curvature product formalism that uses products of Euclidean, hyperbolic, and spherical spaces for embeddings, but use these for GCN operations.