Constant Curvature Graph Convolutional Networks

1 INTRODUCTION . Graph Convolutional Networks . The success of convolutional networks and deep learning for image data has inspired generalizations for graphs for which sharing parameters is consistent with the graph geometry . Bruna et al . ( 2014 ) ; Henaff et al . ( 2015 ) are the pioneers of spectral graph convolutional neural networks in the graph Fourier space using localized spectral filters on graphs . However , in order to reduce the graph-dependency on the Laplacian eigenmodes , Defferrard et al . ( 2016 ) approximate the convolutional filters using Chebyshev polynomials leveraging a result of Hammond et al . ( 2011 ) . The resulting method ( discussed in appendix A ) is computationally efficient and superior in terms of accuracy and complexity . Further , Kipf & Welling ( 2017 ) simplify this approach by considering first-order polynomials approximations obtaining high scalability . The proposed graph convolutional networks ( GCN ) is interpolating node embeddings via a symmetrically normalized adjacency matrix , while this weight sharing can be understood as an efficient diffusion-like regularizer . Recent works extend GCNs to achieve state of the art results for link prediction ( Zhang & Chen , 2018 ) , graph classification ( Hamilton et al. , 2017 ; Xu et al. , 2018 ) and node classification ( Klicpera et al. , 2019 ; Veličković et al. , 2018 ) . Euclidean geometry in ML . In machine learning ( ML ) , data is most often represented in a Euclidean space for various reasons . First , some data is intrinsically Euclidean , such as positions in 3D space in classical mechanics . Second , intuition is easier in such spaces , as they possess an appealing vectorial structure allowing basic arithmetic and a rich theory of linear algebra . Finally , a lot of quantities of interest such as distances and inner-products are known in closed-form formulae and can be computed very efficiently on the existing hardware . These operations are the basic building blocks for most of today ’ s popular machine learning models . Thus , the powerful simplicity and efficiency of Euclidean geometry has led to numerous methods achieving state-of-the-art on tasks as diverse as machine translation ( Bahdanau et al. , 2014 ; Vaswani et al. , 2017 ) , speech recognition ( Graves et al. , 2013 ) , image classification ( He et al. , 2016 ) or recommender systems ( He et al. , 2017 ) . Riemannian ML . In spite of this success , certain types of data ( e.g . hierarchical , scale-free or spherical data ) have been shown to be better represented by non-Euclidean geometries ( Defferrard et al. , 2019 ; Bronstein et al. , 2017 ; Nickel & Kiela , 2017 ; Gu et al. , 2019 ) , leading in particular to the rich theories of manifold learning ( Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ) and information geometry ( Amari & Nagaoka , 2007 ) . The mathematical framework in vigor to manipulate nonEuclidean geometries is known as Riemannian geometry ( Spivak , 1979 ) . Although its theory leads to many strong and elegant results , some of its basic quantities such as the distance function d ( · , · ) are in general not available in closed-form , which can be prohibitive to many computational methods . Representational Advantages of Geometries of Constant Curvature . An interesting trade-off between general Riemannian manifolds and the Euclidean space is given by manifolds of constant sectional curvature . They define together what are called hyperbolic ( negative curvature ) , elliptic ( positive curvature ) and Euclidean ( zero curvature ) geometries . As discussed below and in appendix B , Euclidean spaces have limitations and suffer from large distortion when embedding certain types of data such as trees , e.g . fig . 1 . In these cases , the hyperbolic and spherical spaces have representational advantages providing a better inductive bias for the respective data . The hyperbolic space can be intuitively understood as a continuous tree : the volume of a ball grows exponentially with its radius , similarly as how the number of nodes in a binary tree grows exponentially with its depth . Its tree-likeness properties have long been studied mathematically ( Gromov , 1987 ; Hamann , 2017 ; Ungar , 2008 ) and it was proven to better embed complex networks ( Krioukov et al. , 2010 ) , scale-free graphs and hierarchical data compared to the Euclidean geometry ( Cho et al. , 2019 ; Sala et al. , 2018 ; Ganea et al. , 2018b ; Gu et al. , 2019 ; Nickel & Kiela , 2018 ; 2017 ; Tifrea et al. , 2019 ) . Several important tools or methods found their hyperbolic counterparts , such as variational autoencoders ( Mathieu et al. , 2019 ; Ovinnikov , 2019 ) , attention mechanisms ( Gulcehre et al. , 2018 ) , matrix multiplications , recurrent units and multinomial logistic regression ( Ganea et al. , 2018a ) . Similarly , spherical geometry provides benefits for modeling spherical or cyclical data ( Defferrard et al. , 2019 ; Matousek , 2013 ; Davidson et al. , 2018 ; Xu & Durrett , 2018 ; Gu et al. , 2019 ; Grattarola et al. , 2018 ; Wilson et al. , 2014 ) . Computational Efficiency of Constant Curvature Spaces ( CCS ) . CCS are some of the few Riemannian manifolds to possess closed-form formulae for geometric quantities of interest in computational methods , i.e . distance , geodesics , exponential map , parallel transport and their gradients . We also leverage here the closed expressions for weighted centroids . “ Linear Algebra ” of CCS : Gyrovector Spaces . In order to study the geometry of constant negative curvature in analogy with the Euclidean geometry , Ungar ( 1999 ; 2005 ; 2008 ; 2016 ) proposed the elegant non-associative algebraic formalism of gyrovector spaces . Recently , Ganea et al . ( 2018a ) have linked this framework to the Riemannian geometry of the space , also generalizing the building blocks for non-Euclidean deep learning models operating with hyperbolic data representations . However , it remains unclear how to extend in a principled manner the connection between Riemannian geometry and gyrovector space operations for spaces of constant positive curvature ( spherical ) . By leveraging Euler ’ s formula and complex analysis , we present to our knowledge the first unified gyro framework that smoothly interpolates between geometries of constant curvatures irrespective of their signs . This is possible when working with the Poincaré ball and stereographic spherical projection models of respectively hyperbolic and spherical spaces . How should one adapt graph neural networks to non-flat geometries of constant curvature ? In this work , we propose constant curvature GCNs to model non-Euclidean data . Node embeddings lie in spaces of constant curvature or product of those instead of a Euclidean space , thus leveraging both the representational power of these geometries and the effectiveness of GCNs . Concurrent to our work , Chami et al . ( 2019 ) ; Liu et al . ( 2019 ) propose hyperbolic graph neural networks using tangent space aggregation . 2 THE GEOMETRY OF CONSTANT CURVATURE SPACES . Riemannian Geometry . A manifoldM of dimension d is a generalization to higher dimensions of the notion of surface , and is a space that locally looks like Rd . At each point x ∈M , M can be associated a tangent space TxM , which is a vector space of dimension d that can be understood as a first order approximation ofM around x . A riemannian metric g is given by an inner-product gx ( · , · ) at each tangent space TxM , gx varying smoothly with x . A given g defines the geometry ofM , because it can be used to define the distance between x and y as the infimum of the lengths of smooth paths γ : [ 0 , 1 ] →M from x to y , where the length is defined as ` ( γ ) : = ∫ 1 0 √ gγ ( t ) ( γ̇ ( t ) , γ̇ ( t ) ) dt . Under certain assumptions , a given g also defines a curvature at each point . Unifying all curvatures κ . There exist several models of respectively constant positive and negative curvatures . For positive curvature , we choose the stereographic projection of the sphere , while for negative curvature we choose the Poincaré model which is the stereographic projection of the Lorentz model . As explained below , this choice allows us to generalize the gyrovector space framework and unify spaces of both positive and negative curvature κ into a single model which we call the κ-stereographic model . The κ-stereographic model . For a curvature κ ∈ R and a dimension d ≥ 2 , it is defined as stdκ = { x ∈ Rd | −κ‖x‖22 < 1 } equipped with its Riemannian metric gκx = 4 ( 1+κ||x||2 ) 2 I = : ( λ κ x ) 2I . Note in particular that when κ ≥ 0 , stdκ is Rd , while when κ < 0 it is the open ball of radius 1/ √ −κ . Gyrovector spaces & Riemannian geometry . As discussed in section 1 , the gyrovector space formalism is used to generalize vector spaces to the Poincaré model of hyperbolic geometry ( Ungar , 2005 ; 2008 ) . In addition , important quantities from Riemannian geometry can be rewritten in terms of the Möbius vector addition and scalar-vector multiplication ( Ganea et al. , 2018a ) . We here extend gyrovector spaces to the κ-stereographic model , i.e . allowing positive curvature . For κ > 0 and any point x ∈ stdκ , we will denote by x̃ the unique point of the sphere of radius κ− 1 2 in Rd+1 whose stereographic projection is x . As detailed in appendix C.2.2 , it is given by x̃ : = ( λκxx , κ − 12 ( λκx − 1 ) ) . ( 1 ) For x , y ∈ stdκ , we define the κ-addition , in the κ-stereographic model by : x⊕κ y = ( 1− 2κxTy − κ||y||2 ) x + ( 1 + κ||x||2 ) y 1− 2κxTy + κ2||x||2||y||2 ∈ stdκ . ( 2 ) The κ-addition is defined in all the cases except for spherical geometry and x = y/ ( κ‖y‖2 ) as stated by the following theorem proved in Appendix C.2.1 . Theorem 1 ( Definiteness of κ-addition ) . We have 1 − 2κxTy + κ2||x||2||y||2 = 0 if and only if κ > 0 and x = y/ ( κ‖y‖2 ) . For s ∈ R and x ∈ stdκ ( and |s tan−1κ ‖x‖| < κ 1 2π/2 if κ > 0 ) , the κ-scaling in the κ-stereographic model is given by : s⊗κ x = tanκ ( s · tan−1κ ||x|| ) x ||x|| ∈ stdκ , ( 3 ) where tanκ equals κ−1/2 tan if κ > 0 and ( −κ ) −1/2 tanh if κ < 0 . This formalism yields simple closed-forms for various quantities including the distance function inherited from the Riemannian manifold ( stdκ , g κ ) , the exp and log maps , and geodesics , as shown by the following theorem . Theorem 2 ( Extending gyrovector spaces to positive curvature ) . For x , y ∈ stdκ , x 6= y , v 6= 0 , ( and x 6= −y/ ( κ‖y‖2 ) if κ > 0 ) , the distance function is given bya : dκ ( x , y ) = 2|κ|−1/2 tan−1κ ‖ − x⊕κ y‖ , ( 4 ) the unit-speed geodesic from x to y is unique and given by γx→y ( t ) = x⊕κ ( t⊗κ ( −x⊕κ y ) ) , ( 5 ) and finally the exponential and logarithmic maps are described as : expκx ( v ) = x⊕κ ( tanκ ( |κ| 12 λ κ x||v|| 2 ) v ||v|| ) ; logκx ( y ) = 2|κ|− 12 λκx tan−1κ ||−x⊕κy|| −x⊕κ y || − x⊕k y|| ( 6 ) aWe write −x⊕ y for ( −x ) ⊕ y and not − ( x⊕ y ) .

Summary: This paper builds a new graph convolutional network (GCN) based on hyperbolic representations of the graph nodes: all latent representations of the nodes are points on Poincare disks. The authors adapted the Hyperbolic Neural Networks by Ganea et al. (2018) into the Kipf & Welling's (2016) version of GCN. Specifically, the authors variated the right matrix multiplication in GCN with Ganea et al.'s Mobius matrix-vector multiplication (that can be regarded as a transformation between two Poincare disks). Moreover, as a non-trivial adaptation, the author defined the left matrix multiplication in GCN (that can be regarded as a weighted linear combination of several points on the same Poincare disk) with Ungar's (2010) weighted barycenter, which is from computational geometry but not the machine learning community. The resulting method is tested on a toy problem and semi-supervised node classification of commonly used datasets, showing the possibility of improvement.