Geom-GCN: Geometric Graph Convolutional Networks

1 INTRODUCTION . Message-passing neural networks ( MPNNs ) , such as GNN ( Scarselli et al. , 2008 ) , ChebNet ( Defferrard et al. , 2016 ) , GG-NN ( Li et al. , 2016 ) , GCN ( Kipf & Welling , 2017 ) , are powerful for learning on graphs with various applications ranging from brain networks to online social network ( Gilmer et al. , 2017 ; Wang et al. , 2019 ) . In a layer of MPNNs , each node sends its feature representation , a “ message ” , to the nodes in its neighborhood ; and then updates its feature representation by aggregating all “ messages ” received from the neighborhood . The neighborhood is often defined as the set of adjacent nodes in graph . By adopting permutation-invariant aggregation functions ( e.g. , summation , maximum , and mean ) , MPNNs are able to learn representations which are invariant to isomorphic graphs , i.e. , graphs that are topologically identical . Although existing MPNNs have been successfully applied in a wide variety of scenarios , two fundamental weaknesses of MPNNs ’ aggregators limit their ability to represent graph-structured data . Firstly , the aggregators lose the structural information of nodes in neighborhoods . Permutation invariance is an essential requirement for any graph learning method . To meet it , existing MPNNs adopt permutation-invariant aggregation functions which treat all “ messages ” from neighborhood as ∗This work is conducted partially during his visit at University of Illinois at Urbana-Champaign . †Corresponding author . a set . For instance , GCN simply sums the normalized “ messages ” from all one-hop neighbors ( Kipf & Welling , 2017 ) . Such aggregation loses the structural information of nodes in neighborhood because it does not distinguish the “ messages ” from different nodes . Therefore , after such aggregation , we can not know which node contributes what to the final aggregated output . Without modeling such structural information , as shown in ( Kondor et al. , 2018 ) and ( Xu et al. , 2019 ) , the existing MPNNs can not discriminate between certain non-isomorphic graphs . In those cases , MPNNs may map non-isomorphic graphs to the same feature representations , which is obviously not desirable for graph representation learning . Unlike MPNNs , classical convolutional neural networks ( CNNs ) avoid this problem by using aggregators ( i.e. , convolutional filters ) with a structural receiving filed defined on grids , i.e. , a Euclidean space , and are hence able to distinguish each input unit . As shown by our experiments , such structural information often contains clues regarding topology patterns in graph ( e.g. , hierarchy ) , and should be extracted and used to learn more discriminating representations for graph-structured data . Secondly , the aggregators lack the ability to capture long-range dependencies in disassortative graphs . In MPNNs , the neighborhood is defined as the set of all neighbors one hop away ( e.g. , GCN ) , or all neighbors up to r hops away ( e.g. , ChebNet ) . In other words , only messages from nearby nodes are aggregated . The MPNNs with such aggregation are inclined to learn similar representations for proximal nodes in a graph . This implies that they are probably desirable methods for assortative graphs ( e.g. , citation networks ( Kipf & Welling , 2017 ) and community networks ( Chen et al. , 2019 ) ) where node homophily holds ( i.e. , similar nodes are more likely to be proximal , and vice versa ) , but may be inappropriate to the disassortative graphs ( Newman , 2002 ) where node homophily does not hold . For example , Ribeiro et al . ( 2017 ) shows disassortative graphs where nodes of the same class exhibit high structural similarity but are far apart from each other . In such cases , the representation ability of MPNNs may be limited significantly , since they can not capture the important features from distant but informative nodes . A straightforward strategy to address this limitation is to use a multi-layered architecture so as to receive “ messages ” from distant nodes . For instance , due to the localized nature of convolutional filters in classical CNNs , a single convolutional layer is similarly limited in its representational ability . CNNs typically use multiple layers connected in a hierarchical manner to learn complex and global representations . However , unlike CNNs , it is difficult for multi-layer MPNNs to learn good representations for disassortative graphs because of two reasons . On one hand , relevant messages from distant nodes are mixed indistinguishably with a large number of irrelevant messages from proximal nodes in multi-layer MPNNs , which implies that the relevant information will be “ washed out ” and can not be extracted effectively . On the other hand , the representations of different nodes would become very similar in multi-layer MPNNs , and every node ’ s representation actually carries the information about the entire graph ( Xu et al. , 2018 ) . In this paper , we overcome the aforementioned weaknesses of graph neural networks starting from two basic observations : i ) Classical neural networks effectively address the similar limitations thanks to the stationarity , locality , and compositionality in a continuous space ( Bronstein et al. , 2017 ) ; ii ) The notion of network geometry bridges the gap between continuous space and graph ( Hoff et al. , 2002 ; Muscoloni et al. , 2017 ) . Network geometry aims to understand networks by revealing the latent continuous space underlying them , which assumes that nodes are sampled discretely from a latent continuous space and edges are established according to their distance . In the latent space , complicated topology patterns in graphs can be preserved and presented as intuitive geometry , such as subgraph ( Narayanan et al. , 2016 ) , community ( Ni et al. , 2019 ) , and hierarchy ( Nickel & Kiela , 2017 ; 2018 ) . Inspired by those two observations , we raise an enlightening question about the aggregation scheme in graph neural network . • Can the aggregation on a graph benefit from a continuous latent space , such as using geometry in the space to build structural neighborhoods and capture long-range dependencies in the graph ? To answer the above question , we propose a novel aggregation scheme for graph neural networks , termed the geometric aggregation scheme . In the scheme , we map a graph to a continuous latent space via node embedding , and then use the geometric relationships defined in the latent space to build structural neighborhoods for aggregation . Also , we design a bi-level aggregator operating on the structural neighborhoods to update the feature representations of nodes in graph neural networks , which are able to guarantee permutation invariance for graph-structured data . Compared with exist- ing MPNNs , the scheme extracts more structural information of the graph and can aggregate feature representations from distant nodes via mapping them to neighborhoods defined in the latent space . We then present an implementation of the geometric aggregation scheme in graph convolutional networks , which we call Geom-GCN , to perform transductive learning , node classification , on graphs . We design particular geometric relationships to build the structural neighborhood in Euclidean and hyperbolic embedding space respectively . We choose different embedding methods to map the graph to a suitable latent space for different applications , where suitable topology patterns of graph are preserved . Finally , we empirically validate and analyze Geom-GCN on a wide range of open datasets of graphs , and Geom-GCN achieved the state-of-the-art results . In summary , the contribution of this paper is three-fold : i ) We propose a novel geometric aggregation scheme for graph neural network , which operates in both graph and latent space , to overcome the aforementioned two weaknesses ; ii ) We present an implementation of the scheme , Geom-GCN , for transductive learning in graph ; iii ) We validate and analyze Geom-GCN via extensive comparisons with state-of-the-art methods on several challenging benchmarks . 2 GEOMETRIC AGGREGATION SCHEME . In this section , we start by presenting the geometric aggregation scheme , and then outline its advantages and limitations compared to existing works . As shown in Fig . 1 , the aggregation scheme consists of three modules , node embedding ( panel A1 and A2 ) , structural neighborhood ( panel B1 and B2 ) , and bi-level aggregation ( panel C ) . We will elaborate on them in the following . A. Node embedding . This is a fundamental module which maps the nodes in a graph to a latent continuous space . Let G = ( V , E ) be a graph , where each node v ∈ V has a feature vector xv and each edge e ∈ E connects two nodes . Let f : v → zv be a mapping function from a node in graph to a representation vector . Here , zv ∈ Rd can also be considered as the position of node v in a latent continuous space , and d is the number of dimensions of the space . During the mapping , the structure and properties of graph are preserved and presented as the geometry in the latent space . For instance , hierarchical pattern in graph is presented as the distance to the original in embedding hyperbolic space ( Nickel & Kiela , 2017 ) . One can employ various embedding methods to infer the latent space ( Cai et al. , 2018 ; Wang et al. , 2018 ) . B . Structural neighborhood . Based on the graph and the latent space , we then build a structural neighborhood , N ( v ) = ( { Ng ( v ) , Ns ( v ) } , τ ) , for the next aggregation . The structural neighborhood consists of a set of neighborhood { Ng ( v ) , Ns ( v ) } , and a relational operator on neighborhoods τ . The neighborhood in the graph , Ng ( v ) = { u|u ∈ V , ( u , v ) ∈ E } , is the set of adjacent nodes of v. The neighborhood in the latent space , Ns ( v ) = { u|u ∈ V , d ( zu , zv ) < ρ } , is the set of nodes from which the distance to v is less than a pre-given parameter ρ . The distance function d ( · , · ) depends on the particular metric in the space . Compared with Ng ( v ) , Ns ( v ) may contain nodes which are far from v in the graph , but have a certain similarity with v , and hence are mapped together with v in the latent space though preserving the similarity . By aggregating on such neighborhood Ns ( v ) , the long-range dependencies in disassortative graphs can be captured . The relational operator τ is a function defined in the latent space . It inputs an ordered position pair ( zv , zu ) of nodes v and u , and outputs a discrete variable r which indicates the geometric relationship from v to u in the latent space . For u , v ∈ V , τ : ( zv , zu ) → r ∈ R , where R is the set of the geometric relationships . According to the particular latent space and application , r can be specified as an arbitrary geometric relationship of interest . A requirement on τ is that it should guarantee that each ordered position pair has only one geometric relationship . For example , τ is illustrated in Fig . 1B by a colorful 3 × 3 grid in a 2-dimensional Euclidean space , in which each unit is corresponding to a geometric relationship to node v. C. Bi-level aggregation . With the structural neighborhood N ( v ) , we propose a novel bi-level aggregation scheme for graph neural network to update the hidden features of nodes . The bi-level aggregation consists of two aggregation functions and operates in a neural network layer . It can extract effectively structural information of nodes in neighborhoods as well as guarantee permutation invariance for graph . Let hlv be the hidden features of node v at the l-th layer , and h 0 v = xv be the node features . The l-th layer updates hlv for every v ∈ V by the following . ev , l+1 ( i , r ) = p ( { h l u|u ∈ Ni ( v ) , τ ( zv , zu ) = r } ) , ∀i ∈ { g , s } , ∀r ∈ R ( Low-level aggregation ) ml+1v = q i∈ { g , s } , r∈R ( ( ev , l+1 ( i , r ) , ( i , r ) ) ) ( High-level aggregation ) hl+1v = σ ( Wl ·ml+1v ) ( Non-linear transform ) ( 1 ) In the low-level , the hidden features of nodes that are in the same neighborhood i and have the same geometric relationship r are aggregated to a virtual node via the aggregation function p. The features of the virtual node are ev , l+1 ( i , r ) , and the virtual node is indexed by ( i , r ) which is corresponding to the combination of a neighborhood i and a relationship r. It is required to adopt a permutation-invariant function for p , such as an Lp-norm ( the choice of p = 1 , 2 , or∞ results in average , energy , or max pooling ) . The low level aggregation is illustrated by dashed arrows in Fig . 1C . In the high-level , the features of virtual nodes are further aggregated by function q . The inputs of function q contain both the features of virtual nodes ev , l+1 ( i , r ) and the identity of virtual nodes ( i , r ) . That is , q can be a function that take an ordered object as input , e.g. , concatenation , to distinguish the features of different virtual nodes , thereby extracting the structural information in the neighborhoods explicitly . The output of high-level aggregation is a vector ml+1v . Then new hidden features of v , h ( l+1 ) v , are given by a non-linear transform , wherein Wl is a learnable weight matrix on the l-th layer shared by all nodes , and σ ( · ) is a non-linear activation function , e.g. , a ReLU . Permutation invariance is an essential requirement for aggregators in graph neural networks . Thus , we then prove that the proposed bi-level aggregation , Eq . 1 , is able to guarantee invariance for any permutation of nodes . We firstly give a definition for permutation-invariant mapping of graph . Definition 1 . Let a bijective function ψ : V → V be a permutation for nodes , which renames v ∈ V as ψ ( v ) ∈ V . Let V ′ and E′ be the node and edge set after a permutation ψ , respectively . A mapping of graph , φ ( G ) , is permutation-invariant if , given any permutation ψ , we have φ ( G ) = φ ( G′ ) , G′ = ( V ′ , E′ ) . Lemma 1 . For a composite function φ1 ◦ φ2 ( G ) , if φ2 ( G ) is permutation-invariant , the entire composite function φ1 ◦ φ2 ( G ) is permutation-invariant . Proof . Let G′ be an isomorphic graph of G after a permutation ψ , as defined in Definition 1 . If φ2 ( G ) is permutation-invariant , we have φ2 ( G ) = φ2 ( G ′ ) . Therefore , the entire composite function φ1 ◦ φ2 ( G ) is permutation-invariant because φ1 ◦ φ2 ( G ) = φ1 ◦ φ2 ( G ′ ) . Theorem 1 . Given a graph G = ( V , E ) and its structural neighborhoodN ( v ) , ∀v ∈ V , the bi-level aggregation , Eq . 1 , is a permutation-invariant mapping of graph . Proof . The bi-level aggregation , Eq . 1 , is a composite function , where the low-level aggregation is the input of the high-level aggregation . Thus , Eq . 1 is permutation-invariant if the low-level aggregation is permutation-invariant according to Lemma 1 . We then prove that the low-level aggregation is permutation-invariant . The low-level aggregation consists of 2×|R| sub-aggregations , each of which is corresponding to the nodes in a neighborhood i and with a relationship r to v. Firstly , the input of each sub-aggregations is permutation-invariant because both i ∈ { g , s } and r ∈ R are determined by the given structural neighborhoodN ( v ) , ∀v ∈ V , which is constant for any permutation . Secondly , Eq . 1 adopts a permutation-invariant aggregation function p for the sub-aggregations . Thus the low-level aggregation is permutation-invariant .

Summary: This work proposes geometric aggregation scheme for GCNs, which aims to overcome the limitations in traditional GCNs; those are lacking long distance dependencies and structure information in nodes. In particular, each node is transformed into a latent space. To overcome the first limitation, some nodes that are not directly connected but fall in a near range are also used in aggregation. A relational operator is used to provide position information for each pair of nodes. In this way, the structure information in graph can be used.