A Hierarchy of Graph Neural Networks Based on Learnable Local Features

Graph neural networks ( GNNs ) are a powerful tool to learn representations on graphs by iteratively aggregating features from node neighbourhoods . Many variant models have been proposed , but there is limited understanding on both how to compare different architectures and how to construct GNNs systematically . Here , we propose a hierarchy of GNNs based on their aggregation regions . We derive theoretical results about the discriminative power and feature representation capabilities of each class . Then , we show how this framework can be utilized to systematically construct arbitrarily powerful GNNs . As an example , we construct a simple architecture that exceeds the expressiveness of the Weisfeiler-Lehman graph isomorphism test . We empirically validate our theory on both synthetic and real-world benchmarks , and demonstrate our example ’ s theoretical power translates to strong results on node classification , graph classification , and graph regression tasks . 1 INTRODUCTION . Graphs arise naturally in the world and are key to applications in chemistry , social media , finance , and many other areas . Understanding graphs is important and learning graph representations is a key step . Recently , there has been an explosion of interest in utilizing graph neural networks ( GNNs ) , which have shown outstanding performance across tasks ( e.g . Kipf & Welling ( 2016 ) , Veličković et al . ( 2017 ) ) . Generally , we consider node-feature GNNs which operate recursively to aggregate representations from a neighbouring region ( Gilmer et al. , 2017 ) . In this work , we propose a representational hierarchy of GNNs , and derive the discriminative power and feature representation capabilities in each class . Importantly , while most previous work has focused on GNNs aggregating over vertices in the immediate neighbourhood , we consider GNNs aggregating over arbitrary subgraphs containing the node . We show that , under mild conditions , there is only in fact a small class of subgraphs that are valid aggregation regions . These subgraphs provide a systematic way of defining a hierarchy for GNNs . Using this hierarchy , we can derive theoretical results which provide insight into GNNs . For example , we show that no matter how many layers are added , networks which only aggregate over immediate neighbors can not learn the number of triangles in a node ’ s neighbourhood . We demonstrate that many popular frameworks , including GCN1 ( Kipf & Welling , 2016 ) , GAT ( Veličković et al. , 2017 ) , and N-GCN ( Abu-El-Haija et al. , 2018 ) are unified under our framework . We also compare each class using the Weisfeiler-Lehman ( WL ) isomorphism test ( Weisfeiler & Lehman , 1968 ) , and conclude our hierarchy is able to generate arbitrarily powerful GNNs . Then we utilize it to systematically generate GNNs exceeding the discriminating power of the 1-WL test . Experiments utilize both synthetic datasets and standard GNN benchmarks . We show that the method is able to learn difficult graph properties where standard GCNs fail , even with multiple layers . On benchmark datasets , our proposed GNNs are able to achieve strong results on multiple datasets covering node classification , graph classification , and graph regression . 1Throughout the paper we use GCN to specifically refer to the model proposed in Kipf & Welling ( 2016 ) . 2 RELATED WORK . Numerous works ( see Li et al . ( 2015 ) , Atwood & Towsley ( 2016 ) , Defferrard et al . ( 2016 ) , Kipf & Welling ( 2016 ) , Niepert et al . ( 2016 ) , Santoro et al . ( 2017 ) , Veličković et al . ( 2017 ) , Verma & Zhang ( 2018 ) , Zhang et al . ( 2018a ) , Ivanov & Burnaev ( 2018 ) , Wu et al . ( 2019a ) for examples ) have constructed different architectures to learn graph representations . Collectively , GNNs have pushed the state-of-the-art on many different tasks on graphs , including node classification , and graph classification/regression . However , there are relatively few works that attempt to understand or categorize GNNs theoretically . Scarselli et al . ( 2009 ) presented one of the first works that investigated the capabilities of GNNs . They showed that the GNNs are able to approximate a large class of functions ( those satisfying preservation of the unfolding equivalence ) on graphs arbitrarily well . A recent work by Xu et al . ( 2018 ) also explored the theoretical properties of GNNs . Its definition of GNNs is limited to those that aggregate features in the immediate neighbourhood , and thus is a special case of our general framework . We also show that the paper ’ s conclusion that GNNs are at most as powerful as the Weisfeiler-Lehman test fails to hold in a simple extension . Survey works including Zhou et al . ( 2018 ) and Wu et al . ( 2019b ) give an overview of the current field of research in GNNs , and provide structural classifications of GNNs . We differ in our motivation to categorize GNNs from a computational perspective . We also note that our classification only covers static node feature graphs , though extensions to more general settings are possible . The disadvantages of GNNs using localized filter to propagate information are analyzed in Li et al . ( 2018 ) . One major problem is their incapability of exploring global graph structures . To alleviate this , there has been two important works in expanding neighbourhoods : N-GCN ( Abu-El-Haija et al. , 2018 ) feeds higher-degree polynomials of adjacency matrix to multiple instantiations of GCNs , and Morris et al . ( 2018 ) generalizes GNNs to k-GNNs by constructing a set-based k-WL to consider higher-order neighbourhoods and capture information beyond node-level . We compare architectures constructed using our hierarchy to these previous works in the experiments , and show that systematic construction of higher-order neighbourhoods brings an advantage across different tasks . 3 BACKGROUND . Let G = ( V , E ) denote an undirected and unweighted graph , where |V | = N , and |E| = Ω . Unless otherwise specified , we include self-loops for every node v ∈ V . Let A be the graph ’ s adjacency matrix . Denote d ( u , v ) as the distance between two nodes u and v on a graph , defined as the minimum length of walk between u and v. We further write dv as the degree of node v , and N ( v ) as the set of nodes in the direct neighborhood of v ( including v itself ) . Graph Neural Networks ( GNNs ) utilize the structure of a graph G and node features X ∈ RN×p to learn a refined representation of each node , where p is input feature size , i.e . for each node v ∈ V , we have features Xv ∈ Rp . A GNN is a function that for every layer l at every node v aggregates features over a connected subgraph Gv ⊆ G containing the node v , and updates a hidden representation H ( l ) = [ h ( l ) 1 , · · · , h ( l ) N ] . Formally , we can define the lth layer of a GNN ( with h ( 0 ) v = Xv ) : a ( l ) v = Agg ∣∣ Gv ( H ( l−1 ) ) h ( l ) v = Com ( h ( l−1 ) v , a ( l ) v ) where | is the restriction symbol over the domain Gv , the aggregation subgraph . The aggregation function Agg ( · ) is invariant with respect to the labeling of the nodes . The aggregation function , Agg ( · ) , summarizes information from a neighbouring region Gv , while the combination function Com ( · ) joins such information with the previous hidden features to produce a new representation . For different tasks , these GNNs are combined with an output layer to coerce the final output into an appropriate shape . Examples include fully-connected layers ( Xu et al. , 2018 ) , convolutional layers ( Zhang et al. , 2018a ) , and simple summation ( Verma & Zhang , 2018 ) . These output layers are task-dependent and not graph-dependent , so we would omit these in our framework , and consider the node level output H ( L ) of the final Lth layer as the output of the GNN . We consider three representative GNN variants in terms of this notation , where W ( l ) is a learnable weight matrix at layer l:2 • Graph Convolutional Networks ( GCNs ) ( Kipf & Welling , 2016 ) : Agg ( · ) = ∑ u∈N ( v ) h ( l−1 ) u Com ( · ) = ReLu ( a ( l ) v W ( l ) ) • Graph Attention Networks ( GAT ) ( Veličković et al. , 2017 ) : Agg ( · ) = ∑ u∈N ( v ) softmax u∈N ( v ) ( MLP ( h ( l−1 ) u , h ( l−1 ) v ) ) h ( l−1 ) u Com ( · ) = ReLu ( a ( l ) v W ( l ) ) • N-GCN ( Abu-El-Haija et al. , 2018 ) ( 2-layer case ) : Agg ( · ) = ∑ u1∈N ( v ) ∑ u2∈N ( u1 ) h ( l−1 ) u2 Com ( · ) = ReLu ( a ( l ) v W ( l ) ) 4 HIERARCHICAL FRAMEWORK FOR CONSTRUCTING GNNS . Our proposed framework uses walks to specify a hierarchy of aggregation ranges . The aggregation function over a node v ∈ G is a permutation-invariant function over a connected subgraph Gv . Consider the simplest case , using the neighbouring vertices u ∈ N ( v ) , utilized by many popular architectures ( e.g . GCN , GAT ) . Then Gv in this case is a star-shaped subgraph , as illustrated below in Figure 1 . We refer to that as D1 ( v ) , which in terms of walks , is the union of all edges and nodes in length-2 walks that start and end at v. To build a hierarchy , we consider benefits of longer walks . The next simplest graph feature is the triangles in the neighbourhood of v. Knowledge on connections between the neighbouring nodes of v are necessary for considering triangles . A natural formulation using walks would be length-3 walks that start and end at v. A length-3 returning walk outlines a triangle , and the union of all length-3 returning walks induces a subgraph , formed by all nodes and edges included in those walks . This is illustrated in Figure 1 as L1 ( v ) . Definition 1 . Define the set of all walks of length ≤ m returning to v as Wm ( v ) . For k ∈ Z+ , we define Dk ( v ) as the subgraph formed by all the edges and nodes in W2k ( v ) , while Lk ( v ) is defined as the subgraph formed by all the nodes and edges in W2k+1 ( v ) . Intuitively , Lk ( v ) is a subgraph of G consisting of all nodes and edges in the k-hop neighbourhood of node v , and Dk ( v ) only differs from Lk ( v ) by excluding the edges between the distance-k neighbors of v. We explore this further in Section 5 . An example illustration of the neighbourhoods defined above is shown in Figure 1 . This set of subgraphs naturally induces a hierarchy with increasing aggregation region : Definition 2 . The D-L hierarchy of aggregation regions for a node v , AD−L ( v ) in a graph G is , in increasing order : AD−L ( v ) = { D1 ( v ) , L1 ( v ) , · · · , Dk ( v ) , Lk ( v ) , · · · } ( 1 ) Where D1 ( v ) ⊆ L1 ( v ) ⊆ D2 ( v ) ⊆ L2 ( v ) · · · . Next , we consider the properties of this hierarchy . One important property is completeness - that the hierarchy can classify every possible GNN . Note that there is no meaningful complete hierarchy if Gv is arbitrary . Therefore , we propose to limit our focus to those Gv that can be defined as a function of the distance from v. Absent specific graph structures , distance is a canonical metric between vertices and this definition includes all examples listed in Section 3 . With such assumption , we can show that the D-L hierarchy is complete : 2For simplicity we present the version without feature normalization using node degrees . Theorem 1 . Consider a GNN defined by its action at each layer : a ( l ) v = Agg ∣∣ Gv ( H ( l−1 ) ) h ( l ) v = Com ( h ( l−1 ) v , a ( l ) v ) ( 2 ) Assume Gv can be defined as a univariate function of the distance from v. Then both of the following statements are true for all k ∈ Z+ : • If Dk ( v ) ⊆ Gv ⊆ Lk ( v ) , then Gv ∈ { Dk ( v ) , Lk ( v ) } . • If Lk ( v ) ⊆ Gv ⊆ Dk+1 ( v ) , then Gv ∈ { Lk ( v ) , Dk+1 ( v ) } . This theorem shows that one can not create an aggregation region based on node distance that is `` in between '' the hierarchy defined . With Theorem 1 , we can use the D-L aggregation hierarchy to create a hierarchy of GNNs based on their aggregation regions . Definition 3 . For k ∈ Z+ , G ( Dk ) is the set of all graph neural networks with aggregation region Gv = Dk ( v ) that is not a member of G ( Lk ) . G ( Lk ) is the set of all graph neural networks with aggregation region Gv = Lk ( v ) that is not a member of G ( Dk−1 ) . We explicitly exclude those belonging to a lower aggregation region in order to make the hierarchy well-defined ( otherwise a GNN of order G ( D1 ) is trivially one of order G ( L1 ) ) . We also implicitly define D0 = L0 = ∅ .

Summary: The paper proposes a generalized framework for GNN. It proposes a representational hierarchy of GNNs. (D-1, L-1,..D-n, L-n). L-k is the k-hop neighborhood including all edges. And D-k is L-k without edges between the outermost nodes. The discriminative power of a network using L-1 is > WL-1. On various graph classification/regression tasks, the proposed method shows good performance. Show promising result on QM7b QM9 graph regression task  (counting triangles, cycles etc) which are highly relevant to the proposed method.