Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classification

1 INTRODUCTION . Recent years have seen increasing attention to Graph Neural Nets ( GNNs ) ( Scarselli et al. , 2009 ; Li et al. , 2015 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) , which have achieved superior performance in many graph tasks , such as node classification ( Kipf and Welling , 2016 ; Wu et al. , 2019 ) and graph classification ( Simonovsky and Komodakis , 2017 ; Xinyi and Chen , 2019 ) . Different from traditional neural networks that are defined on regular structures such as sequences or images , graphs provide a more general abstraction for structured data , which subsume regular structures as special cases . The power of GNNs is that they can directly define learnable compositional function on ( arbitrary ) graphs , thus extending classic networks ( e.g . CNNs , RNNs ) to more irregular and general domains . Despite their success , it is unclear what GNNs have learned , and how sophisticated the learned graph functions are . It is shown in ( Zeiler and Fergus , 2014 ) that traditional CNNs used in image recognition have learned complex hierarchical and compositional features , and that deep non-linear computation can be beneficial He et al . ( 2016 ) . Is this also the case when applying GNNs to common graph problems ? Recently , Wu et al . ( 2019 ) showed that , for common node classification benchmarks , non-linearity can be removed in GNNs without suffering much loss of performance . The resulting linear GNNs collapse into a logistic regression on graph propagated features . This raises doubts on the necessity of complex GNNs , which require much more expensive computation , for node classification benchmarks . Here we take a step further dissecting GNNs , and examine the necessity of complex GNN parts on more challenging graph classification benchmarks ( Yanardag and Vishwanathan , 2015 ; Zhang et al. , 2018a ; Xinyi and Chen , 2019 ) . To better understand GNNs on graph classification , we dissect it into two parts/stages : 1 ) the graph filtering part , where graph-based neighbor aggregations are performed , and 2 ) the set function part , where a set of hidden node features are composed for prediction . We aim to test the importance of both parts separately , and seek answers to the following questions . Do we need a sophisticated graph filtering function for a particular task or dataset ? And if we have a powerful set function , is it enough to use a simple graph filtering function ? To answer these questions , we propose to linearize both parts separately . We first linearize graph filtering part , resulting Graph Feature Network ( GFN ) : a simple lightweight neural net defined on a set of graph augmented features . Unlike GNNs , which learn a multi-step neighbor aggregation function on graphs ( Dai et al. , 2016 ; Gilmer et al. , 2017 ) , the GFN only utilizes graphs in constructing its input features . It first augments nodes with graph structural and propagated features , and then learns a neural net directly on the set of nodes ( i.e . a bag of graph pre-processed feature vectors ) , which make it more efficient . We then further linearize set function in GFN , and arrive at Graph Linear Network ( GLN ) , which is a linear function of augmented graph features . Empirically , we perform evaluations on common graph classification benchmarks ( Yanardag and Vishwanathan , 2015 ; Zhang et al. , 2018a ; Xinyi and Chen , 2019 ) , and find that GFN can match or exceed the best accuracies produced by recently proposed GNNs , at a fraction of the computation cost . GLN performs much poorly than both GFN and recent GNNs . This result casts doubts on the necessity of non-linear graph filtering , and suggests that the existing GNNs may not have learned more sophisticated graph functions than linear neighbor aggregation on these benchmarks . Furthermore , we find non-linear set function plays an important role , as its linearization can hurt performance significantly . 2 PRELIMINARIES . Graph classification problem . We use G = ( V , E ) ∈ G to denote a graph , where V is a set of vertices/nodes , and E is a set of edges . We further denote an attributed graph as GX = ( G , X ) ∈ GX , where X ∈ Rn×d are node attributes with n = |V | . It is assumed that each attributed graph is associated with some label y ∈ Y , where Y is a set of pre-defined categories . The goal in graph classification problem is to learn a mapping function f : GX → Y , such that we can predict the target class for unseen graphs accurately . Many real world problems can be formulated as graph classification problems , such as social and biological graph classification Yanardag and Vishwanathan ( 2015 ) ; Kipf and Welling ( 2016 ) . Graph neural networks . Graph Neural Networks ( GNNs ) define functions on the space of attributed graph GX . Typically , the graph function , GNN ( G , X ) , learns a multiple-step transformation of the original attributes/signals for final node level or graph level prediction . In each of the step t , a new node presentation , h ( t ) v is learned . Initially , h ( 1 ) v is initialized with the node attribute vector , and during each subsequent step , a neighbor aggregation function is applied to generate the new node representation . More specifically , common neighbor aggregation functions for the v-th node take the following form : h ( t ) v = f ( h ( t−1 ) v , { h ( t−1 ) u |u ∈ N ( v ) } ) , ( 1 ) whereN ( v ) is a set of neighboring nodes of node v. To instantiate this neighbor aggregation function , ( Kipf and Welling , 2016 ) proposes the Graph Convolutional Network ( GCN ) aggregation scheme as follows . h ( t+1 ) v = σ ( ∑ u∈N ( v ) Ãuv ( W ( t ) ) Th ( t ) u ) , ( 2 ) where W ( t ) ∈ Rd×d′ is the learnable transformation weight , Ã = D̃−1/2 ( A + I ) D̃−1/2 is the normalized adjacency matrix with as a constant ( = 1 in Kipf and Welling ( 2016 ) ) and D̃ii =∑ j Aij + . σ ( · ) is a non-linear activation function , such as ReLU . This transformation can also be written as H ( t+1 ) = σ ( ÃH ( t ) W ( t ) ) , where H ( t ) ∈ Rn×d are the hidden states of all nodes at t-th step . More sophisticated neighbor aggregation schemes are also proposed , such as GraphSAGE ( Hamilton et al. , 2017 ) which allows pooling and recurrent aggregation over neighboring nodes . Most recently , in Graph Isomorphism Network ( GIN ) ( Xu et al. , 2019 ) , a more powerful aggregation function is proposed as follows . h ( t ) v = MLP ( t ) ( ( 1 + ( t ) ) h ( t−1 ) v + ∑ u∈N ( v ) h ( t−1 ) u ) , ( 3 ) where MLP abbreviates for multi-layer perceptrons and ( t ) can either be zero or a learnable parameter . Finally , in order to generate graph level representation hG , a readout function is used , which generally takes the following form : hG = g ( { h ( T ) v |v ∈ G } ) . ( 4 ) This can be instantiated by a global sum pooling , i.e . hG = ∑n v=1 h ( T ) v followed by fully connected layers to generate the categorical or numerical output . 3 APPROACH . 3.1 GRAPH FEATURE NETWORK . Motivated by the question that , with a powerful graph readout function , whether wwe can simplify the sophisticated multi-step neighbor aggregation functions ( such as Eq . 2 and 3 ) . Therefore we propose Graph Feature Network ( GFN ) : a neural set function defined on a set of graph augmented features . Graph augmented features . In GFN , we replace the sophisticated neighbor aggregation functions ( such as Eq . 2 and 3 ) with graph augmented features based on GX . Here we consider two categories as follows : 1 ) graph structural/topological features , which are related to the intrinsic graph structure , such as node degrees , or node centrality scores1 , but do not rely on node attributes ; 2 ) graph propagated features , which leverage the graph as a medium to propagate node attributes . The graph augmented features XG can be seen as the output of a feature extraction function defined on the attributed graph , i.e . XG = γ ( G , X ) , and Eq . 5 below gives a specific form , which combine node degree features and multi-scale graph propagated features as follows : XG = γ ( G , X ) = [ d , X , Ã1X , Ã2X , · · · , ÃKX ] , ( 5 ) where d ∈ Rn×1 is the degree vector for all nodes , and Ã is again the normalized adjacency matrix ( Ã = D̃−1/2 ( A+ I ) D̃−1/2 ) , but other designs of propagation operator are possible ( Klicpera et al. , 2019 ) . Features separated by comma are concatenated to form XG . Neural set function . To build a powerful graph readout function based on graph augmented features XG , we use a neural set function . The neural set function discards the graph structures and learns purely based on the set of augmented node features . Motivated by the general form of a permutationinvariant set function shown in Zaheer et al . ( 2017 ) , we define our neural set function for GFN as follows : GFN ( G , X ) = ρ ( ∑ v∈V φ ( XGv ) ) . ( 6 ) Both φ ( · ) and ρ ( · ) are parameterized by neural networks . Concretely , we parameterize the function φ ( · ) as a multi-layer perceptron ( MLP ) , i.e . φ ( x ) = σ ( σ ( · · ·σ ( xTW ( 1 ) ) · · · ) W ( T ) ) . Note that a single layer of φ ( · ) resembles a graph convolution layer H ( t+1 ) = σ ( ÃH ( t ) W ( t ) ) with the normalized adjacency matrix Ã replaced by identity matrix I ( a.k.a . 1× 1 convolution ) . As for the function ρ ( · ) , we parameterize it with another MLP ( i.e . fully connected layers in this case ) . 1We only use node degree in this work as it is very efficient to calculate during both training and inference . Computation efficiency . GFN provides a way to approximate GNN with less computation overheads , especially during the training process . Since the graph augmented features can be pre-computed before training starts , the graph structures are not involved in the iterative training process . This brings the following advantages . First , since there is no neighbor aggregation step in GFN , it reduces computational complexity . To see this , one can compare a single layer feature transformation function in GFN , i.e . σ ( HW ) , against the neighbor aggregation function in GCN , i.e . σ ( ÃHW ) . Secondly , since graph augmented features of different scales are readily available from the input layer , GFN can leverage them much earlier , thus may require fewer transformation layers . Lastly , it also eases the implementation related overhead , since the neighbor aggregation operation in graphs are typically implemented by sparse matrix operations . Graph Linear Network . When we use a linear set function instead of the generic one used in Eq . 6 , we arrive at graph linear network , which can be expressed as follows . GLN ( G , X ) = σ ( W ∑ v∈V ( XGv ) ) . ( 7 ) Where W is a weight matrix , and σ ( · ) is softmax function produce class probability .

Summary: The paper dissects the importance of two parts in GCN: 1) nonlinear neighborhood aggregation; 2) nonlinear set function by linearizing the two parts and resulting in Graph Feature Network (GFN) and Graph Linear Network (GLN). It shows empirically that GFN achieves almost the same performance while GLN is much worse, suggesting the nonlinear graph neighborhood aggregation step may be unnecessary. Extensive ablation studies are conducted to single out the effects of various factors.