On Graph Neural Networks versus Graph-Augmented MLPs

1 INTRODUCTION . While multi-layer Graph Neural Networks ( GNNs ) have gained popularity for their applications in various fields , recently authors have started to investigate what their true advantages over baselines are , and whether they can be simplified . On one hand , GNNs based on neighborhood-aggregation allows the combination of information present at different nodes , and by increasing the depth of such GNNs , we increase the size of the receptive field . On the other hand , it has been pointed out that deep GNNs can suffer from issues including over-smoothing , exploding or vanishing gradients in training as well as bottleneck effects ( Kipf & Welling , 2016 ; Li et al. , 2018 ; Luan et al. , 2019 ; Oono & Suzuki , 2020 ; Rossi et al. , 2020 ; Alon & Yahav , 2020 ) . Recently , a series of models have attempted at relieving these issues of deep GNNs while retaining their benefit of combining information across nodes , using the approach of firstly augmenting the node features by propagating the original node features through powers of graph operators such as the ( normalized ) adjacency matrix , and secondly applying a node-wise function to the augmented node features , usually realized by a Multi-Layer Perceptron ( MLP ) ( Wu et al. , 2019 ; NT & Maehara , 2019 ; Chen et al. , 2019a ; Rossi et al. , 2020 ) . Because of the usage of graph operators for augmenting the node features , we will refer to such models as Graph-Augmented MLPs ( GA-MLPs ) . These models have achieved competitive performances on various tasks , and moreover enjoy better scalability since the augmented node features can be computed during preprocessing ( Rossi et al. , 2020 ) . Thus , it becomes natural to ask what advantages GNNs have over GA-MLPs . In this work , we ask whether GA-MLPs sacrifice expressive power compared to GNNs while gaining these advantages . A popular measure of the expressive power of GNNs is their ability to distinguish non-isomorphic graphs ( Hamilton et al. , 2017 ; Xu et al. , 2019 ; Morris et al. , 2019 ) . In our work , besides studying the expressive power of GA-MLPs from the viewpoint of graph isomorphism tests , we propose a new perspective that better suits the setting of node-prediction tasks : we analyze the ∗Equal contributions . Code available at https : //github.com/leichen2018/GNN_vs_GAMLP . expressive power of models including GNNs and GA-MLPs as node-level functions , or equivalently , as functions on rooted graphs . Under this perspective , we prove an exponential-in-depth gap between the expressive powers of GNNs and GA-MLPs . We illustrate this gap by finding a broad family of user-friendly functions that can be provably approximated by GNNs but not GA-MLPs , based on counting attributed walks on the graph . Moreover , via the task of community detection , we show a lack of flexibility of GA-MLPs , compared to GNNs , to learn the best operators to use . In summary , our main contributions are : • Finding graph pairs that several GA-MLPs can not distinguish while GNNs can , but also proving there exist simple GA-MLPs that distinguish almost all non-isomorphic graphs . • From the perspective of approximating node-level functions , proving an exponential gap between the expressive power of GNNs and GA-MLPs in terms of the equivalence classes on rooted graphs that they induce . • Showing that the functions that count a particular type of attributed walk among nodes can be approximated by GNNs but not GA-MLPs both in theory and numerically . • Through community detection tasks , demonstrating that GNNs have higher flexibility in learning than GA-MLPs due to the fixed choice of the operator family in the latter . 2 RELATED WORKS . Depth in GNNs Kipf & Welling ( 2016 ) observe that the performance of Graph Convolutional Networks ( GCNs ) degrade as the depth grows too large , and the best performance is achieved with 2 or 3 layers . Along the spectral perspective on GNNs ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Bronstein et al. , 2017 ; NT & Maehara , 2019 ) , Li et al . ( 2018 ) and Wu et al . ( 2019 ) explain the failure of deep GCNs by the over-smoothing of the node features . Oono & Suzuki ( 2020 ) show an exponential loss of expressive power as the depth in GCNs increases in the sense that the hidden node states tend to converge to Laplacian sub-eigenspaces as the depth increases to infinity . Alon & Yahav ( 2020 ) show an over-squashing effect of deep GNNs , in the sense that the width of the hidden states needs to grow exponentially in the depth in order to retain all information about long-range interactions . In comparison , our work focuses on more general GNNs based on neighborhoodaggregation that are not limited in the hidden state widths , and demonstrates the their advantage in expressive power compared to GA-MLP models at finite depth , in terms of distinguishing rooted graphs for node-prediction tasks . On the other hand , there exist examples of useful deep GNNs . Chen et al . ( 2019b ) apply 30-layer GNNs for community detection problems , which uses a family of multi-scale operators as well as normalization steps ( Ioffe & Szegedy , 2015 ; Ulyanov et al. , 2016 ) . Recently , Li et al . ( 2019 ; 2020a ) and Chen et al . ( 2020a ) build deeper GCN architectures with the help of various residual connections ( He et al. , 2016 ) and normalization steps to achieve impressive results in standard datasets , which further highlights the need to study the role of depth in GNNs . Gong et al . ( 2020 ) propose geometrically principled connections , which improve upon vanilla residual connections on graph- and mesh-based tasks . Existing GA-MLP-type models Motivated by better understanding GNNs as well as enhancing computational efficiency , several models of the GA-MLP type have been proposed and they achieve competitive performances on various datasets . Wu et al . ( 2019 ) propose the Simple Graph Convolution ( SGC ) , which removes the intermediary weights and nonlinearities in GCNs . Chen et al . ( 2019a ) propose the Graph Feature Network ( GFN ) , which further adds intermediary powers of the normalized adjacency matrix to the operator family and is applied to graph-prediction tasks . NT & Maehara ( 2019 ) propose the Graph Filter Neural Networks ( gfNN ) , which enhances the SGC in the final MLP step . Rossi et al . ( 2020 ) propose Scalable Inception Graph Neural Networks ( SIGNs ) , which augments the operator family with Personalized-PageRank-based ( Klicpera et al. , 2018 ; 2019 ) and triangle-based ( Monti et al. , 2018 ; Chen et al. , 2019b ) adjacency matrices . Expressive Power of GNNs Xu et al . ( 2019 ) and Morris et al . ( 2019 ) show that GNNs based on neighborhood-aggregation are no more powerful than the Weisfeiler-Lehman ( WL ) test for graph isomorphism ( Weisfeiler & Leman , 1968 ) , in the sense that these GNNs can not distinguish between any pair of non-isomorphic graphs that the WL test can not distinguish . They also propose models that match the expressive power of the WL test . Since then , many attempts have been made to build GNN models whose expressive power are not limited by WL ( Morris et al. , 2019 ; Maron et al. , 2019a ; Chen et al. , 2019c ; Morris & Mutzel , 2019 ; You et al. , 2019 ; Bouritsas et al. , 2020 ; Li et al. , 2020b ; Flam-Shepherd et al. , 2020 ; Sato et al. , 2019 ; 2020 ) . Other perspectives for understanding the expressive power of GNNs include function approximation ( Maron et al. , 2019b ; Chen et al. , 2019c ; Keriven & Peyré , 2019 ) , substructure counting ( Chen et al. , 2020b ) , Turing universality ( Loukas , 2020 ) and the determination of graph properties ( Garg et al. , 2020 ) . Sato ( 2020 ) provides a survey on these topics . In this paper , besides studying the expressive power of GA-MLPs along the line of graph isomorphism tests , we propose a new perspective of approximating functions on rooted graphs , which is motivated by node-prediction tasks , and show a gap between GA-MLPs and GNNs that grows exponentially in the size of the receptive field in terms of the equivalence classes that they induce on rooted graphs . 3 BACKGROUND . 3.1 NOTATIONS . Let G = ( V , E ) denote a graph , with V being the vertex set and E being the edge set . Let n denote the number of nodes in G , A ∈ Rn×n denote the adjacency matrix , D ∈ Rn×n denote the diagonal degree matrix with Dii = di being the degree of node i . We call D− 1 2AD− 1 2 the ( symmetrically ) normalized adjacency matrix , and D−αAD−β a generalized normalized adjacency matrix for any α , β ∈ R. Let X ∈ Rn×d denote the matrix of node features , where Xi denotes the d-dimensional feature that node i possesses . For a node i ∈ V , letN ( i ) denote the set of neighbors of i . We assume that the edges do not possess features . In a node prediction task , the labels are given by Y ∈ Rn . For a positive integer K , we let [ K ] = { 1 , ... , K } . We use { ... } m to denote a multiset , which allows repeated elements . We say a function f ( K ) is doubly-exponential in K if log log f ( K ) is polynomial in K , and poly-exponential in K if log f ( K ) is polynomial in K , as K tends to infinity . 3.2 GRAPH NEURAL NETWORKS ( GNNS ) . Following the notations in Xu et al . ( 2019 ) , we consider K-layer GNNs defined generically as follows . For k ∈ [ K ] , we compute the hidden node states H ∈ Rn×d ( k ) iteratively as M ( k ) i = AGGREGATE ( k ) ( { H ( k−1 ) j : j ∈ N ( i ) } ) , H ( k ) i = COMBINE ( k ) ( H ( k−1 ) i , M ( k ) i ) , ( 1 ) where we set H ( 0 ) = X to be the node features . If a graph-level output is desired , we finally let ZG = READOUT ( { H ( K ) i : i ∈ V } ) , ( 2 ) Different choices of the trainable COMBINE , AGGREGATE and READOUT functions result in different GNN models , though usually AGGREGATE and READOUT are chosen to be permutationinvariant . As graph-level functions , it is shown in Xu et al . ( 2019 ) and Morris et al . ( 2019 ) that the maximal expressive power of models of this type coincides with running K iterations of the WL test for graph isomorphism , in the sense that any two non-isomorphic graphs that can not be distinguished by the latter can not be distinguished by the K-layer GNNs , either . For this reason , we will not distinguish between GNN and WL in discussions on expressive powers .

Summary: The paper compares graph neural networks (GNNs) with graph-augmented multi-layer perceptrons (GA-MLPs) where GA-MLPs are MLPs over nodes with additional node features computed over the graph. The paper contains theoretical results and experimental results for graph isomorphism testing and for node level functions. In the overflow of papers studying the expressivity of GNNs, the originality comes from the study of GA-MLPs and from the comparison for node level functions.