On Graph Neural Networks versus Graph-Augmented MLPs

1 INTRODUCTION . While multi-layer Graph Neural Networks ( GNNs ) have gained popularity for their applications in various fields , recently authors have started to investigate what their true advantages over baselines are , and whether they can be simplified . On one hand , GNNs based on neighborhood-aggregation allows the combination of information present at different nodes , and by increasing the depth of such GNNs , we increase the size of the receptive field . On the other hand , it has been pointed out that deep GNNs can suffer from issues including over-smoothing , exploding or vanishing gradients in training as well as bottleneck effects ( Kipf & Welling , 2016 ; Li et al. , 2018 ; Luan et al. , 2019 ; Oono & Suzuki , 2020 ; Rossi et al. , 2020 ; Alon & Yahav , 2020 ) . Recently , a series of models have attempted at relieving these issues of deep GNNs while retaining their benefit of combining information across nodes , using the approach of firstly augmenting the node features by propagating the original node features through powers of graph operators such as the ( normalized ) adjacency matrix , and secondly applying a node-wise function to the augmented node features , usually realized by a Multi-Layer Perceptron ( MLP ) ( Wu et al. , 2019 ; NT & Maehara , 2019 ; Chen et al. , 2019a ; Rossi et al. , 2020 ) . Because of the usage of graph operators for augmenting the node features , we will refer to such models as Graph-Augmented MLPs ( GA-MLPs ) . These models have achieved competitive performances on various tasks , and moreover enjoy better scalability since the augmented node features can be computed during preprocessing ( Rossi et al. , 2020 ) . Thus , it becomes natural to ask what advantages GNNs have over GA-MLPs . In this work , we ask whether GA-MLPs sacrifice expressive power compared to GNNs while gaining these advantages . A popular measure of the expressive power of GNNs is their ability to distinguish non-isomorphic graphs ( Hamilton et al. , 2017 ; Xu et al. , 2019 ; Morris et al. , 2019 ) . In our work , besides studying the expressive power of GA-MLPs from the viewpoint of graph isomorphism tests , we propose a new perspective that better suits the setting of node-prediction tasks : we analyze the ∗Equal contributions . Code available at https : //github.com/leichen2018/GNN_vs_GAMLP . expressive power of models including GNNs and GA-MLPs as node-level functions , or equivalently , as functions on rooted graphs . Under this perspective , we prove an exponential-in-depth gap between the expressive powers of GNNs and GA-MLPs . We illustrate this gap by finding a broad family of user-friendly functions that can be provably approximated by GNNs but not GA-MLPs , based on counting attributed walks on the graph . Moreover , via the task of community detection , we show a lack of flexibility of GA-MLPs , compared to GNNs , to learn the best operators to use . In summary , our main contributions are : • Finding graph pairs that several GA-MLPs can not distinguish while GNNs can , but also proving there exist simple GA-MLPs that distinguish almost all non-isomorphic graphs . • From the perspective of approximating node-level functions , proving an exponential gap between the expressive power of GNNs and GA-MLPs in terms of the equivalence classes on rooted graphs that they induce . • Showing that the functions that count a particular type of attributed walk among nodes can be approximated by GNNs but not GA-MLPs both in theory and numerically . • Through community detection tasks , demonstrating that GNNs have higher flexibility in learning than GA-MLPs due to the fixed choice of the operator family in the latter . 2 RELATED WORKS . Depth in GNNs Kipf & Welling ( 2016 ) observe that the performance of Graph Convolutional Networks ( GCNs ) degrade as the depth grows too large , and the best performance is achieved with 2 or 3 layers . Along the spectral perspective on GNNs ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Bronstein et al. , 2017 ; NT & Maehara , 2019 ) , Li et al . ( 2018 ) and Wu et al . ( 2019 ) explain the failure of deep GCNs by the over-smoothing of the node features . Oono & Suzuki ( 2020 ) show an exponential loss of expressive power as the depth in GCNs increases in the sense that the hidden node states tend to converge to Laplacian sub-eigenspaces as the depth increases to infinity . Alon & Yahav ( 2020 ) show an over-squashing effect of deep GNNs , in the sense that the width of the hidden states needs to grow exponentially in the depth in order to retain all information about long-range interactions . In comparison , our work focuses on more general GNNs based on neighborhoodaggregation that are not limited in the hidden state widths , and demonstrates the their advantage in expressive power compared to GA-MLP models at finite depth , in terms of distinguishing rooted graphs for node-prediction tasks . On the other hand , there exist examples of useful deep GNNs . Chen et al . ( 2019b ) apply 30-layer GNNs for community detection problems , which uses a family of multi-scale operators as well as normalization steps ( Ioffe & Szegedy , 2015 ; Ulyanov et al. , 2016 ) . Recently , Li et al . ( 2019 ; 2020a ) and Chen et al . ( 2020a ) build deeper GCN architectures with the help of various residual connections ( He et al. , 2016 ) and normalization steps to achieve impressive results in standard datasets , which further highlights the need to study the role of depth in GNNs . Gong et al . ( 2020 ) propose geometrically principled connections , which improve upon vanilla residual connections on graph- and mesh-based tasks . Existing GA-MLP-type models Motivated by better understanding GNNs as well as enhancing computational efficiency , several models of the GA-MLP type have been proposed and they achieve competitive performances on various datasets . Wu et al . ( 2019 ) propose the Simple Graph Convolution ( SGC ) , which removes the intermediary weights and nonlinearities in GCNs . Chen et al . ( 2019a ) propose the Graph Feature Network ( GFN ) , which further adds intermediary powers of the normalized adjacency matrix to the operator family and is applied to graph-prediction tasks . NT & Maehara ( 2019 ) propose the Graph Filter Neural Networks ( gfNN ) , which enhances the SGC in the final MLP step . Rossi et al . ( 2020 ) propose Scalable Inception Graph Neural Networks ( SIGNs ) , which augments the operator family with Personalized-PageRank-based ( Klicpera et al. , 2018 ; 2019 ) and triangle-based ( Monti et al. , 2018 ; Chen et al. , 2019b ) adjacency matrices . Expressive Power of GNNs Xu et al . ( 2019 ) and Morris et al . ( 2019 ) show that GNNs based on neighborhood-aggregation are no more powerful than the Weisfeiler-Lehman ( WL ) test for graph isomorphism ( Weisfeiler & Leman , 1968 ) , in the sense that these GNNs can not distinguish between any pair of non-isomorphic graphs that the WL test can not distinguish . They also propose models that match the expressive power of the WL test . Since then , many attempts have been made to build GNN models whose expressive power are not limited by WL ( Morris et al. , 2019 ; Maron et al. , 2019a ; Chen et al. , 2019c ; Morris & Mutzel , 2019 ; You et al. , 2019 ; Bouritsas et al. , 2020 ; Li et al. , 2020b ; Flam-Shepherd et al. , 2020 ; Sato et al. , 2019 ; 2020 ) . Other perspectives for understanding the expressive power of GNNs include function approximation ( Maron et al. , 2019b ; Chen et al. , 2019c ; Keriven & Peyré , 2019 ) , substructure counting ( Chen et al. , 2020b ) , Turing universality ( Loukas , 2020 ) and the determination of graph properties ( Garg et al. , 2020 ) . Sato ( 2020 ) provides a survey on these topics . In this paper , besides studying the expressive power of GA-MLPs along the line of graph isomorphism tests , we propose a new perspective of approximating functions on rooted graphs , which is motivated by node-prediction tasks , and show a gap between GA-MLPs and GNNs that grows exponentially in the size of the receptive field in terms of the equivalence classes that they induce on rooted graphs . 3 BACKGROUND . 3.1 NOTATIONS . Let G = ( V , E ) denote a graph , with V being the vertex set and E being the edge set . Let n denote the number of nodes in G , A ∈ Rn×n denote the adjacency matrix , D ∈ Rn×n denote the diagonal degree matrix with Dii = di being the degree of node i . We call D− 1 2AD− 1 2 the ( symmetrically ) normalized adjacency matrix , and D−αAD−β a generalized normalized adjacency matrix for any α , β ∈ R. Let X ∈ Rn×d denote the matrix of node features , where Xi denotes the d-dimensional feature that node i possesses . For a node i ∈ V , letN ( i ) denote the set of neighbors of i . We assume that the edges do not possess features . In a node prediction task , the labels are given by Y ∈ Rn . For a positive integer K , we let [ K ] = { 1 , ... , K } . We use { ... } m to denote a multiset , which allows repeated elements . We say a function f ( K ) is doubly-exponential in K if log log f ( K ) is polynomial in K , and poly-exponential in K if log f ( K ) is polynomial in K , as K tends to infinity . 3.2 GRAPH NEURAL NETWORKS ( GNNS ) . Following the notations in Xu et al . ( 2019 ) , we consider K-layer GNNs defined generically as follows . For k ∈ [ K ] , we compute the hidden node states H ∈ Rn×d ( k ) iteratively as M ( k ) i = AGGREGATE ( k ) ( { H ( k−1 ) j : j ∈ N ( i ) } ) , H ( k ) i = COMBINE ( k ) ( H ( k−1 ) i , M ( k ) i ) , ( 1 ) where we set H ( 0 ) = X to be the node features . If a graph-level output is desired , we finally let ZG = READOUT ( { H ( K ) i : i ∈ V } ) , ( 2 ) Different choices of the trainable COMBINE , AGGREGATE and READOUT functions result in different GNN models , though usually AGGREGATE and READOUT are chosen to be permutationinvariant . As graph-level functions , it is shown in Xu et al . ( 2019 ) and Morris et al . ( 2019 ) that the maximal expressive power of models of this type coincides with running K iterations of the WL test for graph isomorphism , in the sense that any two non-isomorphic graphs that can not be distinguished by the latter can not be distinguished by the K-layer GNNs , either . For this reason , we will not distinguish between GNN and WL in discussions on expressive powers .

Summary: The paper studies a variant of Graph Neural Networks (GNNs) namely, Graph Augmented MLPs (GA-MLPs). Unlike in GNNs where nodes send messages to neighbors, and aggregate received messages via non-linear MLPs,  GA-MLPs rely on a single augmented embedding computed once and then applying an MLP to the new embeddings. The augmented embeddings can be obtained by applying linear transformations of the form A, A^2, …, A^k to the input representations, thereby capturing larger neighborhoods. The main goal of the paper is to demonstrate a fundamental weakness when using GA-MLPs for solving graph problems as compared to GNNs. Along these line the paper the main results can be characterized as follows: