Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

1 INTRODUCTION . Motivated by the success of Deep Learning ( DL ) , several attempts have been made to apply DL models to non-Euclidean data , particularly , graph-structured data such as chemical compounds , social networks , and polygons . Recently , Graph Neural Networks ( graph NNs ) ( Duvenaud et al. , 2015 ; Li et al. , 2016 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Kipf & Welling , 2017 ; Nguyen et al. , 2017 ; Schlichtkrull et al. , 2018 ; Battaglia et al. , 2018 ; Xu et al. , 2019 ; Wu et al. , 2019a ) have emerged as a promising approach . However , despite their practical popularity , theoretical research of graph NNs has not been explored extensively . The characterization of DL model expressive power , i.e. , to identify what function classes DL models can ( approximately ) represent , is a fundamental question in theoretical research of DL . Many studies have been conducted for Fully Connected Neural Networks ( FNNs ) ( Cybenko , 1989 ; Hornik , 1991 ; Hornik et al. , 1989 ; Barron , 1993 ; Mhaskar , 1993 ; Sonoda & Murata , 2017 ; Yarotsky , 2017 ) and Convolutional Neural Networks ( CNNs ) ( Petersen & Voigtlaender , 2018 ; Zhou , 2018 ; Oono & Suzuki , 2019 ) . For such models , we have theoretical and empirical justification that deep and nonlinear architectures can enhance representation power ( Telgarsky , 2016 ; Chen et al. , 2018b ; Zhou & Feng , 2018 ) . However , for graph NNs , several papers have reported that node representations go indistinguishable ( known as over-smoothing ) and prediction performances severely degrade when we stack many layers ( Kipf & Welling , 2017 ; Wu et al. , 2019b ; Li et al. , 2018 ) . Besides , Wu et al . ( 2019a ) reported that graph NNs achieved comparable performance even if they removed intermediate non-linear functions . These studies posed a question about the current architecture and made us aware of the need for the theoretical analysis of the graph NN expressive power . 1Code is available at https : //github.com/delta2323/gnn-asymptotics . In this paper , we investigate the expressive power of graph NNs by analyzing their asymptotic behaviors as the layer size goes to infinity . Our theory gives new theoretical conditions under which neither layer stacking nor non-linearity contributes to improving expressive power . We consider a specific dynamics that includes a transition defining a Markov process and the forward propagation of a Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) , which is one of the most popular graph NN variants , as special cases . We prove that under certain conditions , the dynamics exponentially approaches a subspace that is invariant under the dynamics . In the case of GCN , the invariant space is a set of signals that correspond to the lowest frequency of graph spectra and that have “ no information ” other than connected components and node degrees for a node classification task whose goal is to predict the nodes ’ properties in a graph . The rate of the distance between the output and the invariant space is O ( ( sλ ) L ) where s is the maximum singular values of weights , λ is typically a quantity determined by the spectra of the ( augmented ) normalized Laplacian , and L is the layer size . See Sections 3.3 ( general case ) and 4 ( GCN case ) for precise statements . We can interpret our theorem as the generalization of the well-known property that if a finite and discrete Markov process is irreducible and aperiodic , it exponentially converges to a unique equilibrium and the eigenvalues of its transition matrix determine the convergence rate ( see , e.g. , Chung & Graham ( 1997 ) ) . Different from the Markov process case , which is linear , the existence of intermediate non-linear functions complicates the analysis . We overcame this problem by leveraging the combination of the ReLU activation function ( Krizhevsky et al. , 2012 ) and the positivity of eigenvectors of the Laplacian associated with the smallest positive eigenvalues . Our theory enables us to investigate asymptotic behaviors of graph NNs via the spectral distribution of the underlying graphs . To demonstrate this , we take GCNs defined on the Erdős – Rényi graph GN , p , which has N nodes and each edge appears independently with probability p , for an example . We prove that if logNpN = o ( 1 ) as a function of N , any GCN whose weights have maximum singular values at most C √ Np log ( N/ε ) approaches the “ information-less ” invariant space with probability at least 1− ε , where C is a universal constant . Intuitively , if the graph on which we define graph NNs is sufficiently dense , graph-convolution operations mix signals on nodes fast and hence the feature maps lose information for distinguishing nodes quickly . Our contributions are as follows : • We relate asymptotic behaviors of graph NNs with the topological information of underlying graphs via the spectral distribution of the ( augmented ) normalized Laplacian . • We prove that if the weights of a GCN satisfy conditions determined by the graph spectra , the output of the GCN carries no information other than the node degrees and connected components for discriminating nodes when the layer size goes to infinity ( Theorems 1 , 2 ) . • We apply our theory to Erdős – Rényi graphs as an example and show that when the graph is sufficiently dense and large , many GCNs suffer from the information loss ( Theorem 3 ) . • We propose a principled guideline for weight normalization of graph NNs and empirically confirm it using real data . 2 RELATED WORK . MPNN-type Graph NNs . Since many graph NN variants have been proposed , there are several unified formulations of graph NNs ( Gilmer et al. , 2017 ; Battaglia et al. , 2018 ) . Our approach is the closest to the formulation of Message Passing Neural Network ( MPNN ) ( Gilmer et al. , 2017 ) , which unified graph NNs in terms of the update and readout operations . Many graph NNs fall into this formulation such as Duvenaud et al . ( 2015 ) , Li et al . ( 2016 ) , and Veličković et al . ( 2018 ) . Among others , GCN ( Kipf & Welling , 2017 ) is an important application of our theory because it is one of the most widely used graph NNs . In addition , GCNs are interesting from a theoretical research perspective because , in addition to an MPNN-type graph NN , we can interpret GCNs as a simplification of spectral-type graph NNs ( Henaff et al. , 2015 ; Defferrard et al. , 2016 ) , that make use of the graph Laplacian . Our approach , which considers the asymptotic behaviors graph NNs as the layer size goes to infinity , is similar to Scarselli et al . ( 2009 ) , one of the earliest works about graph NNs . They obtained node representations by iterating message passing between nodes until convergence . Their formulation is general in that we can use any local aggregation operation as long as it is a contraction map . Our theory differs from theirs in that we proved that the output of a graph NN approaches a certain space even if the local aggregation function is not necessarily a contraction map . Expressive Power of Graph NNs . Several studies have focused on theoretical analysis and the improvement of graph NN expressive power . For example , Xu et al . ( 2019 ) proved that graph NNs are no more powerful than the Weisfeiler – Lehman ( WL ) isomorphism test ( Weisfeiler & A.A. , 1968 ) and proposed a Graph Isomorphism Network ( GIN ) , that is approximately as powerful as the WL test . Although they experimentally showed that GIN has improved accuracy in supervised learning tasks , their analysis was restricted to the graph isomorphism problem . Xu et al . ( 2018 ) analyzed the non-asymptotic properties of GCNs through the lens of random walk theory . They proved the limitations of GCNs in expander-like graphs and proposed a Jumping Knowledge Network ( JKNet ) to address the issue . To handle the non-linearity , they linearized networks by a randomization assumption ( Choromanska et al. , 2015 ) . We take a different strategy and make use of the interpretation of ReLU as a projection onto a cone . Recently , NT & Maehara ( 2019 ) showed that a GCN approximately works as a low-pass filter plus an MLP in a certain setting . Although they analyzed finite-depth GCNs , our theory has similar spirits with theirs because our “ information-less ” space corresponds to the lowest frequency of a graph Laplacian . Another point is that they imposed assumptions that input signals consist of low-frequent true signals and high-frequent noise , whereas we need not such an assumption . Role of Deep and Non-linear Structures . For ordinal DL models such as FNNs and CNNs , we have both theoretical and empirical justification of deep and non-linear architectures for enhancing of the expressive power ( e.g. , Telgarsky ( 2016 ) ; Petersen & Voigtlaender ( 2018 ) ; Oono & Suzuki ( 2019 ) ) . In contrast , several studies have witnessed severe performance degradation when stacking many layers on graph NNs ( Kipf & Welling , 2017 ; Wu et al. , 2019b ) . Li et al . ( 2018 ) reported that feature vectors on nodes in a graph go indistinguishable as we increase layers in several tasks . They named this phenomenon over-smoothing . Regarding non-linearity , Wu et al . ( 2019a ) empirically showed that graph NNs achieve comparable performance even if we omit intermediate non-linearity . These observations gave us questions about the current models of deep graph NNs in terms of their expressive power . Several studies gave theoretical explanations of the over-smoothing phenomena for linear GNNs ( Li et al. , 2018 ; Zhang , 2019 ; Zhao & Akoglu , 2020 ) . We can think of our theory as an extension of their results to non-linear GNNs . 3 PROBLEM SETTING AND MAIN RESULT . 3.1 NOTATION . Let N+ be the set of positive integers . For N ∈ N+ , we denote [ N ] : = { 1 , . . . , N } . For a vector v ∈ RN , we write v ≥ 0 if and only if vn ≥ 0 for all n ∈ [ N ] . Similarly , for a matrix X ∈ RN×C , we write X ≥ 0 if and only if Xnc ≥ 0 for all n ∈ [ N ] and c ∈ [ C ] . We say such a vector and matrix is non-negative . 〈· , ·〉 denotes the inner product of vectors or matrices , depending on the context : 〈u , v〉 : = u > v for u , v ∈ RN and 〈X , Y 〉 : = tr ( XTY ) for X , Y ∈ RN×C . 1P equals to 1 if the proposition P is true else 0 . For vectors v ∈ RN and w ∈ RC , v ⊗ w ∈ RN×C denotes the Kronecker product of v and w defined by ( v⊗w ) nc : = vnwc . For X ∈ RN×C , ‖X‖F : = 〈X , X〉 1 2 denotes the Frobenius norm of X . For a vector v ∈ RN , diag ( v ) : = ( vnδnm ) n , m∈ [ N ] ∈ RN×N denotes the diagonalization of v. IN ∈ RN×N denotes the identity matrix of size N . For a linear operator P : RN → RM and a subset V ⊂ RN , we denote the restriction of P to V by P |V .

Summary: The paper studies why graph NNs lose the expressive power as additional layers are added. A dynamical system perspective is adopted and used to show that under certain conditions on the weights, the expressiveness of the network deteriorates. This is since the network's output eventually only carries information about superficial graph properties for distinguishing nodes, and nothing else. A case study of Erdos-Renyi graph is provided, showing that for dense graphs information loss indeed occurs. Guidelines to try and deal with this in practice were devised and empirically examined.