Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective

1 INTRODUCTION . Over the last five years , many Graph Neural Networks ( GNNs ) have been proposed in the literature of geometric deep learning ( Veličković et al. , 2018 ; Gilmer et al. , 2017 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) , in order to generalize the very efficient deep learning paradigm into the world of graphs . This large number of contributions explains a new challenge recently tackled by the community , which consists in assessing the expressive power of GNNs . In this area of research , there is a consensus to evaluate the theoretic expressive power of GNNs according to equivalence of Weisfeiler-Lehman ( WL ) test order ( Morris et al. , 2019 ; Xu et al. , 2019 ; Maron et al. , 2019b ; a ) . Hence , GNNs models are frequently classified as ” as powerful as 1-WL ” , ” as powerful as 2-WL ” , . . . , ” as powerful as k-WL ” . However , this perspective can not make differences between two methods if they are as powerful as the same WL test order . Moreover , it does not always explain success or failure of any GNN on common benchmark datasets . In this paper , we claim that analyzing theoretically and experimentally GNNs with a spectral point of view can bring a new perspective on their expressive power . So far , GNNs have been generally studied separately as spectral based or as spatial based ( Wu et al. , 2019b ; Chami et al. , 2020 ) . To the best of our knowledge , Message Passing Neural Networks ( MPNNs ) ( Gilmer et al. , 2017 ) and GraphNets ( Battaglia et al. , 2018 ) are the only attempts to merge ∗muhammetbalcilar @ gmail.com both approaches in the same framework . However , these models are not able to generalize custom designed spectral filters , as well as the effect of each convolution support in a multi convolution case . The spatial-spectral connection is also mentioned indirectly in several cornerstone studies by Defferrard et al . ( 2016 ) ; Kipf & Welling ( 2017 ) ; Levie et al . ( 2019 ) . Since the spectral-spatial interchangeability is missing , they did not propose to show spectral behavior of any graph convolution . Recent studies have also attempted to show , for a limited number of spatial GNNs , that they act as low-pass filters ( NT & Maehara , 2019 ; Wu et al. , 2019a ) . NT & Maehara ( 2019 ) concluded that using adjacency induces low-pass effects , while Wu et al . ( 2019a ) studied a single spatial GNN ’ s spectral behavior by assuming adding self-connection changes the given topology of the graph . In this paper , we bridge the gap between spectral and spatial domains for GNNs . Our first contribution consists in demonstrating the equivalence of convolution processes regardless if they are defined as spatial or as spectral GNN . Using this connection , we propose a new general framework and taxonomy for GNNs as the second contribution . Taking advantage of this equivalence , our third contribution is to provide a spectral analysis of any GNN model . This spectral analysis is another perspective for the analysis of expressive power of GNNs . Our theoretical spectral analysis is confirmed by experiments on various well-known graph datasets . Furthermore , we show the necessity of high and/or band-pass filters in our experiments , while the majority of GNNs are limited to only low-pass filters and thus inevitably fail when dealing with these problems . The code used in this paper is available at https : //github.com/balcilar/gnn-spectral-expressive-power . The remainder of this paper is organized as follows . Section 2 introduces convolutional GNNs and presents existing approaches . In Section 3 and Section 4 , we describe the main contributions mentioned above . Section 5 presents a series of experiments and results which validate our propositions . Finally , Section 6 concludes this paper . 2 PROBLEM STATEMENT AND STATE OF THE ART . Let G be a graph with n nodes and an arbitrary number of edges . Connectivity is given by the adjacency matrix A ∈ { 0 , 1 } n×n and features are defined on nodes by X ∈ Rn×f0 , with f0 the length of feature vectors . For any matrix X , we used Xi , X : j and Xi , j to refer its i-th column vector , j-th row vector and scalar value on ( i , j ) location , respectively . A graph Laplacian is L = D − A ( or L = I −D−1/2AD−1/2 ) where D ∈ Rn×n is the diagonal degree matrix and I is the identity . Through eigendecomposition , L can be written by L = Udiag ( λ ) UT where each column of U ∈ Rn×n is an eigenvector of L , λ ∈ Rn gathers the eigenvalues of L and diag ( . ) function creates a diagonal matrix whose diagonal elements are from a given vector . We use superscript to refer same kind variable as base . For instance , H ( l ) ∈ Rn×fl refers node representation on layer l whose feature dimension is fl . A Graph Convolution layer takes the node representation of the previous layer H ( l−1 ) as input and produces a new representation H ( l ) , with H ( 0 ) = X . 2.1 SPECTRAL APPROACHES . Spectral GNNs rely on the spectral graph theory ( Chung , 1997 ) . In this framework , signals on graphs are filtered using the eigendecomposition of graph Laplacian ( Shuman et al. , 2013 ) . By transposing the convolution theorem to graphs , the spectral filtering in the frequency domain can be defined by xflt = Udiag ( Φ ( λ ) ) U > x , where Φ ( . ) is the desired filter function . As a consequence , a graph convolution layer in spectral domain can be written by a sum of filtered signals followed by an activation function as in ( Bruna et al. , 2013 ) , namely H ( l+1 ) j = σ ( fl∑ i=1 Udiag ( F ( l , j ) i ) U > H ( l ) i ) , for j ∈ { 1 , . . . , fl+1 } . ( 1 ) Here , σ is the activation function , F ( l , j ) ∈ Rn×fl is the corresponding weight vector to be tuned as used in ( Henaff et al. , 2015 ) for the single-graph problem known as non-parametric spectral GNN . A first drawback is the necessity of Fourier and inverse Fourier transform by matrix multiplication of U and UT . Another drawback occurs when generalizing the approach to multi-graph learning problems . Indeed , the k-th element of the vector F ( l , j ) i weights the contribution of the k-th eigenvector to the output . Those weights are not shareable between graphs of different sizes , which means a different length of F ( l , j ) i is needed . Moreover , even though the graphs have the same number of nodes , their eigenvalues will be different if their structures differ . To overcome these issues , a few spatially-localized filters have been defined such as cubic B-spline ( Bruna et al. , 2013 ) , polynomial and Chebyshev polynomial ( Defferrard et al. , 2016 ) and Cayley polynomial parameterization ( Levie et al. , 2019 ) . With such approaches , trainable parameters are defined by F ( l , j ) i = B [ W ( l,1 ) i , j , . . . , W ( l , se ) i , j ] > , where each column in B ∈ Rn×se is designed as a function of eigenvalues , namely Bk , s = Φs ( λk ) , where k = 1 , . . . , n denotes eigenvalue index , s = 1 , . . . , se denotes index of filters and se is the number of desired filters . Here , W ( l , s ) ∈ Rfl×fl+1 is the trainable matrix for the l-th layer ’ s s-th filter ’ s . 2.2 SPATIAL APPROACHES . Spatial GNNs consider an agg operator , which aggregates the neighborhood nodes , and an upd operator , which updates the concerned node as follows : H ( l+1 ) : v = upd ( g0 ( H ( l ) : v ) , agg ( g1 ( H ( l ) : u ) : u ∈ N ( v ) ) ) , ( 2 ) whereN ( v ) is the set of neighborhood nodes and g0 , g1 : Rn×fl → Rn×fl+1 trainable models . The choice of agg , upd , g0 , g1 , and even N ( v ) , determines the capability of model . The vanilla GNN ( known by GIN-0 in ( Xu et al. , 2019 ) ) uses the same weights in g0 and g1 . N ( v ) is the set of connected nodes to v , agg is the sum of all connected node values and upd ( x , y ) : = σ ( x+ y ) where σ is an elementwise nonlinearity . GCN has the same selection but normalizes features as in ( Kipf & Welling , 2017 ) . Hamilton et al . ( 2017 ) used separated weights in g0 and g1 , which means that two sets of trainable weights are applied on self feature and neighbor nodes . Other approaches defined multi neighborhood and used different gi for different kind of neighborhood . For instance , Duvenaud et al . ( 2015 ) defined the neighborhood according to node label and/or degree , Niepert et al . ( 2016 ) reordered the neighbor nodes and used the same model gi to neighbors according to their order . These spatial GNNs use sum or normalized sum over gi in equation 2 . Other methods weighted this summation by another trainable parameter , where the weights can be written by the function of node and/or edge features in order to make the convolutions more productive , such as graph attention networks ( Veličković et al. , 2018 ) , MoNet ( Monti et al. , 2017 ) , GatedGCN ( Bresson & Laurent , 2018 ) and SplineCNN ( Fey et al. , 2018 ) . 3 BRIDGING SPATIAL AND SPECTRAL GNNS . In this section , we define a general framework which includes most of the well-know GNN models , including euclidean convolution and models which use anisotropic update schema such as in Veličković et al . ( 2018 ) ; Bresson & Laurent ( 2018 ) . When upd ( x , y ) = σ ( x + y ) , agg is a sum ( or weighted sum ) of the defined neighborhood nodes contributions and gi applies linear transformation , one can trivially show that mentioned spatial GNNs can be generalized as propagation of the node features to the neighboring nodes followed by feature transformation and activation function of the form H ( l+1 ) = σ ( ∑ s C ( s ) H ( l ) W ( l , s ) ) , ( 3 ) where C ( s ) ∈ Rn×n is the s-th convolution support that defines how the node features are propagated to the neighboring nodes . Within this generalization , GNNs differ from each other by the choice of convolution supports C ( s ) . This formulation generalizes many different kinds of Graph Convolutions , as well as Euclidean domain convolutions , which can be seen in Appendix A with the detailed schema . Definition 1 . A Trainable-support is a Graph Convolution Support C ( s ) with at least one trainable parameter that can be tuned during training . If C ( s ) has no trainable parameters , i.e . when the supports are pre-designed , it is called a fixed-support graph convolution . In the trainable support case , supports can be different in each layer , which can be shown by C ( l , s ) for the s-th support in layer l. Formally , we can define a trainable support by : ( C ( l , s ) ) v , u = hs , l ( H ( l ) : v , H ( l ) : u , E ( l ) v , u , A ) , ( 4 ) where E ( l ) v , u shows edge features on layer l from node v to node u if it is available and h ( . ) is any trainable model parametrized by ( s , l ) . Theorem 1 . Spectral GNN parameterized with B of entries Bi , j = Φj ( λi ) , defined as H ( l+1 ) j =σ ( fl∑ i=1 U diag ( B [ W ( l,1 ) i , j , . . . , W ( l , se ) i , j ] > ) U > H ( l ) i ) , ( 5 ) is a particular case of framework in equation 3 with the convolution kernel set to C ( s ) = U diag ( Φs ( λ ) ) U > . ( 6 ) The proof can be found in Appendix B . This theorem is general and it covers many well-known spectral GNNs , such as non-parametric spectral graph convolution ( Henaff et al. , 2015 ) , polynomial parameterization ( Defferrard et al. , 2016 ) , cubic B-spline parameterization ( Bruna et al. , 2013 ) , CayleyNet ( Levie et al. , 2019 ) and also any custom designed graph convolution . From Theorem 1 , one can see that the spatial and spectral GNNs work all the same way . Therefore , Fourier calculations are not necessary when convolutions are parameterized by B . As a consequence of Theorem 1 , one can see that the separation of spectral and spatial GNNs is just an interpretation . The only difference is the way convolution supports are designed : either in the spectral domain or in the spatial one . Definition 2 . A Spectral-designed graph convolution refers to a convolution where supports are written as a function of eigenvalues ( Φs ( λ ) ) and eigenvectors ( U ) of the corresponding graph Laplacian ( equation 6 ) . Thus , each convolution supportC ( s ) has the same frequency response Φs ( λ ) over different graphs . Graph convolution out of this definition is called spatial-designed graph convolution . Corollary 1.1 . The frequency profile of any given graph convolution support C ( s ) can be defined in spectral domain by Φs ( λ ) = diag−1 ( U > C ( s ) U ) . ( 7 ) where diag−1 ( . ) returns the vector made of the diagonal elements from the given matrix . The proof of this corollary is given in Appendix C. This corollary leads to the spectral analysis of any given graph convolution support , including spatial-designed convolutions . Since the spatialdesigned convolutions do not fit into equation 6 , U > C ( s ) U is not a diagonal matrix . Therefore , we also compute the full frequency profile by Φs = U > C ( s ) U , which includes all eigenvectors pairwise contributions for spatial-designed convolutions .

Summary: This work studies the performance of different Graph Neural Network (GNN) from a spectral perspective. In particular, it shows the kernel for all kinds of proposed GNN models can be expressed in a general form with a specific frequency response definition, which indicates the spectral property (spectrum) of the kernel. Based on this definition, this work empirically studies the band-pass property for kernels in different models and demonstrate the importance of such spectral perspective.