Frequency Analysis for Graph Convolution Network

In this work , we develop quantitative results to the learnability of a two-layers Graph Convolutional Network ( GCN ) . Instead of analyzing GCN under some classes of functions , our approach provides a quantitative gap between a two-layers GCN and a two-layers MLP model . Our analysis is based on the graph signal processing ( GSP ) approach , which can provide much more useful insights than the message-passing computational model . Interestingly , based on our analysis , we have been able to empirically demonstrate a few cases when GCN and other state-of-the-art models can not learn even when true vertex features are extremely low-dimensional . To demonstrate our theoretical findings and propose a solution to the aforementioned adversarial cases , we build a proof of concept graph neural network model with stacked filters named Graph Filters Neural Network ( gfNN ) . 1 INTRODUCTION . Graph neural networks ( GNN ) is a class of neural networks which can learn from graph-structured data . Recently , graph neural networks for vertex classification and graph isomorphism test have achieved excellent results on several benchmark datasets and continuously set new state-of-the-art performance ( Kipf and Welling , 2017 ; Veličković et al. , 2019 ; Wu et al. , 2019 ; Klicpera et al. , 2019 ) . Many variants of GNN have been proposed to solve problems in social networks ( Hamilton et al. , 2017 ; Zhang et al. , 2018a ) , biology ( Veličković et al. , 2017 ; 2019 ) , chemistry ( Fout et al. , 2017 ; Gilmer et al. , 2017 ) , natural language processing ( Bastings et al. , 2017 ; Zhang et al. , 2018b ; Wu et al. , 2019 ) , reasoning for vision ( Santoro et al. , 2017 ) , and few/zero-shot learning ( Garcia and Bruna , 2017 ; Kampffmeyer et al. , 2019 ) . Due to the rise of GNNs in machine learning applications , understanding GNNs theoretically has gathered lots of attention in the machine learning community . While most theoretical results of GNNs are based on the message-passing model ( Xu et al. , 2019 ; Keriven and Peyré , 2019 ) , there are a limited number of theoretical results for the filtering approach . However , in practice , the graph filtering view have inspired many computationally fast and high accuracy models such as ChebNet ( Defferrard et al. , 2016 ) , GCN ( Kipf and Welling , 2017 ) , SplineConv ( Fey et al. , 2018 ) , LanczosNet ( Liao et al. , 2019 ) , and label efficient models ( Li et al. , 2019 ) . In this work , we theoretically study the GCN model ( Kipf and Welling , 2017 ) and the SGC model ( Wu et al. , 2019 ) . Instead of showing which class of function a GCN or GNN model can theoretically learn , we develop quantitative bounds for the gaps between GCN/SGC and a two-layers fully connected neural network . Our theoretical results imply a few cases where SGC and GCN would fail to work : high-frequency labels , noisy features , and complex features . Interestingly , we also find other stateof-the-art GNN models such as GraphSAGE Hamilton et al . ( 2017 ) or DGI Veličković et al . ( 2019 ) failed to perform in our high-frequency labels experiments . Recently , Wu et al . ( 2019 ) showed that graph convolutional networks ( GCN ) can be broken down into two steps : low-pass filtering and feature learning . Such simplification not only improves GCN ’ s computational speed and accuracy in a wide range of problems but also allows a better understanding of the GCN model . To show the low-pass filtering nature of GCN-like models , Wu et al . ( 2019 ) used the Rayleigh quotient to bound the maximum eigenvalue when self-loops are added to the graph , which means adding self-loops created a stronger low-pass filter ( Wu et al. , 2019 , Lemma 3 , Theorem 1 ) . Similar low-pass observation and results have also been mentioned by other works by Li et al . ( 2018 ) , Qiu et al . ( 2018 ) , and Li et al . ( 2019 ) . We adopt SGC ’ s two steps simplification and gf ( A ) X W1 gf ( A ) σ * * W2 softmax gf ( A ) X W1 k * softmax gf ( A ) X W1 k * softmax σ W2 GCN SGC gfNN Figure 2 : A simple realization of gfNN the graph signal processing perspective ( Ortega et al. , 2018 ) to our work and further extend their implications to demonstrate when GCN or SGC does not perform well . We claim the following contributions : • Our first contribution is another proof for ( Wu et al. , 2019 , Theorem 1 ) using the CourantFisher-Weyl ’ s min-max principle . Our result ( Theorem 3 , Section 4 ) has the advantage of showing the monotone shrinking of all eigenvalues . In addtion , our proof on generalized eigenvalues is simpler and shorter . • Our second and main contribution is the exploration of cases when SGC or GCN fails to work and the following performance gap Theorem : Theorem 1 ( Informal , see Theorem 7 , 8 ) . Under Assumption 2 , the outcomes of SGC , GCN , and gfNN are similar to those of the corresponding NNs using “ true features ” . • Our third contribution is to empirically verify Assumption 2 since our Theorem 7 and 8 are based on this assumption ( stated below ) . Assumption 2 . Input features consist of frequency-limited true features and noise . The true features have sufficient information for the machine learning task . As a base model for analysis , we build a proof of concept model named gfNN ( Figure 2 ) . This model adopts the two steps simplification from SGC with two major differences : 1 . We use a two-layers MLP as the classifier ; and 2 . We use two different filters ( low-pass and high-pass ) . Our theorem 7 implies that , under Assumption 2 , both gfNN ( with appropriate filter ) and GCN Kipf and Welling ( 2017 ) have similar high performance . Since gfNN does not require multiplications of the adjacency matrix during the learning phase , it is much faster than GCN . Besides , gfNN maintains an expected behavior in the aforementioned cases when other GNN models do not work . 2 GRAPH SIGNAL PROCESSING . Graph signal processing ( GSP ) regards data on the vertices as signals and applies signal processing techniques to understand the signal characteristics . By combining signals ( feature vectors ) and graph structure ( adjacency matrix or its transformations ) , GSP has inspired the development of learning algorithms on graph-structured data Shuman et al . ( 2012 ) . In a standard signal processing problem , it is common to assume the observations contain some noise and the underlying “ true signal ” has low-frequency Rabiner and Gold ( 1975 ) . Our Assumption 2 is of similar nature . Many recent GNNs were built upon results from graph signal processing . The most common practice is to multiply the ( augmented ) normalized adjacency matrix I − L̃ with the feature matrix X . The product ( I − L̃ ) X is understood as features averaging and propagation . In graph signal processing literature , such operation filters signals on the graph without explicitly performing eigendecomposition on the normalized Laplacian matrix , which requires O ( n3 ) time Vaseghi ( 2008 ) . Here , we refer to this augmented normalized adjacency matrix and its variants as graph filters and propagation matrices interchangeably . In this section , we introduce the basic concepts of graph signal processing . We adopt a recent formulation Girault et al . ( 2018 ) of graph Fourier transform on irregular graphs . Let G = ( V , E ) be a simple undirected graph , where V = { 1 , . . . , n } be the set of n ∈ Z vertices and E be the set of edges.1 Let A = ( aij ) ∈ Rn×n be the adjacency matrix of G , D = diag ( d ( 1 ) , . . . , d ( n ) ) ∈ Rn×n be the degree matrix of G , where d ( i ) = ∑ j∈V a ( i , j ) is the degree of vertex i ∈ V . L = D − A ∈ Rn×n be the combinatorial Laplacian of G , L = I −D−1/2AD−1/2 be the normalized Laplacian of G , where I ∈ Rn×n is the identity matrix , andLrw = I−D−1A be the random walk Laplacian ofG . Also , for γ ∈ R with γ > 0 , let Ã = A+γI be the augmented adjacency matrix , which is obtained by adding γ self loops to G , D̃ = D + γI be the corresponding augmented degree matrix , and L̃ = D̃ − Ã = L , L̃ = I − D̃−1/2ÃD̃−1/2 , L̃rw = I − D̃−1Ã be the corresponding augmented combinatorial , normalized , and random walk Laplacian matrices . A vector x ∈ Rn defined on the vertices of the graph is called a graph signal . To introduce a graph Fourier transform , we need to define two operations , variation and inner product , on the space of graph signals . Here , we define the variation ∆ : Rn → R and the D̃-inner product by ∆ ( x ) : = ∑ ( i , j ) ∈E ( x ( i ) − x ( j ) ) 2 = x > Lx ; ( x , y ) D̃ : = ∑ i∈V ( d ( i ) + γ ) x ( i ) y ( i ) = x > D̃y . ( 1 ) We denote by ‖x‖D̃ : = √ ( x , x ) D̃ the norm induced by D̃ . Intuitively , the variation ∆ and the inner product ( · , · ) D̃ specify how to measure the smoothness and importance of the signal , respectively . In particular , our inner product puts more importance on high-degree vertices , where larger γ closes the importance more uniformly . We then consider the generalized eigenvalue problem ( variational form ) : Find u1 , . . . , un ∈ Rn such that for each i ∈ { 1 , . . . , n } , ui is a solution to the following optimization problem : minimize ∆ ( u ) subject to ( u , u ) D̃ = 1 , ( u , uj ) D̃ = 0 , j ∈ { 1 , . . . , n } . ( 2 ) The solution ui is called an i-th generalized eigenvector and the corresponding objective value λi : = ∆ ( ui ) is called the i-th generalized eigenvalue . The generalized eigenvalues and eigenvectors are also the solutions to the following generalized eigenvalue problem ( equation form ) : Lu = λD̃u . ( 3 ) Thus , if ( λ , u ) is a generalized eigenpair then ( λ , D̃1/2u ) is an eigenpair of L̃ . A generalized eigenvector with a smaller generalized eigenvalue is smoother in terms of the variation ∆ . Hence , the generalized eigenvalues are referred to as the frequency of the graph . The graph Fourier transform is a basis expansion by the generalized eigenvectors . Let U = [ u1 , . . . , un ] be the matrix whose columns are the generalized eigenvectors . Then , the graph Fourier transform F : Rn → Rn is defined by Fx = x̂ : = U > D̃x , and the inverse graph Fourier transform F−1 is defined by F−1x̂ = Ux̂ . Note that these are actually the inverse transforms since FF−1 = U > D̃U = I . The Parseval identity relates the norms of the data and its Fourier transform : ‖x‖D̃ = ‖x̂‖2 . ( 4 ) Let h : R→ R be an analytic function . The graph filter specified by h is a mapping x 7→ y defined by the relation in the frequency domain : ŷ ( λ ) = h ( λ ) x̂ ( λ ) . In the spatial domain , the above equation is equivalent to y = h ( L̃rw ) x. where h ( L̃rw ) is defined via the Taylor expansion of h ; see Higham ( 2008 ) for the detail of matrix functions . In a machine learning problem on a graph , each vertex i ∈ V has a d-dimensional feature x ( i ) ∈ Rd . We regard the features as d graph signals and define the graph Fourier transform of the features by the graph Fourier transform of each signal . Let X = [ x ( 1 ) ; . . . , x ( n ) ] > be the feature matrix . Then , the graph Fourier transform is represented by FX = X̂ = : U > D̃X and the inverse transform is F−1X̂ = UX̂ . We denote X̂ = [ x̂ ( λ1 ) ; . . . ; x̂ ( λn ) ] > as the frequency components of X . 1We only consider unweighted edges but it is easily adopted to positively weighted edges .

Summary: The authors extend the existing work SGC [1] to a nonlinear version, which addresses the limitations of dealing with nonlinear feature. It further extends the theoretical finding in [1] about the low-pass filtering functionality of graph convolutional networks and shows its advantage in dealing with graph signal processing problem. Evaluation of the proposed method is performed on 7 datasets for node classification task.