A Unified Framework for Convolution-based Graph Neural Networks

1 INTRODUCTION . Recent years have witnessed a fast development in graph processing by generalizing convolution operation to graph-structured data , which is known as Graph Convolutional Networks ( GCNs ) ( Kipf & Welling , 2017 ) . Due to the great success , numerous variants of GCNs have been developed and extensively adopted in the field of social network analysis ( Hamilton et al. , 2017 ; Wu et al. , 2019a ; Veličković et al. , 2018 ) , biology ( Zitnik et al. , 2018 ) , transportation forecasting ( Li et al. , 2017 ) and natural language processing ( Wu et al. , 2019b ; Yao et al. , 2019 ) . Inspired by GCN , a wide variety of convolution-based graph learning approaches are proposed to enhance the generalization performance of graph neural networks . Several research aim to achieve higher expressiveness by exploring higher-order information or introducing additional learning mechanisms like attention modules . Although proposed from different perspectives , their exist some connections between these approaches . For example , attention-based GCNs like GAT ( Veličković et al. , 2018 ) and AGNN ( Thekumparampil et al. , 2018 ) share the similar intention by adjusting the adjacency matrix with a function of edge and node features . Similarly , TAGCN ( Du et al. , 2017 ) and MixHop ( Kapoor et al. , 2019 ) can be viewed as particular instances of PPNP ( Klicpera et al. , 2018 ) under certain approximation . However , the relations among these graph learning models are rarely studied and the comparisons are still limited in analyzing generalization performances on public datasets . As a consequence , we still lack a systematic view of different GCN models and deep understanding of the relations among them . In this paper , we resort to the techniques in graph signal processing and attempt to understand GCN-based approaches from a general perspective . Specifically , we present a unified graph convolution framework by establishing graph convolution operations with optimization problems in the graph Fourier domain . We consider a Laplacian regularized least squares optimization problem and show that most of the convolution-based approaches can be interpreted in this framework by adding carefully designed regularizers . Besides vanilla GCNs , we also extend our framework to formulating non-convolutional operations ( Xu et al. , 2018a ; Hamilton et al. , 2017 ) , attention-based GCNs ( Veličković et al. , 2018 ; Thekumparampil et al. , 2018 ) and topology-based GCNs ( Klicpera et al. , 2018 ; Kapoor et al. , 2019 ) , which cover a large fraction of the state-of-the-art graph learning ap- proaches . This novel perspective provides a re-interpretation of graph convolution operations and enables a better understanding of the similarities and differences among many widely used GCNs , and may inspire new approaches for designing better models . As a conclusion , we summarize our contributions as follow : 1 . We introduce a unified framework for convolution-based graph neural networks and interpret various convolution filters as carefully designed regularizers in the graph Fourier domain , which provides a general methodology for evaluating and relating different graph learning modules . 2 . Based on the proposed framework , we provide new insights on understanding the limitations of GCNs and show new directions to tackle common problems and improve the generalization performance of current graph neural networks in the graph Fourier domain . Additionally , the unified framework can serve as a once-for-all platform for expert-designed modules on convolution-based approaches , where newly designed modules can be easily implemented on other networks as a plugin module with trivial adaptations . We believe that our framework can provide convenience for designing new graph learning modules and searching for better combinations . 3 . As a showcase , we present a novel regularization technique under the proposed framework to alleviate the oversmoothing problem in graph representation learning . As shown in Section 4 , the newly designed regularizer can be implemented on several convolution-based networks and effectively improve the generalization performance of graph learning models . 2 PRELIMINARY . We start with an overview of the basic concepts of graph signal processing . Let G = ( V , A ) denote a graph with node feature vectors where V represents the vertex set consisting of nodes { v1 , v2 , . . . , vN } and A = ( aij ) ∈ RN×N is the adjacency matrix implying the connectivity between nodes in the graph . Let D = diag ( d ( 1 ) , . . . , d ( N ) ) ∈ RN×N be the degree matrix of A where d ( i ) = ∑ j∈V aij is the degree of vertex i . Then , L = D−A is the combinatorial Laplacian and L̃ = I−D ( −1/2 ) AD ( −1/2 ) is the normalized Laplacian of G. Additionally , we let Ã = A+I and D̃ = D + I denote the augmented adjacency and degree matrices with added self-loops . Then L̃sym = I − D̃−1/2ÃD̃−1/2 ( Ãsym = D̃−1/2ÃD̃−1/2 ) and L̃rw = I − D̃−1Ã ( Ãrw = D̃−1Ã ) are the augmented symmetric normalized and random walk normalized Laplacian ( augmented adjacency matrices ) of G , respectively . Let x ∈ RN be a signal on the vertices of the graph . The spectral convolution is defined as a function of a filter gθ parameterized in the Fourier domain ( Kipf & Welling , 2017 ) : gθ ? x = Ugθ ( Λ ) U Tx , ( 1 ) where U and Λ are the eigenvectors and eigenvalues of the normalized Laplacian L̃ . Also , we follow Hoang & Maehara ( 2019 ) and define the variation ∆ and D̃-inner product as : ∆ ( x ) = ∑ i , j∈V aij ( x ( i ) − x ( j ) ) 2 = xTLx , ( x , y ) D̃ = ∑ i∈V ( d ( i ) + 1 ) x ( i ) y ( i ) = xT D̃y , ( 2 ) which specifies the smoothness and importance of the signal respectively . 3 UNIFIED GRAPH CONVOLUTION FRAMEWORK . With the success of GCNs , a wide variety of convolution-based approaches are proposed which progressively enhance the expressive power and generalization performance of graph neural networks . Despite the effectiveness of GCN and its derivatives on specific tasks , there still lack a comprehensive understanding on the relations and differences among various graph learning modules . Graph signal processing is a powerful technique which has been adopted in several graph learning researches ( Kipf & Welling , 2017 ; Hoang & Maehara , 2019 ; Zhao & Akoglu , 2019 ) . However , existing researches mainly focus on analyzing the properties of GCNs while ignore the connections between different graph learning modules . Innovatively , in this work , we consider interpreting convolution-based approaches from a general perspective with graph signal processing techniques . In specific , we establish the connections between graph convolution operations and optimization problems in graph Fourier space , showing the effect of each module explicitly with specific regularizers . This novel perspective provides a systematic view of different GCN models and deep understanding of the relations among them . 3.1 UNIFIED GRAPH CONVOLUTION FRAMEWORK . Several researches have proved that , in the field of graph signal processing , the representative features are mostly preserved in the low-frequency signals while noises are mostly contained in the high-frequency signals ( Hoang & Maehara , 2019 ) . Based on this observation , numerous graph representation learning methods are designed to decrease the high-frequency components , which can be viewed as low-pass filters in the graph Fourier space . With similar inspiration , we consider a Laplacian regularized least squares optimization problem with graph signal regularizers and attempt to build connections with these filters . Definition 1 Unified Graph Convolution Framework . Graph convolution filters can be achieved by solving the following Laplacian regularized least squares optimization : min X̄ ∑ i∈V ‖x̄ ( i ) − x ( i ) ‖2 D̃ + λLreg , ( 3 ) where ‖x‖D̃ = √ ( x , x ) D̃ denotes the norm induced by D̃ . In the following sections , we will show that a wide range of convolution-based graph neural networks can be derived from Definition 1 with different carefully designed regularizers , and provide new insights on understanding different graph learning modules from the graph signal perspective . 3.1.1 GRAPH CONVOLUTIONAL NETWORKS . Graph convolutional networks ( GCNs ) ( Kipf & Welling , 2017 ) are the foundation of numerous graph learning models and have received widespread concerns . Several researches have demonstrated that the vanilla GCN is essentially a type of Laplacian smoothing over the whole graph , which makes the features of the connected nodes similar . Therefore , to reformulate GCNs in the graph Fourier space , we consider utilizing the variation ∆ ( x ) as the regularizer . Definition 2 Vanilla GCNs . Let x̄ ( i ) i∈V be the estimation of the input observation x ( i ) i∈V . A low-pass filter : X̄ = ÃrwX , ( 4 ) is the first-order approximation of the optimal solution of the following optimization : min X̄ ∑ i∈V ‖x̄ ( i ) − x ( i ) ‖2 D̃ + ∑ i , j∈V aij‖x̄ ( i ) − x̄ ( j ) ‖22 . ( 5 ) Derivations of the definitions are presented in Appendix A . As the eigenvalues of the approximated filter Ãrw are bounded by 1 , it resembles a low-pass filter that removes the high-frequency signals . By exchanging Ãrw with Ãsym ( which has the same eigenvalues as Ãrw ) , we obtain the same formulation adopted in GCNs . It has been stated that the second term ∆ ( x ) in Eq . ( 5 ) measures the variation of the estimation x̄ over the graph structure . By adding this regularizer to the objective function , the obtained filter emphasizes the low-frequency signals through minimizing the variation over the local graph structure , while keeping the estimation close to the input in the graph Fourier space . 3.1.2 NON-CONVOLUTIONAL OPERATIONS . Residual Connection . Residual connection is first proposed by He et al . ( 2016 ) and has been widely adopted in graph representation learning approaches . In the vanilla GCNs , norms of the eigenvalues of the filter Ãrw ( or Ãsym ) are bounded by 1 which ensures numerical stability in the training procedure . However , on the other hand , signals in all frequency band will shrink as the convolution layer stacks , leading to a consistent information loss . Therefore , adding the residual connection is deemed to preserve the strength of the input signal . Definition 3 Residual Connection . A graph convolution filter with residual connection : X̄ = ÃrwX + X , ( 6 ) where > 0 controls the strength of residual connection , is the first-order approximation of the optimal solution of the following optimization : min X̄ ∑ i∈V ( ‖x̄ ( i ) − x ( i ) ‖2 D̃ − ‖x̄ ( i ) ‖2 D̃ ) + ∑ i , j∈V aij‖x̄ ( i ) − x̄ ( j ) ‖22 . ( 7 ) By adding the negative regularizer to penalize the estimations with small norms , we can induce the same formulation as the vanilla graph convolution with residual connection . Concatenation . Concatenation is practically a residual connection with different learning weights . Definition 3 ’ Concatenation . A graph convolution filter concatenating with the input signal : X̄ = ÃrwX + XΘΘ T , ( 8 ) is the first-order approximation of the optimal solution of the following optimization : min X̄ ∑ i∈V ( ‖x̄ ( i ) − x ( i ) ‖2 D̃ − ‖x̄ ( i ) Θ‖2 D̃ ) + ∑ i , j∈V aij‖x̄ ( i ) − x̄ ( j ) ‖22 , ( 9 ) where > 0 controls the strength of concatenation and Θ is the learning coefficient . Although the learning weights ΘΘT has a constrained expressive capability , it can be compensated by the following feature learning modules .

Summary: This paper presents a unified framework for graph convolutional neural networks based on regularized optimization, connecting different  variants of graph neural networks including vanilla, attention-based, and topology-based approaches. The authors also propose  a novel regularization technique to approach the oversmoothing problem in graph convolution. Experiments on the standard settings of node classification on Citeseer, Cora, and Pubmed prove the effectiveness of the proposed regularization techniques.