LanczosNet: Multi-Scale Deep Graph Convolutional Networks

1 INTRODUCTION . Graph-structured data is ubiquitous in real world applications , social networks , gene expression regulatory networks , protein-protein interactions , and many other physical systems . How to model such data using machine learning , especially deep learning , has become a central research question [ 1 ] . For supervised and semi-supervised tasks such as graph or node classification and regression , learning based models can be roughly categorized into two classes , formulated either in terms of graph convolutions [ 2 ] or recurrent neural networks [ 3 ] . Methods based on recurrent neural networks ( RNN ) , especially graph neural networks ( GNN ) [ 3 ] , repeatedly unroll a message passing process over the graph by exchanging information between the nodes . In theory , a GNN can have as large a model capacity as its convolutional counterpart . However , due to the instability of RNN dynamics and difficulty of optimization , GNN and its variants are generally slower and harder to train . In this paper we focus on graph convolution based methods . Built on top of the graph signal processing ( GSP ) approaches [ 4 ] , these methods extend convolution operators to graphs by leveraging spectral graph theory , graph wavelet theory , etc . Graph convolutions can be stacked and combined with nonlinear activation functions to build deep models , just as in regular convolutional neural networks ( CNN ) . They often have large model capacity and achieve promising results . Also , graph convolution can be easily implemented with modern scientific computing libraries . There are two main issues with current graph convolution approaches . First , it is not clear how to efficiently leverage multi-scale information except by directly stacking multiple layers . Having an effective multi-scale scheme is key for enabling the model to be invariant to scale changes , and to capture many intrinsic regularities [ 5 , 6 ] . Graph coarsening methods have been proposed to form a hierarchy of multi-scale graphs [ 7 ] , but this coarsening process is fixed during both inference and learning which may cause some bias . Alternatively , the graph signal can be multiplied by the exponentiated graph Laplacian , where the exponent indicates the scale of the diffusion process on the graph [ 8 ] . Unfortunately , the computation and memory cost increases linearly with the exponent , which prohibits the exploitation of long scale diffusion in practice . Other fast methods for computing matrix power such as exponentiating by squaring are very memory intensive , even for moderately large graphs . Second , spectral filters within current graph convolution based models are mostly fixed . In the context of image processing , using a Gaussian kernel along with a spectral filter f ( λ ) = 2λ − λ2 corresponds to running forward the heat equation ( blurring ) followed by running it backwards ( sharpening ) [ 9 ] . Multi-scale kernels introduced in [ 10 ] extends the idea of forward-backward diffusion process and can be represented as polynomials of matrices related to a Gaussian kernel . Learning the spectral filters is thus beneficial since it learns the stochastic processes on the graph which produce useful representations for particular tasks . However , how to learn spectral filters which have large model capacity is largely underexplored . In this paper , we propose the Lanczos network ( LanczosNet ) to overcome the aforementioned issues . First , based on the tridiagonal decomposition implied by the Lanczos algorithm , our model exploits the low rank approximation of the graph Laplacian . This approximation facilitates efficient computation of matrix powers thus gathering multi-scale information easily . Second , we design learnable spectral filters based on the approximation which effectively increase model capacity . In scenarios where one wants to learn the graph kernel and/or node embeddings , we propose another variant , i.e. , adaptive Lanczos network ( AdaLanczosNet ) , which back-propagates through the Lanczos algorithm . We show that our proposed model is closely related to graph based manifold learning approaches such as diffusion maps which could potentially inspire more work from the intersection between deep graph networks and manifold learning . We benchmark against 9 recent deep graph networks , including both convolutional and RNN based methods , on citation networks and a quantum chemistry graph regression problem , and achieve state-of-the-art results in most tasks . 2 BACKGROUND . In this section , we introduce some background material . A graph G with N nodes is denoted as G = ( V , E , A ) , where A ∈ RN×N is an adjacency matrix which could either be binary or real valued . X ∈ RN×F is the compact representation of node features ( or graph signal in the GSP literature ) . For any node v ∈ V , we denote its feature as a row vector Xv , : ∈ R1×F . We use X : ,i to denote the i-th column of X . Graph Fourier Transform Given input node features X , we now discuss how to perform a graph convolution . Based on the adjacency matrix A , we compute the graph Laplacian L which can be defined in different ways : ( 1 ) L = D − A ; ( 2 ) L = I −D−1A ; ( 3 ) L = I −D− 12AD− 12 , where D is a diagonal degree matrix and Di , i = ∑N j=1Ai , j . The definition ( 3 ) is often used in the GSP literature due to the fact that it is real symmetric , positive semi-definite ( PSD ) and has eigenvalues lying in [ 0 , 2 ] . In certain applications [ 11 ] , it was found that adding self-loops , i.e. , changing A to A + I , and using the affinity matrix S = D− 1 2AD− 1 2 instead of L gives better results . Since S is real symmetric , based on spectral decomposition , we have S = UΛU > where U is an orthogonal matrix and its column vectors are the eigenvectors of S. The diagonal matrix Λ contains the sorted eigenvalues where Λi , i = λi and 1 ≥ λ1 ≥ · · · ≥ λN ≥ −1 . Based on the eigenbasis , we can define the graph Fourier transform Y = U > X and its inverse transform X̂ = UY following [ 12 ] . Note that L = I −D− 12AD− 12 shares the same eigenvectors with S = D− 12AD− 12 and the eigenvalues of L are µi = 1− λi . Therefore , L and S share the same graph Fourier transform which justifies the usages of S. Different forms of filters can be further constructed in the spectral domain . Localized Polynomial Filter A τ -localized polynomial filter is typically adopted in GSP literature [ 12 ] , gw ( Λ ) = ∑τ−1 t=0 wtΛ t , where w = [ w0 , w1 , . . . , wτ−1 ] ∈ Rτ×1 is the filter coefficient , i.e. , learnable parameter . The filter is τ -localized in the sense that the filtering leverages information from nodes which are at most τ -hops away . One prominent example of this class is the Chebyshev polynomial filter [ 7 ] . Here the graph Laplacian is modified to L̃ = 2L/λmax− I such that its eigenvalues fall into [ −1 , 1 ] . Then the Chebyshev polynomial recursion is applied : X̃ ( t ) = 2L̃X̃ ( t− 1 ) − X̃ ( t− 2 ) where X̃ ( 0 ) = X and X̃ ( 1 ) = L̃X . For a pair of input and output channels ( i , j ) , the final filtering becomes , yi , j = [ X̃ ( 0 ) : ,i , . . . , X̃ ( τ − 1 ) : ,i ] wi , j , where [ · ] means concatenation along columns and wi , j ∈ Rτ×1 . Chebyshev polynomials provide two benefits : they form an orthogonal basis of L2 ( [ −1 , 1 ] , dy/ √ 1− y2 ) and one avoids the spectral decomposition of L̃ in the filtering . However , the functional form of the spectral filter is not learnable , and can not adapt to the data . In this paper , instead of using the modified graph Laplacian L̃ , we use the aforementioned S. Therefore , we can write the localized polynomial filtering in a more general form as , Y = τ−1∑ t=0 gt ( S , . . . , S t , X ) Wt , ( 1 ) where gt is a function that takes node features X and powers of the affinity matrices up to the t-th order as input and outputs a N × F matrix . Wt ∈ RF×O is the corresponding filter coefficient and Y ∈ RN×O is the output . One can easily verify that in the Chebyshev polynomial filter , any i-th column of the corresponding gt ( X , S , . . . , St ) lies in the Krylov subspace Kt+1 ( S , X : ,i ) ≡ span { X : ,i , SX : ,i , . . . , StX : ,i } . This naturally motivates the usage of Krylov subspace methods , like the Lanczos algorithm [ 13 ] , since it provides an orthonormal basis for the above Krylov subspace , thus making the filter coefficients compact . 3 LANCZOS NETWORKS . In this section , we first introduce the Lanczos algorithm which approximates the affinity matrix S. We present our first model , called Lanczos network ( LanczosNet ) , in which we execute the Lanczos algorithm once per graph and fix the basis throughout inference and learning . Then we introduce the adaptive Lanczos network ( AdaLanczosNet ) in which we learn the graph kernel and/or node embedding by back-propagating through the Lanczos algorithm . Algorithm 1 : Lanczos Algorithm 1 : Input : S , x , K , 2 : Initialization : β0 = 0 , q0 = 0 , and q1 = x/‖x‖ 3 : For j = 1 , 2 , . . . , K : 4 : z = Sqj 5 : γj = q > j z 6 : z = z − γjqj − βj−1qj−1 7 : βj = ‖z‖2 8 : If βj < , quit 9 : qj+1 = z/βj 10 : 11 : Q = [ q1 , q2 , · · · , qK ] 12 : Construct T following Eq . ( 2 ) 13 : Eigen decomposition T = BRB > 14 : Return V = QB and R. Algorithm 2 : LanczosNet 1 : Input : Signal X , Lanczos output V and R , scale index sets S and I , 2 : Initialization : Y0 = X 3 : For ` = 1 , 2 , . . . , ` c : 4 : Z = Y ` −1 , Z = { ∅ } 5 : For j = 1 , 2 , . . . , max ( S ) : 6 : Z = SZ 7 : If j ∈ S : 8 : Z = Z ∪ Z 9 : For i ∈ I : 10 : Z = Z ∪ V R̂ ( Ii ) V > Y ` −1 11 : Y ` = concat ( Z ) W ` 12 : If ` < L 13 : Y ` = Dropout ( σ ( Y ` ) ) 14 : Return Y ` c .

Summary: The authors propose a novel method for learning graph convolutional networks. The core idea is to use the Lanczos algorithm to obtain a low-rank approximation of the graph Laplacian. The authors propose two ways to include the Lanczos algorithm. First, as a preprocessing step where the algorithm is applied once on the input graph and the resulting approximation is fixed during learning. Second, by including a differentiable version of the algorithm into an end-to-end trainable model.