Graph Wavelet Neural Network

We present graph wavelet neural network ( GWNN ) , a novel graph convolutional neural network ( CNN ) , leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform . Different from graph Fourier transform , graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost . Moreover , graph wavelets are sparse and localized in vertex domain , offering high efficiency and good interpretability for graph convolution . The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets : Cora , Citeseer and Pubmed . 1 INTRODUCTION . Convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) have been successfully used in many machine learning problems , such as image classification ( He et al. , 2016 ) and speech recognition ( Hinton et al. , 2012 ) , where there is an underlying Euclidean structure . The success of CNNs lies in their ability to leverage the statistical properties of Euclidean data , e.g. , translation invariance . However , in many research areas , data are naturally located in a non-Euclidean space , with graph or network being one typical case . The non-Euclidean nature of graph is the main obstacle or challenge when we attempt to generalize CNNs to graph . For example , convolution is not well defined in graph , due to that the size of neighborhood for each node varies dramatically ( Bronstein et al. , 2017 ) . Existing methods attempting to generalize CNNs to graph data fall into two categories , spatial methods and spectral methods , according to the way that convolution is defined . Spatial methods define convolution directly on the vertex domain , following the practice of the conventional CNN . For each vertex , convolution is defined as a weighted average function over all vertices located in its neighborhood , with the weighting function characterizing the influence exerting to the target vertex by its neighbors ( Monti et al. , 2017 ) . The main challenge is to define a convolution operator that can handle neighborhood with different sizes and maintain the weight sharing property of CNN . Although spatial methods gain some initial success and offer us a flexible framework to generalize CNNs to graph , it is still elusive to determine appropriate neighborhood . Spectral methods define convolution via graph Fourier transform and convolution theorem . Spectral methods leverage graph Fourier transform to convert signals defined in vertex domain into spectral domain , e.g. , the space spanned by the eigenvectors of the graph Laplacian matrix , and then filter is defined in spectral domain , maintaining the weight sharing property of CNN . As the pioneering work of spectral methods , spectral CNN ( Bruna et al. , 2014 ) exploited graph data with the graph Fourier transform to implement convolution operator using convolution theorem . Some subsequent works make spectral methods spectrum-free ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Khasanova & Frossard , 2017 ) , achieving locality in spatial domain and avoiding high computational cost of the eigendecomposition of Laplacian matrix . In this paper , we present graph wavelet neural network to implement efficient convolution on graph data . We take graph wavelets instead of the eigenvectors of graph Laplacian as a set of bases , and define the convolution operator via wavelet transform and convolution theorem . Graph wavelet neural network distinguishes itself from spectral CNN by its three desirable properties : ( 1 ) Graph wavelets can be obtained via a fast algorithm without requiring the eigendecomposition of Laplacian matrix , and thus is efficient ; ( 2 ) Graph wavelets are sparse , while eigenvectors of Laplacian matrix are dense . As a result , graph wavelet transform is much more efficient than graph Fourier transform ; ( 3 ) Graph wavelets are localized in vertex domain , reflecting the information diffusion centered at each node ( Tremblay & Borgnat , 2014 ) . This property eases the understanding of graph convolution defined by graph wavelets . We develop an efficient implementation of the proposed graph wavelet neural network . Convolution in conventional CNN learns an individual convolution kernel for each pair of input feature and output feature , causing a huge number of parameters especially when the number of features is high . We detach the feature transformation from convolution and learn a sole convolution kernel among all features , substantially reducing the number of parameters . Finally , we validate the effectiveness of the proposed graph wavelet neural network by applying it to graph-based semi-supervised classification . Experimental results demonstrate that our method consistently outperforms previous spectral CNNs on three benchmark datasets , i.e. , Cora , Citeseer , and Pubmed . 2 OUR METHOD . 2.1 PRELIMINARY . Let G = { V , E , A } be an undirected graph , where V is the set of nodes with |V| = n , E is the set of edges , and A is adjacency matrix withAi , j = Aj , i to define the connection between node i and node j . The graph Laplacian matrix L is defined as L = D−A where D is a diagonal degree matrix with Di , i = ∑ j Ai , j , and the normalized Laplacian matrix is L = In−D−1/2AD−1/2 where In is the identity matrix . Since L is a real symmetric matrix , it has a complete set of orthonormal eigenvectors U = ( u1 , u2 , ... , un ) , known as Laplacian eigenvectors . These eigenvectors have associated real , non-negative eigenvalues { λl } nl=1 , identified as the frequencies of graph . Eigenvectors associated with smaller eigenvalues carry slow varying signals , indicating that connected nodes share similar values . In contrast , eigenvectors associated with larger eigenvalues carry faster varying signals across connected nodes . 2.2 GRAPH FOURIER TRANSFORM . Taking the eigenvectors of normalized Laplacian matrix as a set of bases , graph Fourier transform of a signal x ∈ Rn on graph G is defined as x̂ = U > x , and the inverse graph Fourier transform is x = Ux̂ ( Shuman et al. , 2013 ) . Graph Fourier transform , according to convolution theorem , offers us a way to define the graph convolution operator , denoted as ∗G . Denoting with y the convolution kernel , ∗G is defined as x ∗G y = U ( ( U > y ) ( U > x ) ) , ( 1 ) where is the element-wise Hadamard product . Replacing the vector U > y by a diagonal matrix gθ , then Hadamard product can be written in the form of matrix multiplication . Filtering the signal x by the filter gθ , we can write Equation ( 1 ) as UgθU > x . However , there are some limitations when using Fourier transform to implement graph convolution : ( 1 ) Eigendecomposition of Laplacian matrix to obtain Fourier basis U is of high computational cost with O ( n3 ) ; ( 2 ) Graph Fourier transform is inefficient , since it involves the multiplication between a dense matrix U and the signal x ; ( 3 ) Graph convolution defined through Fourier transform is not localized in vertex domain , i.e. , the influence to the signal on one node is not localized in its neighborhood . To address these limitations , ChebyNet ( Defferrard et al. , 2016 ) restricts convolution kernel gθ to a polynomial expansion gθ = K−1∑ k=0 θkΛ k , ( 2 ) where K is a hyper-parameter to determine the range of node neighborhoods via the shortest path distance , θ ∈ RK is a vector of polynomial coefficients , and Λ =diag ( { λl } nl=1 ) . However , such a polynomial approximation limits the flexibility to define appropriate convolution on graph , i.e. , with a smaller K , it ’ s hard to approximate the diagonal matrix gθ with n free parameters . While with a largerK , locality is no longer guaranteed . Different from ChebyNet , we address the aforementioned three limitations through replacing graph Fourier transform with graph wavelet transform . 2.3 GRAPH WAVELET TRANSFORM . Similar to graph Fourier transform , graph wavelet transform projects graph signal from vertex domain into spectral domain . Graph wavelet transform employs a set of wavelets as bases , defined as ψs = ( ψs1 , ψs2 , ... , ψsn ) , where each wavelet ψsi corresponds to a signal on graph diffused away from node i and s is a scaling parameter . Mathematically , ψsi can be written as ψs = UGsU > , ( 3 ) where U is Laplacian eigenvectors , Gs=diag ( g ( sλ1 ) , ... , g ( sλn ) ) is a scaling matrix and g ( sλi ) = eλis . Using graph wavelets as bases , graph wavelet transform of a signal x on graph is defined as x̂ = ψ−1s x and the inverse graph wavelet transform is x = ψsx̂ . Note that ψ −1 s can be obtained by simply replacing the g ( sλi ) in ψs with g ( −sλi ) corresponding to a heat kernel ( Donnat et al. , 2018 ) . Replacing the graph Fourier transform in Equation ( 1 ) with graph wavelet transform , we obtain the graph convolution as x ∗G y = ψs ( ( ψ−1s y ) ( ψ−1s x ) ) . ( 4 ) Compared to graph Fourier transform , graph wavelet transform has the following benefits when being used to define graph convolution : 1 . High efficiency : graph wavelets can be obtained via a fast algorithm without requiring the eigendecomposition of Laplacian matrix . In Hammond et al . ( 2011 ) , a method is proposed to use Chebyshev polynomials to efficiently approximate ψs and ψ−1s , with the computational complexity O ( m× |E| ) , where |E| is the number of edges and m is the order of Chebyshev polynomials . 2 . High spareness : the matrix ψs and ψ−1s are both sparse for real world networks , given that these networks are usually sparse . Therefore , graph wavelet transform is much more computationally efficient than graph Fourier transform . For example , in the Cora dataset , more than 97 % elements in ψ−1s are zero while only less than 1 % elements in U > are zero ( Table 4 ) . 3 . Localized convolution : each wavelet corresponds to a signal on graph diffused away from a centered node , highly localized in vertex domain . As a result , the graph convolution defined in Equation ( 4 ) is localized in vertex domain . We show the localization property of graph convolution in Appendix A . It is the localization property that explains why graph wavelet transform outperforms Fourier transform in defining graph convolution and the associated tasks like graph-based semisupervised learning . 4 . Flexible neighborhood : graph wavelets are more flexible to adjust node ’ s neighborhoods . Different from previous methods which constrain neighborhoods by the discrete shortest path distance , our method leverages a continuous manner , i.e. , varying the scaling parameter s. A small value of s generally corresponds to a smaller neighborhood . Figure 1 shows two wavelet bases at different scale on an example network , depicted using GSP toolbox ( Perraudin et al. , 2014 ) . 2.4 GRAPH WAVELET NEURAL NETWORK . Replacing Fourier transform with wavelet transform , graph wavelet neural network ( GWNN ) is a multi-layer convolutional neural network . The structure of the m-th layer is Xm+1 [ : ,j ] = h ( ψs p∑ i=1 Fmi , jψ −1 s X m [ : ,i ] ) j = 1 , · · · , q , ( 5 ) where ψs is wavelet bases , ψ−1s is the graph wavelet transform matrix at scale s which projects signal in vertex domain into spectral domain , Xm [ : ,i ] with dimensions n × 1 is the i-th column of Xm , Fmi , j is a diagonal filter matrix learned in spectral domain , and h is a non-linear activation function . This layer transforms an input tensor Xm with dimensions n × p into an output tensor Xm+1 with dimensions n× q . In this paper , we consider a two-layer GWNN for semi-supervised node classification on graph . The formulation of our model is first layer : X2 [ : ,j ] = ReLU ( ψs p∑ i=1 F 1i , jψ −1 s X 1 [ : ,i ] ) j = 1 , · · · , q , ( 6 ) second layer : Zj = softmax ( ψs q∑ i=1 F 2i , jψ −1 s X 2 [ : , , i ] ) j = 1 , · · · , c , ( 7 ) where c is the number of classes in node classification , Z of dimensions n × c is the prediction result . The loss function is the cross-entropy error over all labeled examples : Loss = − ∑ l∈yL c∑ i=1 YlilnZli , ( 8 ) where yL is the labeled node set , Yli = 1 if the label of node l is i , and Yli = 0 otherwise . The weights F are trained using gradient descent .

Summary: This paper proposes to learn graph wavelet kernels through a neural network. This idea is interesting, even if it is not a real surprise, given the interesting features graph wavelets, and the explosion of proposals for graph neural networks. Yet, the paper is interesting, pretty complete, and the proposed algorithm is shown to be relatively effective.