Deconvolutional Networks on Graph Data

1 Introduction . Graph Convolutional Networks ( GCNs ) [ 6 , 20 ] have been widely used to transform an input graph structure A and feature matrix X into graph representations H , which can be taken as a mapping function f : ( A , X ) → H . In this paper , we consider its inverse problem g : ( A , H ) → X , i.e. , given the graph representations smoothed by GCNs , how can we reconstruct the input graph signal ? More specifically , we initiate a systematic study of Graph Deconvolutional Networks ( GDNs ) , to reconstruct the signals smoothed by GCNs . A good GDN component could benefit many applications such as reconstruction [ 44 ] and generation [ 19 ] . For examples in computer vision , several studies [ 45 , 29 ] have demonstrated that the performance of autoencoders on reconstruction and generation can be further improved by encoding with Convolutional Networks and decoding with Deconvolutional Networks [ 46 ] . However , extending this symmetric autoencoder framework to graph-structured data requires graph deconvolutional operations , which remains open-ended and hasn ’ t been well studied as opposed to the large body of solutions that have already been proposed for GCNs . Most GCNs proposed by prior works , e.g. , Cheby-GCN [ 6 ] and GCN [ 20 ] , exploit spectral graph convolutions [ 33 ] and Chebyshev polynomials [ 15 ] to retain coarse-grained information and avoid explicit eigen-decomposition of the graph Laplacian . Until recently , [ 39 ] and [ 8 ] have noticed that GCN [ 20 ] acts as a low pass filter in spectral domain and retains smoothed representations . Inspired by prior works in the domain of signal deconvolution [ 22 , 9 , 13 ] , we are motivated to design a GDN by an inverse operation of the low pass filters embodied in GCNs . Furthermore , we observe directly reversing a low pass filter results in a high frequency amplifier in spectral domain and may amplify the noise . In this regard , we introduce a spectral-wavelet GDN to decode the smoothed representations into the input graph signals . The proposed spectral-wavelet GDN employs spectral graph convolutions with an inverse filter to obtain inversed signals and then de-noises the inversed signals in wavelet domain . In addition , we apply Maclaurin series as a fast approximation technique to compute both inverse filters and wavelet kernels [ 8 ] . We evaluate the effectiveness of the proposed GDN with two tasks : graph feature imputation [ 35 , 44 ] and graph structure generation [ 19 , 14 ] . For the former task , we further propose a graph autoencoder ( GAE ) framework that resembles the symmetric fashion of architectures [ 29 ] . The proposed GAE outperforms the state-of-the-arts on six benchmarks . For the latter task , our proposed GDN can 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . enhance the generation performance of popular variational autoencoder frameworks including VGAE [ 19 ] and Graphite [ 14 ] . The contributions of this work are summarized as follows : • We propose Graph Deconvolutional Network ( GDN ) , the opposite operation of Graph Convolutional Network ( GCN ) that reconstructs graph signals from smoothed node representations . • We study GDN from the perspective of inverse operations in spectral domain and identity the inverse operation results in a high frequency amplifier and may amplify the noise . To solve the noise issue , a de-noising process based on graph wavelet is further proposed . 2 Preliminaries . In this work , we are given an undirected , unweighted graph G = ( V , A , X ) . V is the node set and N = |V | denotes the number of nodes . The adjacency matrix A ∈ RN×N represents the graph structure . The feature matrix X ∈ RN×d represents the node attributes . 2.1 Convolution on graphs . The convolutional layers are used to derive smoothed node representations , such that nodes that are similar in topological space should be close enough in Euclidean space . Formally , the spectral graph convolution on a signal x ∈ RN is defined as : h = gc ∗ x = Udiag ( gc ( λ1 ) , . . . , gc ( λN ) ) U > x , ( 1 ) where { λi } Ni=1 are the eigenvalues of the normalized Laplacian matrix Lsym = D− 1 2LD− 1 2 , L and D are the Laplacian and degree matrices of the input graph A respectively , and U is the eigenvector matrix of Lsym = UΛU > . In this work , we focus on GCN [ 20 ] , one of the popular convolution operations on graphs : H = GCN ( A , X ) , ( 2 ) where H ∈ RN×v denotes smoothed node representations . Specifically , [ 39 ] show that GCN is a low pass filter in spectral domain with gc ( λi ) = 1− λi , i.e. , hc = σ ( Udiag ( gc ( λ ) ) U > x ) , ( 3 ) where σ ( · ) is the activation function . 2.2 Deconvolution on graphs . The deconvolutional layers are used to recover the original graph features given smoothed node representations . Formally , the spectral graph deconvolution on the smoothed representation h ∈ RN is defined as : x′ = gd ∗ h = Udiag ( gd ( λ1 ) , . . . , gd ( λN ) ) U > h , ( 4 ) With this deconvolution operation , we can use GDNs to produce fine-grained graph signals from the encoded H . X ′ = GDN ( A , H ) , ( 5 ) where X ′ ∈ RN×d denotes the recovered graph features . We shall further discuss our design of GDN in Section 3 . From the perspective of encoder-decoder , the convolution operation could be used as an encoder and the deconvolution operation as a decoder . We show a schematic diagram of this encoder-decoder perspective in Figure 1 . 3 Graph Deconvolutional Network . In this section , we present our design of GDN . Motivated by prior works in signal deconvolution [ 16 ] , a naive deconvolutional net can be obtained using the inverse filter g−1c ( λi ) = 1 1−λi in spectral domain . Unfortunately , the naive deconvolutional net results in a high frequency amplifier and may amplify the noise [ 9 , 13 ] . This point is obvious for g−1c ( λi ) as the signal fluctuates near λi = 1 if noise exists . 3.1 Noise analysis . Our target is to recover the original graph features given smoothed node representations hc encoded by GCN [ 20 ] . Without the noise , it is naive to obtain the closed-form recovery relationship by the inverse operation g−1c ( λ ) = 1 1−λ directly , i.e. , x = Udiag ( g−1c ( λ ) ) U > σ−1 ( hc ) , ( 6 ) Here , we assume the activation function σ ( · ) is an inverse operator , e.g. , Leaky ReLU , sigmoid , tanh . Actually , it is quite reasonable to consider such a prototypical model of invertible networks in the theoretical analysis . Otherwise , we need to address an ill-posed inverse problem instead as we automatically lose some information ( e.g. , null space ) . Hence , it is impossible to recover the ground-truth even under the noiseless model , see [ 1 ] for details . Graph noise is ubiquitous in real-world applications . Following the convention in signal deconvolution [ 9 ] and image restoration [ 13 ] , let ’ s consider the white noise model , i.e. , ĥ is the smoothed node representations mixed with graph convolution and white noise , ĥ = σ ( Udiag ( gc ( λ ) ) U > x ) + , ( 7 ) where is the white Gaussian noise ( i.e. , ∼ N ( 0 , σ2IN ) ) . Next , we show that it will amplify the noise if we apply the inverse operation to the white noise model directly . That is , x′ = Udiag ( g−1c ( λ ) ) U > σ−1 ( ĥ ) , ( 8 ) Proposition 1 . Considering the independent white noise model ( 7 ) and ( 8 ) , if the inversed activation function is deferentiable which satisfies ∂σ−1 ( 0 ) ≥ 1 , the variance of x′ is larger than the white noised added on the generative model . It was worthwhile mentioning that the condition regarding the inverse operator ∂σ−1 ( 0 ) ≥ 1 is quite reasonable . Actually , it holds for many representative activation functions , including sigmoid , tanh and Leaky ReLU . Please refer to Appendix A for details . Proposition 1 shows the proposed inverse operation may introduce undesirable noise into the output graph . Thus , a de-noising process is necessary thereafter . 3.2 The proposed GDN . To ameliorate the noise issue , we propose an efficient , hybrid spectral-wavelet deconvolutional network that performs inverse signal recovery in spectral domain first , and then conducts a de-noising step in wavelet domain to remove the amplified noise [ 28 ] . 3.2.1 Inverse operator . Regarding the inverse operation in ( 6 ) , the core is to explicitly compute eigen-decomposition . We propose to apply Maclaurin series approximation on g−1c ( Λ ) = ∑∞ n=0 Λ n and get a fast algorithm as below : Udiag ( g−1c ( λ ) ) U = ∑∞ n=0 L n sym . ( 9 ) A detailed derivative of ( 9 ) can be found in Appendix E. When truncated low order approximation is used in practice with n ≤ 3 [ 6 ] , we derive a spectral filter with g−1c ( λi ) = 1 + λi + λ2i + λ3i . Following [ 29 ] , we can use another activation function and trainable parameter matrix to estimate σ−1 , to avoid the potential non-invertible problem . Thus , the inverse operator becomes : M = σ ( ( IN + 3∑ n=1 Lnsym ) HW3 ) , ( 10 ) where W3 is the parameter set to be learned . We contrast the difference between inverse operator in this work and other reconstruction methods including [ 47 ] and GALA [ 30 ] . Inverse operator vs. GCN decoder [ 47 ] Compared with GCN decoder [ 47 ] which directly uses GCN for signal reconstruction , the proposed inverse operator demonstrates its efficacy in recovering the high frequency signals of the graph , as shown in Figure 3 ( b ) . We shall further discuss this point in Section 3.3 . Inverse operator vs. GALA GALA [ 30 ] can be viewed as the first-order approximation towards inverse operator in this work , with a spectral kernel of 1 + λi . The difference is that our inverse operator can well approximate the inverse filter g−1c ( λi ) = 1 1−λi with third-order polynomials . 3.2.2 Wavelet de-noising . Wavelet-based methods have a strong impact on the field of de-noising , especially in image restoration when noise is amplified by inverse filters [ 28 ] . In the literature , many wavelet de-noising methods have been proposed , e.g. , SureShrink [ 9 ] , BayesShrink [ 5 ] , and they differ in how they estimate the separating threshold . Our method generalizes these threshold ideas and automatically separates the noise from the useful information with a learning-based approach . Consider the input signal of wavelet system is mixed with Gaussian noise , the reasons why wavelet denoising works are : ( 1 ) a graph wavelet basis holds the majority of signal energy into a small number of large coefficients , and ( 2 ) Gaussian noise in the original graph domain is again a Gaussian noise in wavelet domain ( with the same amplitude ) , see [ 16 ] for details . The core thus becomes how we can separate a small number of large coefficients and the noise in wavelet domain . Given the latter is unknown , we adopt a trainable parameter matrix to linearly transform the noise under zero and useful coefficients bigger than zero , where ReLU is used to eliminate the noise . Consider a set of wavelet bases Ψs = ( Ψs1 , . . . , ΨsN ) , where each one Ψsi denotes a signal on graph diffused away from node i and s is a scaling parameter [ 40 ] , the wavelet bases can be written as Ψs = Ugs ( Λ ) U > and gs ( · ) is a filter kernel . Following previous works GWNN [ 40 ] and GRAPHWAVE [ 8 ] , we use the heat kernel gs ( λi ) = e−sλi , as heat kernel admits easy calculation of inverse wavelet transform with g−1s ( λi ) = e sλi . In addition , we can apply Maclaurin series approximation on heat kernel and neglect explicit eigen-decomposition by : Ψs = ∑∞ n=0 ( −1 ) n n ! s nLnsym , ( 11 ) Ψ−1s = ∑∞ n=0 sn n ! L n sym , ( 12 ) To cope with the noise threshold estimation problem , we introduce a parameterized matrix in wavelet domain and eliminate these noise coefficients via a ReLU function , in the hope of learning a separating threshold from the data itself . Together with the representation M raised in Section 3.2.1 , we derive the reconstructed signal as : X ′ = ΨsReLU ( Ψ−1s MW4 ) W5 , ( 13 ) where W4 and W5 are trainable parameters . We then contrast the difference between Wavelet Neural Network ( WNN ) in this work and other related WNNs including GWNN [ 40 ] and GRAPHWAVE [ 8 ] . Scalability An important issue in WNN is that one needs to avoid explicit eigen-decomposition to compute wavelet bases for large graphs . Both GRAPHWAVE and GWNN , though trying to avoid eigen-decomposition by exploiting Chebyshev polynomials , still rely integral operations ( see Appendix D in [ 40 ] ) and there is no clear way to scale up . Differently , we use Maclaurin series to approximate wavelet bases , which has explicit polynomial coefficients and can resemble the heat kernel well when n = 3 . Please refer to Figure 2 for more details . De-noising The purpose of both GWNN and GRAPHWAVE is to derive node presentations with localized graph convolution and flexible neighborhood , such that downstream tasks like classification can be simplified . On the contrary , our work implements wavelet neural network in the purpose of detaching the useful information and the noise amplified by inverse operator . Due to the different purpose , our work applies the activation function in wavelet domain while GWNN in the original vertex domain .

Summary: This paper proposed GDN, the opposite of GCN that recovers graph signals from smoothed representations. The introduced GDN uses spectral graph convolutions with a high pass filter to obtain inversed signals and then de-noises the inversed signals in wavelet domain. Effectiveness of GDN is validated on graph feature imputation and graph structure generation tasks.