Deconvolutional Networks on Graph Data

1 Introduction . Graph Convolutional Networks ( GCNs ) [ 6 , 20 ] have been widely used to transform an input graph structure A and feature matrix X into graph representations H , which can be taken as a mapping function f : ( A , X ) → H . In this paper , we consider its inverse problem g : ( A , H ) → X , i.e. , given the graph representations smoothed by GCNs , how can we reconstruct the input graph signal ? More specifically , we initiate a systematic study of Graph Deconvolutional Networks ( GDNs ) , to reconstruct the signals smoothed by GCNs . A good GDN component could benefit many applications such as reconstruction [ 44 ] and generation [ 19 ] . For examples in computer vision , several studies [ 45 , 29 ] have demonstrated that the performance of autoencoders on reconstruction and generation can be further improved by encoding with Convolutional Networks and decoding with Deconvolutional Networks [ 46 ] . However , extending this symmetric autoencoder framework to graph-structured data requires graph deconvolutional operations , which remains open-ended and hasn ’ t been well studied as opposed to the large body of solutions that have already been proposed for GCNs . Most GCNs proposed by prior works , e.g. , Cheby-GCN [ 6 ] and GCN [ 20 ] , exploit spectral graph convolutions [ 33 ] and Chebyshev polynomials [ 15 ] to retain coarse-grained information and avoid explicit eigen-decomposition of the graph Laplacian . Until recently , [ 39 ] and [ 8 ] have noticed that GCN [ 20 ] acts as a low pass filter in spectral domain and retains smoothed representations . Inspired by prior works in the domain of signal deconvolution [ 22 , 9 , 13 ] , we are motivated to design a GDN by an inverse operation of the low pass filters embodied in GCNs . Furthermore , we observe directly reversing a low pass filter results in a high frequency amplifier in spectral domain and may amplify the noise . In this regard , we introduce a spectral-wavelet GDN to decode the smoothed representations into the input graph signals . The proposed spectral-wavelet GDN employs spectral graph convolutions with an inverse filter to obtain inversed signals and then de-noises the inversed signals in wavelet domain . In addition , we apply Maclaurin series as a fast approximation technique to compute both inverse filters and wavelet kernels [ 8 ] . We evaluate the effectiveness of the proposed GDN with two tasks : graph feature imputation [ 35 , 44 ] and graph structure generation [ 19 , 14 ] . For the former task , we further propose a graph autoencoder ( GAE ) framework that resembles the symmetric fashion of architectures [ 29 ] . The proposed GAE outperforms the state-of-the-arts on six benchmarks . For the latter task , our proposed GDN can 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . enhance the generation performance of popular variational autoencoder frameworks including VGAE [ 19 ] and Graphite [ 14 ] . The contributions of this work are summarized as follows : • We propose Graph Deconvolutional Network ( GDN ) , the opposite operation of Graph Convolutional Network ( GCN ) that reconstructs graph signals from smoothed node representations . • We study GDN from the perspective of inverse operations in spectral domain and identity the inverse operation results in a high frequency amplifier and may amplify the noise . To solve the noise issue , a de-noising process based on graph wavelet is further proposed . 2 Preliminaries . In this work , we are given an undirected , unweighted graph G = ( V , A , X ) . V is the node set and N = |V | denotes the number of nodes . The adjacency matrix A ∈ RN×N represents the graph structure . The feature matrix X ∈ RN×d represents the node attributes . 2.1 Convolution on graphs . The convolutional layers are used to derive smoothed node representations , such that nodes that are similar in topological space should be close enough in Euclidean space . Formally , the spectral graph convolution on a signal x ∈ RN is defined as : h = gc ∗ x = Udiag ( gc ( λ1 ) , . . . , gc ( λN ) ) U > x , ( 1 ) where { λi } Ni=1 are the eigenvalues of the normalized Laplacian matrix Lsym = D− 1 2LD− 1 2 , L and D are the Laplacian and degree matrices of the input graph A respectively , and U is the eigenvector matrix of Lsym = UΛU > . In this work , we focus on GCN [ 20 ] , one of the popular convolution operations on graphs : H = GCN ( A , X ) , ( 2 ) where H ∈ RN×v denotes smoothed node representations . Specifically , [ 39 ] show that GCN is a low pass filter in spectral domain with gc ( λi ) = 1− λi , i.e. , hc = σ ( Udiag ( gc ( λ ) ) U > x ) , ( 3 ) where σ ( · ) is the activation function . 2.2 Deconvolution on graphs . The deconvolutional layers are used to recover the original graph features given smoothed node representations . Formally , the spectral graph deconvolution on the smoothed representation h ∈ RN is defined as : x′ = gd ∗ h = Udiag ( gd ( λ1 ) , . . . , gd ( λN ) ) U > h , ( 4 ) With this deconvolution operation , we can use GDNs to produce fine-grained graph signals from the encoded H . X ′ = GDN ( A , H ) , ( 5 ) where X ′ ∈ RN×d denotes the recovered graph features . We shall further discuss our design of GDN in Section 3 . From the perspective of encoder-decoder , the convolution operation could be used as an encoder and the deconvolution operation as a decoder . We show a schematic diagram of this encoder-decoder perspective in Figure 1 . 3 Graph Deconvolutional Network . In this section , we present our design of GDN . Motivated by prior works in signal deconvolution [ 16 ] , a naive deconvolutional net can be obtained using the inverse filter g−1c ( λi ) = 1 1−λi in spectral domain . Unfortunately , the naive deconvolutional net results in a high frequency amplifier and may amplify the noise [ 9 , 13 ] . This point is obvious for g−1c ( λi ) as the signal fluctuates near λi = 1 if noise exists . 3.1 Noise analysis . Our target is to recover the original graph features given smoothed node representations hc encoded by GCN [ 20 ] . Without the noise , it is naive to obtain the closed-form recovery relationship by the inverse operation g−1c ( λ ) = 1 1−λ directly , i.e. , x = Udiag ( g−1c ( λ ) ) U > σ−1 ( hc ) , ( 6 ) Here , we assume the activation function σ ( · ) is an inverse operator , e.g. , Leaky ReLU , sigmoid , tanh . Actually , it is quite reasonable to consider such a prototypical model of invertible networks in the theoretical analysis . Otherwise , we need to address an ill-posed inverse problem instead as we automatically lose some information ( e.g. , null space ) . Hence , it is impossible to recover the ground-truth even under the noiseless model , see [ 1 ] for details . Graph noise is ubiquitous in real-world applications . Following the convention in signal deconvolution [ 9 ] and image restoration [ 13 ] , let ’ s consider the white noise model , i.e. , ĥ is the smoothed node representations mixed with graph convolution and white noise , ĥ = σ ( Udiag ( gc ( λ ) ) U > x ) + , ( 7 ) where is the white Gaussian noise ( i.e. , ∼ N ( 0 , σ2IN ) ) . Next , we show that it will amplify the noise if we apply the inverse operation to the white noise model directly . That is , x′ = Udiag ( g−1c ( λ ) ) U > σ−1 ( ĥ ) , ( 8 ) Proposition 1 . Considering the independent white noise model ( 7 ) and ( 8 ) , if the inversed activation function is deferentiable which satisfies ∂σ−1 ( 0 ) ≥ 1 , the variance of x′ is larger than the white noised added on the generative model . It was worthwhile mentioning that the condition regarding the inverse operator ∂σ−1 ( 0 ) ≥ 1 is quite reasonable . Actually , it holds for many representative activation functions , including sigmoid , tanh and Leaky ReLU . Please refer to Appendix A for details . Proposition 1 shows the proposed inverse operation may introduce undesirable noise into the output graph . Thus , a de-noising process is necessary thereafter . 3.2 The proposed GDN . To ameliorate the noise issue , we propose an efficient , hybrid spectral-wavelet deconvolutional network that performs inverse signal recovery in spectral domain first , and then conducts a de-noising step in wavelet domain to remove the amplified noise [ 28 ] . 3.2.1 Inverse operator . Regarding the inverse operation in ( 6 ) , the core is to explicitly compute eigen-decomposition . We propose to apply Maclaurin series approximation on g−1c ( Λ ) = ∑∞ n=0 Λ n and get a fast algorithm as below : Udiag ( g−1c ( λ ) ) U = ∑∞ n=0 L n sym . ( 9 ) A detailed derivative of ( 9 ) can be found in Appendix E. When truncated low order approximation is used in practice with n ≤ 3 [ 6 ] , we derive a spectral filter with g−1c ( λi ) = 1 + λi + λ2i + λ3i . Following [ 29 ] , we can use another activation function and trainable parameter matrix to estimate σ−1 , to avoid the potential non-invertible problem . Thus , the inverse operator becomes : M = σ ( ( IN + 3∑ n=1 Lnsym ) HW3 ) , ( 10 ) where W3 is the parameter set to be learned . We contrast the difference between inverse operator in this work and other reconstruction methods including [ 47 ] and GALA [ 30 ] . Inverse operator vs. GCN decoder [ 47 ] Compared with GCN decoder [ 47 ] which directly uses GCN for signal reconstruction , the proposed inverse operator demonstrates its efficacy in recovering the high frequency signals of the graph , as shown in Figure 3 ( b ) . We shall further discuss this point in Section 3.3 . Inverse operator vs. GALA GALA [ 30 ] can be viewed as the first-order approximation towards inverse operator in this work , with a spectral kernel of 1 + λi . The difference is that our inverse operator can well approximate the inverse filter g−1c ( λi ) = 1 1−λi with third-order polynomials . 3.2.2 Wavelet de-noising . Wavelet-based methods have a strong impact on the field of de-noising , especially in image restoration when noise is amplified by inverse filters [ 28 ] . In the literature , many wavelet de-noising methods have been proposed , e.g. , SureShrink [ 9 ] , BayesShrink [ 5 ] , and they differ in how they estimate the separating threshold . Our method generalizes these threshold ideas and automatically separates the noise from the useful information with a learning-based approach . Consider the input signal of wavelet system is mixed with Gaussian noise , the reasons why wavelet denoising works are : ( 1 ) a graph wavelet basis holds the majority of signal energy into a small number of large coefficients , and ( 2 ) Gaussian noise in the original graph domain is again a Gaussian noise in wavelet domain ( with the same amplitude ) , see [ 16 ] for details . The core thus becomes how we can separate a small number of large coefficients and the noise in wavelet domain . Given the latter is unknown , we adopt a trainable parameter matrix to linearly transform the noise under zero and useful coefficients bigger than zero , where ReLU is used to eliminate the noise . Consider a set of wavelet bases Ψs = ( Ψs1 , . . . , ΨsN ) , where each one Ψsi denotes a signal on graph diffused away from node i and s is a scaling parameter [ 40 ] , the wavelet bases can be written as Ψs = Ugs ( Λ ) U > and gs ( · ) is a filter kernel . Following previous works GWNN [ 40 ] and GRAPHWAVE [ 8 ] , we use the heat kernel gs ( λi ) = e−sλi , as heat kernel admits easy calculation of inverse wavelet transform with g−1s ( λi ) = e sλi . In addition , we can apply Maclaurin series approximation on heat kernel and neglect explicit eigen-decomposition by : Ψs = ∑∞ n=0 ( −1 ) n n ! s nLnsym , ( 11 ) Ψ−1s = ∑∞ n=0 sn n ! L n sym , ( 12 ) To cope with the noise threshold estimation problem , we introduce a parameterized matrix in wavelet domain and eliminate these noise coefficients via a ReLU function , in the hope of learning a separating threshold from the data itself . Together with the representation M raised in Section 3.2.1 , we derive the reconstructed signal as : X ′ = ΨsReLU ( Ψ−1s MW4 ) W5 , ( 13 ) where W4 and W5 are trainable parameters . We then contrast the difference between Wavelet Neural Network ( WNN ) in this work and other related WNNs including GWNN [ 40 ] and GRAPHWAVE [ 8 ] . Scalability An important issue in WNN is that one needs to avoid explicit eigen-decomposition to compute wavelet bases for large graphs . Both GRAPHWAVE and GWNN , though trying to avoid eigen-decomposition by exploiting Chebyshev polynomials , still rely integral operations ( see Appendix D in [ 40 ] ) and there is no clear way to scale up . Differently , we use Maclaurin series to approximate wavelet bases , which has explicit polynomial coefficients and can resemble the heat kernel well when n = 3 . Please refer to Figure 2 for more details . De-noising The purpose of both GWNN and GRAPHWAVE is to derive node presentations with localized graph convolution and flexible neighborhood , such that downstream tasks like classification can be simplified . On the contrary , our work implements wavelet neural network in the purpose of detaching the useful information and the noise amplified by inverse operator . Due to the different purpose , our work applies the activation function in wavelet domain while GWNN in the original vertex domain .

Summary: This paper derives an inverse operator, which is said to be better than other methods in ref [50] and GALA. The work also points out the inverse operation results in a high pass filter and may amplify the noise. Motivated by this observation, a de-noising layer is designed and introduced into the proposed network. The graph deconvolution uses the wavelet transforms on the graph, which is implemented based on heat kernel and the series in terms of symmetric graph Laplacian. Theoretical analysis was provided for the derivation of the deconvolutional layer.