Dissecting the Diffusion Process in Linear Graph Convolutional Networks

1 Introduction . Recently , Graph Convolutional Networks ( GCNs ) have successfully extended the powerful representation learning ability of modern Convolutional Neural Networks ( CNNs ) to the graph data [ 7 ] . A graph convolutional layer typically consists of two stages : linear feature propagation and nonlinear feature transformation . Simple Graph Convolution ( SGC ) [ 21 ] simplifies GCNs by removing the nonlinearities between GCN layers and collapsing the resulting function into a single linear transformation , which is followed by a single linear classification layer and then becomes a linear GCN . SGC can achieve comparable performance to canonical GCNs while being much more computationally efficient and using significantly fewer parameters . Thus , we mainly focus on linear GCNs in this paper . Although being comparable to canonical GCNs , SGC still suffers from a similar issue as non-linear GCNs , that is , more ( linear ) feature propagation steps K will degrade the performance catastrophically . This issue is widely characterized as the “ over-smoothing ” phenomenon . Namely , node features become smoothed out and indistinguishable after too many feature propagation steps [ 10 ] . In this work , through a dissection of the diffusion process of linear GCNs , we characterize a fundamental limitation of SGC . Specifically , we point out that its feature propagation step amounts to a very coarse finite difference with a fixed step size ∆t = 1 , which results in a large numerical error . ∗Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . And because the step size is fixed , more feature propagation steps will inevitably lead to a large terminal time T = K ·∆t→∞ that over-smooths the node features . To address these issues , we propose Decoupled Graph Convolution ( DGC ) by decoupling the terminal time T and propagation stepsK . In particular , we can flexibly choose a continuous terminal time T for the optimal tradeoff between under-smoothing and over-smoothing , and then fix the terminal time while adopting more propagation steps K. In this way , different from SGC that over-smooths with more propagation steps , our proposed DGC can obtain a more fine-grained finite difference approximation with more propagation steps , which contributes to the final performance both theoretically and empirically . Extensive experiments show that DGC ( as a linear GCN ) improves over SGC significantly and obtains state-of-the-art results that are comparable to many modern non-linear GCNs . Our main contributions are summarized as follows : • We investigate SGC by dissecting its diffusion process from a continuous perspective , and characterize why it can not benefit from more propagation steps . • We propose Decoupled Graph Convolution ( DGC ) that decouples the terminal time T and the propagation steps K , which enables us to choose a continuous terminal time flexibly while benefiting from more propagation steps from both theoretical and empirical aspects . • Experiments show that DGC outperforms canonical GCNs significantly and obtains stateof-the-art ( SOTA ) results among linear GCNs , which is even comparable to many competitive non-linear GCNs . We think DGC can serve as a strong baseline for the future research . 2 Related Work . Graph convolutional networks ( GCNs ) . To deal with non-Euclidean graph data , GCNs are proposed for direct convolution operation over graph , and have drawn interests from various domains . GCN is firstly introduced for a spectral perspective [ 26 , 7 ] , but soon it becomes popular as a general message passing algorithm in the spatial domain . Many variants have been proposed to improve its performance , such as GraphSAGE [ 5 ] with LSTM and GAT with attention mechanism [ 19 ] . Over-smoothing issue . GCNs face a fundamental problem compared to standard CNNs , i.e. , the over-smoothing problem . Li et al . [ 10 ] offer a theoretical characterization of over-smoothing based on linear feature propagation . After that , many researchers have tried to incorporate effective mechanisms in CNNs to alleviate over-smoothing . DeepGCNs [ 9 ] shows that residual connection and dilated convolution can make GCNs go as deep as CNNs , although increased depth does not contribute much . Methods like APPNP [ 8 ] and JKNet [ 25 ] avoid over-smoothing by aggregating multiscale information from the first hidden layer . DropEdge [ 16 ] applies dropout to graph edges and find it enables training GCNs with more layers . PairNorm [ 27 ] regularizes the feature distance to be close to the input distance , which will not fail catastrophically but still decrease with more layers . Continuous GCNs . Deep CNNs have been widely interpreted from a continuous perspective , e.g. , ResNet [ 6 ] as the Euler discretization of Neural ODEs [ 11 , 3 ] . This viewpoint has recently been borrowed to understand and improve GCNs . GCDE [ 15 ] directly extends GCNs to a Neural ODE , while CGNN [ 22 ] devises a GCN variant inspired by a new continuous diffusion . Our method is also inspired by the connection between discrete and continuous graph diffusion , but alternatively , we focus on their numerical gap and characterize how it affects the final performance . Linear GCNs . SGC [ 21 ] simplifies and separates the two stages of GCNs : feature propagation and ( non-linear ) feature transformation . It finds that utilizing only a simple logistic regression after feature propagation ( removing the non-linearities ) , which makes it a linear GCN , can obtain comparable performance to canonical GCNs . In this paper , we further show that a properly designed linear GCN ( DGC ) can be on-par with state-of-the-art non-linear GCNs while possessing many desirable properties . For example , as a linear model , DGC requires much fewer parameters than non-linear GCNs , which makes it very memory efficient , and meanwhile , its training is also much faster ( ∼ 100× ) than non-linear models as it could preprocess all features before training . 3 Dissecting Linear GCNs from Continuous Dynamics . In this section , we make a brief review of SGC [ 21 ] in the context of semi-supervised node classification task , and further point out its fundamental limitations . 3.1 Review of Simple Graph Convolution ( SGC ) . Define a graph as G = ( V , A ) , where V = { v1 , . . . , vn } denotes the vertex set of n nodes , and A ∈ Rn×n is an adjacency matrix where aij denotes the edge weight between node vi and vj . The degree matrix D = diag ( d1 , . . . , dn ) of A is a diagonal matrix with its i-th diagonal entry as di = ∑ j aij . Each node vi is represented by a d-dimensional feature vector xi ∈ Rd , and we denote the feature matrix as X ∈ Rn×d = [ x1 , . . . , xn ] . Each node belongs to one out of C classes , denoted by a one-hot vector yi ∈ { 0 , 1 } C . In node classification problems , only a subset of nodes Vl ⊂ V is labeled and we want to predict the labels of the rest nodes Vu = V\Vl . SGC shows that we can obtain similar performance with a simplified GCN , ŶSGC = softmax ( SKXΘ ) , ( 1 ) which pre-processes the node features X with K linear propagation steps , and then applies a linear classifier with parameter Θ . Specifically , at the step k , each feature xi is computed by aggregating features in its local neighborhood , which can be done in parallel over the whole graph for K steps , X ( k ) ← SX ( k−1 ) , where S = D̃− 12 ÃD̃− 12 =⇒ X ( K ) = SKX . ( 2 ) Here Ã = A + I is the adjacency matrix augmented with the self-loop I , D̃ is the degree matrix of Ã , and S denotes the symmetrically normalized adjacency matrix . This step exploits the local graph structure to smooth out the noise in each node . At last , SGC applies a multinomial logistic regression ( a.k.a . softmax regression ) with parameter Θ to predict the node labels ŶSGC from the node features of the last propagation step X ( K ) : ŶSGC = softmax ( X ( K ) Θ ) . ( 3 ) Because both the feature propagation ( SKX ) and classification ( X ( K ) Θ ) steps are linear , SGC is essentially a linear version of GCN that only relies on linear features from the input . 3.2 Equivalence between SGC and Graph Heat Equation . Previous analysis of linear GCNs focuses on their asymptotic behavior as propagation stepsK →∞ ( discrete ) , known as the over-smoothing phenomenon [ 10 ] . In this work , we instead provide a novel non-asymptotic characterization of linear GCNs from the corresponding continuous dynamics , graph heat equation [ 4 ] . A key insight is that we notice that the propagation of SGC can be seen equivalently as a ( coarse ) numerical discretization of the graph diffusion equation , as we show below . Graph Heat Equation ( GHE ) is a well-known generalization of the heat equation on graph data , which is widely used to model graph dynamics with applications in spectral graph theory [ 4 ] , time series [ 12 ] , combinational problems [ 13 ] , etc . In general , GHE can be formulated as follows : { dXt dt = −LXt , X0 = X , ( 4 ) where Xt ( t ≥ 0 ) refers to the evolved input features at time t , and L refers to the graph Laplacian matrix . Here , for the brevity of analysis , we take the symmetrically normalized graph Laplacian for the augmented adjacency Ã and overload the notation as L = D̃− 1 2 ( D̃− Ã ) D̃− 1 2 = I− S. As GHE is a continuous dynamics , in practice we need to rely on numerical methods to solve it . We find that SGC can be seen as a coarse finite difference of GHE . Specifically , we apply the forward Euler method to Eq . ( 4 ) with an interval ∆t : X̂t+∆t =X̂t −∆tLX̂t = X̂t −∆t ( I− S ) X̂t = [ ( 1−∆t ) I + ∆tS ] X̂t . ( 5 ) By involving the update rule for K forward steps , we will get the final features X̂T at the terminal time T = K ·∆t : X̂T = [ S ( ∆t ) ] KX , where S ( ∆t ) = ( 1−∆t ) I + ∆tS . ( 6 ) Comparing to Eq . ( 2 ) , we can see that the Euler discretization of GHE becomes SGC when the step size ∆t = 1 . Specifically , the diffusion matrix S ( ∆t ) reduces to the SGC diffusion matrix S and the final node features , X̂T and X ( K ) , become equivalent . Therefore , SGC with K propagation steps is essentially a finite difference approximation to GHE with K forward steps ( step size ∆t = 1 and terminal time T = K ) . 3.3 Revealing the Fundamental Limitations of SGC . Based on the above analysis , we theoretically characterize several fundamental limitations of SGC : feature over-smoothing , large numerical errors and large learning risks . Proofs are in Appendix B. Theorem 1 ( Oversmoothing from a spectral view ) . Assume that the eigendecomposition of the Laplacian matrix as L = ∑n i=1 λiuiu > i , with eigenvalues λi and eigenvectors ui . Then , the heat equation ( Eq . ( 4 ) ) admits a closed-form solution at time t , known as the heat kernel Ht = e−tL =∑n i=1 e −λituiu > i . As t → ∞ , Ht asymptotically converges to a non-informative equilibrium as t→∞ , due to the non-trivial ( i.e. , positive ) eigenvalues vanishing : lim t→∞ e−λit = { 0 , if λi > 0 1 , if λi = 0 , i = 1 , . . . , n. ( 7 ) Remark 1 . In SGC , T = K ·∆t = K. Thus , according to Theorem 1 , a large number of propagation steps K →∞ will inevitably lead to over-smoothed non-informative features . Theorem 2 ( Numerical errors ) . For the initial value problem in Eq . ( 4 ) with finite terminal time T , the numerical error of the forward Euler method in Eq . ( 5 ) with K steps can be upper bounded by∥∥∥e ( K ) T ∥∥∥ ≤ T‖L‖‖X0‖2K ( eT‖L‖ − 1 ) . ( 8 ) Remark 2 . Since T = K in SGC , the upper bound reduces to c · ( eT‖L‖ − 1 ) ( c is a constant ) . We can see that the numerical error will increase exponentially with more propagation steps . Theorem 3 ( Learning risks ) . Consider a simple linear regression problem ( X , Y ) on graph , where the observed input features X are generated by corrupting the ground truth features Xc with the following inverse graph diffusion with time T ∗ : dX̃t dt = LX̃t , where X̃0 = Xc and X̃T∗ = X . ( 9 ) Denote the population risk with ground truth features as R ( W ) = E ‖Y −XcW‖2 and that of Euler method applied input X ( Eq . ( 5 ) ) as R̂ ( W ) = E ∥∥∥Y − [ S ( ∆t ) ] K XW∥∥∥2 . Supposing that E‖Xc‖2 = M < ∞ , we have the following upper bound : R̂ ( W ) < R ( W ) + 2‖W‖2 ( E ∥∥∥e ( K ) T̂ ∥∥∥2 +M ∥∥∥eT ? L∥∥∥2 · ∥∥∥e−T ? L − e−T̂L∥∥∥2 ) . ( 10 ) Remark 3 . Following Theorem 3 , we can see that the upper bound can be minimized by finding an optimal terminal time such that T̂ = T ? and minimizing the numerical error ∥∥∥e ( K ) T̂ ∥∥∥ . While SGC fixes the step size ∆t = 1 , thus T and K are coupled together , which makes it less flexible to minimize the risk in Eq . ( 10 ) .

Summary: This work studies linear GCNs and tries to provide a solution to find the optimal tradeoff between under-smoothing and over-smoothing. It first pointed out the limitations of SGC, then propose the Decoupled Graph Convolution (DGC) model to address the limitations. The key idea is to view the feature propagation step in linear GCN from a perspective of continuous graph diffusion, and dissect the terminal time and the propagation steps to enable choosing a flexible continuous terminal time (and therefore can benefit from more propagation steps).