Dissecting the Diffusion Process in Linear Graph Convolutional Networks

1 Introduction . Recently , Graph Convolutional Networks ( GCNs ) have successfully extended the powerful representation learning ability of modern Convolutional Neural Networks ( CNNs ) to the graph data [ 7 ] . A graph convolutional layer typically consists of two stages : linear feature propagation and nonlinear feature transformation . Simple Graph Convolution ( SGC ) [ 21 ] simplifies GCNs by removing the nonlinearities between GCN layers and collapsing the resulting function into a single linear transformation , which is followed by a single linear classification layer and then becomes a linear GCN . SGC can achieve comparable performance to canonical GCNs while being much more computationally efficient and using significantly fewer parameters . Thus , we mainly focus on linear GCNs in this paper . Although being comparable to canonical GCNs , SGC still suffers from a similar issue as non-linear GCNs , that is , more ( linear ) feature propagation steps K will degrade the performance catastrophically . This issue is widely characterized as the “ over-smoothing ” phenomenon . Namely , node features become smoothed out and indistinguishable after too many feature propagation steps [ 10 ] . In this work , through a dissection of the diffusion process of linear GCNs , we characterize a fundamental limitation of SGC . Specifically , we point out that its feature propagation step amounts to a very coarse finite difference with a fixed step size ∆t = 1 , which results in a large numerical error . ∗Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . And because the step size is fixed , more feature propagation steps will inevitably lead to a large terminal time T = K ·∆t→∞ that over-smooths the node features . To address these issues , we propose Decoupled Graph Convolution ( DGC ) by decoupling the terminal time T and propagation stepsK . In particular , we can flexibly choose a continuous terminal time T for the optimal tradeoff between under-smoothing and over-smoothing , and then fix the terminal time while adopting more propagation steps K. In this way , different from SGC that over-smooths with more propagation steps , our proposed DGC can obtain a more fine-grained finite difference approximation with more propagation steps , which contributes to the final performance both theoretically and empirically . Extensive experiments show that DGC ( as a linear GCN ) improves over SGC significantly and obtains state-of-the-art results that are comparable to many modern non-linear GCNs . Our main contributions are summarized as follows : • We investigate SGC by dissecting its diffusion process from a continuous perspective , and characterize why it can not benefit from more propagation steps . • We propose Decoupled Graph Convolution ( DGC ) that decouples the terminal time T and the propagation steps K , which enables us to choose a continuous terminal time flexibly while benefiting from more propagation steps from both theoretical and empirical aspects . • Experiments show that DGC outperforms canonical GCNs significantly and obtains stateof-the-art ( SOTA ) results among linear GCNs , which is even comparable to many competitive non-linear GCNs . We think DGC can serve as a strong baseline for the future research . 2 Related Work . Graph convolutional networks ( GCNs ) . To deal with non-Euclidean graph data , GCNs are proposed for direct convolution operation over graph , and have drawn interests from various domains . GCN is firstly introduced for a spectral perspective [ 26 , 7 ] , but soon it becomes popular as a general message passing algorithm in the spatial domain . Many variants have been proposed to improve its performance , such as GraphSAGE [ 5 ] with LSTM and GAT with attention mechanism [ 19 ] . Over-smoothing issue . GCNs face a fundamental problem compared to standard CNNs , i.e. , the over-smoothing problem . Li et al . [ 10 ] offer a theoretical characterization of over-smoothing based on linear feature propagation . After that , many researchers have tried to incorporate effective mechanisms in CNNs to alleviate over-smoothing . DeepGCNs [ 9 ] shows that residual connection and dilated convolution can make GCNs go as deep as CNNs , although increased depth does not contribute much . Methods like APPNP [ 8 ] and JKNet [ 25 ] avoid over-smoothing by aggregating multiscale information from the first hidden layer . DropEdge [ 16 ] applies dropout to graph edges and find it enables training GCNs with more layers . PairNorm [ 27 ] regularizes the feature distance to be close to the input distance , which will not fail catastrophically but still decrease with more layers . Continuous GCNs . Deep CNNs have been widely interpreted from a continuous perspective , e.g. , ResNet [ 6 ] as the Euler discretization of Neural ODEs [ 11 , 3 ] . This viewpoint has recently been borrowed to understand and improve GCNs . GCDE [ 15 ] directly extends GCNs to a Neural ODE , while CGNN [ 22 ] devises a GCN variant inspired by a new continuous diffusion . Our method is also inspired by the connection between discrete and continuous graph diffusion , but alternatively , we focus on their numerical gap and characterize how it affects the final performance . Linear GCNs . SGC [ 21 ] simplifies and separates the two stages of GCNs : feature propagation and ( non-linear ) feature transformation . It finds that utilizing only a simple logistic regression after feature propagation ( removing the non-linearities ) , which makes it a linear GCN , can obtain comparable performance to canonical GCNs . In this paper , we further show that a properly designed linear GCN ( DGC ) can be on-par with state-of-the-art non-linear GCNs while possessing many desirable properties . For example , as a linear model , DGC requires much fewer parameters than non-linear GCNs , which makes it very memory efficient , and meanwhile , its training is also much faster ( ∼ 100× ) than non-linear models as it could preprocess all features before training . 3 Dissecting Linear GCNs from Continuous Dynamics . In this section , we make a brief review of SGC [ 21 ] in the context of semi-supervised node classification task , and further point out its fundamental limitations . 3.1 Review of Simple Graph Convolution ( SGC ) . Define a graph as G = ( V , A ) , where V = { v1 , . . . , vn } denotes the vertex set of n nodes , and A ∈ Rn×n is an adjacency matrix where aij denotes the edge weight between node vi and vj . The degree matrix D = diag ( d1 , . . . , dn ) of A is a diagonal matrix with its i-th diagonal entry as di = ∑ j aij . Each node vi is represented by a d-dimensional feature vector xi ∈ Rd , and we denote the feature matrix as X ∈ Rn×d = [ x1 , . . . , xn ] . Each node belongs to one out of C classes , denoted by a one-hot vector yi ∈ { 0 , 1 } C . In node classification problems , only a subset of nodes Vl ⊂ V is labeled and we want to predict the labels of the rest nodes Vu = V\Vl . SGC shows that we can obtain similar performance with a simplified GCN , ŶSGC = softmax ( SKXΘ ) , ( 1 ) which pre-processes the node features X with K linear propagation steps , and then applies a linear classifier with parameter Θ . Specifically , at the step k , each feature xi is computed by aggregating features in its local neighborhood , which can be done in parallel over the whole graph for K steps , X ( k ) ← SX ( k−1 ) , where S = D̃− 12 ÃD̃− 12 =⇒ X ( K ) = SKX . ( 2 ) Here Ã = A + I is the adjacency matrix augmented with the self-loop I , D̃ is the degree matrix of Ã , and S denotes the symmetrically normalized adjacency matrix . This step exploits the local graph structure to smooth out the noise in each node . At last , SGC applies a multinomial logistic regression ( a.k.a . softmax regression ) with parameter Θ to predict the node labels ŶSGC from the node features of the last propagation step X ( K ) : ŶSGC = softmax ( X ( K ) Θ ) . ( 3 ) Because both the feature propagation ( SKX ) and classification ( X ( K ) Θ ) steps are linear , SGC is essentially a linear version of GCN that only relies on linear features from the input . 3.2 Equivalence between SGC and Graph Heat Equation . Previous analysis of linear GCNs focuses on their asymptotic behavior as propagation stepsK →∞ ( discrete ) , known as the over-smoothing phenomenon [ 10 ] . In this work , we instead provide a novel non-asymptotic characterization of linear GCNs from the corresponding continuous dynamics , graph heat equation [ 4 ] . A key insight is that we notice that the propagation of SGC can be seen equivalently as a ( coarse ) numerical discretization of the graph diffusion equation , as we show below . Graph Heat Equation ( GHE ) is a well-known generalization of the heat equation on graph data , which is widely used to model graph dynamics with applications in spectral graph theory [ 4 ] , time series [ 12 ] , combinational problems [ 13 ] , etc . In general , GHE can be formulated as follows : { dXt dt = −LXt , X0 = X , ( 4 ) where Xt ( t ≥ 0 ) refers to the evolved input features at time t , and L refers to the graph Laplacian matrix . Here , for the brevity of analysis , we take the symmetrically normalized graph Laplacian for the augmented adjacency Ã and overload the notation as L = D̃− 1 2 ( D̃− Ã ) D̃− 1 2 = I− S. As GHE is a continuous dynamics , in practice we need to rely on numerical methods to solve it . We find that SGC can be seen as a coarse finite difference of GHE . Specifically , we apply the forward Euler method to Eq . ( 4 ) with an interval ∆t : X̂t+∆t =X̂t −∆tLX̂t = X̂t −∆t ( I− S ) X̂t = [ ( 1−∆t ) I + ∆tS ] X̂t . ( 5 ) By involving the update rule for K forward steps , we will get the final features X̂T at the terminal time T = K ·∆t : X̂T = [ S ( ∆t ) ] KX , where S ( ∆t ) = ( 1−∆t ) I + ∆tS . ( 6 ) Comparing to Eq . ( 2 ) , we can see that the Euler discretization of GHE becomes SGC when the step size ∆t = 1 . Specifically , the diffusion matrix S ( ∆t ) reduces to the SGC diffusion matrix S and the final node features , X̂T and X ( K ) , become equivalent . Therefore , SGC with K propagation steps is essentially a finite difference approximation to GHE with K forward steps ( step size ∆t = 1 and terminal time T = K ) . 3.3 Revealing the Fundamental Limitations of SGC . Based on the above analysis , we theoretically characterize several fundamental limitations of SGC : feature over-smoothing , large numerical errors and large learning risks . Proofs are in Appendix B. Theorem 1 ( Oversmoothing from a spectral view ) . Assume that the eigendecomposition of the Laplacian matrix as L = ∑n i=1 λiuiu > i , with eigenvalues λi and eigenvectors ui . Then , the heat equation ( Eq . ( 4 ) ) admits a closed-form solution at time t , known as the heat kernel Ht = e−tL =∑n i=1 e −λituiu > i . As t → ∞ , Ht asymptotically converges to a non-informative equilibrium as t→∞ , due to the non-trivial ( i.e. , positive ) eigenvalues vanishing : lim t→∞ e−λit = { 0 , if λi > 0 1 , if λi = 0 , i = 1 , . . . , n. ( 7 ) Remark 1 . In SGC , T = K ·∆t = K. Thus , according to Theorem 1 , a large number of propagation steps K →∞ will inevitably lead to over-smoothed non-informative features . Theorem 2 ( Numerical errors ) . For the initial value problem in Eq . ( 4 ) with finite terminal time T , the numerical error of the forward Euler method in Eq . ( 5 ) with K steps can be upper bounded by∥∥∥e ( K ) T ∥∥∥ ≤ T‖L‖‖X0‖2K ( eT‖L‖ − 1 ) . ( 8 ) Remark 2 . Since T = K in SGC , the upper bound reduces to c · ( eT‖L‖ − 1 ) ( c is a constant ) . We can see that the numerical error will increase exponentially with more propagation steps . Theorem 3 ( Learning risks ) . Consider a simple linear regression problem ( X , Y ) on graph , where the observed input features X are generated by corrupting the ground truth features Xc with the following inverse graph diffusion with time T ∗ : dX̃t dt = LX̃t , where X̃0 = Xc and X̃T∗ = X . ( 9 ) Denote the population risk with ground truth features as R ( W ) = E ‖Y −XcW‖2 and that of Euler method applied input X ( Eq . ( 5 ) ) as R̂ ( W ) = E ∥∥∥Y − [ S ( ∆t ) ] K XW∥∥∥2 . Supposing that E‖Xc‖2 = M < ∞ , we have the following upper bound : R̂ ( W ) < R ( W ) + 2‖W‖2 ( E ∥∥∥e ( K ) T̂ ∥∥∥2 +M ∥∥∥eT ? L∥∥∥2 · ∥∥∥e−T ? L − e−T̂L∥∥∥2 ) . ( 10 ) Remark 3 . Following Theorem 3 , we can see that the upper bound can be minimized by finding an optimal terminal time such that T̂ = T ? and minimizing the numerical error ∥∥∥e ( K ) T̂ ∥∥∥ . While SGC fixes the step size ∆t = 1 , thus T and K are coupled together , which makes it less flexible to minimize the risk in Eq . ( 10 ) .

Summary: Non-linear GCNs have attracted a lot of attention recently, and many methods are proposed around them. However, this paper finds and investigates that current GCN variants still have not shown significant advantages over a properly designed linear GCN (their proposal). This is significant to the whole field and can help shape a new direction to propose real powerful non-linear GCNs.