GRAND++: Graph Neural Diffusion with A Source Term

1 INTRODUCTION . Graph neural networks ( GNNs ) are the backbone for deep learning on graphs . Recent GNN architectures include graph convolutional networks ( GCNs ) [ 30 ] , ChebyNet [ 16 ] , GraphSAGE [ 29 ] , neural graph fingerprints [ 20 ] , message passing neural networks [ 28 ] , and graph attention networks ( GATs ) [ 54 ] . These graph deep networks have achieved success in many applications , including computational physics and computational chemistry [ 20 , 28 , 3 ] , recommender systems [ 41 , 62 ] , and social networks [ 63 , 47 ] . Hyperbolic GNNs have also been proposed to enable certain kinds of data embedding with much smaller distortion [ 11 , 37 ] . See [ 6 ] for some recent advances of GNN algorithm development and applications . A well-known problem of GNNs is that increasing the depth of GNNs often results in a significant drop in performance on various graph learning tasks . This performance degradation has been widely interpreted as the over-smoothing issue of GNNs [ 35 , 44 , 12 ] . Intuitively , GNN layers update the node representation by taking a weighted average of its neighbors ’ features , making representations for neighboring nodes to be similar . As the GNN architecture gets deeper , all nodes ’ representation will become indistinguishable resulting in over-smoothing . In Sec . 2 , we briefly show that certain GNNs have a diffusive nature which makes over-smoothing inevitable . Another interesting interpretation of the GNN performance degradation is via a bottleneck [ 1 ] , since a GNN tends to represent exponentially growing information from neighbors with fixed-size vectors . Several algorithms have been proposed to mitigate the over-smoothing of GNNs , including skip connection and dilated convolution [ 33 ] , Jumping Knowledge [ 60 ] , DropEdge [ 49 ] , PairNorm [ 64 ] , graph neural diffusion ( GRAND ) [ 10 ] , and wave equation motivated GNNs [ 21 ] . Nevertheless , developing deep GNN architectures is still in its infancy compared to the development of other deep networks . Besides suffering from over-smoothing , we notice that the accuracy of existing GNNs drops severely when they are trained with a limited labeled data . As illustrated in Fig . 1 , the test accuracy of several ∗Correspond to wangbaonj @ gmail.com or matthew.thorpe-2 @ manchester.ac.uk celebrated GNN architectures , including GCN , GAT , and GraphSage , drops rapidly when they are trained with fewer labeled data . Moreover , the variance of classification accuracy grows significantly as number of labeled nodes drops . Indeed , semi-supervised graph learning with very low-labeling rates has been studied in the Laplace learning and graph deep learning settings , see , e.g. , [ 36 , 9 , 23 ] ; one question is can we develop new GNN architectures to improve the performance of graph deep learning in low-labeling rate regimes ? 1.1 OUR CONTRIBUTION . With the above GNN problems in mind , we focus on developing new continuous-depth GNNs to overcome over-smoothing and boost the accuracy of GNNs with a limited number of labeled data . We first present a random walk interpretation of the GRAND model [ 10 ] , revealing a potentially inevitable over-smoothing phenomenon when GRAND is implicitly very deep . Based on the random walk viewpoint of GRAND , we then propose graph neural diffusion with a source term ( GRAND++ ) that corrects the bias arising from the diffusion process , see Sec . 5 for details . GRAND++ theoretically guarantees that : ( i ) under GRAND++ dynamics , the graph node features do not converge to a constant vector over all nodes even as the time goes to infinity , and ( ii ) GRAND++ can provide accurate prediction even when it is trained with a limited number of labeled nodes . Moreover , these theoretical results resonate with the practical advantages of GRAND++ . We summarize the major practical advantages of GRAND++ below . • GRAND++ can effectively overcome the over-smoothing issue ; it is remarkably more accurate than existing GNNs when the architecture is very deep . • GRAND++ is suitable for graph deep learning when only a few nodes are labeled as training data . Moreover , in the low-labeling rates , GRAND++ can be more accurate when the network is deeper . • GRAND++ inherits the continuous-depth merit from GRAND , which defines the network depth implicitly and enables memory-efficient training by using the adjoint method . 1.2 RELATED WORK . Diffusion on graphs and continuous-depth graph neural networks . Diffusion has been defined on graphs , see , e.g. , [ 25 , 24 ] , and used in various applications , including data clustering and dimension reduction [ 15 , 4 ] , image processing [ 27 , 22 , 17 , 38 ] , and semi-supervised graph nodes classification [ 67 , 65 ] . From the numerical viewpoint , fast algorithms have been proposed for using diffusion on graphs to solve penalized graph cut problems [ 26 ] . The connection between GNNs and diffusion on graphs has been studied substantially . For instance , GNN has been interpreted as a diffusion process on graphs , which performs low-pass filtering on the input features [ 43 ] . Moreover , insights from the diffusion process on graphs have been used to improve the performance of GNNs , see , e.g. , [ 2 , 36 , 31 , 57 ] . Leveraging neural ordinary differential equations ( ODEs ) [ 13 ] , continuous-depth GNNs have been proposed , see , e.g. , [ 45 , 58 , 68 ] . One recent work is GRAND [ 10 ] , which parameterizes the diffusion equation on graphs with a neural network . See Sec . 3 for a brief review of GRAND . Neural ODEs . Neural ODEs [ 13 ] are a class of continuous-depth neural networks whose depth is defined implicitly . Training neural ODEs using the adjoint method [ 46 ] is more memory efficient than training other neural networks using backpropagation . We provide a brief review of neural ODEs and the adjoint method in Appendix C. GRANDs [ 10 ] are a class of neural partial differential equations ( PDEs ) on graphs that can also be considered as a coupled system of neural ODEs . Furthermore , GRANDs are also trained by using the adjoint method . Laplace learning and Poisson learning . Laplace learning has been used for semi-supervised data classification [ 67 , 65 , 56 ] , image processing [ 7 , 27 ] , etc . Direct application of Laplace learning with Gaussian weights [ 5 ] or locally linear embedding weights [ 50 ] for the above tasks may cause inference inconsistency when only a limited number of graph nodes are labeled , resulting in poor performance . Several algorithms address the inference inconsistency at low labeling rate . They include up-weighting the weights of the labeled data [ 52 ] and the p-Laplacian [ 8 , 48 , 66 ] . In [ 9 ] , the authors have proposed Poisson learning for improving Laplace learning at extremely low-labeling rate regimes . Poisson learning augments Laplace learning with a Green ’ s function at each labeled data , enabling accurate node classification when only a few labeled data are available . Compared to Laplace learning , Poisson learning adds Green ’ s function to the label of each labeled node and then performs label propagation to predict the label for unlabeled graph nodes . GRAND and GRAND++ both learn graph node representations and perform prediction by activating the node representations , which are fundamentally different from Laplace and Poisson learning . 1.3 NOTATION . We denote scalars by lower- or upper-case letters and vectors and matrices by lower- and upper-case boldface letters , respectively . For a matrix A , we denote its transpose as A > and its Hadamard product with another matrix B as A B , i.e. , the entrywise multiplication of A and B . We write the set { 1 , 2 , · · · , n } as [ n ] . We denote the probability and expectation of a given random variable x as P ( x ) and E [ x ] , respectively . The meaning of other notations can be inferred from the context . 1.4 ORGANIZATION . The paper is organized as follows : In Sec . 2 , we review diffusion equation on graphs and its connection to GNNs . In Secs . 3 and 4 , we briefly review GRAND and present a random walk interpretation of GRAND , respectively . Leveraging the random walk viewpoint of GRAND , we propose GRAND++ for deep graph learning with theoretical guarantees in Sec . 5 . We verify the efficacy of GRAND++ in Sec . 6 . Technical proofs and more validations are provided in the appendix . 2 BACKGROUND . Diffusion equation on graphs . Let G = ( X , W ) represent an undirected graph with n nodes , where X = ( [ x ( 1 ) ] > , · · · , [ x ( n ) ] > ) > ∈ Rn×d with each row x ( i ) ∈ Rd a feature vector and W : = ( Wij ) a n × n matrix with Wij representing the similarity ( edge weight ) between the ith and jth feature vectors , and we assume Wij = Wji . Consider the following diffusion process that evolves the feature matrix X on the graph ( see Appendix A for a brief review of calculus on graphs ) : ∂X ( t ) ∂t = div ( G ( X ( t ) , t ) ∇X ( t ) ) , ( 1 ) where X ( t ) = ( [ x ( 1 ) ( t ) ] > , · · · , [ x ( n ) ( t ) ] > ) > ∈ Rn×d with x ( i ) ( 0 ) = x ( i ) , ∇ and div are the gradient and divergence operators , respectively . The matrix G ( X ( t ) , t ) is chosen such that W G is right-stochastic , i.e. , each row of W G summing to 1 . In the machine learning setting , we can parameterize G with learnable parameters θ which we denote by G ( X ( t ) , t , θ ) . The initial features are evolved under the diffusion dynamics ( 1 ) from t = 0 to T to learn the final representation X ( T ) for further machine learning tasks . In the simplest case when G ( X ( t ) , t ) is only dependent on the initial node features X , i.e. , G is time-independent , right-stochasticity implies ∑ jWijGij = 1 for all i , and so we focus on the particular case when Gij = 1/di with di = ∑n j=1Wij . In this case the right-hand side of ( 1 ) reduces to the negative of the random-walk Laplacian applied to X ( t ) and ( 1 ) becomes ∂X ( t ) ∂t = div ( G ( X ( t ) , t ) ∇X ( t ) ) = −LX ( t ) , ( 2 ) where L = I −D−1W : = I −A ( A : = A ( X ) ) is the random walk Laplacian and D is diagonal with Dii = di . See [ 14 , 25 , 24 ] for more about random walk Laplacian and diffusion on graphs . Graph neural networks . Applying forward Euler discretization , with step size δt < 1 , of ( 2 ) gives X ( kδt ) = X ( ( k − 1 ) δt ) − δtLX ( ( k − 1 ) δt ) : = L̃X ( ( k − 1 ) δt ) , for k = 1 , 2 , · · · , K , ( 3 ) T = Kδt , and X ( 0 ) = X . Note that the matrix L̃ is the discretization of the diffusion operator , which is a special low-pass filter . Equation ( 3 ) is a prototype for motivating GNNs : by introducing weights W ( k ) ∈ Rd×d and a nonlinearity σ , e.g. , ReLU , into ( 3 ) , we have X ( ( k + 1 ) δt ) = σ ( L̃X ( kδt ) W ( k ) ) . ( 4 ) The model in ( 4 ) is similar to the well-established GCN architecture proposed in [ 30 ] . The diffusive nature of the GNN architecture in ( 4 ) further explains the over-smoothing issue of training deep GNNs ; the deeper the network architecture is , the more the node features diffuse . Eventually , all nodes share similar features and become indistinguishable . See Sec . 5 for a detailed analysis .

Summary: The paper introduces GRAND++, GRAph Neural Diffusion with a source term. As its predecessor GRAND, this method is focused on the development of a new continuous-depth GNN to tackle the over-smoothing and the bottleneck issues which are typical of deep GNNs. The authors introduce a random-walk interpretation of GRAND, which shows that when the network is very deep it is still prone to over-smoothing. To address this problem they add a source term to the problem formulation, which is shown to provide better results in very deep models and in classification tasks which are characterised by a low amount of labeled data.