PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations

1 Introduction . In recent years , Graph Convolutional Networks ( GCNs ) [ 1 , 2 , 3 ] have drawn the attention of researchers and practitioners in a variety of domains and applications , ranging from computer vision and graphics [ 4 , 5 , 6 , 7 , 8 ] to computational biology [ 9 , 10 , 11 ] , recommendation systems [ 12 ] and social network analysis [ 13 , 14 ] . However , GCNs still suffer from two main problems . First , they are usually shallow as opposed to the concept of deep convolutional neural networks ( CNNs ) [ 15 , 16 ] due to the over-smoothing phenomenon [ 17 , 18 , 19 ] , where the node feature vectors become almost identical , such that they are indistinguishable , which yields non-optimal performance . Furthermore , GCNs are typically customized to a specific domain and application . That is , as we demonstrate in Sec . 4.1 , a successful point-cloud classification network [ 5 ] can perform poorly on a citation graph node-classification problem [ 19 ] , and vice-versa . Furthermore , because many GCNs lack theoretical guarantees , it is difficult to reason about their success on one problem and lack on others . These observations motivate us to develop a profound understanding of graph networks and their dynamics . To this end , we suggest a novel , universal approach to the design of GCN architectures . Our inspiration stems from the similarities and equivalence between Partial Differential Equations ( PDEs ) and deep networks explored in [ 20 , 21 , 22 ] . Furthermore , as GCNs can be thought of as a generalization of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . CNNs , and a standard convolution can be represented as a combination of differential operators on a structured grid [ 22 ] , we adopt this interpretation to explore versions of GCNs as PDEs on graphs or manifolds . We therefore call our network architectures PDE-GCN , and demonstrate that our approach is general with respect to the given task . That is , our architectures behave similarly for different problems , and their performance is on par or better than other domain-specific GCNs . Furthermore , our family of architectures are backed by theoretical guarantees that allow us to explain the behavior of the GCNs in some of the results that we present . To be more specific , our contribution is as follows : • We introduce and implement general graph convolution operators , based on graph gradient and divergence . This abstraction of the spatial operation on the graph leads to a more general and flexible approach for architecture design . • We propose treating a variety of graph related problems as discretized PDEs , and formulate the dynamics that match different problems such as node-classification and dense shape-correspondence . This is in direct effort to propose a family of networks that can solve multiple problems , instead of GCNs which are tailored for a specific application . • Our method allows constructing a deep GCN without over-smoothing , with theoretical guarantees . • We validate our method by conducting numerous experiments on multiple datasets and applications , achieving significantly better or similar accuracy compared to state-of-the-art models . 2 Related work . Graph Convolutional Networks : GCNs are typically divided into spectral [ 1 , 3 , 2 ] and spatial [ 23 , 24 , 25 , 4 ] categories . Most of those can be implemented using the Message-Passing Neural Network paradigm [ 25 ] , where each node aggregates features ( messages ) from its neighbors , according to some scheme . The works [ 3 , 2 ] use polynomials of the graph Laplacian to parameterize the convolution operator . DGCNN [ 5 ] constructs a k-nearest-neighbors graph from point-clouds and dynamically updates it . MoNet [ 4 ] learns a Gaussian mixture model to weight the edges of meshes for shape analysis tasks . Works like GraphSAGE [ 26 ] and GAT [ 27 ] propose methods for inductive and transductive learning on non-geometrical graphs . Several of the methods above suffer from over-smoothing [ 17 , 19 ] , leading to undesired node features similarity for deep networks . To overcome this problem , some approaches rely on imposing regularization and augmentation . For example , PairNorm [ 17 ] introduces a novel normalization layer , and DropEdge [ 28 ] randomly removes edges to decrease the degree of nodes . Other methods prevent over-smoothing by dedicated construction . For instance , JKNet [ 29 ] combines all intermediate representations at each stage of the network . APPNP [ 30 ] replaces learnt convolutional layers with a pre-defined kernel based on PageRank , yielding a shallow network which preserves locality , making it robust to over-smoothing . GCNII [ 19 ] proposes to add an identity residual where the initial features of the networks are added to the features of the l-th layer , scaled by some coefficient . Another approach is to construct a network that inherently does not over-smooth , as we suggest in this work . Our network is based on discretized PDEs , hence we are able to motivate our choices by well studied theory and numerical experiments [ 31 ] . On a similar note , the recent DiffGCN [ 8 ] also makes use of discretized operators to approximate the graph gradient and Laplacian . However , DiffGCN is specifically tailored for geometric tasks since it projects its components on the x , y , z axes , and is applied using a ResNet [ 16 ] ( diffusion ) structure only . The recent GRAND [ 32 ] applies attention mechanism with diffusive dynamics , using several integration schemes . Here we propose a network for both geometric and non-geometric tasks , and also utilize both the diffusion or hyperbolic layer dynamics , including a mixture of the two . PDEs and CNNs : In a recent series of works , the connection between PDEs and CNNs was studied [ 20 , 21 , 22 , 33 , 34 , 35 , 36 , 37 ] . It was shown that it is possible to treat a deep neural network as a dynamical system driven by some PDE , where each convolution layer is considered a time step of a discretized PDE . The connection between PDEs and CNNs was also used to reduce the computational burden [ 38 ] . Besides the interpretation of CNNs as a PDE solver , it was shown that it is also possible to learn a symbolic representation of PDEs in a task-and data driven approach in [ 39 , 40 ] . In the context of GCNs and PDEs , it was recently shown [ 41 , 42 ] that GCNs can be utilized to enhance the solution of PDEs for problems like fluid flow dynamics . However , in this work , we harness PDE concepts to design and construct GCNs for a variety of applications . 3 Methods . 3.1 Partial Differential Equations on manifolds . We show now that GCNs can be viewed as discretizations of PDEs on manifolds , similarly to CNNs that are viewed as discretized PDEs on regular grids [ 22 ] . Consider a general manifoldM where a vector function f resides ( also dubbed the features function ) , along with its continuous differential operators such as the gradient ∇ , divergence ∇· and the Laplacian ∆ that reside on the manifoldM . Given these differential operators , one can model different processes onM . In particular , we consider two PDEs – the non-linear diffusion and the non-linear hyperbolic equations ft = ∇ ·K∗σ ( K∇f ) , f ( t = 0 ) = f0 , t ∈ [ 0 , T ] , ( 1 ) ftt = ∇ ·K∗σ ( K∇f ) , f ( t = 0 ) = f0 , ft ( t = 0 ) = 0 , t ∈ [ 0 , T ] , ( 2 ) respectively , equipped with appropriate boundary conditions . Here K is a coefficient matrix that can change in time and represents the propagation over the manifoldM , K∗ is its conjugate transpose and σ ( · ) is a non-linear activation function . Eq . ( 1 ) - ( 2 ) define a non-linear operator that takes initial feature vectors f0 at time 0 and propagates them to time T , yielding fT where they can be used for different tasks . We now provide two theorems that characterize the behavior of Eq . ( 1 ) - ( 2 ) , based on ideas from [ 22 ] 1 . Theorem 1 . If the activation function σ ( · ) is monotonically non-decreasing and sign-preserving , then the forward propagation through the diffusive PDE in ( 1 ) for t ∈ [ 0 , ∞ ) yields a non-increasing feature norm , that is , ∂ ∂t ‖f‖2 ≤ 0 . Theorem 2 . Assume that the activation function σ ( · ) is monotonically non-decreasing , signpreserving and satisfies |σ ( x ) | ≤ |x| , and define energy Enet = ‖ft‖2 + ( K∇f , σ ( K∇f ) ) , 1See proofs in Appendix A. then the forward propagation through the hyperbolic PDE in ( 2 ) satisfies Enet ≤ cK , where cK is a constant that depends on K but independent of time . The outcome of those theorems is that the dynamics described in Eq . ( 1 ) is smoothing , while the one in Eq . ( 2 ) is bounded by a conserving mapping . An illustration of this behavior is presented in Fig . 1 . In the physical world , diffusion and hyperbolic equations are used for different applications . Similarly , many computational models for image segmentation [ 43 ] , denoising [ 44 ] and deblurring are based on anisotropic diffusion which are similar to the model in Eq . ( 1 ) . On the other hand , applications that require conservation such as volume/distance preservation as in the dense shape correspondence task [ 45 ] and protein folding [ 11 ] , are typically better treated using a hyperbolic equation as in Eq . ( 2 ) . Those insights motivate us to construct two types of layers according to Eq . ( 1 ) - ( 2 ) using discretized differential operators on graphs . 3.2 Discretized differential operators on graphs . The models ( 1 ) and ( 2 ) reside in a continuous manifoldM , on which a continuous function vector f is defined . A graph can be thought of as a discretization of that manifold to a finite space . Assume we are given an undirected graph G = ( V , E ) where V ∈ M is the set of n vertices of the graph and E is the set of m edges of the graph . Let us denote by f i ∈ Rc the value of the discrete version of f , on the i-th node of G. c is the number of channels , which is the width of the neural network . We define G , the discrete gradient operator on the graph , also known as the incidence matrix , as follows : ( Gf ) ij = Wij ( fi − fj ) , ( 3 ) where nodes i and j are connected via the ( i , j ) -th edge , Wij is an edge weight matrix which can be learnt , and fi and fj are the features on the i-th and j-th nodes , respectively . The gradient operator can be thought of as a weighted directional derivative of the function f in the direction defined by the nodes i and j . Furthermore , if we choose the scaled identity matrix Wij = d−1ij I , where dij is the distance between the two nodes , then the discrete gradient is a second order approximation to the true gradient of the function f on the edges of the graph . In this work , we use Wij = γijI , where the scale γij is the geometric mean of the degree of the nodes i , j . Note , that the gradient operator is a mapping from the vertex space to the edge space . Given the gradient matrix , it is possible to define the divergence matrix [ 46 ] , which is an approach that is used extensively in mimetic discretizations of PDEs . To this end , we define the inner product between an edge feature vector q and the gradient of a node feature vector f as ( q , Gf ) = q > Gf = f > G > q . ( 4 ) The divergence is naturally defined as the operator that maps edge operator q to the node space , that is ∇· ≈ −G > . As usual , the graph Laplacian operator can be obtained by taking the divergence of the gradient . In graph theory it is defined as a positive matrix that is , ∆ ≈ G > G 2 . We also define the weighted line integral over an edge . Similarly to Eq . ( 3 ) - ( 4 ) , we define ( Af ) ij = 1 2 Wij ( fi + fj ) , ( q , Af ) = q > Af = f > A > q . ( 5 ) The operator A approximates the mass operator on the edges . The right equation in Eq . ( 5 ) suggests that an appropriate averaging operator for edge features is the transpose of the nodal edge average . The advantage of defining such operators is that we are able to design networks with architectures that mimic the continuous operators ( 1 ) and ( 2 ) on the discrete level , as we show in the next section .

Summary: The authors propose a new class of graph neural networks motivated by numerical methods for solving PDEs on manifolds. The idea is that there is a correspondence between specific GNNs (e.g., GNNs that learn the weight of edges such as GATs) and discretized PDEs on manifolds. Here, the discretization is spatially. For time (i.e., the depth of the GNN) one can use numerical solvers for PDEs similar to what was proposed by the original Neural ODE paper. The main motivation is that there are types of PDEs that do not suffer from the oversmoothing problem and, therefore, that their discretized versions might be a better fit for some problem domains where oversmoothing is a problem.