PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
Authors: Moshe Eliasof
Eldad Haber
Eran Treister
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Networks
Neural Network Architectures
PDES  in Deep Learning
Partial Differential Equations

Abstract: Graph neural networks are increasingly becoming the go-to approach in various fields such as computer vision, computational biology and chemistry, where data are naturally explained by graphs. However, unlike traditional convolutional neural networks, deep graph networks do not necessarily yield better performance than shallow graph networks. 
This behavior usually stems from the over-smoothing phenomenon. In this work, we propose a family of architectures
to control this behavior by design. Our networks are motivated by numerical methods for solving Partial Differential Equations (PDEs) on manifolds, and as such, their behavior can be explained by similar analysis. Moreover, as we demonstrate using an extensive set of experiments, our PDE-motivated networks can generalize and be effective for various types of problems from different fields. Our architectures obtain better or on par with the current state-of-the-art results for problems that are typically approached using different architectures.

PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
Moshe Eliasof
Eldad Haber
Department of Computer Science Ben-Gurion University of the Negev
Department of Earth, Ocean and Atmospheric Sciences
University of British Columbia
Beer-Sheva, Israel eliasof@post, bgu.ac. .c.il
Vancouver, Canada
ehaber wbc.
Eran Treister
Department of Computer Science Ben-Gurion University of the Negev
Beer-Sheva, Israel erantêcs. bgu. ac.
Abstract
Graph neural networks are increasingly becoming the go-to approach in various fields such as computer vision, computational biology and chemistry, where data are naturally explained by graphs. However, unlike traditional convolutional neural networks, deep graph networks do not necessarily yield better performance than shallow graph networks. This behavior usually stems from the over-smoothing phenomenon. In this work, we propose family of architectures to control this behavior by design. Our networks are motivated by numerical methods for solving Partial Differential Equations (PDEs) manifolds, and such, their behavior explained by similar analysis. Morcover, as we demonstrate using an extensive set of experiments, our PDE-motivated networks can generalize and be effective for various types of problems from different fields. Our architectures obtain better par with the current state-of-the-ae art results for problems that typically approached using different architectures.
Introduction
In recent years, Graph Convolutional Networks (GCNs) II
3 have drawn the attention of re- searchers and practitioners in a variety of domains and applications, ranging from computer vision and graphics [ 3
6 to computational biology O 10 11. recommendation systems [1] and
social network analysis 13 [4] However, GCNs still suffer from two main problems. First, they
usually shallow as opposed to the concept deep convolutional neural networks (CNNs) [3] [6]
due to the over-smoothing phenomenon [17 [8 19] where the node feature vectors become almost identical, such that they are indistinguishable which yields non-optimal performance. Furthermore, GCNs typically customized to a specific domain and application. That is, Sec. 4.1 a successful point-cloud classification network (5) can perform poorly on a citation graph node-llassifictton problem [19 and vice-versa. Furthermore, because many GCNs lack theoretical guarantees, it is difficult to reason about their success on one problem and lack on others. These observations motivate to develop profound understanding graph networks and their dynamics.
demonstrate in
To this end, we suggest novel, universal approach to the design of GCN architectures. Our inspiration stems from the similarities and equivalence between Partial Differential Equations (PDEs) and deep networks explored in 20 21 22]. Furthermore, as GCNs can be thought of as generalization of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

19, leading to undesired node CNNs, and a standard convolution can be represented as a combination of differential operators on structured grid [22]. we adopt this interpretation explore versions of GCNs as PDEs on graphs or manifolds. We therefore call our network architectures DCN, and demonstrate that our approach general with respect to the given task. That is, our architectures behave similarly for different problems, and their performance is on par or better than other domain-specific GCNs. Furthermore, our family of architectures are backed by theoretical guarantees that allow us to explain the behavior of the GCNs in some of the results that we present. To be more specific, our contribution is follows:
We introduce and implement general graph convolution operators, based on graph gradient and divergence. This abstraction of the spatial operation on the graph leads more general and flexible approach for architecture design.
We propose treating a variety of graph related problems as discretized PDEs, and formulate the dy- namics that match different problems such as node-classification and dense shape-correspondenee This is in direct effort to propose family of networks that can solve multiple problems, instead of achieving significantly better or similar accuracy compared to state-of-the models.
GCNs which are tailored for specific application.
Our method allows constructing a deep GCN without over-smoothing, with theoretical guarantees.
We validate our method by conducting numerous experiments on multiple datasets and applications,
2
Related work
Graph Convolutional Networks: GCNs are typically divided into spectral 1 and spatial 23 24 25 4 categories. Most of those can be implemented using the Dessage-Passing Neural Network paradigm (25,, where each node aggregates features (messages) from its neighbors, according to some use polynomials of the graph Laplacian parameterize the convolution operator. DGCNN 5 constructs a k-nearest-neighbors graph from point-clouds and dynamically updates it. MoNet learns a Gaussian mixture model to weight the edges of meshes for shape analysis tasks. Works like GraphSAGE [26) and GAT 27 propose methods for inductive and transductive learning on non-geometrical graphs.
scheme. The works B [
Several of the methods above suffer from over-smoothing
features similarity for deep networks. To overcome this problem, some approaches rely on imposing regularization and augmentation. For example, PairNorm [17] introduces a novel normalization layer, and DropEdge (28) randomly removes edges to decrease the degree of nodes. Other methods prevent over-smoothing by dedicated construction. For instance, JKNet (2 combines all intermediate representations at each stage of the network. APPNP (30] replaces learnt convolutional layers with pre-defined kernel based on PageRank, yielding a shallow network which preserves locality, making it robust to wver-smoothing. GCNII [19] proposes to add an identity residual where the initial features of the networks are added to the features of the 1-th layer, scaled by some coefficient.
Another approach is to construct network that inherently does not overts this work. Our network is based on discretized PDEs, hence we are able to motivate our choices by well studied theory and numerical experiments [3 On similar note, the recent DiffGCN [ also makes use of discretized operators to approximate the graph gradient and Laplacian. However, DiffGCN is specifically tailored for geometric tasks since it projects its components on the x, y, axes, and is applied using a ResNet (16] (diffusion) structure only. The recent GRAND [32] applics attention mechanism with diffusive dynamics, using several integration schemes. Here propose network for both geometric and non gengeeeettric tasks, and also utilize both the diffusion or hyperbolic layer dynamics, including mixture of the two.
suggest in
In a recent series of works, the connection between PDEs and CNNs was studied 35 36 37]. It was shown that it is possible treat a deep neural network as dynamical system driven by some PDE, where each convolution layer is considered time step of a discretized PDE. The connection between PDEs and CNNs was also used to reduce the computational burden [38]. Besides the interpretation of CNNs as PDE solver, it was shown that it is also possible to learn a symbolic representation of PDEs
PDEs and CNNs:
20 21 20
33
34
task-and data driven approach in [9 40). In the
context of GCNs and PDEs, it was recently shown [1] 42 that GCNs can be utilized to enhance the solution of PDEs for problems like fluid flow dynamics. However, in this work, we harness PDE concepts to design and construct GCNs for variety of applications.

---

t=0
Figure 1: Feature evolvement on an input mesh (left). Propagation in time is from left to right. Hyperbolic and diffusion equation dynamics a diffusive graph network (similar to most common GCNs that rely on ResNets) smooths the information on the manifold, the hyperbolic network yields non-uniform field.
the top and bottom row, respectively. While
3 Methods
3.1 Partial Differential Equations on manifolds
We show now that GCNs can be viewed as discretizations of PDEs on manifolds, similarly to CNNs that are viewed as discretized PDEs on regular grids [22]] Consider general manifold M where vector function f resides (also dubbed the features function), along with its continuous differential operators such as the gradient V, divergence V. and the Laplacian A that reside on the manifold M.
Given these differential operators, one can model different processes on M. In particular, we consider two PDEs the non-linear diffusion and the non-linear hyperbolic equations
ft V K* K (KV))), fu - K'V(K))) 0) fi(t 0) =0, t0
f(t = 0) f-, E [0,
(2)
respectively, equipped with appropriate boundary conditions. Here K is coefficient matrix that can change in time and represents the propagation over the manifold M, K** is its conjugate transpose and (-) is non-linear activation function. Eq. (1-( define a non-linear operator that takes initial feature vectors fd at time 0 and propagates them to time T, yielding f where they can used for different tasks. We now provide two theorems that characterize the behavior of Eq.
based on
ideas from 122
Theorem 1. If the activation function o(:) is monotonically noc-decrgasing and sign-preserving. then the forward propagation through the diffusive PDE in for t E (0) yields non-increasing feature norm, that is,
Ōt 1f1 5.
Theorem 2. Assume that the activation function a(.) is monotonically non-decreasing, sign- preserving and satisfies [a(x)] < [z), and define energy
Enet || (KV
'See proofs in Appendix A.

---

then the forward propagation through the hyperbolic PDE in (2) satisfies Enet S Cg. where eg is constant that depends on K but independent of time.
The outcome of those theorems is that the dynamics described in Eq. ①I is smoothing, while the one in Eq. 22 bounded by a conserving mapping. An illustration of this behavior presented Fig. [
In the physical world, diffusion and hyperbolic equations are used for different applications. Similarly, many computational models for image segmentation [R] denoising 44 and deblurring based on
anisotropic diffusion which are similar to the model in Eq. [ On the other hand, applications that require conservation such as volume/distance preservation in the dense shape correspondence task [45] and protein folding .. are typically better treated using hyperbolic equation as in Eq. (2)
Those insights motivate us to construct two types of layers according to Eq. (I- using discretized differential operators on graphs.
3.2 Discretized differential operators on graphs
The models (T) and 2 reside in a continuous manifold M, which continuous function vector defined. A graph can be thought of as a discretization of that manifold to a finite space. Assume we are given an undirected graph 9 = (V,E) where V E .M the set of n vertices of the graph and is the set of m edges of the graph. Let us denote by f; E Ie the value of the discrete version of f, on the i-th node of G. is the number of channels, which is the width of the neural network. We define G, the discrete gradient operator on the graph, also known as the incidence matrix, as follows:
(Gf) = Wi (f, -f),,
where nodes i and j are connected via the ( -th edge, W;; is an edge weight matrix which can be learnt, and f; and fj are the features on the i-th and j-th nodes, respectively. The gradient operator can be thought of as weighted directional derivative of the function ʃ in the direction defined by the nodes and j. Furthermore, if we choose the scaled identity matrix Wij dij I, where dæj is the distance between the two nodes, then the discrete gradient a second order approximation to the true gradient of the function f on the edges of the graph. In this work, we use W Is where the scale YYi the geometric mean the degree of the nodes j. Note, that the gradient operator is mapping from the verfex space to the edge space.
Given the gradient matrix, is possible to define the divergence matrix [46], which is an approach that is used extensively in mimetic discretizations of PDEs. To this end, we define the inner product between edge feature vector q and the gradient of node feature vector f as
(q. Gf) = Gf = G
The divergence is naturally defined as the operator that maps edge operator q to the node space, that V. = G As usual. the graph Laplacian operator can be obtained by taking the divergence of the gradient. In graph theory it is defined as a positive matrix that is, A z G
We also define the weighted line integral
edge. Similarly to Eq.
3
define
1 w, i (f. f,,, (q. = q Af f A T = 2
(Af).;
q.
(S)
The operator A approximates the mass operator on the edges. The right equation in Eq. 5 suggests that an appropriate averaging operator for edge features is the transpose of the nodal edge average.
The advantage of defining such operators is that that mimic the continuous operators and (2 on the discrete level, as we show in the next section.
able to design networks with architectures
3.3 PDE-GCN: Graph Convolutional Networks by Partial Differential Equations
In order to use the computational models in Eq.
T 22. we form their discrete versions:
fl++1) = = - J.G T K (KGGP()
(6)
p(++)) 2t( - KJG (KKGP()
h2
(7)
'In the tield of numerical PDEs, the Laplacian is defined as - G G. but the combinatorial Laplacian is detined as G G.

---

Here, in Eq. 6 we use the forward Euler to discretize Eq. ① and in Eq. 7 we discretize the second order time derivative in Eq. 2, using the leapfrog method. In both cases, f()) are the node features and K is a
trainable convolution of the (-th layer. The similarity between (6) and ResNet is well documented in the context of CNNs (22). The hyper-parameter h is the step-size, and is chosen such that the stability of the discretization is kept. We use tanh for the activation function as it yields slightly better results in our experiments. although other functions such as ReLU can also be used, as reported in Sec. 4.7 Each of Eq. O 7 defines PDE- block. We denote the former (diffusive equation) by GCN and the latter (byperbolic equation) by PDE- DCN-
To complete the description of network, few more details are required, follows:
The (opening) embedding layer. The input vertex features Hy are fed through an embedding (1 x 1 convolution) layer K, to obtain the initial features fo of our PDE-GCN network: fo K, uy. In cases where input edge attributes (features) ug are available (as in the experiment in Sec. 4.6)). we transform them to the vertex space by taking their divergence and average, and concatenate them to the input of the embedding layer K. as follows: fu = Ku @ A ugg + G ug).
The (closing) embedding layer. Given the final vertex features f(L)) of our PDE-GCN with layers, we obtain the output of the network by performing: Nout K,P(L), Here K2 is 1 x 1 convolution layer mapping the hidden feature space to the output shape.
Initialization. The 1 x 1 convolutions K, in Eq. (7) are initialized as identity. This way, the network begins from a diffusion/hyperboliic equation, which serves as prior and guides the network to initially behave like classical methods [43 455 and to further improve during training.
The choice of dynamics. For some applications, anisotropic diffusion is appropriate, while for others conservation is more important. However, in some applications this may not be clear. To this end, it possible to combine Eq.
(-( to obtain the continuous process
afu + (1 -1 = K K (KV)), -) -) =
fe(t = 0) = 0 t E [o,T],
(8)
where a = sigmoid(B). meaning 0 I a I 1, and 8 is single trainable parameter. The discretization of this PDE leads to the following network dynamics:
a(p(++1) 2F()) + f (1-1)) + h(1 - ()(((++) p()) = -h OT
(9)
where p(++1) is updated by the known p(t) and p(t-1), We denote a layer that is governed by Eq. by PDE-GCNM (where M stands for mixture). Note, that it is also possible to learn combination coefficient a; per layer, although we did not read a benefit from such scheme.
(
As we show in our numerical experiments, learning yields results that are consistent with our understanding of the problem, that is, graph cose-classification gravitates towards no second order derivatives while applications that require conservation gravitate towards the hyperbolic equation.
4 Experiments
In this section demonstrate our approach various problems from different fields and applications ranging from 3D chape-clasiification [47] to protein-protein interaction (26) and node- classifsiiiin
[8] The experiments also vary in their output type. Node classification is similar to segmentation problems that are typically solved by anisotropic diffusion while the dense shape correspondence problem is conservative and therefore can be thought of as problem that does not require smoothing.
In all experiments, we use the suitable PDE-GCN (D, H or M) block as described in Sec. 3. with various depths (number of layers) and widths (number of channels), well as the appropriate final convolution steps, depending on the task at hand. A detailed description of the architectures used in our experiments is given in Appendix B. We use the Adam 44) optimizer in all experiments, and perform grid search over the hyper- parameters of our network. The selected reported in Appendix C. Our objective function in all experiments is the cross-entropy loss, besides inductive learning on PPI where we use the binary cross eross-entropy loss. Our code implemented using PyTorch (50). trained
are
Nvidia Titan RTX GPU.
We show that for all the considered tasks and datasets, our method is either remarkably better or on par with state-of-the-ar models.
5

---

Table I: Generalization of GCNs different domains and applications. (L) denotes layers.
Method
Dataset
Accuracy (%)
DGCNN (4) [
ModelNet-10
92.8
DGCNN (2) (4)
Cora
34.9 / 25.2
DGCNN + Diffusion (2) (4)
Cora
71.0 / 66.1
GCNII (4) 19
ModelNee-10
65.4
Cora
82.6
PDE-GCND(4) (Ours)
ModelNet-10
92.2
Cora
83.6
0.5
0.5
Accuracy
0.8
Correspondence
03
06
0.2
0.80
011
0.60
0.75
0.5
070
100
200
400
500
20
300
Epoch #
Epoch
(a) Graph node-classification (Cora)
(b) Dense shape correspondence (FAUST)
Figure 2: Learnt mixture of the hyperbolic equation a, to the dynamics of the network. The diffusion
equation contribution is a.
4.1 Generalization of GCNs to different applications
To gain deeper understanding about the effectiveness and generalization of various GCN methods to different tasks, we start by picking two datasets ModelNet-10 10 [47 for 3D shape-clasiffiation. and Cora for semi-supervised node classification. Those datasets are not only different in terms of application (global versus local classification), but also stem from different domains. While in 1o the data has geometrical meaning, the data in Cora has no obvious geometrical suggest that success in both applications should be obtained from
interpretation. Therefore, generalizable GCN, We compare our PDE-GCNp with two recent and popular networks DGCNN randomly sample 1,024 points from
[5] and GCNII [19]. For ModelNet-10 thape-classification each shape to form a point cloud, and connect its points using k-nearest-neing neighbss (k-nn) algorithm with &: = 10 obtain graph and follow the training scheme of [5]. On Cora, we follow the same procedure as in 19 We evaluate all models with 4 layers, as well as 2 layers for DGCNN on Cora.
Our results, reported in Tab. D suggest that while each of the considered methods obtains high accuracy on the dataset it originally was tested on (Cora for GCNII and ModelNet- 10 for DGCNN), obtaining similar measure of success
different dataset same networks. Additionally, while our attempts to add a diffusion-equation dynamics to DGCNN (i.e., updating features as in Eq. (6)) showed an increase in performance a large gap state-of-the- art model still exists. On top of that, we also see that DGCNN suffers from over-smoothing, as its accuracy significantly decreases when adding more layers. Last but not least, observe that our PDE-GCNp obtains high accuracy on both datasets, similar or better than state-of-the-an models.
not possible when using the very
4.2 Learning PDE network dynamics
In this experiment, we delve on the ability to learn the appropriate PDE that better models given problem. To this end, we use the mixture model from Eq. 4 so that the resulting PDE combination of the diffusion and hyperbolic dynamics. We use a layer mixed PDE-GCN, starting with a = 0.5, such that balanced between a PDE-GCNp and PDE-GCNB- By learning the parameter in (9. allow to choose mixed PDE between purely conservative network and

---

diffusive one. We consider two problems: semi-supervised node classification on Cora, and dense shape correspondence on FAUST [5T].
Our results, reported in Fig. 2 suggest that just as in classical works 43 52
problems like node-
classification obtain better performance with anisotropic diffusion like in Eq. (6. and for problems involving dense-corresse cowreeooorresseeeessss like in [45 [ that tend to conserve the energy of the underlying problem, hyperbolic equation type of PDE in Eq. 7 is more appropriate.
Table 2: Statistics of datasets used in our semi-and fully supervised node-classification experiments.
Dataset
Cora
CiteSeer
PubMed
Chameleon
Cornell
Texas
Wisconsin
Classes
7
3
5
5
5
121
Nodes
2,708
3,327
19,717
2,277
183
183
251
56,944
Edges
5,429
4,732
44,338
36,101
295
309
499
818,716
Features
1,433
3,703
500
2,325
1,703
1,703
1,703
50
4.3 Semi-supervised node classification
In this set of experiments we use three datasets Cora, Citeseer and PubMed 48 For all datasets we use the standard traiiingvaliddaiionnestingg split as in [53), with 20 nodes per class for training. 500 validation nodes and 1,000 testing nodes and follow the training scheme of []9]. The statistics of the datasets are reported in Tab. 2 We compare our results using PDE-GCNp with recent and popular models like GCN [. GAT (77), APPNP 30) JKNet [29] and DropEdge (28) We note that our network does not ovor-smooth, as an increase in the number of layers does not cause performance degradation. For example, CiteScer,
obtain 75.6% accuracy with 32 layers, compared to 74.6% with two layers] Overall, our results in Tab. show that our GCND achieves similar or better accuracy than the considered methods.
4.4 Fully-supervised node classification
We follow [54] and use 7 datasets: Cora, CiteSeer, PubMed, Chameleon, Cornell, Texas and Wisconsin. We also use the same train/validation/tesl splits of 60%, 20%, 20%. respectively. In addition, we report the average performance over 10 random splits from [54]. We fix the number of channels to 64 and perform grid search to determine the hyper-paramette reported in Appendix C. We compare [54], APPNP, JKNet, Incep and GCNII in Tab. 4 Our experiments read either similar or better than the state-of-thee of-the--rt
parameters, which are network with GCN, GAT, three variants of Geom GCN
Cora, CiteSeer and PubMed datasets. On Chameleon [55], Cornell, Texas and Wisconsin datasets, we improve state-of-the- accuracy by a significant margin. For example, we obtain 93. 83.24% accuracy Texas with PDE-GCNM, compared 77.84% with GCNII* Similar improvements hold for Cornell and Wisconsin datasets. The common factor for these datasets is their small size, as depicted from Tab. 2 We argue that the success of our network stems from its capability of apriori extracting features from graphs, due to its utilization of discretized differential operators and PDE guided construction. On Chameleon [55] using PDE-GCNM, we improve the current state-of-te of-the-art accuracy of GCNII* from 62.48% to 66.01 %. Also, we note that unlike in the semi-supers cumi-super case, where some of the labels are missing, here is possible to obtain meaningful results with the hyperbolic equation based PDE-GCNH as we do not have unknown nodes in the sully-sueervised case, which would be otherwise preserved using the hyperbolic equation dynamics.
4.5 Inductive Learning
19 and employ the PPI dataset (26] for the inductive learning task. We use 8 layer PDE- IDE-GCND network, without dropout or weight-decay. and learning rate of 0.001. We compare our results with methods like GraphSAGE, GAT, JKNet, GeniePath. GCNII and others. As reported in Tab. 5 our PDE-GCN achieves 99.07 Micro-averane Fl score, superior to methods like GAT, JKNet and GeniePath, also close ofte-of-the-ar GCNII* with a score of 99.58.
We follow
Note that this result is also ofate-of-the-at accuracy.

---

Table Semi-supervised node classification accuracy (%).
indicates not available results.
Layers
Dataset
Method
2
8
16
32
64
Cora
GCN
81.1
80.4
69.5
64.9
60.3
28.7
GCN (Drop)
28
82.8
82.0
75.8
75.7
62.5
49.5
JKNet
80.2
80.7
80.2
81.1
71.5
JKNet (Drop)
[2
83.3
82.6
83.0
82.5
83.2
Incep [28]
77.6
76.5
81.7
81.7
80.0
Incep (Drop) 28
82.9
82.5
83.1
83.1
83.5
GCNII [99
82.2
82.6
84.2
84.6
85.4
85.5
GCNII" 19
80.2
82.3
82.8
83.5
84.9
85.3
PDE-GCNp (Ours)
82.0
83.6
84.0
84.2
84.3
84.3
Citeseer
GCN 3
70.8
67.6
30.2
18.3
25.0
20.0
GCN (Drop)
72.3
70.6
61.4
57.2
41.6
34.4
JKNet 29
68.7
67.7
69.8
68.2
63.4
JKNet (Drop)
72.6
71.8
72.6
70.8
72.2
Incep 28
69.3
68.4
70.2
68.0
67.5
Incep (Drop) 2X
72.7
71.4
72.5
72.6
71.0
GCNII [19
68.2
68.8
70.6
72.9
73.4
73.4
GCNII 19
66.
66.7
70.6
72.0
73.2
73.1
PDE-GCNp (Ours)
74.6
75.0
75.2
75.5
75.6
75.5
Pubmed
GCN 3
79.0
76.5
61.2
40.9
22.4
35.3
GCN (Drop)
28
79.6
79.4
78.1
78.5
77.0
61.5
JKNet 29
78.0
78.1
72.6
72.4
74.5
JKNet (Drop)
228
78.7
78.7
79.7
79.2
78.9
Incep 28
77.7
77.9
74.9
Incep (Drop)
28
79.5
78.6
79.0
GCNII 19
78.2
78.8
79.3
80.2
79.8
79.7
GCNII 19
77.7
78.2
78.8
80.3
79.8
80.1
PDE-GCNp (Ours)
79.3
80.6
80.1
80.4
80.2
80.3
4.6 Dense shape correspondence
Finding dense correspondences between shapes a classical experiment for hyperbolic dynamics, as we are interested in modeling local motion dynamics. In essence, learning to find correspondences between shapes similar learning a transformation from one shape the other. To this end, we use the FAUST dataset [51) containing 10 scanned human shapes in 10 different poses with 6,890 nodes each. We follow the train and test split from 4 where the first 80 subjects are used for training and the remaining 20 subjects for testing. Our metric is the correspondence percentage with zero geodesic error, i.c., the percentage of perfectly matched shapes from all our test cases. We follow the pre-processing and training scheme of (62) where use Cartesian coordinates describe distances between nodes, with initial features of a constant 1 E ET where n the number of nodes. We use layer PDE-GCNp and PDE-GCN, variants, both with constant learning rate of 0.001 with weight-decay or dropout, and compare to recent and popular methods like ACNN 660, MoNet [4] FMNet [61] and SplineCNN [62] As expected from the discussion in Sec. [.I and reported in Tab. 6, the hyperbolic equation proves to be a better fit for this kind problem. Also, our DCDG achieves promising 99.9% correspondence rate with zero geodesic error, outperforming the rest of the considered methods.
4.7 Ablation study
Our method has two main dynamics diffusion and hyperbolic. To verify our proposal in Sec. 3.1 we examine the performance of the hyperbolic PDE-GCNg on semi-supervised node-classification on Cora and CiteSeer datasets. Our results Tab. show that indeed for problems where we wish to obtain piccewise-constant prediction, the diffusive formulation of our network, PDE-GCNp. is

---

Table 4: Fully-supervised node classification accuracy (%). (L) indicates a layers network.
Method
Cora
Cite.
Pubm.
Cham.
Corn.
Texas
Wisc.
GCN
13
85.77
73.68
88,15 13
28.18
52.70
52.16
45.88
GAT 27
86.37
74.32
87.62
42.93
54.32
58.38
49.41
Geom-GCN-1
55
85.19
77.99
90.05
60.31
56.76
57.58
58,24
Geom-GCN-P
54
84.93
75.14
88,09
60.90
60.81
67.57
64.12
Geom-GCN-S
54
85.27
74.71
84.75
59.96
55.68
59.73
56.67
APPNP 30
87.87
76.53
89.40
54.30
73.51
65.41
69.02
JKNet 29
85,25 (16)
75.85 (8)
88.94 (64)
60.07 (32)
57.30 (4)
56.49 (32)
48.82 (8)
JKNet (Drop)
28
87.46 (16)
75.96 (8)
89.45 (64)
62.08 (32)
61.08 (4)
57.30 (32)
50.59 (8)
Incep (Drop)
28
86.86 (8)
76.83 (8)
89.18 (4)
61.71 (8)
61,62 (16)
57.84 (8)
50.20 (8)
GCNI 19
88,49 (64)
77.08 (64)
89.57 (64)
60.61 (8)
74.86 (16)
69.46 (32)
74.12 (16)
GCNII*
88.01 (64)
77.13 (64)
90.30 (64)
62.48 (8)
76.49 (16)
77.84 (32)
81.57 (16)
PDE-GCN, (Ours)
88.51 (16)
78.36 (64)
89.6 (64)
64.12 (8)
89.19 (2)
90.81 (8)
90.39 (8)
PDE-GCN, (Ours)
87.71 (32)
78.15 18.13 (16)
89.16 (16)
61.57 (64)
89.45 (64)
92.16 (64)
91.37 (16)
PDE-GCNg (Ours)
88.60 (16)
78.48 (32)
89.93 (16)
66.01 (16)
89.73 (64)
93.24 (32)
91.76 (16)
Table Protein-protein interaction (PPI). Results
reported in micro-averaged F1 score.
Method
Micro-averaged
Table 6: Dense shape correspondence
(%) with zero geodesic error
GraphSAGE 266
61.20
VR-GCN [66]
97.80
Method
Faust
GaAN 157
98.71
ACNN
660
63.8
GAT 22
97.30
MoNet
4
89.1
JKNet 29
97.60
FMNet
6
98.2
GeniePath 58
98.50
SplineCNN 66
99.2
Cluster-Gee GCN [59]
99.36
GCNII 19
99.54
PDE-GCND (Ours)
64.2
GCNII 19
99.58
PDE-GCNH (Ours)
99.9
PDE-GCNp (Ours)
99.07
PDE-GCNM (Ours)
99.18
more suitable. Furthermore, study the importance of the positíve-semi definiteness of our learnt operator as described in Eq. (6)-(7 by removing the K" term from the dynamics equations. This yields non-symmetric operator that does not guarantee the enforcement of the latter is important to obtain higher accuracy which is improved by to 3% with the introduction of positive semi-definite operator. Also, report on the use of a Re LU, from the discussion in Sec. 3.3 where we favor tanh as an activation function, and the ability of our PDE-GCN reproduce the results of PDE-GCNp the case of semi-supervised learning.
saatiie-semiidefiiinnessss We note that
5 Summary
In this paper we explored new architectures for graph neural networks. Our motivation stems from the similarities between graph networks and time dependent partial differential equations that discretized on manifolds and graphs. By adopting an appropriate PDE, and embedding the finite graph in an infinite manifold, we are able to define networks that are either diffusive, conservative, or a combination of both.
Not all natural phenomena are solved using the same PDE and we should not expect that all graph problems should be solved by the same network dynamics. To this end we allow the data to choose which type of network is appropriate for the solution of the problem (diffusive or hyperbolic). Indeed, numerical experiments show that the network gravitates towards a hyperbolic one for problems where conservation is required, and towards a diffusive one when anisotropic diffusion is favorable.

---

Table Ablation study of PDE- GCN accuracy (%) on semi-supervised code-classificatton
Layers
Method
Dataset
2
8
16
32
64
PDE-GCNH
Cora
79
79.3
78.0
78.0
77.8
77.5
CiteSeer
70.9
71.7
72.1
72.3
72.5
72.4
PDE-GCND (non-symmetric)
Cora CiteSeer
83.5
83.3
83.6
83.1
82.7
81
74.3
74.5
74.8
75.0
73.9
73.3
PDE-GCND (a = ReIUU
Cora CiteSeer
80.3
81.8
82.6
83.0
83.4
83.5
73.1
73.2
72.8
73.3
73.6
74.0
PDE-GCNM
Cora
82.0
83.4
83.9
84.2
84.3
84.5
CiteSeer
74.2
74.8
75.3
75.4
75.4
75.8
Finally, we showed that the proposed networks can be made deep without over-smoothing and, can deliver the state-of-the-art performance or improve it for virtually every problem we worked with. In particular, our network dramatically improved the stateoof-the-arr for problems that are data-poor. We believe that for such problems the structure imposed by our dynamics and operators regularizes the network and therefore yields implicit regularization.
Acknowledgments and Disclosure of oufuum Funding
The research reported in this paper was supported by grant no. 2018209 from the United States Israel Binational Science Foundation (BSF), Jerusalem, Israel. ME is supported by Kreitman High-tech scholarship.
References
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 2013.
[2] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844- 3852, 2016.
[3] Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 0009000990 2016.
[4] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 5115-5124, 2017.
[5] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael Bronstein. and Justin Solomon. Dynamic graph cun for learning on point clouds. arXiv preprint arXiv: 007829, 2018.
[6] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep cnns? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9267-9276. 2019.
[7] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshcnn: a network with an edge. ACM Transactions Graphics (TOG). 38(4):90, 2019.
[8] Moshe Eliasof and Treister. Diffggen: Graph convolutional networks via differential operators and algebraic multigrid pooling. 34th Conference on Neural Information Processing Systems (NewrIPS 2020), Vancouver Canada., 2020.
Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Perez-Riba, and Philip M. Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):002 -112--11-000 1.1, 2020.
[10] Thomas Gaudelet, Ben Day, Arian R Jamasb, Jyothish Soman, Cristian Regep, Gertrude Liu, Jeremy BR Hayter, Richard Vickers, Charles Roberts. Jian Tang, al. Utilising graph machine learning within drug discovery and development. arXiv preprint arXiv:2012.055:
2020.
[11] Moshe Eliasof, Tue Boesen, Eldad Haber, Chen Keasar, and Eran Treister. Mimetic neural networks: A unified framework for protein design and folding. arXiv preprint arXiv:2102.0388/ 2021.
10

---

[12] Rex Ying. Ruining He. Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983, 2018.
[13] Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jic Tang. Deepinf: Social influence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2110-2119, 2018.
[14] Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection news media. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2594-2604 2019.
[15] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, 61:1097-1105, 2012.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778. 2016.
[17] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In Internationall Conference on Learning Representations, 2020.
[18] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smootiing problem for graph neural networks from the topological view. Proceedings of the AAAI Conference on Artificial Intelligence, 34:3438- 1334333344,, 04 2020.
[19] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph con- volutional networks. Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1725-1735. PMLR, 13-18 Jul 2020.
[20] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1), 2017.
[21] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Shirty-Second AAAI Conference on Artificial Intelligence, 2018.
[22] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 1-13, 2019.
[23] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3693-3702, 2017.
[24] Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pages 37-45, 2015.
[25] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1263 1272. JMLR. org, 2017.
[26] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
[27] Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio.
Graph Attention Networks. International Conference on Learning Representations, 2018.
[28] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph con- volutional networks on node classification. In International Conference on Learning Representations, 2020.
[29] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarahayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages
PMLR. 10-15 Jul 2018.
[30] Johannes Kliepera, Alcksandar Bojchevski, and Stephan Günnemann. Combining neural networks with per- sonalized pagerank for classification on graphs. International Conference on Learning Representations, 2019.
[31] Uri M Ascher. Numerical methods for evolutionary differential equations. SIAM, 2008
[32] Benjamin Paul Chamberlain, James Rowbottom, Maria Gorinova, Stefan Webb, Emanuele Rossi, and Michael M Bronstein. Grand: Graph neural diffusion. arXiv preprint 2rX1v:2106.10934 2021.
11

---

[33] E Weinan. A Proposal on Machine Learning via Dynamical Systems. Communications in Muthematics and Statisticx, 5(1):1-11, March 2017.
[34] Pratik Chaudhari, Adam Oberman, Stanley Osher, Stefano Soatto, and Guillaume Carlier. Deep relaxation: partial differential equations for optimizing deep neural networks. Research in the Mathematical Sciences, 5(3):30, 2018.
[35] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning (ICMI.), 2018.
[36] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary diffcrential equations. In Advances in Neural Information Processing Systems, pages 6571-6583, 2018.
[37] Eldad Haber, Keegan Lensink, Eran Triester, and Lars Ruthotto. Imexnet: A forward stable deep neural network. arXiv preprint arXiv: 1903,020339,, 2019.
[38] Jonathan Fphrrth, Moshe Fliasof, Lars Ruthotto, Eldad Haber, and Fran Treister. Leanconvnets: Low-cost yet effective convolutional neural networks, IEFE Journal of Selected Topics in Signal Processing, 2020.
[39] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 16631))15334-1554
2019.
[40] Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric-symbolle hybrid deep network. Journal of Computational Physics, page 108925, 2019.
[41] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli. Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXiv:2006.09553 2020.
[42] Filipe de Avila Belbute-Peree Thomas D. Economon, and J. Zico Kolter. Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction. International Conference on Machine Learning (JCML), 2020.
[43] Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. International journal of computer vision, 22(1):61-79, 1997.
[44] Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1):259-28) 2228-268,, 1992.
[45] Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The wave kemel signature: A quantum mechanical approach shape analysis. In 2011 IEEE international conference computer vision workshops (ICCV workshops), pages 1633. IEEE, 2011.
[46] Hyman, J Morel, M Shashkov, and Stanly Steinberg. Mimetic finite difference methods for diffusion equations. Computational Geosciences, 6(3):333-33 352, 2002.
[47] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 1912-1920, 2015.
[48] Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 93, 2008
[49] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rX2v:1412.69900 2014.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bis Buc. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[51] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA, June 2014. IEEE
[52] Ross T Whitaker. A level-set approach to 3d reconstruction from range data. International journal of computer vision, 29(3):003-31,, 1998.
[53] Zhilin Yang. William Cohen, and Ruslan Salakhudinov. Revisiting supersupe learning with graph embeddings. In International conference machine learning, pages 40-48. PMLR, 2016.
[54] Hongbin Pei, Bingzhe Wei, Kevin Mhen-Chuan Chang, Yu Lei, and Bo Yang. Geom gen: Geometric graph convolutional networks. In International Conference Learning Representations, 2020.
12

---

[55] Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-Scale Attributed Node Embedding. Journal of Complex Networks, 9(2), 2021.
[56] Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 942-99999 950. PMLR. 10-15 Jul 2018.
[57] Jiani Zhang, Xingjian Shi, Junyuan Xic, Hao Ma, Irwin King, and Dit- Yan Yeung. Gaan; Gated attention networks for learning large and spatiotemporal graphs. In Proceedings of the Thirty- Fourth Conference Uncertainty in Artificial Intelligence, pages 339-349, 2018.
[58] Ziqi Liu, Chaochao Chen, Longfei Li, Jun Zhou, Xiaolong Li, and Le Song. Geniepath: Graph neural networks with adaptive receptive paths. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 02 2018.
[59] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019.
[60] Davide Boscaini, Jonathan Masci, Emanueic Rodolà, and Michael Bronstein. Learning shape correspon- dence with anisotropic convolutional neural networks. 05 2016.
[61] Or Litany, Tal Remez, Emanuele Rodolà, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. pages 5660-5668, 10 2017.
[62] Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Our theorems state that the
activation function must be sign-preserving bounded and monotone.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We did not find any societal impacts of the work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are provided in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and the links to the (public) data sets are provided in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "lsWSWsCvNb8", "reviewer": "Reviewer_Buis", "summary": "This paper proposed using discretized PDE operator on the graph to generalize current graph convolutions. With diffusion and hyperbolic equations, the proposedd method has the flexibility to fit the unseen graph signals.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed discretized PDE to generalize graph convolution. \n\n### Motivation \n- The used PDE include non-linear diffusion and hyperbolic. However, the justification for using such two equations is not clear. (Why using these two rather than the other first/second-order PDE?)\n- several related works are missing, like _Combining differentiable PDE solvers and graph neural networks for fluid flow prediction_, ICML 20 _Continuous Graph Neural Networks_, ICML 20\n\n### Technical analysis\n- Some related work generalized discretized graph convolution into continuous operators. This work generalized graph convolution as discretized operators. In theory, continuous operators should be more flexible than being discretized. So this methodology may need to be discussed. \n- If the motivation of using diffusion and hyperbolic is to adapt unseen data, especially conserving and/or smoothing the signals. Then many existing works regarding solving over-smoothing have similar logic, such as _DeepGCN_, ICCV 19.\n\n### Experiment\n- The node classification didn't show significant and consistent superiority in performance (many baselines take the lead) \n- The protein-protein interaction task showed that the proposed method is among the best, but it is still not the best.\n- In Table 7, the proposed method seems also has an over-smoothing issue: performance decrease as layers go deeper", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Justify the motivation of the proposed method over the related works, including continuous PDE, ResNet for GCN, etc.\n- Many graph data are suitable for diffusion since the signals are in low-frequency. It's better to use proper datasets to show the advantage of the proposed method, such as a mix of high and low-frequency.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "b5rber2jRg", "reviewer": "Reviewer_zFMQ", "summary": "The authors propose a new class of graph neural networks motivated by numerical methods for solving PDEs on manifolds. The idea is that there is a correspondence between specific GNNs (e.g., GNNs that learn the weight of edges such as GATs) and discretized PDEs on manifolds. Here, the discretization is spatially. For time (i.e., the depth of the GNN) one can use numerical solvers for PDEs similar to what was proposed by the original Neural ODE paper. The main motivation is that there are types of PDEs that do not suffer from the oversmoothing problem and, therefore, that their discretized versions might be a better fit for some problem domains where oversmoothing is a problem. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all I want to say that the idea of drawing connections between PDEs/ODEs and GNNs as spatially discretized PDEs is really interesting. I also think that the experiments are comprehensive and while the improvements are marginal at best, they still validate the idea of relating PDEs on manifolds and GNNs as one that can lead to interesting new architectures. \n\nUnfortunately, I have some concerns when it comes to positioning the work in the context of existing methods. There are several papers that should be discussed here:  \n\n[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In NeurIPS, 2020b.\n\n[2] Belbute-Peres, F. d. A., Economon, T., and Kolter, Z. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In ICML. PMLR, 2020\n\nI would ask the authors to discuss the differences in their rebuttal. Especially the latter seems highly related but perhaps more focused on a particular dynamical system? Moreover, there is an ICML 2021 paper that showed up on arxiv about a month ago:\n\n[3] https://arxiv.org/abs/2106.10934\n\nWhile [3] appeared on arxiv *after* the NeurIPS submission deadline and therefore, is  not a reason to reject the current submission!, it would still be beneficial to write a few sentences outlining the differences and similarities. One difference I see is the way the edge weight matrix is modelled (in the submission denoted by W and modelled as the geometric mean of the degree of adjacent nodes, in [3] denoted by A and modelled as a GAT). There are also differences in the numerical solvers used. \n[3] also compares to several other *ODE* based graph neural networks which, at least, should be discussed in the related work section of the submission.\n\nAgain, I don’t think there is a problem with several papers coming out at roughly the same time with roughly the same ideas, as it often points to an important topic ripe for dissemination. But please make sure you end of discussing prior work (even if only motivated by ODEs) on combining differential equations in some form with GNNs. \n\nBesides the related work section, the one area where I see a lot of room for improvement is the presentation. For instance, the order in which you introduce parts of the paper seems odd and counterintuitive. For instance, you state two theorems on the third page. At this point, however, they seem out of context and, in their statement, use terms such as PDE-GCN that have not been introduced before. Either you state these theorems as standalone background theory about PDEs on manifolds, or you first introduce the concepts needed to understand their premises.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that the limitations of the paper are sufficiently discussed since \"Our theorems state that the activation function must be sign-preserving, bounded and monotone.\"\n\nWhat I am wondering about: what about efficiency of the method? What about the dependence on defining W the way you did?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "H9u80rJFKMk", "reviewer": "Reviewer_Pd3f", "summary": "Motivated by the connections between PDEs and conventional convolutional networks, this paper proposes the view of graph convolutional networks (GCNs) as discretizations of PDEs on manifolds. Using this view, the authors propose a a family of graph convolutional architectures that utilize discrete graph operators as their basis and argue that this approach can address existing issues with graph convolutional networks, such as over-smoothing. Moreover, the authors also argue that this approach can have a more even performance accorss a diverse range of tasks, instead of being specialized to only a specific type of problem.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is presented clearly and the paper is easy to follow and understand.\n\nThe authors present an analysis of the two types of dynamics (diffusion and hyperbolic) they propose as a basis for their graph convolutional architecture. They connect this analysis to the issue of over-smoothing and the different performance of architectures on different type of tasks. It is this analysis that forms the basis for the proposed architecture\n\nThe empirical evaluations presented in the paper are comprehensive, including diverse relevant baselines. The proposed method does perform well across a variety of tasks, either matching or surpassing existing methods.\n\nNevertheless, even though the choice is argued as being motivated by the characteristics of the task at hand, the authors do have the advantage point of proposing two methods and having the possibility of choosing between them for the convenience of the results. Since a mixture model was proposed and evaluated on a few tasks, it would be interesting to know if that single \"unified\" model is able to maintain the performance of the separate models in the cases where one significantly outperforms the other.\n\nAs a final remark, it would be interesting if the authors could expand on the comment on line 229 (\"We argue that the success of our network stems from its capability of a priori extracting features from graphs, due to its utilization of discretized differential operators and PDE guided construction\"). What specifically in the formulation allows it to better extract information a priori? A better discussion, even if not possible to completely ground it in theory of empirical results, would be important when making such a claim.\n\nOverall, despite some of the comments above, the paper does present an interesting analysis and a good empirical evaluation with good results on important datasets when compared to standard existing methods. \n\n-------------------\n\nUpdate post authors' response:\n\nThe reported experiments with the \"unified\" model strengthen the experimental evaluation, by showing that the results are not a consequence of arbitrary model-task combinations. \n\nOverall, I originally had only small issues with this paper, and these were addressed by the authors. My previous assessment already reflected the evaluation that \"the paper does present an interesting analysis and a good empirical evaluation with good results on important datasets\". The authors' response confirms my positive evaluation and I will thus maintain my evaluation and score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the potential negative societal impact of their work.\n\nAs the authors state, they do address some limitations, in the form of the conditions for their theorems to apply. Further discussion of limitations (e.g. practical aspects) might have been of interest, but is not a significant detriment to the paper. The empirical demonstration in Section 4.2 that this argument is reflected in practice (when a mixture model is given a choice of which type of dynamics to use for different types of problems) is an interesting connection to the theoretical analysis.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "I do not review my assigned papers sequentially and thus have not estimated how much time was spent on each paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All ratings were (weak) "accept". 

Generally the work is novel and timely. A related paper appeared on arxiv in late June (GRAND), which one reviewer pointed to, but obviously this paper was submitted before that appeared, and doesn't factor into this paper's evaluation (as the reviewer appropriately recognized). To me this supports the relevance and importance of this work. My sense was that the reviewers' ratings were low given the feedback they provided, and the authors addressed the reviewers thoroughly and appropriately, causing one to raise their score.

If accepted, my suggestion to the authors is that they do a careful revision of the text to emphasize the broader scope and applicability of this work, in case some of the reviewers' 6 ratings were due to not clearly seeing connections to other areas.
Decision: Accept (Poster)