On the Robustness of Graph Neural Diffusion to Topology Perturbations
Authors: Yang Song
QIYU KANG
Sijie Wang
Zhao Kai
Wee Peng Tay
Venue: NeurIPS 2022 Accept (2022)

Abstract: Neural diffusion on graphs is a novel class of graph neural networks that has attracted increasing attention recently. The capability of graph neural partial differential equations (PDEs) in addressing common hurdles of graph neural networks (GNNs), such as the problems of over-smoothing and bottlenecks, has been investigated but not their robustness to adversarial attacks. In this work, we explore the robustness properties of graph neural PDEs. We empirically demonstrate that graph neural PDEs are intrinsically more robust against topology perturbation as compared to other GNNs. We provide insights into this phenomenon by exploiting the stability of the heat semigroup under graph topology perturbations. We discuss various graph diffusion operators and relate them to existing graph neural PDEs. Furthermore, we propose a general graph neural PDE framework based on which a new class of robust GNNs can be defined. We verify that the new model achieves comparable state-of-the-art performance on several benchmark datasets.

On the Robustness of Graph Neural Diffusion to Topology Perturbations
Yang Song"
Qiyu Kang"
Nanyang Technological University C3 AI
Nanyang Technological University kang00800e ntt. edu. sg
yang song@c3. ai
Sijie Wang' Nanyang Technological University wang16798e ntu. edu.
Kai Zhao
Nanyang Technological University kai zhao@ntu. edu. sg
Wee Peng Tay Nanyang Technological University #ptayEntu. edu.sg
Abstract
Neural diffusion on graphs novel class of graph neural networks that has attracted increasing attention recently. The capability of graph neural partial differ- ential equations (PDEs) in addressing common hurdles of graph neural networks (GNNs), such as the problems of over-smoothing and bottlenecks, has been in- vestigated but not their robustness to adversarial attacks. In this work, we explore the robustness properties of graph neural PDEs. We empirically demonstrate that graph neural PDEs are intrinsically more robust against topology perturbation as compared to other GNNs. We provide insights into this phenomenon by exploiting the stability of the heat semigroup under graph topology perturbations. We discuss various graph diffusion operators and relate them existing graph neural PDEs. Furthermore, we propose a general graph neural PDE framework based on which a new class of robust GNNs can be defined. We verify that the new model achieves comparable state-of-the-art performance on several benchmark datasets.
Introduction
Deep learning on graphs and Graph Neural Networks (GNNs), in particular, have achieved remarkable success in variety of application areas such as those related to chemistry (molecules), finance (trading networks) and the social media (the Facebook friend network) [1-5]. GNNs have been applied to various tasks including node classification [3], link prediction [6]. and recommender systems [7]. The key to the success of GNNs is the neural message passing scheme [8] where messages are propagated along edges and optimized toward downstream task.
While aggregating neighboring nodes' information is powerful principle of representation learning, the way that GNNs exchange information between nodes makes them vulnerable to adversarial attacks [9]. Adversaries can perturb a graph's topology by adding or removing edges [10-12] or by injecting malicious nodes to the original graph [13-15]. Another common attack is to perturb node attributes [9, 16-18]. Our paper will mainly tackle graph topology perturbation. Adversaries who can
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

inject nodes to the original graph while not modifying the original graph directly are called injection attacks [13-15]. Adversaries who can directly modify the original graph including edges and node features are called modification attacks [10-12, 9, 16-18). To defend against adversarial attacks, several robust GNN models have been proposed. Examples include RobustGCN [19], GRAND [20], and ProGNN [21]. In addition, pre-processing based defenders include GNN-SVD [22] and GNNGuard [23].
Recent studies [24-27] have applied neural Ordinary Differential Equations (ODEs) [28] to defend against adversarial attacks. Some works like [24, 27] have revealed interesting intrinsic properties of ODEs that make them more stable than conventional convolutional neural networks (CNNs). Neural Partial Differential Equations (PDEs) have also been applied to graph-structured data [29, 30J. Some papers like [29] (GRAND) and [30] (BLEND) approach deep learning on graphs as continuous diffusion process and treat GNNs as spatial discretizations of underlying PDE. However, robustness to adversarial attack has not been studied on such graph neural PDEs. In this work, we demonstrate that graph neural PDEs are intrinsically more robust against adversarial topological perturbations compared to other GNNs. We investigate the heat diffusion show that the diffusion process is essentially stable under small perturbations of the manifold metric. In doing we provide insights into the underlining reasons why graph neural PDEs are stable under graph topological perturbations. Such insights indicate that further improvements to the design of graph neural PDEs are possible. We develop several such improved models under a general graph neural PDE framework and show that these models are also robust to node attribute perturbations.
general Riemannian manifold and
Main contributions. In this paper, our objective is to develop a general diffusion framework on graphs and study the robustness properties of the induced graph neural PDEs. Our main contributions are summarized as follows:
We review the notion of heat diffusion on Riemannian manifolds and the stability of its semigroup. We present analogous concepts of gradient, divergence, and Laplacian operators for heat diffusion on a graph.
We generalize heat flow to more general flow schemes, including mean curvature flow and Beltrami flow, which are able to preserve inter-class edges in diffusion. A novel class of graph neural PDEs is thereby induced.
We show that the proposed graph neural PDEs are intrinsically robust to graph topology per- turbations. We verify that the new model achieves comparable state-of-the-ar performance several benchmark datasets.
The rest of this paper is organized as follows. In Section 2, we start with preliminaries on continuous diffusion over a Riemannian manifold and the discrete graph. In Section 3, we present our main results on the stability properties of heat flow graphs and generalize heat flow to edge-preserving flows from which a new class of graph neural PDEs is proposed. Our proposed model architecture is detailed in Section 4. We summarize experimental results in Section 5 and conclude the paper in Section 6. The proofs for all theoretical results in this paper are given in the supplementary material, where more experiments are also presented.
2 Preliminaries
Similar to [29,30) we consider graph discretization of Riemannian manifold. We introduce concepts and notations for flows diffused over general manifold and a discrete graph. Readers are referred to [31] for more details. The stability of the heat kernel and heat semigroup of the heat diffusion equation under perturbations of the manifold metric is considered. This discussion sheds light on the diffusion stability of graphs under perturbation of the graph topology in the next section.
2.1 Heat Equation and Solution Stability on Manifold
Riemannian manifold (M, .) is a smooth manifold M endowed with a Riemannian metric g. where the norms of tangent vectors and the angles between them are defined by an inner product. It the natural generalization of Euclidean space and correspondingly the divergence, gradient, and I Laplace operators are well-defined, analogous to their corresponding concepts in Euclidean space. Let Gg (M) denote the space of smooth functions on M with compact support and o the composition operation.

---

Definition 1. The Laplace operator A (M) H (M) on d-dimensional Riemannian manifold (M, g) defined as
= div
where div is the divergence defined for the Co vector on M and V is the gradient operator. In local chart U with coordinates (x* x² ,...,"")) ). function f and vector field v, we have
(V
= 9 j=1
Of dry
div v
à
(Vdot gol))
(1)
det g des
j=1
where vi is the j-th component of v. ged the components of the inverse metric of g, and det is the determinant operator of a matrix. We also have
a det g Oz
(Viet ggو
Oz3
The classical Cauchy problem associated with the heat diffusion equation to find a function 2t), C (R+ x M) such that
{
0e Ap 1>0,
(3)
P =
where we only consider e C8 and R+ the space of positive real numbers.
We can extend the Laplace operator to the generalized Dirichlet Laplace operator, denoted as C = A and defined on larger domain [31, section 4.2]. This operator is self-adjoint and non- negative definite on L (MM,, and can be shown that the above problem (3) is solved by means of the following family {Pi)220 of operators:
Pt Pf:=
Japred f.
e
22 dEA = S - dE
(4)
where { Xespecc in (MM is the unique spectral resolution of the Dirichlet Laplace operator and spec& is the spectrum ser of C. The family (Pth220 is called the heat semigroup associated with A, and the solution is given by Ptf. It is also well-known that the solution the above problem has integration form via the (minimal) heat kernel function (2 v) R M x M + [31, Theorem 7.13]:
Pzf(x)) = =J ke (y) dp(v).
where du is the Riemannian measure [31, Section 3.4] on M.
For a Riemannian manifold M, different metrics g give rise to different structures. Let Â be the Laplacian associated with another metric ğ on M such that, for some 2 1,
S g S ag.
The new Á is said be quasi-isomerie [32] g, and can be viewed as a perturbation of the metric g. A can be also viewed as uniformly elliptic operator [32] with respect (w.r..) to g. The stability of the heat semigroup and the heat kernel under perturbations of the Laplace operator (i.e., changes of g) well studied in [33]. the special case of uniformly elliptic operators on Euclidean manifold Rd the uniformly elliptic operator can be written div (AV) with symmetric measurable coefficients A(æ) = (a., (z)) such that Σn, a:y (z)(2) 2 > for all & E Bd and for some constant À > 0 independent of æ. The reference [33] shows the following theorem:
Theorem 1. On the Euclidean manifold IR", let (P:)e2 and [Pth2o be two diffusiion semigroups
on IR. (d 2 2) associated with uniformly elliptic operators div(AV) and div(AV) with symmetric measurable coefficients A(x) (ai)(x) (x)) and A(x) (āzz(x)), respectively. The corresponding heat kernels are denoted by po (x. y) and px(x, u). We then have that
2L is defined on the larger Sobolev space W [31], and is the (negative) extension of A. We have - C

---

There is bounded, piecewise continuous function Fi(t, on F x R with limz-10 Fi Ft, for each on d and such that
0 and constant both of which depend only
(- (_1xx-
(-1mm-yl U113 ct
Pe(a, y) pr(2, v) I st-d/² cxp
Fi (1, AllA-AA All
for any (t, 2, v) E R x R x R || 2l2 is the Euclidean norm, and || I1ee is the local LL-norm distance defined in [33].
it. Furthermore, we also have an L -P-operator norm bound for Pt Pt in terms of the local I.2 n,2-norm distance between asj and āzy. There bounded, piecewise continuous function (2 z) on R+ Iet with limz-0 F2 2) = 0 for each t O that depends only on d and a, such that for any p E [1, 00), we have
Pt 1.
S F2 (t.11A- Allezo.)
The above theorem shows the pointwise convergence of the heat kernel under perturbations of the matrix A. It proves the stability of the solution under small perturbations of the Laplace operator. These results can also be extended to general Riemannian manifolds [34] since every Riemannian manifold be isometrically embedded into some Euclidean space [35] and the kernel is isometrically invariant [31, theorem 9.12]. We therefore know that if the difference between A and A (or more generally 9 and g) are small, the final solutions to (3) have small difference. An intuitive explanation of the stability of the solution comes from the fact that the heat kernel ky (ar, y) is the transition density function of a Brownian motion on the manifold [36] since Brownian motion can be viewed as a diffusion generated by half of the Laplace operator. This means that 4e (ar, y) weighted average over all possible paths between x and y at time f, which does not change dramatically under small perturbations of g. More specifically, for a Brownian motion on a domain M, if perturb the g over a subset P C M then only the paths passing through P will be affected.
If go one step further from the uniformly elliptic operators div(AV) defined in Theorem by extending A(x) to A(x, t, s,, Vi), we get the following general quasilinear parabolic equation:
{
0g div(A(r, sp Vy)), t + =
(5)
where the principle part A(x, f, s,, Vip) is equipped with additional structure conditions as shown in [37]. When A(z,t, s,, Vi) A(x, is time-dependent diffusion process, analogous bounds to Theorem are shown in [38]. leave the analysis of general A(x, L, sp, Vip) for future work.
2.2 Gradient and Divergence Operators on Graphs
We consider graph discretization of a Riemannian manifold. More specifically, consider an undirected graph
(V.E) consisting of finite set of vertices, together with a subset & of edges. A graph is weighted when it is associated with a function w : + R which is symmetric, i... w[[u, u)) = w(( [[v, u)), for all [u, e] E.
Let H(V) denote the Hilbert space on V of real-valued functions with the inner product defined (a, b)() = Ever (())(),) for all a, b E H(V). Similarly, ((,,,)))) u)), for all e, H(E). We next define the gradient and divergence operators on graphs analogue the general continuous manifold defined in Definition I as follows [39].
define a Hilbert space H(E)
with inner product (c, d)H(S)) Ei, o([u, v))d((w [M,
Definition 2. Given an undirected graph G (V,E), we define the the following:
The graph gradient is operator V +H(V) + H(E) defined by
Vip(lu,)) = (ulu w))
(p()) -v Nollu, w))
plu), V [u, e E
(6)
h(v))
h(u)
where h(v) Eos,ee w(lu, u)) is the degree of node

---

ii. The graph divergence is operator div M(EE) + H(V) defined by
w(u, vl)
(div ()(
E
(v>(1, ul) vi( m]))
h(w))
[u,ajEE
ii.. The graph Laplacian is operator A H(V) + H(V) defined by
A = divivVVp))
(8)
2
The graph Laplacian defined above identical to the normalized Laplacian matrix, i.e.,
A = D -112(/ W)D 1/2
(9)
where D is diagonal matrix with D(v, u) h(v), and W is the adjacency matrix satisfying W(u, v) = w((u, uI) if [u, r| and W(u, v) = 0 otherwise. Note that (8) we have included the negative sign. The analogue of the graph Laplacian A in Section 2.1 is the Dirichlet Laplace operator L. Note the use the same notation A for graphs. The manifold Laplace operator and graph Laplacian A will be apparent from the context.
3 Neural Diffusion and Stability on Graphs
We now consider neural diffusion on graphs by making use of concepts from Section 2. Various parabolic-type equations on graphs are studied in this section. The solution stability against adversarial attacks linked to the stability of the solution for the heat diffusion equation under perturbation of the manifold metric, introduced in Section 2.1. The general framework we consider is based on parabolic- type equations on graphs:
Op(u,t) div(A(u, V()),,
(10)
with pu, 0) being the initial node attribute at node u and where A(u, t, sp, Vip) can take different forms. We next provide some examples.
3.1 Continuous Diffusion on Graphs
Definition 3 (Heat diffusion). The heat diffusion on graphs is defined by
@e(u,,) t) 1 divv(V)))(u,))
(11)
Ōt
Definition 4 (GRAND/BLEND [29, 30]). According to [29, eq (1)] and [30, eq (7) and (9)], the GRAND/BLEND flow is defined by
@e(uu,t) 1
(u,,
(12)
Ōt 2
where V, is an adaptive Laplace operator (with possible graph rewiring) depending on the evolved node feature ip(,tt).
The gradient operator defined in GRAND/BLEND assumes constant edge weight. In Definition 4 the weight function w([u, v|) defined in (6) is incorporated into the gradient definition, so it can be
absorbed the time-dependent term V. Note that BLEND degenerates to GRAND [29] when there no positional encoding. In this paper, for fair comparison, do not use positional encoding for all GNNs.
Analogous to [40], we can define mean curvature flow and Beltrami flow follows.
Definition 5. The mean curvature diffusion on graphs is defined by
a<(u, -iv
VpP
(u, t),
(13)
Øt
|
1 div Vi [[vi)]] is a discrete analogue of the mean curvature operator, ||Vpl|
where -2
1/2 and [V 1)ll = (Eeou)ee (Ve((u, o),1)) 1/2 H(E)
PV 1/2
5

---

Definition 6. The Beltrami diffussinn on graphs is defined by
Ōp(u, t) 1
V
div
| (u,
(14)
01 2 V
Intuitively, the term [Vp(u, t)]l, which appears in (13) and (14) but not in (11), measures the smoothness of the signals in the neighborhood around vertex u at time t. The diffusion using (13) or (14) vertex is small when [V,,,,, I)] is large, i.e., signals are less smooth around vertex Hence, mean curvature flow and Beltrami flow are able to preserve the non-smooth graph signals. This phenomenon is visualized in Fig. 1, where mean curvature flow and Beltrami flow capable of preserving inter-class edges whose weights are much larger than those in heat flow.
Figure 1: Effect of using different diffusion equations visualized using t-SNE with attention weights. Left: heat flow, middle: mean curvature flow, right: Beltrami flow. The darker the edge, the larger the attention weight.
3.2 Stability against Graph Topology Perturbation
Attackers can modify the original graph by adding or removing edges [10-12] and perturbing node attributes. Adding or removing edges leads to a different graph Laplacian A defined in (8). Consequently, the solution to (11) is altered. Using the language of Section 2. the edge perturbations correspond to the perturbation of the metric (or equivalently A in Theorem 1). According to Theorem the stability of the heat semigroup and the solutions under small perturbations of the Laplace operator is guaranteed. Consequently, the solution is not affected significantly by the edge perturbations. This is different from the notion of stability in [20], where the output p(u, t) is shown to be stable after perturbation of node attributes (pu 0) with fixed graph edges. The stability studied in [20] called Lyapunov stability, which is addressed in Proposition These two notions of stability combined lead to graph neural PDE in Section 4 that is robust against both edge and node attacks.
The following result shows the stability of the solution of (11), i.e., 0p(u.t) - Av, !), under graph topology or Laplacian perturbation. the change in the graph topology with respect to any matrix norm is small, the semigroup perturbation can be bounded similarly as the result in Theorem 1. This result considers only the case where the Laplace operator is time-invariant. The time-variant analogy of [38] discussed after (5) is provided in the supplementary material.
Proposition 1. Consider A D-1/2(D W)D 1/2 in (9) and suppose Á -
W)D 1/2 with D being the diagonal degree matrix defined analogously to (9r W W W+D E. Suppose = Ell = o(1). for a matrix norm ||I-II,, and and are non-singular. Then, || t)
G(u, ()]l (e))
3.3 Neural Flows on Graphs
Substituting (6) and (7) into (14), and ignoring the degree variable h(u) for simplicity, we propose the neural Beltrami flow as
tp(u,t)
w[[u, w))
(ivo(u)) + ( (v(u) =(v)),
(15)
01 2 |ilill
where [v(l = v (V. 2.))a. Here, we further assume (p(u), for all nodes u are time-independent for the sake of simplicity, Suppose | The function ;p V + R maps each

---

node to a feature vector. Stacking all the feature vectors together, we obtain Z E Raxd Let the weight function w([u [u, vI) be the scaled dot product attention function [41] given by
v|) softmax (W ((W.)
w((w
(16)
Vdk
where WK and We are the key and query learnable matrices, respectively, and dg denotes the number of rows W k has. If multi-head attention is applied and denote A,, as the attention matrix associated with head hi then (Z) = ,. >. A (Z) with A, (u, v) = wn((u, e!) where wn the weight function for head h. The diffusion equation (15) can be compactly written in matrix form as
(A()(t)) (t) V (Z(t))) (tt)
(17)
where @ denotes element-wist multiplication, B v) softmax
1 [[
+ (لcاio
and (Z))) is diagonal matrix with (u, u) Σ, (A B) (u, v).
Similarly, the diffusion equations using mean curvature flow (13) and heat flow (11) can be written as:
OZ(() = (A(Z()) @ B(Z()) v (Z(()) Z()).
(18)
ōt
where Bu,, r) softmax
[Volao] + [[v(o)]
1
and
OZ(1)
Øt (A(Z())) I)Z(
(19)
respectively. Although the BLEND model in [30] is inspired from Beltrami flow, its final formulation (equation (8) in [30] where Ztt) contains positional and node feature embeddings) is indeed (19),
i.c., heat flow using attention weight function.
Proposition Diffusion equations (17), (18) and (19) are all Lyapunov stable [42].
4 Model Architecture with Lipschitz Constraint
In this section, we propose a general graph neural PDE network based on the discussion in Section 3. A layer of graph neural PDE is illustrated in Fig. 2. Since may stack up multiple such layers, the diffusion is performed in a hierarchical manner, where the high-dimensiones features are diffused over the graph at the front layers and the low-dimensional features are diffused at the back layers.
One important ingredient in our model that we perform spectral normalization on WK k and Wo Q in (16). This is motivated by the fact that attention models have poor performance when the depth increases. In other words, attention weights tend to be uniformly distributed for excessive message exchanges. This phenomenon becomes even more obvious when the nodes' features are diffused according to (17), (18) or (19) as solving such a PDE normally requires many discrete steps. To overcome this over-smoothing problem, we utilize the strategy proposed in [43] to enforce Lipschitz continuity by normalizing the attention scores. Besides, enforcing Lipschitz continuity can also help improve the robustness of the model because the Lipschitz constant controls the perturbation of the output given bounded input perturbation. Fig. illustrates that attention weights become overly smooth if no spectral normalization is applied.
7-1
Z
ΦZ 11
Z(0)
OZ(() dt
QZ(()
Z' Z(T)
Figure 2: Graph neural PDE at the i-th layer, where each node's feature vector are linearly transformed before being difffused over the graph. We have Q (((())))B)(()))))))))) flow and Beltrami flow, while Q = A(((()) for heat flow.
for mean curvature
Experiments
In this section, we compare graph neural PDEs under different flows to popular GNN architectures: GAT [41], GraphSAGE [45], GIN [46], APPNP [47], and the state-of -h- GNN defenders: Robust- GCN [19], GNNGuard [23], GCNSVD [22), on standard node classification benchmarks. In our ex- periments, 3 we use the following datasets: Cora (citation networks) [44], Citeseer (citation networks)
Our experiments
GeForce RTX 3090 GPU.
7

---

Figure 3: Impact of applying spectral normalization visualized using t-SNE with attention weights using Cora dataset [44]. Left: graph neural PDE with no spectral normalization, right: graph neural PDE with spectral normalization. The darker the edge, the larger the attention weight.
[48] and PubMed (biomedical literature) [49]. We use refined version of these datasets provided by [50]. We refer the readers to the supplementary material for more details. Our experiment codes are provided in https: /tttps///ittub ctt./cccccccai http://oooouootssssssseeeee of of -oraaaph---rrr-hh---- G
Table I: Node classification accuracy (%) on adversarial examples using different GNNs. The implicit Adam PDE solver with step size is used for Beltrami. We denote those experiments that are computationally too heavy to run by "-", The best and the second-best result for each criterion are highlighted in red and blue respectively.
Ditaset
Attack
Beltrami (17)
RobustGINN
GNNGGurd
GCNSVD
GAT
GintiSAGEE
GIN
APPNP
cloan
75.93 + 1.46
81.34 +: 0.66
79.44 + 1.18
09.28
79.74 1.59
76.25 + 1.52
76.79 + 1.35
83.06 + 1.06
SPEIT
61.87
36.16
78.50 2:27
31.50 0,74
38.10
35.82
0.01
35.82
0.01
36,79 4: 0.61
40.77
70.14
69 1.83
66.93
1,31
70:75 4: 0.86
Cileseer
SPRIT
66.46
28.56
69.73:1:
21
1.32
11.14
19:75 1.82
23.54 5,30
23.09 0.86
TDGIA
65.77 +: 1.28
38.81
10.84
1: 1.56
20.77 2,52
19.63 1: 6.53
28.77
7,73
28.65
5.08
$4.48 11 8.56
cinon
86.94 + 0.25
75.55 0.30
84.30 0.51
$4.91 0.76
89:22 = 0.25
76.71 1: 0.14
77:50 0.54
PubMee
SPEIT
86.66
0.68
75.54 0.54
84.36 +: 0:58
40.94 2.47
30,22 # 0.00
76.71 # 0.14
77.55 4 0.54
TDGIA
85.56
84:00 +
39.78
11.23
+ 0.68
5.1 Attack Setup
We apply the setup introduced in the graph robustness benchmark (GRB) [50]. Based on the assump- tion that nodes with lower degrees are easier to attack, GRB constructs three test subsets nodes with different degree distributions. According to the average degrees, GRB defines these subsets of nodes as Easy, Medium, Hard, or Full. In the Easy subset, attacks are easy to succeed and hence the worst performance is expected. The remaining nodes are divided into train set (60%) and val set (10%). for training and validation respectively. In our experiments, we choose the Easy subset, i.c., the most challenging mode.
Following GRB, in this paper, we mainly consider the following cal-world adversarial attack settings: Black-box: The attacker does not know the defender's 's method and vice versa. Attackers firstly attack a pre-trained GCN [3] and then transfer the perturbed graphs to the target model. Evasion: Attacks will only happen during the inference phase. Inductive: GNNs are used to classify unseen data (e.g.. new users), i.e., validation or test data are unseen during training. Injection: Attackers can only inject new nodes but not modify the target nodes directly. This reflects applications like in online social usually hard to hack into users' accounts and modify their profiles. However, is casier to create fake accounts and connect them to existing users. For other attack settings such as white-box attacks and modification attacks, relevant experiments and discussions are provided in the supplementary material.
networks where
Table 2: Node classification accuracy (%) adversarial examples using BeltramiGuard.
Model
Attack
Cora
Citeseer
PubMed
clean
73.01 + 2.01
69.90 + 0.44
87.77 + 0.14 87.68 + 0.03
BeltramiGuard
SPEIT
73.01 + 2.62
69.90 + 0.44
TDGIA
72.14 + 1.20
69.90 + 0.44
88.16 0.61

---

We apply two ofate-of-the-an art injection attack methods: SPEIT [14] and Topological Defective Graph Injection Attack (TDGIA) [15]. These two attacks are the two strongest attacks reported in the GRB. For both attacks, both the maximum allowable injected nodes and the maximum allowable injected edges are set to 50 for Cora and Citeseer, and 300 for PubMed. We provide more details about how these two attacks work under evasion, black-box, and injection setting in the supplemental material.
Table indicates that graph neural PDE induced from Beltrami flow more robust than all other GNNs except for GNNGuard, which is specifically designed to remove malicious edges and is thus robust against injection attacks. This suggests that the output features from a graph PDE are stable under topology perturbations, as suggested by Proposition 1.
Furthermore, the heat diffusion process on manifold tends to diffuse differently under different geometries. For example, it diffuses slower at points with positive curvature, and faster at points with negative curvature [34, 51]. In [52] and [53], the authors show different datasets have different geometric properties like hyperbolicity distribution or Balanced Forman curvature. Our theoretical analysis only shows loose uniform bound in terms of the adjaceney matrix, while the performance is very much dataset-dependent. More advanced theoretical analysis for different datasets is highly non-trivial and needs further investigations.
Our graph neural PDE can be combined with GNNGuard. We denote this model as BeltramiGuard. Table 2 demonstrates that BeltramiGuard renders attacks in vain and exceeds GNNGuard on the Citeseer and PubMed datasets. Note from Table that the vanilla Beltrami model already surpasses GNNGuard on the large PubMed dataset. More details provided in the supplementary material.
5.2 Ablation Studies
Diffusion Schemes
We compare different diffusion equations: heat flow (19) where w[[u, v)) for all uu. E, GRAND/BLEND [29, 30] which is another heat flow where w(uu u|) in (19) is the attention function defined in (16), mean curvature flow (18), and Beltrami flow (17). For GRAND/BLEND, we stack three neural PDE layers using the architecture proposed in Fig. 2 since the original GRAND/BLEND in [29, 30], which has only one PDE layer, does not perform well.
Table Node classification accuracy (%) on adversarial examples using graph neural PDEs induced from different flows, where implicit Adam PDE solver with step size 2 is used.
Dataset
Attack
Heat
GRAND/BLEND
Mean Curvature
Beltrami
clean
78.86 + 1.78
74.89 + 1.29
76.01 = 2.20
75.93 + 1.46
Cora
SPEIT
38.66 + 2.40
55.67 : 3.60
60.67 = 1.31
61.87 + 0.49
TDGIA
60.26 + 5.19
59.55 1 6.41
62.01 = 2.37
62.84 + 1.17
clean
69.47 + 1.22
69.45 + 0.91
70.50 + 1.63
70.14 + 1.80
Citeseer
SPEIT
22.95 + 5.07
39.94 + 6.94
65.39 + 1.62
66.46 + 1.33
TDGIA
52.42 1: 11.2
54.92 1 7.64
66.83 + 1.59
65.77 1 1.28
clean
86.31 + 0.46
88.89 + 0.38
88.45 + 0.32
86.94 0.25
PubMed
SPEIT
40.77 + 1.78
39.39 = 0.26
87.13 - 0.33
86.66 - 0.68
TDGIA
42.63 + 5.28
63.91 + 11.61
85.79 - 0.82
85,56 + 0.91
From Table 3, we observe that even the vanilla time-invarian heat flow preserves some robustness as compared to non-PDE GNNs in Table 1. This further validates theoretical analysis in Proposition 1, which suggests that the topology perturbation is bounded, the learned representations are close to those under the "clean" scenario. We can observe that 1) the flows which are capable of preserving non-smooth features are generally more robust than heat flow, 2) the proposed mean curvature flow and Beltrami flow are more robust than GRAND/BLEND, and 3) these two flows generally suffer less performance variance than the other methods.
More Experiments
Due to space constraint, we refer the reader to the supplementary material for more ablation studies in- cluding the impact of the Lipschitz constraint, PDE solvers, the number of layers, and time complexity. The model performance under other attacks is also presented in the supplementary material.

---

Conclusion
In this paper, we have introduced a general graph neural PDE framework from which several graph neural PDEs are proposed. We analyzed the robustness of the graph neural PDEs and showed that graph neural PDEs are inherently robust against topology perturbations and are also Lyapunov stable. We provided theoretical evidence showing that the robustness of graph neural PDEs stems from the stability of the heat kernel and semigroup during the diffusion process. Moreover, we conducted extensive experiments that empirically verify the robustness of the proposed graph neural PDEs when compared with an existing graph neural PDE induced from approximated heat flow, popular GNNs and the state-of-the-art GNN defenders.
Acknowledgement
This research is supported by the Singapore Ministry of Education Academic Research Fund Tier 2 grant NO0E-12EP202200000 and A under its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund Pre Positioning (IAF-PP) (Grant No. A A19D6a0053).
References
[1] X. Yue, Z. Wang. J. Huang. S. Parthasarathy, S. Moosavinasab, Y. Huang. S. M. Lin, W. Zhang, P. Zhang, and H. Sun, "Graph embedding biomedical networks: methods, applications and evaluations," Bioinformatics, vol. 36, no. 4, pp. 1241-1251, 2019.
[2] H. Ashoor, X. Chen, W. Rosikiewicz, J. Wang, A. Cheng, P. Wang, Y. Ruan, and S. Li, "Graph embedding and unsupervised learning predict genomic sub-compartments from hic chromatin interaction data," Nat. Commun. vol. 11, 2020.
[3] T. N. Kipf and M. Welling, "Sperissue classification with graph convolutional networks," in Proc. Int. Conf. Learn. Representations, 2017.
[4] Z. Zhang, P. Cui, and W. Zhu, "Deep learning on graphs: A survey," IEEE Trans. Knowl. Data Eng., vol. 34, no. pp. 249-270, Jan 2022.
[5] Z. Wu, Pan, F, Chen, G. Long, C. Zhang, and P. S. Yu, "A comprehensive survey on graph neural networks," IEEE Trans. Neural Netw. Learn. Syst. vol. 32, 1, pp. 4-24, 2021.
[6] T. N. Kipf and M. Welling, "Variational graph auto-encoders," in Advances Neural Inf. Process. Syst. Workshop, 2016.
[7] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, "Graph convo- lutional neural networks for web-scale recommender systems," in Proc. Int. Conf. Knowledge Discovery and Data Mining, 2018.
[8] Gilmer, S. S. Schoenholz, F. Riley, O. Vinyals, and G. Dahl, "Neural message passing for quantum chemistry," in Proc. Int. Conf. Mach. Learn,, 2017.
[9] D. Zügner, A. Akbarnejad, and S. Günnemann, "Adversarial attacks on neural networks for graph data," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2018.
[10] Chen. Y. Wu, X. Xu, Y. Chen, H. Zheng, and Q. Xuan, "Fast gradient attack on network embedding." ArXiv, 2018.
[11] M. Waniek, T. Michalak, M. J. Wooldridge, and T. Rahwan, "Hiding individuals and commu- nities in social network," Nature Human Behaviour, vol. 2, no. 1, pp. 139-147, 2018.
[12] Du, S. Zhang, G. Wu, M. F. Moura, and S. Kar, "Topology adaptive graph convolutional networks," ArXiv, vol. Abb/1770..0070, 2017.
[13] Wang, M. Luo, F. Suya, Li, Z. Yang, and Q. Zheng, "Scalable attack on graph data by injecting vicious nodes," Data Mining Knowl. Discovery, pp. - 27, 2020.
10

---

[14] Q. Zheng, Y. Fei, Li, Q. Liu, M. Hu, and Q. Sun. Kdd cup 2020 ml track adversarial attacks and defense on academic graph 1st place solution. Accessed: May 1, 2022. [Online]. Available: http://ggtubb.com/ttatails/////
CUP SBIP__220____ ML.rrack2_SPEII
[15] X. Zou, Q. Zheng, Y. Dong, X. Guan, E. Kharlamov, J. Lu, and J. Tang, "Tdgia: Effective injection attacks on graph neural networks," Proc. Int. Conf. Knowl. Discovery Data Mining, 2021, 2461-2471.
[16] D. Zügner and S. Günnemann, "Adversarial attacks on graph neural networks via meta learning"" in Proc. Int. Conf. Learn. Representations, 2019.
[17] Y. Ma, S. Wang. T. Derr, L. Wu, and J. Tang. "Graph adversarial attack via rewiring," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2021, p. 1161-1169.
[18] Y. Sun, S. Wang. X. Tang, T.-Y. Hsich, and V. Honavar, "Adversarial attacks graph neural networks via node injections: A hierarchical reinforcement learning approach," in Proc. Web Conf. 2020, p. 673-683.
[19] D. Zhu, Z. Zhang, P. Cui, and W. Zhu, "Robust graph convolutional networks against adversarial attacks," in Proc. Inl. Conf. Knowl. Discovery Data Mining, 2019, 1399-1407.
[20] W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, "Graph random neural networks for semi-supervised learning on graphs," in Advances Neural Inf. Process. Syst., 2020.
[21] W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang, "Graph structure learning for robust graph neural networks," " in Proc. Int. Conf. Knowl. Discovery Data Mining, 2020, p. 66-74.
[22] N. Entezari, S. A. Al-Sayouri, A. Darvishzadeh, and E. Papalexakis. "All you need is low (rank): Defending against adversarial attacks on graphs," in Proc. Int. Conf. Web Search Data Mining, 2020, p. 169-177.
[23] X. Zhang and M. Zitnik, "GNNGUARD: Defending graph neural networks against adversarial attacks," in Advances Neural Inf. Process. Syst., 2020.
[24] H. Yan, J. Du, V. Y. Tan, and J. Feng, "On robustness of neural ordinary differential equations," in Advances Neural Inf. Process. Syst., 2018, pp. 1-13,
[25] E. Haber and L. Ruthotto, "Stable architectures for deep neural networks." " Inverse Problems, vol. 34, no. 1, pp. 1-23, Dec. 2017.
[26] X. Liu, S. Si, Q. Cao. S. Kumar, and C.-J. Hsich, "How does noise help robustness? Explanation and exploration under the neural sde framework," " in Proc. Conf. Comput. Vision Pattern Recognition, 2020, pp. 282-290.
[27] Q. Kang, Song, Q. Ding, and W. Tay, "Stable neural ODE with Lyapunov-stable equilibrium points for defending against adversarial attacks," in Advances Neural Inf. Process. Syst., 2021.
[28] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, "Neural ordinary differential equations," arXiv preprint wrXiv:1806.07366 2018
[29] B. Chamberlain, Rowbottom, M. Goronova, S. Webb, E. Rossi, and M. M. Bronstein, "Grand: Graph neural diffusionn" in Proc. Int. Conf. Mach. Learn., 2021.
[30] B. P. Chamberlain, J. Rowbottom, D. Eynard. F. Di Giovanni, D. Xiaowen, and M. M. Bronstein, "Beltrami flow and neural diffusion on graphs," in Advances Neural Inf. Process. Syst., 2021.
[31] A. Grigoryan, Heat kernel and analysis on manifolds. Soc., 2009.
Providence: American Mathematical
[32] L. Saloff-Costt "Uniformly elliptic operators on Riemannian manifolds," J. Differ: Geometry, vol. 36, no. 2, pp. 417-450, 1992.
[33] Z.-Q. Chen, Z. Qian, Hu, and W. Zheng, "Stability and approximations of symmetric diffusion semigroups and kernels," J. functional anal., vol. 152, no. 1, pp. 255-280, 1998.

---

[34] J. Sun, M. Ovsjanikov, and L. Guibas, "A concise and provably informative multi-scale signature based on heat diffusion," Computer graphics forum, vol. 28, no. pp. 2009.
[35] J. Nash, "The imbedding problem for Riemannian manifolds," Ann. math., vol. 63, no. 1, pp. 20-63, 1956.
[36] E. P. Hsu, Stochastic analysis on manifolds. Providence: American Mathematical Soc., 2002.
[37] E. DiBenedetto, U. Gianazza, and V. Vespri, Harnack's 's inequality for degenerate and singular parabolic equations. London: Springer Science & Business Media, 2011.
[38] W. Zheng. "Stability of time-dependent diffusion semigroups and kernels," Acta Math. Sinica, vol. 15, no. 4, pp. 575-586, 1999.
[39] D. Zhou and B. Schölkopf. "Regularization on discrete spaces," in Joint Pattern Recognition Symposium. Springer, 2005, pp. 361-368.
[40] N. Sochen, R. Kimmel, and R. Malladi, "A general framework for low level vision," " IEEE Trans. Image Process., vol. 7, no. 3. pp. 310-318, 1998.
[41] Veličković, G. Cucurull, A. Casanova, A. Romero, Liò, and Y. Bengio, "Graph attention networks," in Proc. Ini. Conf. Learn. Representations, 2018, pp. 1-12.
[42] C.-T. Chen and B. Shafai, Linear system theory and design. press New York, 1999.
New York: Oxford university
[43] G. Dasoulas, K. Scaman, and A. Virmaux, "Lipschitz normalization for self-attention layers with application to graph neural networks," in Proc. Int. Conf. Mach. Learning, 2021, pp. 2456-2466.
[44] A. McCallum, K. Nigam, J. D. M. Rennie, and K. Seymore, "Automating the construction of internet portals with machine learning," Information Retrieval, vol. 3, pp. 127-163, 2004.
[45] W. L. Hamilton, R. Ying, and J. Leskovec, "Inductive representation learning on large graphs," in Advances Neural Inf. Process. Syst., 2017.
[46] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, "How powerful are graph neural networks?" in Proc. Int. Conf. Learn. Representations, 2019.
[47] J. Klicpera, A. Bojchevski, and S. Günnemann, "Predict then propagate: Graph neural networks meet personalized pagerank," in Proc. Int. Conf. Learning Representations, 2019.
[48] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, "Collective classifica- tion in network data," Al Magazine, vol. 29, no. 3, p. 93, Sep. 2008
[49] G. M. Namata, B. London, L. Getoor, and B. Huang, "Query-driven active surveying for collective classification." Workshop Mining and Learn. with Graphs, 2012.
[50] Q. Zheng, X. Zou, Y. Dong, Y. Cen, D. Yin, J. Xu, Y. Yang, and J. Tang, "Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning," Neural Information Processing Systems Track on Datasets and Benchmarks 2021, 2021.
[51] D. Bhaskar, K. MacDonald, D. Thomas, S. Zhao, K. You, J. Paige, Y. Aizenbud, B. Rieck, M. Adelstein, and S. Krishnaswamy, "Diffusion-based methods for estimating curvature in data," in ICLR Workshop Geometrical Topological Representation Learn., 2022.
[52] J. Topping, F. D. Giovanni, B. Chamberlain, X. Dong, and M. M. Bronstein, "Understanding over-squashing and bottlenecks on graphs via curvature," in Proc. Int. Conf. Learn. Representa- lions, 2022.
[53] S. Zhu, S. Pan, C. Zhou, J. Wu, Y. Cao, and B. Wang, "Graph geometry interaction learning." " in Advances Neural Inf. Process. Syst., 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? (Yes]. We have left more complicated problems future work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the Broader Impact section in supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either the supplemental material or as a URL)? [Yes] See supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]. See Section 5 and supplementary material.
(c) Did report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We run each experiment for times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section for the open-soures datasets and models we have used from GitHub.
(b) Did you mention the license the assets? [No]. Please see the licenses given in the GitHub link or datasets websites.
(c) Did you include any new assets either in the supplemental material or URL? [No].
No new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]. The datasets used in the paper are all open-source datasets. (c) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [No] There is no identifiable information or offensive content the datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "SdTqjmJuDL", "reviewer": "Reviewer_UKSj", "summary": "Through the analysis of heat flow, this paper proposes a more general graph neural diffusion scheme, where the authors induce a new class of PDE-based graph neural networks. \nThe authors further explore the graph neural PDEs’ robustness properties from a theoretical point of view and show it is Lyapunov stable. \nThe numerical experiments show certain robustness of the induced graph neural PDEs (including mean curvature flow-based and Beltrami flow-based methods) against topology perturbations.  \n", "questions": "The following questions mainly related to the above weaknesses:\n1. Are the proposed methods or theoretical analysis suitable for the node attribute to perturb scenario?\n2. How do the experiments reflect the setting mentioned in Line 251-258?\n3. How long does it need to run the proposed model?\n\nIt would be appreciated if the authors could also answer other questions or problems listed in the weakness.\n", "limitations": "Related to the above weakness and Questions, one of the limitations I can see is that the discussion in the paper may not be suitable for all the attack scenarios, such as node attribute perturbation.  \nAnother limitation is that the theory guard may not be valid once the perturbation is significant or bigger than a certain “threshold”. Those two potential limitations may need lots of effort and explorations in the future. \n\nTo the best of my knowledge, there is no potential negative societal impact.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper first reviews a couple of math concepts and notions, including differential geometry, diffusion equations, gradient/divergence operators, etc. Those preliminaries are introduced systematically, and close to the following sections. Therefore, it helps the reader quickly get the main idea of each section instead of being stuck in the learning curve. \n2. The paper explores graph neural PDEs' robustness property and mathematically shows they are inherently Lyapunov stable and robust against topology perturbations.\n3. The author proposed a model framework and several insights to design a new class of GNNs based on heat flows. The numerical experiments also validate the above analysis to some degree. Lastly, the related codes are also open-sourced, benefiting the community. \n\nWeaknesses:\n1. One of my concerns comes from the title and some paragraphs (esp. Line 24-27 and Line 251-258). The paper seems to target the \"robustness\" topic, which covers a large area (including topology modification, node attribute perturbation, different attack settings, and so on) as the above sentence/paragraphs mentioned in the paper. However, the proposed theoretical analysis (i.e., propositions 1 and 2) and numerical experiments only focus on topology perturbation, which is just a portion of the big robustness topic. \nIn other words, I am unclear the roles of Line 24-27 and 251-258. For example, Line 251-258 discuss four different attack settings, but I didn't see how the two methods (SPEIT and TDGIA) used in the experiments tie back to those four settings.\n2. The experiment results are not well analyzed. They should connect to the theory in the previous section closer. For example, the reason for good or bad performance is not well illustrated as we can see the proposed method performance in Table 1 and Table S2 is not always outperforming others. Moreover, the propositions require a small perturb assumption. It will also be interesting to see more discussions about the impact of different perturb degrees on model performance.\n3. Lastly, there are some other minor issues:\n(1) It is better if the author can report the training and test speed, or do some algorithm/memory complexity analysis, especially for the section “PDE Solvers”. In this way, the paper can demonstrate better in its practical value.\n(2) It would be better if some table can make consistency. For example, Table 4 and Table S2 don’t report the results under the TDGIA setting. And I find the Table 4 results are not match with any other previous tables, so what is the PDE solver used in Table 1?\n(3) Since the author mentioned the over-smoothing problem and stacking multiple layers, it would be better if the author could provide more information about it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R41zapmdEKS", "reviewer": "Reviewer_BHjT", "summary": "- The authors provide theoretical and empirical insights into the robustness properties of graph neural PDE networks.\n- They show that the heat kernel is stable under small perturbations of the Laplace operator and link this to the graph models by viewing a graph as a discretization of a Riemannian manifold\n- They introduce a general graph PDE framework and instantiate three different diffusion processes to analyze their robustness (which are already introduced in related work)", "questions": "- What of the robustness is attributed to the Lipschitzness of the network? As a Lipschitz constraint can improve robustness itself, an ablation study would be important.\n- The Heat kernel e.g. does not perform better than the APPNP or GAT (for SPEIT attack) while the main robustness result is attributed to the stability of the heat kernel. Can you please explain this result.\n- Why do you use so many layers? In general, on graphs, not many layers are used so the motivation for the Lipschitz constraint is unclear to me.\n- Why do you need more layers for BLEND / GRAND than in their paper?\n- For the inductive setting: are the edges to the unseen data for classification removed before training? Otherwise, due to the increased number of layers, there might be information flow in the training phase\n", "limitations": "-", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The theoretical findings on the robustness of the heat kernel are interesting and novel\n- It is the first analysis of neural graph PDE model robustness\n- As neural graph diffusion gains importance recently a robustness analysis is important to guide the development of these models\n    \nWeaknesses:\n- The major weakness is that the analysis itself could be more in-depth, i.e. (i) I recommend to analyze failure modes of that robustness? (ii) Are there specific attacks that these models don’t defend well. (iii) I would suggest evaluation more threat models. Even though the chosen threat model represents attacks in the practice of social networks, for analysis it would be still beneficial to see exposure to other threat models and attack types. In particular, it seems unfair that no attack is introduced specifically targeting the new model.\n- It is a bit unclear what is proposed by the authors and what is a new contribution on a first glance. It could be written clearer in that regard.\n- Some unclear/open aspects regarding the experiments (see also questions below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KoNRnIesfi4", "reviewer": "Reviewer_w639", "summary": "The authors address the important property of robustness to adversarial attacks in the context of graph neural PDEs. This is the first work to do so. The authors demonstrate that graph neural PDEs are more robust than other GNNs. The authors propose a theoretical reason for this robustness property. ", "questions": "How do lines 92-94 follow from [31, theorem 7.1]? Perhaps it should be theorem 7.13?\n\nWould Figure 1 make sense/be easier to compare if you used t-SNE on one instance and then used the same position to plot the other graphs? (similarly, figure 3).\n\nThe codebase is huge. Has it been forked from an existing repo?", "limitations": "I address the main limitations in the strengths/weaknesses section. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and well organised. \n\nThe authors show that heat diffusion on a Riemannian manifold is robust to small perturbations of the manifold metric. The authors use this as a justification for why graph neural PDEs are robust to changes in the underlying graph. I think the authors should justify why small perturbations to the underlying graph give small perturbations to the graph metric. This seems to be assumed in lines 186-189. In the case of the shortest path distance, this can change quite radically with just a single edge removal/addition. As an example, consider a cycle graph. By removing an edge, the distance between nodes close to the removed edge goes from very small to very large. By increasing the size of the graph, you can make this change arbitrarily drastic. \n\nThe experiments are extensive and convincing. However, the black box attacks trains a surrogate model which is closer to many of the benchmark models (message passing GNNs). This may give an unfair advantage to the method presented in the paper. \n\n“Table 2 shows that BeltramiGuard makes attacks’ efforts totally in vain and exceeds GNNGuard in Citeseer and PubMed datasets.”  The citeseer results are very similar (specially given the error on table 1 for GNNGuard). Can the authors check table 2 is correct for Citeseer, as the values are identical.\n\nThe datasets used are all very small, it may be more relevant to try a larger dataset although given the large combination of models/dataset this could be forgiven on computational grounds. Is it feasible to run experiments on a larger dataset in the review period?\n\nMinor comments:\n- Table 1 should be centred.\n- The paper would be more self-contained and easier to read if the authors defined symmetric measurable coefficients used in Theorem 1.\n- The codebase is quite complex with just a small README.md It would be useful to have instructions to reproduce the main figures/tables in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept