Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach
Authors: Kai Zhao
Qiyu Kang
Yang Song
Rui She
Sijie Wang
Wee Peng Tay
Venue: NeurIPS 2023 spotlight (2023)
Keywords: adversarial robustness
graph neural networks

Abstract: Graph neural networks (GNNs) are vulnerable to adversarial perturbations, including those that affect both node features and graph topology. This paper investigates GNNs derived from diverse neural flows, concentrating on their connection to various stability notions such as BIBO stability, Lyapunov stability, structural stability, and conservative stability. We argue that Lyapunov stability, despite its common use, does not necessarily ensure adversarial robustness. Inspired by physics principles, we advocate for the use of conservative Hamiltonian neural flows to construct GNNs that are robust to adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically compared on several benchmark datasets under a variety of adversarial attacks. Extensive numerical experiments demonstrate that GNNs leveraging conservative Hamiltonian flows with Lyapunov stability substantially improve robustness against adversarial perturbations. The implementation code of experiments  is available at \url{https://github.com/zknus/NeurIPS-2023-HANG-Robustness}.

Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach
Kai Zhao" Nanyang Technological University
Qiyu Kang*1
Yang Song" C3 Al, Singapore
Nanyang Technological University
Rui She
Sijie Wang
Nanyang Technological University
Nanyang Technological University
Wee Peng Tay
Nanyang Technological University
Abstract
Graph neural networks (GNNs) are vulnerable adversarial perturbations, in- cluding those that affect both node features and graph topology. This paper in- vestigates GNNs derived from diverse neural flows, concentrating on their con- nection to various stability notions such as BIBO stability, Lyapunov stability, structural stability, and conservative stability. We argue that Lyapunov stability, despite its common use, does not necessarily ensure adversarial robustness. In- spired by physics principles, we advocate for the use of conservative Hamilto- nian neural flows to construct GNNs that are robust to adversarial attacks. The adversarial robustness of different neural flow GNNs empirically compared several benchmark datasets under variety of adversarial attacks. Extensive numerical experiments demonstrate that GNNs leveraging conservative Hamil- tonian flows with Lyapunov stability substantially improve robustness against adversarial perturbations. The implementation code experiments available at https://gi tht..tthhhb.....
202---AN2----------- Robustness.
1
Introduction
Graph neural networks (GNNs) [1-8] have achieved great success in inference tasks involving graph- structured data, including applications from social media networks, molecular chemistry, and mobility networks. However, GNNs are known to be vulnerable to adversarial attacks [9]. To fool trained GNN, adversaries can either add new nodes to the graph during the inference phase or remove/add edges from/to the graph. The former is called an injection attack [10-13]. and the latter is called modification attack [14-1]]. In some works [9, 17], node feature perturbations are also considered to enable stronger modification attacks.
Neural ordinary differential equation (ODE) networks [18] have recently gained popularity due to their inherent robustness [19-24]. Neural ODEs can be considered as continuous analog of ResNet [25]. Many neural ODE networks have since been proposed, including but not limited to 219,20, 26-28]. Using neural ODEs, we can constrain the input and output of a neural network to follow certain physics laws. Injecting physics constraints to black-box neural networks improves neural networks' explainability. More recently, neural ODEs have also been successfully applied to GNNs by modeling
First three authors contributed equally to this work. Correspondence to: Qiyu Kang <kang00800 .ntuu.ddu.sg>
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the way nodes exchange information given the adjacency structure of the underlying graph. We call them graph neural flows which enables the interpretation of GNNs as evolutionary dynamical systems. These system equations can be learned by instantiating them using neural ODEs [29-34]. For instance, [29, 30] models the message-passe the heat diffusion, while [31,32] model the message passing process as the Beltrami diffusion. The reference [35] models the graph nodes as coupled oscillators with a coupled oscillating ODE guiding the message-passing process.
process, i.e., feature exchanges between nodes, as
Although adversarial robustness of GNNs has been investigated in various works, including [9,13, 17], robustness study of graph neural flows is still in its fancy. To the best of our knowledge, only the recent paper [32] has started formulate theoretical insights into why graph neural diffusion generally more robust against topology perturbation than conventional GNNs. The concept of Lyapunov stability was used in [20, 21 However, there are many different notions of stability in the dynamic system literature [36,37]. In this paper, we focus on the study of different notions of stability for graph neural flows and investigate which notion is most strongly connected to adversarial robustness. We impose an energy conservation constraint graph neural flows that lead to a Hamiltonian graph neural flow. We find that the robustness the most as compared to other existing stable graph neural flows.
graph Hamiltonian flows endowed with Lyapunov stability improve
Main contributions. This research is centered on examining various stability notions within the realm of graph neural flows, especially as they relate to adversarial robustness. Our main contributions are summarized as follows:
1. We revisit the definitions of stability from the perspective of dynamical systems as applicable to graph neural flows. We argue that vanilla Lyapunov stability does not necessarily confer adversarial robustness and provide rationale for this observation.
2. We propose ianiltooian-inspiree graph neural ODEs, noted for their energy-conservative nature. We perform comprehensive numerical experiments to verify their performance standard bench- mark datasets. Crucially, our results demonstrate that Hamiltonian flow GNNs present enhanced robustness against various adversarial perturbations. Moreover, it found that the effectiveness of Lyapunov stability becomes pronounced when layered on top of Hamiltonian flow GNNs, thereby fortifying their adversarial robustness.
The rest of this paper is organized as follows. We introduce various stability notions from a dynamical system perspective in Section 2. A review of existing graph neural flows is presented in Section with links to the stability notions defined in Section 2. We present a new type of graph neural flow inspired by the Hamiltonian system with energy conservation in Section Two different variants of this model are proposed in Section 5. Section 6 details our extensive experimental outcomes. The supplementary section provides an overview of related studies, an exhaustive outline of the algorithm, further insights into model robustness, supplementary experimental data, and the proofs for the theoretical propositions made throughout the paper.
2 Stability in Dynamical Systems
It is well known that a small perturbation at the input of an unstable dynamical system will result in large distortion in the system's 's output. In this section, we first introduce various types of stability in dynamical physical systems and then relate them to graph neural flows. We consider the evolution of a dynamical system that is described as the following autonomous nonlinear differential equation:
dz(()
dt fo(2(())
where So R + T denotes the system dynamics, which may be non-linear in general, 0 denotes the system parameters, and z : [0, oo) + TR²h represents the n-dimensional system state.
We first introduce the stability notions from dynamical systems perspective that is related to the GNN robustness against node feature perturbation.
Definition 1 (BIBO stability). The system is called BIBO (bounded input bounded output) stable if for any bounded input, there exists a constant M s.t. the output [|z(()]] < M, 2 0.
Suppose f has equilibrium at ze so that fo (zz) 0. We can define the stability notion for Ze.

---

Definition 2 (Lyapunov stability and asymptotically stable [38]). The equilibrium Ze is Lyapumov stable for every E O. there exists a 5 > 0 such that. if [lz(0) zel < ö, then for every? 2 0 we have [|((() - zell < e. Furthermore, the equilibrium point Ze is said to be asymptotically stable if is Lyapunov stable and there exists a 8' > such that if [z(0)- - zel < 8', then lime-+oo [|((t)- =
Remark 1. Lyapunov stability indicates that the solutions whose initial points are near an equilibrium point Be stay near Ze forever. For the special linear ane-invariant system dz(t)/ dt Az(t) with constant matrix A, it is Lyapunov stable if and only if all eigenvalues of A have non-positive real parts and those with zero real parts are the simple roots of the minimal polynomial of A [39, 40].
Asymptotically stable means that not only do trajectories stay near Ze for all time (Lyapunov stability), but trajectories also converge to Ze as time goes to infinity (asymptotic stability).
We next introduce the concept of structural stability from dynamical systems theory, which is related to the robustness of GNNs against graph topological perturbation. It describes the sensitivity of the qualitative features of a solution to changes in parameters O. The definition of structural stability requires the introduction of a topology on the space of in (1), which we do not however present here rigorously due to space constraints and not to distract the reader with too much mathematical details. Instead, we provide qualitative description of structural stability to elucidate how it can indicate the robustness of a graph neural flow against topology perturbations.
Definition 3 (Structural stability). Unlike Lyapunov stability, which considers perturbations of initial conditions for a fixed fe. structural stability deals with perturbations of the dynamic function fo by perturbing the parameter 0. The qualitative behavior of the solution is unaffected by small perturbations of fe in the sense that there is a homeomorphism that globally maps the original solution to the solution under perturbation.
Remark 2. In the graph neural flows to be detailed in Section and Section 4, the parameter includes the graph topology (i.e., the adjacency matrix) and learnable neural network weights. Unlike adversarial attacks on other deep learning neural networks where the attacker targets only the input it is worth noting that adversaries for GNNs can also attack the graph topology, which forms part of 0. If there are different Lyapunov stable equilibrium points, one for each class of nodes, one intuitive example of breaking structural stability in graph neural flows is by perturbing the graph topology in such a way that there are strictly fewer equilibrium points than the number of classes.
In this study, we will propose GNNs drawing inspiration from Hamiltonian mechanics. In a Hamilto- nian system, (q.p) p) IR refers to the generalized coordinates, with g and p corresponding to the generalized position and momentum, respectively. The dynamical system is characterized by the following nonlinear differential equation:
dz (t)
(VH(((())
dt
(
0 -1 ) is the 2n x 2n
!
where (z) is the gradient of a scalar function H at z and = skew-symmetric matrix with 7 being the n x n identity matrix.
We now turn our attention to the notion of conservative stability in dynamical systems. It is worth noting that a general dynamical system, as characterized in (1), might not consistently resonate with traditional perspectives on energy and conservation, especially when compared to physics-inspired neural networks, like (2).
Definition 4 (Conservative stability). In dynamical system inspired by physical principles, such as (2), a conserved quantity might be present. This quantity, which frequently embodies the notion of the system's energy, remains invariant along the system's 's evolution trajectory ztt).
Our focus in this work on graph neural flows that can be described by either (1) or (2). Chamberlain et al. [29] postulate that many GNN architectures such as GAT can be construed as discrete versions of (1) via different choices of the function fo and discretization schemes. Therefore, the stability definitions provided above can offer additional insights into many popular GNNs. Most existing graph neural flows only scrutinize the yaauuoo stability of their system. For instance, GRAND [29] proposes BIBO/Lyyapnov stability against node feature perturbation over z. However, the more fundamental structural stability in graph neural flows, which is related to the robustness against graph topological changes, remains largely unexplored. Some models, such as GraphCON 25]3 exhibit conservative stability under certain conditions. We direct the reader to Section and Table for a comprchensive discussion of the stability properties of each model.
3

---

3 Existing Graph Neural Flows and Stability
Consider an undirected, weighted graph 9 = (V,E) where a finite set of vertices and & C denotes the set of edges. The adjacency matrix of the graph is denoted as (W(u, u|) = ([r, u)) for all [u, E E. Let (() E I Vixr represent the features associated with the vertices at time 1. The feature vector for the i-th node
at time can be represented as the i-th row of ((), indicated by x (t). In this section, we introduce several graph neural flows on g, categorizing them according to the stability concepts outlined in Section 2.
GRAND: Inspired by the heat diffusion equation, GRAND [29] employs the following dynamical system:
d)(()) = (X) = (AG (X(()) - (t)
dt
with the initial condition (0). Within this model, (x(((t) is either a time-invariant static matrix, represented as GRAND-I, or trainable time-variant attention matrix (ag (x,(t,, xy (4)), labeled as GRAND-nl, reflecting the graph's 's evolutionary features. The function (-(() calculates similarity for pairs of vertices, and is an identity matrix with dimensions that fit the context. In [29], or is set to be 1. Let D be the diagonal node degree matrix where D(u, u] = EVEy W Wu,
Theorem 1. We can prove the following stability:
1). For GRAND-ul, if the atlention matrix Ag (X))) is set as doubly stochastic atlention [41), J. we have BIBO stability and Lyapunov stability for any a 2 1. When a > 1, it reaches global asymptotic stability under any perturbation.
2). Within the GRAND-I setting, if Ag is set as a constant column- or row-stochastic matrix, such as the normalized adjacency matrices WD-1 1 or D-'W. global asymptotic stability is achieved for a > 1 under any perturbation. If the graph is additionally assumed to be strongly connected [42.[Secc...3], BIBO and Lyapunov stability are realized for a 1. 3). Furthermore, when Ag is specifically a constant
-toumn--tochattcc matrix like WD-1 and a 1, GRAND conserves a quantity that can be interpreted as energy. Furthermore, in this settring. asymptotic stability is attained when the graph is aperiodic and strongly connected and the perturbations on (() ensure unaltered column summations.
In comparison to GRAND, BLEND [31] introduces the use of positional encodings. Following a similar line of reasoning to that used for GRAND, BLEND also exhibits BIBO/Lyapunov stability stated in Theorem 1. Moreover. it is noteworthy that if positional features U(t) are eliminated, for instance by setting them as a constant, BLEND simplifies to the GRAND model.
BLEND:
GraphCON: Inspired by oscillator dynamical systems, GraphCON is graph neural flow proposed in [35] and defined as
dY (F,()(()) - aY)),, axit) dt = Y(),,
where Fo(-) is a learnable 1-neighborhood coupling funetion, Yt)) is an auxiliary velocity variable, denotes an activation function, and y and a are tunable parameters.
As described in [35, Proposition 3.1], under specific settings where is the identity function and F (x(X((),)) = (t( with A being a constant matrix, GraphCON conserves Dirichlet energy (11), thereby demonstrating conservative stability.
GraphBel: Generalizing the Beltrami flow, mean curvature flow and heat flow, a stable graph neural flow [32] is designed
dx(t) (As (()) O Bx(((()) - V (()(())
(5)
dt
where @ the element- multiplication. (-(:) and (-(() are learnable attention function and normalized vector map, respectively. &e (X(t)) is a diagonal matrix in which V (x). = Σx, (A
B)(x;, Xz).
Analogous to BLEND, under certain conditions with & (X())) (x) = I, GraphBel simplifies to the GRAND model. Consequently, it exhibits BIBO/Lyapunov stability in certain scenarios.

---

Nea SENE GEORAND OFGRAND
44
clais
BUEGBAND
Class
aNC LEGIBAND
misc ifsccatotiiicttoon
(a)
Example
(b) Before GIA attack
(c) After GIA attack
Figure 1: (a): We plot the vector field and the system solution trajectories of Example (b) and (c): In GRAND, the node features' energy tends to converge towards each other. In HANG, we observe the node features' energy remains relatively stable over time. Two nodes are from different classes.
The incorporation of ODEs via graph neural flows may enhance the stability of graph feature representations. A summarized relationship between model stability and these graph neural flows can be found in Table I.
3.1 Lyapunov Stability vs. Node Classification Robustness:
At first glance, Lyapunov stability has a strong correlation with node classification robustness against feature perturbations. However, before diving into experimental evidence, we point out an important conclusion: Lyapunov stability by itself does not necessarily imply adversarial robustness. Consider a scenario where a graph neural flow has only one equilibrium point Ze. while the node features are derived from more than one class. In a Lyapunov asymptotically stable graph neural flow, such as GRAND (as shown Theorem 1), node features across different classes would inevitably converge to a single point Z, due to global contraction. We note that this is why the model in [20] requires a diversity-promotte
layer to ensure that different classes converge to different Lyapunov-stabe equilibrium points.
Example 1. We provide an example to demonstrate our claim. Consider the following Lyapunov stable ODE
-1 0 0 -5
x(1)
x(1)
(6)
with initial condition x(0) = [z] (0), x2)( The solution to this ODE is given by x(t) Ii =1(0)e (1, x2 =2)(0)=551 5t 1)" For all initial points in IR,, we have x(t) + 0 as t +
Furthermore, as + oc, the trajectory x(t) for any initial point is approximately parallel to the x-axis. We draw the phase plane in Fig. la.
Assume that the points on the upper half y-axis belongs to class I while we have a linear classifier that seperates class I and class 2 as shown in Fig. la. We observe that for the initial point A belonging to class 1, the solution from a small perturbed initial point A + E misclassified as class 2 for a large enough for any linear classifier. We see from this example that Lyapunov stability iteself does not imply adversarial robustness in graph neural flow models.
This example indicates that Lyapunov stability does not guarantee node classification robustness. Additionally, for a system exhibiting global contraction a single equilibrium point, structural stability may also be ensured. For instance, in the case of GRAND, even if the edges are perturbed, the system maintains the same number of equilibrium points with global contraction. We conclude that even an amalgamation of both Lyapunov stability and structural stability may not help the graph's s adversarial robustness for node classification.
In the example shown in Fig. 1, we observe that in the case of GRAND when > 1, the node features from different classes tend become closer to each other as time progresses. This phenomenon can potentially create more vulnerability to adversarial attacks.
5

---

p(0) 9(0)
p(7)
Ps(7
Pu003 92200
9697
f'())
Ps
de
Pj(0) 9,100
Hnet
PcTT
9:07
dHset
(1(1)
9
P400) 9.000
Hamiltonian Information Propagation
Py(77
Figure 2: The model architecture: each node is assigned a learnable "momentum" vector at time t = 0 which initializes the evolution of the system together with node features. The graph features evolve on following a learnable law (10) derived from the Huet- At the time t = T, we use ((T) as the final node feature. Host (g(()) is learnable graph energy function.
4 Hamiltonian-Inppree Graph Neural Flow
Drawing inspiration from the principles Hamiltonian classical mechanics, we introduce a novel graph neural flow paradigm, namely HamiltoniAN Graph diffusion (HANG). In Hamiltonian me- chanics, the notation (g, p) is traditionally used to represent the properties of nodes (position and momentum). Hence, in this section, adopt this notation instead of X (as used in Section 3) to denote node features.
4.1 Physics Philosophy Behind HANG
In Hamiltonian mechanics, the state evolution of multi-object physical system, such as an electro- magnetic field, double pendulum, or spring network [43,44], adheres to well-established physical laws. For instance, in system of charged particles, each particle generates an electromagnetic field that influences other particles. The Hamiltonian for such a system includes terms representing the kinetic and potential energy of each particle, along with terms representing the interactions between particles via their electromagnetic fields. In this paper, we propose a novel concept of information propagation between graph nodes, where interactions follow similar Hamiltonian style.
In Hamiltonian system, the position q and momentum p together constitute the phase space (g. p)- which comprehensively characterizes the system's 's evolution. In our HANG model, we process the raw node features of the graph using linear input layer, yielding 2r-dimensional vectors. Following the methodologies introduced in the GNN work [31, we split the 2r dimensions into two equal halves, with the first half serving as the feature (position) vector and the second half "momentum" vector that guides the system evolution. Concretely, each node l is associated with an r-dimensional feature vector qu (0) = ... (h) and an r-dimensional momentum vector Px (0) = (p......,p) Subsequently, (t) and (t) will evolve along with the propagation of information between graph nodes, with q (0) and (a (0) serving as the initial conditions.
Following the modeling conventions in physical systems, we concatenate the feature positions of all [V] vertices into single vector, treating as the system's generalized coordinate within dill-dimes manifold, process that involves index relabeling".
q(t) (t)....... (+)) (q) (t))
This dimmeniiaal coordinate representation at each time instance provides snapshot of the state of the graph system. Similarly. we concatenate all the "momentum" vectors at time construct an rlD)--dimnsioonl vector:
p(t) (p (t) =
(p.
In multilinear algebra and tensor computation, vector components employ upper indices, while covector components use lower indices. We adhere to this convention.

---

which can be interpreted as a generalized momentum vector for the entire graph system.
In physics, the system evolves in accordance with fundamental physical laws, and conserved quantity function (q, p) remains constant along the system's 's evolution trajectory. This conserved quantity is typically interpreted as the "system energy". In our HANG model, instead of defining an explicit energy function H (p, q) from fixed physical law, we utilize a learnable energy function Inet + R+ parameterized by a neural network, referred as the Hamiltonian energy function:
Hee 9 + IR+
We allow the graph features evolve according to learnable Hamiltonian law analogous basic physical laws. More specifically, we model the feature evolution trajectory the following canonical Hamilton's 's equations, which is a restatement of (2):
((t) = Huet
H 0q
p(t)
(10)
Op
with the initial features ((()). E 221 at time t = 0 being the vectors after the raw node features transformation.
The neural ODE given by (10) can be trained and solved through integration to obtain the trajectory (g(t), p()). At the terminal time point 1 T, the system's 's solution is represented as (q(T), p(T).
We then apply the canonical projection map я to extract the nodes' concatenated feature vector ((T') as follows: x((((T), (TT)) q(T). This concatenated feature vector ((T)) is subsequently decompressed into individual node features for utilization in downstream tasks. For this study, we employ backpropagation to minimize the cross-entropy in node classification tasks. The complete model architecture is depicted in Fig. 2, while a comprehensive summary of the full algorithm can be found in the Appendix G.
4.2 Hamiltonian Energy Conservation
Referring to [45], it is known that the total Hamiltonian energy Huet remains constant along the tra- jectory of its induced Hamiltonian flow. This principle is recognized as the law of energy conservation in Hamiltonian system.
Theorem 2. the graph system evolves in accordance with (10), the total energy Hne (q(t), p(t)) of the system remains constant. BIBO stability is achieved if Hnet remains bounded for all bounded imputs and, as (4,p) 0C, Hnee (g.p(g.)) + oo.
In light of Theorem 2, if our system evolves following (10), it adheres to the law of energy conser- vation. As a result, our model guarantees conservative stability. Compared to GraphCON, which conserves Dirichlet energy over time & under specific conditions, the notion of Hamiltonian energy conservation is broader in scope. Under the settings delincated in Section 3, GraphCON can be considered particular instance of HANG when H is set represent Dirichlet energy E(q):
Definition (Dirichlet energy [35]). The Dirichlet energy defined on node features q()) at time of an undirected graph G as
= 1 3 (1:(()-a;;0)))
E(((1))
M
(11)
JEN())
where N(i) = [i.5] E {} is the set of neighbors adjacent to node i in the graph.
5 Different Hamiltonian Energy Functions
In physical systems, the system is often depicted as a graph where two neighboring vertices with mass are connected by spring of given stiffness and length [44]. The system's 's energy is thus related to the graph's 's topology. Similarly. in our graph system, the energy function Hnot involves interactions between neighboring nodes, signifying the importance of the graph's 's topology. There exist multiple ways to learn the energy function, and we present two examples below.
7

---

Table 1: The stability summary for different graph neural flows where the thenodes that stability is affirmed under additional conditions.
Graph Neural Flows BIBO stabiliy Lyspunor stabilly
Siructural sabiliy
Conservtiive stabiliy
GRAND
<
BLEND
GraphCON
GraphBel
HANG
x
X
HANN quad
5.1 Vanilla HANG
We define Huet composition of two graph convolutional layers:
Hnt [ggenn tanh (ggens) (q. ()
(12)
where gren; 2rx [V] + Rdxxv]] and Igeng Rdx + R VI are two GCN [3] layers with different
hidden dimensions. A tanh activation function is applied between the two GCN layers, and || ||2 denotes the l norm. We concatenate qu and Pa: for each node & at the input of the above composite function, resulting in input dimension of 2r per node. From Theorem 2, it follows that HANG exhibits BIBO stability. If (q(t), p(t)) were unbounded, the value of Huet would also become unbounded, contradicting the energy conservation principle. In subsequent discussions, this invariant referred to as HANG.
5.2
Quadratic HANG (HANG-quad)
The general vanilla HANG with conservative stability does not possess Lyapunov stability. Other additional conditions may be required for Lyapunov stability. The following Lagrange-Dinee Dirichlet Theorem provides sufficient condition.
Theorem 3 (Lagrange-Dirichle Theorem [46]). Let ze be locally quadratic equilibrium of the natural Hamiltonian (2) with
H == + U(q),
(13)
where I is positive definite, quadratic function Then ze is Lyapunov stable if the position of a strict local minimum of U(q).
Theorem 3 implies that we can design an energy function Hnet such that the induced graph neural flow is both Lyapunov stable and energy conservative. For instance, we can define T
T(g. Lh, +A,
(14)
where h, = EjeN() ag (qi, q;) Py is the aggregation of p, through the attention mechanism ag (qt, q;) calculated based g (or directly use adjacency matrix with ag (qi, q;) W j.. The term is a small positive number included to ensure the positive definiteness. If U(g) has only single global minimum, such U(q) |/q/| with an norm, this stability may not ensure adversarial robustness as discussed in Section 3.1. An alternative is setting U() = sin (q) ||- which fosters Lyapunov stability across multiple local equilibrium points. However, this choice considerably restricts the form that U can take and may consequently limit the model's capacity. the implementation, we set the function (g) in Hl to be a single GAT [6] layer Ogat with a sin activation function followed by & norm, i.e., U(g) (9)ll2 The stability of HANG and HANG-quad summarized in Table
Rrxiv| + x
Experiments
In this section, we conduct comprehensive evaluation of our theoretical findings and assess the
robustness of two conservative stable models: HANG and LAAG-Quuad We compare their perfor- mance against various benchmark GNN models, including GAT [47], GraphSAGE [48], GCN [3] and other prevalent graph neural flows. We incorporate different types of graph adversarial attacks as described in Section 6.1 and Section 6.3. These attacks are conducted in black-box setting, where surrogate model is trained to generate perturbed graphs or features. For more experiments, we direct readers to Appendix C.
8

---

Table 2: Node classification accuracy (%) on graph injection, evasion, non-targeted attack in inductive learning. The best and the second-best result for each criterion are highlighted in red and blue respectively.
Albac
HANG
HANG-qud
GaplCTON
GraphBel
GRAND
GAT
GriphAGG
GCN
clom
87.1340.36
79.822062
86.27-00.51
86. 16.33-0511
87.53=0.59
87.58420.64
86.65 1.51
88.31=0.48
Cura
TDGIA
PGD
18
79.76::0:99
79:05-0:42
79. 79.5420.65
42.81=00330
430020 24
59 10::0
40.04:0.08
41.77:2.70
39.65=1.32
38.27-22.73
39.39::7557
35.45-:5005
34.3822.05
35.83=00.71
33.05 1.0
MotaGiA
17.48::0002
08.28
42,50400.33
$9.9340..59
39.36
39,49::4119
28.14
35.84=0073
clom
74.11+00.22
71885=0.48
74.84+00.49
69.62+00.66
74, 0449
67,87444.97
63.224-9114
72.63=1.14
Citeseer
PGD
12.314:1.16
10.07
40.56::0.36
55.67 5:535
36.08 L.05
32.65
32.70 5.11
30,69 2.339
TDGIA
72.12-40.52
71.
3667-1125
14.17:44.00
36.67-1.25
21.10=22.35
MetaGiAA
72.92-00.66
71. 0..40
48.36--2.122
45 45.00:5-4.31
46.23=22.11
37.08-5-4.0
35.75 50.55
35.86 25.86-00.68
cleon
96.16-1:0.09
95.27:10.12
95, 95.10:1:.12
91.91:10.88
05.08:00:12
92.81-10.41
93.0:10.39
0,.33:20.37
Commhoers
PGD
94. 04.30110.33
95.00 05.08:2000 0:223
12.68:10.113
71.35:17.29
74.69
34,22
7.29 2.58
11.02:5..04
TDGIA
95.40+4-0.13
0.099
7,92+44.11
43.13-4-.58
5.05+1145
18.08
6.47-544255
3.11+1.77
MetaGIA
95.85:1:0.31
67.79:1:.04
8329
41.21:1-.26
52.01
24.221
7.82::..66
1..90:33.95
efean
19:91:1:0:27
58 10:10.33
88.78:0:46
86 36.97:0:377
88 0::44
87.11::.77
08.31
58 08.46:0020
Puhmed
TDGIA
PGD
81:81:1::44
86.62:1:1.05
87 0:55
00.55
450021.55
16.30:111.56
52:21:1:068
06:200077
44,99
44.61
47.56:13.11
48.94-12.99
4..91
44.62:16.49
42.64::.41
39.03:2011
MetaGIA
87.58:1:0.75
87. 0..40:20.22
45.55:11118
50.0110644
1..3621.20
44,75:2..55
42.39:20.53
40.42:0.17
Table 3: Node classification accuracy (%) on graph injection, evasion, targeted attack in inductive learning.
Dulasel
Atback
HANG
quadG-qud
GnapoCON
GraphBel
GRAND
GAT
GripSSAGE
GCN
PGD
00
87.53=0.99
74.01:2:487
50.13:20.56
65 65.75=25.00
10.72
18.33=0.57
17 17.80:20.06
Computers
TDGIA
90.88:0.00
83.75=0.55
80.11:33211
20.63
77.1827.00
16.09:11.007
50,89_1.55
66.51
MenaGIA
90/73:1:064
09.32=0048
87.95:1:265
89.27:1::57
81.85:24666
60.28:2:11 11.11
37.51
36.19:20.10
PGD
3334111333
52.95-0:33
63
45.0743887
42.75::4.55
25.48=5.65
29.7642..56 56
oghn-arsiv
TDGIA
66 (6.36:22.50
61.95-0:94
29.77:2..001
00M
22.69
17.76::130
4.78:22.76
28.43:25..0
MenKiLA
64.09-+1.77
64.83=1.21
57.91-44999
00M
50.75#3.511
57.07
33.84=2.51
37.36-50088
Table 4: Node classification accuracy (%) under modification, poisoning, non-targeted attack (Metattack) in transductive learning.
Datnset
Pib Rote('))
HANG
HANG-quad
GraphCON
GriphBel
GRAND
GAT
GCN
RGCN
GCN-SVD
Pro-GNN
94.77 1.07
94.63- 1.06
93.14:20.84
85.12
95.57-40.44
9,,25
95.69-00.38
95.22-4-0.44
95.31400.18
93.20:1-0.4
2..19:2.52
94.38:00222
70.244:2.71
3..34
77.58-13.66
13.69=21445
23.07=0.00
24.34
89000:0222
13
Polblogs
15
71,65
1.34
92.46-1155
90.8-112.43
7187-11.71
69.09-0.90
53.41 3:308
73.84++-.46 1.46
76.32-085
L.14
20.72=1.1 1.15
64.96 1.91
67.28 0.33
3..70-33.73
81244-0.99
$9442-1009
26.04:12221
20
6..27:33.9
30.19::3772
60.46
52.18-00554
19.14:1:32
$1.50=1.66 1.63
51,27=1.25 1.25
59.89900334
57.33-33315
79 59.56-25.68
23.30
86.89-3..00
1.87
$1.89:::336
11.19:1.49
49.25 1.36
48.66-9999
63.18-44400
85,08 0:020
85.23-10.14
86.65
84.02 1:0.26
85.06:1:026
83.73-10.40
87.19
86.16:10.188
83.441-0.21
57.33-10.188
15.08-10.118
85.12:100.88
86.52:10.14
81.11:10.00
78.00:10.44
83.09
81.08:10.200
6341100115
$7.25:10.99
Puhmed
15
10
85, 4:122
85.01::0.2
05.15:10..17
$5.05-2-0119
86.41
0:..1
$4.83 1:0.0
84.62-4-0.20
83.74:00.34
84:24-00188
74.93-00.88
01.13:00.51
78.66
$1.21
27.51 4-0.27
73.91
852774-021
83.1011.118
51.25-00099
$7.20:1-0009
15.20
85.03-:0.19
86.20-200188
84.89-2:0.45
83.58-002200
06996
17.35-0019
71.18:0.331
:3.01:0:222
07.09-00110
85.05-00.177
84.99-4-0.16
86.04-20.14
85 +-011
83.66-+-0225
65.41=00.77
75.50-60.17
67.95-t0.115
82/72-00.18
86.71+00.09
6.1 Graph Injection Attacks (GIA)
We implement various GIA algorithms following the methodology in [49]. This framework consists of node injection and feature update procedures. Node injection involves generating new edges for injected nodes using gradient information or heuristics. We use the PGD-GIA method [49] to randomly
inject nodes and determine their features with the PGD algorithm [50]. TDGIA [12] identifies topological vulnerabilities to guide edge generation and optimize a smooth loss function for feature generation. MetaGIA [49] performes iterative updates of the adjacency matrix and node features using gradient information. Our datasets include citation networks (Cora, Citeseer, Pubmed) [51], the Coauthor academic network [52], an Amazon co-purchase network (Computers) [52], and the Ogbn-Ars Arxiv dataset [53]. For inductive learning, we follow the data splitting method in the GRB framework [54], with 60% for training. 10% for validation, and 20% for testing. Details on data statistics and attack budgets can be found in Appendix C. Targeted attacks are applied to the Ogbn- Arxiv and Computers datasets [49]. Additional results for various attack strengths and white-box attacks can be found in Appendix C.4 and Appendix C.3, respectively.
9

---

6.2 Performance Results Under GIAs
Upon examining the results in Table 2 and Table 3 pertaining to experiments conducted under GIA conditions, the robustness of our proposed HANG and HANG-quad, is notably prominent across different GIA scenarios. Interestingly, GRAND, despite its Lyapunov stability as analyzed in Theo- rem does not significantly outperform GAT under certain attacks. In contrast, HANG consistently displays robustness against attacks. Notably, HANG-quad exhibits superior performance to HANG on the Pubmed dataset under GIA perturbations, underscoring the effectiveness of integrating both Lyapunov stability and Hamiltonian mechanics to boost robustness. Although other graph neural flows might demonstrate range of improved performance compared to conventional GNN models under GIA, the degree of improvement is not consistently distinct. Despite the pronounced association between conservative stability in Hamiltonian systems and adversarial robustness, a clear relationship between adversarial robustness with the other stability of the graph neural flows outlined in Table is not immediately discernible. The performance differential between HANG variants and other graph neural flows further underscores the potential of our proposed Hamiltonian models in enhancing robustness against GIA attacks.
6.3 Graph Modification Attacks
To evaluate the robustness of our proposed conservative models, we conducted graph modification adversarial attacks using the Metattack method [17]. We followed the attack setting described in the Pro-GNN [55] and utilized the perturbed graph provided by the library [56] ensure a fair comparison. The perturbation rate, which indicates the proportion of altered edges, was incrementally varied in 5% inerements from 0% 25%. For comparison, we also considered other defense models for GNNs, namely Pro-GNN [55], RGCN [57], and GCN-SVD [58]. We report the results of baseline models from [55].
6.4 Performance Results Under MalfficetoonPooionnngg//aaaaattt Attacks
In the case of the Polblogs dataset [59], as shown in Table 4, our proposed HANG-quad model demonstrates superior performance compared to other methods, including existing defense models. This result indicates that incorporating Lyapunov stability indeed enhances HANG's robustness against graph modification and poisoning attacks. For the Pubmed dataset, we note that the impact of Meta-attacks of varying strengths on all graph neural flows, including our proposed ones, is negligible. Conversely, traditional GNN models such as GAT, GCN, and RGCN are marginally affected as the attack strength escalates. This observation underlines the robustness of graph neural flows, including our proposed models, against Meta-attacks on this dataset.
6.5 Combination with other defense mechanisms
It merits noting that our models, HANG and HANG-quad, can be readily integrated with additional defense mechanisms against adversarial attacks. These include Adversarial Training (AT) [60] and other preprocessing methods such as GNNGUARD [61]. This integration can further bolster the robustness of the HANG model. To validate this enhancement, extensive experiments are conducted, with results detailed in Appendix C.8 and Appendix C.9.
7 Conclusion
In this paper, we conducted a comprehensive study on stability notions in the context of graph neural flows and made significant findings. While Lyapunov stability is frequently employed, it alone may suffice in guaranteeing robustness against adversarial attacks. With a grounding in foundational physics principles, we proposed shift towards conservative Hamiltonian neural flows for crafting GNNs resilient against adversarial attacks. Our empirical comparisons across diverse neural flow GNNs, as tested on multiple benchmark datasets subjected to a range of adversarial attacks, have further corroborated this proposition. Notably, GNNs that amalgamate conservative Hamiltonian flows with Lyapunov stability exhibited marked enhancement in their robustness metrics. We are optimistic that our work will inspire further research into marrying physics principles with machine learning paradigms for enhanced security.
10

---

8 Acknowledgments and Disclosure of Funding
This research supported by the Singapore Ministry of Education Academic Research Fund Tier grant MOE-T2EP20220-0000 and the National Research Foundation, Singapore and Infocomm Me- dia Development Authority under its Future Communications Research and Development Programme. To improve the readability, parts of this paper have been grammatically revised using ChatGPT [62].
References
[1] X. Yue, Z. Wang. J. Huang. S. Parthasarathy, S. Moosavinasab, Y. Huang. S. M. Lin, W. Zhang, Zhang, and H. Sun, "Graph embedding on biomedical networks: methods, applications and evaluations," Bioinformatics, vol. 36, no. 4, pp. 1241-1251, 2019.
[2] H. Ashoor, X. Chen, W. Rosikiewicz, J. Wang, A. Cheng. P. Wang, Y. Ruan, and S. Li, "Graph embedding and unsupervised learning predict genomic sub-compartments from hic chromatin interaction data," Nat. Commun., vol. 11, 2020.
[3] T. N. Kipf and M. Welling, "Semiisuperrvied classification with graph convolutional networks," in Proc. Int. Conf. Learn. Representations, 2017.
[4] Z. Zhang, P. Cui, and W. Zhu, "Deep learning graphs: A survey," IEEE Trans. Knowl. Data Eng., vol. 34, no. pp. 249-270, Jan 2022.
[5] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, "A comprehensive survey graph neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 1, pp. 4-24, 2021.
[6] Veličković, Cucurull, Casanova, A. Romero, P. Lio, and Y. Bengio, "Graph attention networks," in Proc. Int. Conf. Learn. Representations, 2018, pp. 1-12.
[7] Ji, S. H. Lee, H. Meng, K. Zhao, J. Yang. and W. P. Tay, "Leveraging label non-uniformity for node classification in graph neural networks," in Proc. Int. Conf. Mach. Learn., vol. 202, Jul. 2023, pp. 14 889-6694460 885.
[8] H. Lee, F. Ji, and W. Tay, "SGAT: Simplicial graph attention network," in Proc. Inter. Joint Conf. Artificial Intell., Jul. 2022.
[9] D. Zügner, A. Akbarnejad, and S. Günnemann, "Adversarial attacks on neural networks for graph data," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2018.
[10] J. Wang, M. Luo, F. Suya, J. Li, Z. Yang, and Q. Zheng, "Scalable attack on graph data by injecting vicious nodes," Data Mining Knowl. Discovery, pp. 27, 2020.
[11] Q. Zheng, Y. Fei, Y. Li, Q. Liu, M. Hu, and Q. Sun. Kdd cup 2020 ml track adversarial attacks and defense on academic graph 1st place solution. Accessed: May 1, 2022. [Online]. Available: https://github.c Sot.aubboooomSSaaaaas/Ssii.
CUP _020000002 (LOO0__TTrcck2__P___0
[12] X. Zou, Q. Zheng, Y. Dong, X. Guan, E. Kharlamov, J. Lu, and J. Tang, "Tdgia: Effective injection attacks on graph neural networks," " in Proc. Int. Conf. Knowl, Discovery Data Mining, 2021, 2461-22444 4611-477.
[13] H. Hussain, M. Cao, S. Sikdar, D. Helic, E. Lex, M. Strohmaier, and R. Kern, "Adversarial inter-gne grour- link injection degrades the faimess of graph neural networks," in Prof. Int. Conf. Data Mining, 2022, pp. 975-980.
[14] Chen, Y. Wu, X. Xu, Y. Chen, H. Zheng, and Q. Xuan, "Fast gradient attack on network embedding," arXiv preprint arXiv: 02797, 2018.
[15] M. Wanick, T. P. Michalak, M. J. Wooldridge, and T. Rahwan, "Hiding individuals and commu- nities in social network." Nature Human Behaviour, vol. 2, no. I, pp. 139-147, 2018.
[16] J. Du, S. Zhang, G. Wu, J. M. F. Moura, and S. Kar, "Topology adaptive graph convolutional networks." ArXiv, vol. abs/17100 2017.
[17] D. Zügner and S. Günnemann, "Adversarial attacks on graph neural networks via meta learning." in Proc. Int. Conf. Learn. Representations, 2019.
[18) R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, "Neural ordinary differential equations," in Advances Neural Inf. Process. Syst., 2018.
[19] H. Yan, J. Du, Y. Tan, and J. Feng, "On robustness of neural ordinary differential equations." in Advances Neural Inf. Process. Syst., 2018, pp. 1-13,

---

[20] Q. Kang, Y. Song, Q. Ding, and W. P. Tay. "Stable neural ODE with Lyapunov stable equilibrium points for defending against adversarial attacks," in Advances Neural Inf. Process. Syst., 2021.
[21] I. D. J. Rodriguez, A. Ames, and Y. Yue, "Lyanet: A Iyapunov framework for training neural odes," in Proc. Int. Conf. Mach. Learn., 2022, pp. 18 687-18 703.
[22] R. She, Q. Kang, S. Wang, Y.-R. Yáng. K. Zhao, Y. Song, and W. P. Tay, "Robustmat: Neural diffusion for street landmark patch matching under challenging environments." IEEE Trans. Image Process.. 2023.
[23] S. Wang. Q. Kang, R. She, W. P. Tay, A. Hartmannsgruber, and D. N. Navarro, "RobustLoe: Robust camera pose regression in challenging driving environments," in Proc. AAAI Conference on Artificial Intelligence, Feb. 2023.
[24] R. She, Q. Kang, S. Wang. W. P. Tay, Y. L. Guan, D. N. Navarro, and A. Hartmannsgruber, "Image patch-matching with graph-based learning in street scenes," IEEE Trans. Image Process.. 2023.
vol. 32, pp.
[25] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," " in Proc. IEEE Int. Conf. Compur. Vision, 2016, pp. 770-778.
[26] Y. D. Zhong, B. Dey, and A. Chakraborty, "Symplectic ode-net: Learning hamiltonian dynamics with control," in Proc. Int. Conf. Learn. Representations, 2020.
[27] Y. Chen, T. Matsubara, and T. Yaguchi, "Neural symplectic form: Learning hamiltonian equations on general coordinate systems," in Advances Neural Inf. Process. Syst., 2021.
[28] Y. Huang, Y. Yu, H. Zhang, Y. Ma, and Y. Yao, "Adversarial robustness of stabilized neural ode might be from obfuscated gradients," in Proc. Math. Sci. Mach. Learn. Conf. J. Bruna,
J. Hesthaven, and L. Zdeborova, Eds., 2022, pp. 497-515.
[29] B. Chamberlain, J. Rowbottom, M. Goronova, S. Webb, E. Rossi, and M. M. Bronstein, "Grand: Graph neural diffusion," in Proc. Inf. Conf. Mach. Learn., 2021.
[30] M. Thorpe, T. M. Nguyen, H. Xia, T. Strohmer, A. Bertozzi, S. Osher, and B. Wang, "Grand++; Graph neural diffusion with source term," in Proc. Inf. Conf. Learn. Representations, 2021.
[31] B. P. Chamberlain, J. Rowbottom, D. Bynard, F. Di Giovanni, D. Xiaowen, and M. M. Bronstein, "Beltrami flow and neural diffusion on graphs," in Advances Neural Inf. Process. Syst., 2021.
[32] Y. Song, Q. Kang, S. Wang, K. Zhao, and W. P. Tay, "On the robustness of graph neural diffusion to topology perturbations," in Advances Neural Inf. Process. Syst., New Orleans, USA, Nov. 2022.
[33] K. Zhao, Q. Kang. Y. Song, R. She, S. Wang, and W. P. Tay, "Graph neural convection-diffuss with heterophily," in Proc. Inter. Joint Conf. Artificial Intell., Aug. 2023.
[34] Q Kang, K. Zhao, Y. Song, S. Wang, and W. P. Tay, "Node embedding from neural Hamiltonian orbits in graph neural networks," in Proc. Int. Conf. Mach. Learn., vol. 202, Jul. 2023, pp. 15 7766-15 15 808.
[35] T. K. Rusch, B. P. Chamberlain, J. Rowbottom, Mishra, and M. M. Bronstein, "Graph-coupled oscillator networks," in Proc. Int. Conf. Mach. Learn., 2022.
[36] W. E. Boyce and R. C. a. DiPrima, Elementary Differential Equations and Boundary Value Problems. Wiley, New York, 1997, v.l.
[37] M. W. Hirsch and S. Smale, Differential Equations, Dynamical Systems and an Introduction to Chaos. Academic Press/Elsevier, San Diego 2004.
[38] A. Katok and B. Hasselblatt, Introduction to the modern theory of dynamical systems. bridge University Press, 1995, no. 54,
Cam-
[39] C.-T. Chen and B. Shafai, Linear system theory and design. New York: Oxford university press New York, 1999.
[40] D. Arrowsmith and C. M. Place, Dynamical systems: differential equations, maps, and chaotic behaviour. London: CRC Press, 1992.
[41] M. E. Sander, P. Ablin, M. Blondel, and G. Peyré, "Sinkformers: Transformers with dou- bly stochastic attention," " in International Conference on Artificial Intelligence and Statistics.
PMLR, 2022, pp. 3515-3530.
12

---

[42] R. A. Horn and C. R. Johnson, Matrix analysis. New York: Cambridge university press, 2012.
[43] M. De León and P. R. Rodrigues, Generalized Classical Mechanics and Field Theory: a geometrical approach of Lagrangian and Hamiltonian formalisms involving higher order derivatives. Elsevier, 2011.
[44] W. Curtin and H. Scher, "Mechanics modeling using a spring network," Mater. Research, vol. 5, no. 3, pp. 554-562, 1990.
[45] A. C. Da Silva and A. C. Da Salva, Lectures on symplectic geometry. Springer, 2008, vol. 3575.
[46] V. I. Amold, V. V. Kozlov, and A. I. Neishtadt, Mathematical aspects of classical and celestial mechanics. Springer Science & Business Media, 2007, vol. 3.
[47] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, "Graph attention networks," in Proc. Int. Conf. Learn. Representations, 2018, pp. 1-12.
[48] W. L. Hamilton, Ying, and J. Leskovec, "Inductive representation learning on large graphs," in Advances Neural Inf. Process. Syst., 2017.
[49] Y. Chen, H. Yang, Y. Zhang, K. Ma, T. Liu, B. Han, and Cheng, "Understanding and improving graph injection attack by promoting unnoticeabilityy" in Proc. Int. Conf. Learn. Representations, 2022.
[50] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," in Proc. Int. Conf. Learn. Representations, 2018.
[51] Z. Yang, W. Cohen, and R. Salakhudinov, "Revisiting semi-supervised learning with graph embeddings," in Proc. Int. Conf. Mach. Learn., 2016, pp. 40-48.
[52] O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, "Pitfalls of graph neural network evaluation," Relational Representation Learning Workshop, Advances Neural Inf. Process. Syst. 2018.
[53] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, "Open graph benchmark: Datasets for machine learning graphs," Advances Neural Inf. Process. Syst., vol. 33, pp. 22 118-22 133, 2020.
[54] Q. Zheng, X. Zou, Y. Dong, Y. Cen, D. Yin, Xu, Yang, and J. Tang, "Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning," Advances Neural Inf. Process. Syst. Track Datasets Benchmarks, 2021.
[55] W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang. "Graph structure learning for robust graph neural networks," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2020, pp. 66-74.
[56] Y. Li, W. Jin, H. Xu, and J. Tang, "Deeprobust: pytorch library for adversarial attacks and defenses," arXiv preprint 1eXivv2005.06111
2020.
[57] D. Zhu, Z. Zhang, P. Cui, and W. Zhu, "Robust graph convolutional networks against adversarial attacks," in Proc. Inf. Conf. Knowl. Discovery Data Mining, 2019, pp. 1399-1407.
[58) N. Entezari, S. A. Al-Sayouri, A. Darvishzadeh, and E. F. Papalexakis, "All you need is low (rank): Defending against adversarial attacks on graphs," " in Proc. Int. Conf. Web Search Data Mining, 2020, P. 169-177.
[59] L. A. Adamic and N. Glance, "The political blogosphere and the 2004 us election: divided they blog," in Proc. Int. Workshop Link Discovery, 2005, pp. 36-43.
[60] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint arXiv: 10w.17066000000
2017.
[61] X. Zhang and M. Zitikk "Gunguard: Defending graph neural networks against adversarial attacks," Advances Neural Inf. Process. Syst. vol. 33, pp. 9263-9275, 2020.
[62] OpenAI, "Chatgpt-4," 2022, available at: https//wwww.opnna..com (Accessed: 26 September 2023).
[63] Y. Ma, S. Wang, T. Derr, L. Wu, and J. Tang, "Graph adversarial attack via rewiring," " in Proc. Int. Conf. Knowl. Discovery Data Mining, 2021, p. 1161-1169.
[64] Y. Sun, S. Wang, X. Tang, T.-Y. Hsieh, and V. Honavar, "Adversarial attacks on graph neural networks via node injections: A hierarchical reinforcement learning approach," in Proc. Web Conf., 2020, P. 673-683.
13

---

[65] X. Wan, H. Kenlay, B. Ru, A. Blaas, M. A. Osbore,, and X. Dong, "Adversarial attacks on graph classification via bayesian optimisation," " arXiv preprint arXhv:2111.02842, 2021.
[66] S. Geisler, T. Schmidt, H. Sirin, D. Zügger,, A. Bojchevski, and S. Günnemann, "Robustness of graph neural networks at scale," Advances Neural Inf. Process. Syst. vol. 34, pp. 7637-7649, 2021.
[67] J. Ma, J. Deng, and Q. Mei, "Adversarial attack graph neural networks influence maximization problem," " in Prof. Int. Conf. Web Search Data Mining, 2022, pp. 675-685.
[68] B. Finkelshtein, C. Baskin, E. Zheltonozhskii, and U. Alon, "Single-node attacks for fooling graph neural networks," Neurocomputing, vol. 513, pp. 1-12, 2022.
[69] D. Zhu, Z. Zhang, P. Cui, and W. Zhu, "Robust graph convolutional networks against adversarial attacks," in Proc. Inl. Conf. Knowl. Discovery Data Mining, 2019, p. 1399-1407.
[70] W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, "Graph random neural networks for semi-super learning on graphs," " in Proc. Advances Neural Inf. Process. Syst., 2020.
[71] W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang, "Graph structure learning for robust graph neural networks," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2020, p. 66-74.
[72] X. Gao, W. Hu, and Z. Guo, "Exploring structure-adaptive graph learning for robust semi- supervised classification," in 2020 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2020, pp. 1-6.
[73] T. Zhao, Y. Liu, L. Neves, O. Woodford, M. Jiang, and N. Shah, "Data augmentation for graph neural networks," in Proc. AAAI Conference on Artificial Intelligence vol. 35, no. 12, 2021, pp.
11 015-11000 023.
[74] K. Li, Y. Liu, X. Ao, J. Chi, J. Feng, H. Yang, and Q He, "Reliable representations make stronger defender: Unsupervised structure refinement for robust gnn," in Proc. Int. Conf. Knowl. Discovery Data Mining, 2022, pp. 925-935.
[75] B. Runwal, S. Kumar et al. "Robust graph neural networks using weighted graph laplacian." arXiv preprint arXiv:2208.00 0/853, 2022.
[76] X. Zhang and M. Zitnik, "Gnnguard: Defending graph neural networks against adversarial attacks," in Proc. Advances Neural Inf. Process. Syst. 2020.
[77] S. Greydanus, M. Dzamba, and J. Yosinski, "Hamiltonian neural networks," in Advances Neural Inf. Process. Syst., 2019.
[78] Z. Chen, J. Zhang, M. Arjovsky, and L. Bottou, "Symplectic recurrent neural networks," in Proc. Int. Conf. Learn. Representations, 2020.
[79] A. Choudhary, J. F. Lindner, E. G. Holliday, S. T. Miller, S. Sinha, and W. L. Ditto, "Physics- enhanced neural networks learn order and chaos," Physical Review E, vol. 101, no. 6, p. 062207,
2020.
[80] E. Haber and L. Ruthotto, "Stable architectures for deep neural networks." Inverse Problems, vol. 34, no. 1, pp. 1-23, Dec. 2017.
[81] Li, Peng, Chen, Z. Zheng, T. Liang, and Q. Ling, "Spectral adversarial training for robust graph neural network," IEEE Transactions on Knowledge and Data Engineering. 2022.
[82] C. Deng, X. Li, Z. Feng, and Z. Zhang, "Garnet: Reduced-rank topology learning for robust and scalable graph neural networks," in Learning on Graphs Conference. PMLR, 2022, pp. 3-1.
[83] V. Nikiforov, "Revisiting schur's bound on the largest singular value," arXiv preprint math/0702722, 2007.
[84] H. K. Khalil, "Nonlinear control," 2015, available at: hutp://www..g..msuuedd--- hatteaaanooooo/SSiiieesssllle.e 2.pddf (Accessed: 26 September 2023).
[85] R. Durrett, Probability: theory and examples. Cambridge university press, 2019, vol. 49.
14

---

A Summary of Supplement
In this supplement, we expand on material from the main paper, addressing each point follows:
1. An in-depth comparison with related work covered in Appendix
2. Appendix C presents the datasets and details of the attackers, and includes additional experimental results, inference time, and model size to reinforce model's effectiveness.
3. Theoretical proofs supporting assertions made in the main paper are presented in Appendix E.
4. Further discussions on the robustness insights of our model are covered in Appendix D.
5. The complete summary of the algorithm can found in Appendix G.
6. Lastly, we address the limitations of our work and discuss its broader impact.
B Related Work
In what follows, we briefly review a few concepts closely related to our work.
Graph Adversarial Attacks and Defenses. In modification attacks, adversaries can perturb a graph's topology by adding or removing edges [9, 14-17, 63-66]. To improve the modification attack performance, adversaries are also permitted to perturb node attributes [9. 17,63,, 63, 64, 67, 68]. In injection attacks, adversaries can only inject malicious nodes into the original graph [10-13] while the edges and nodes inside the original graph are not allowed to be perturbed. For defense methods against adversarial attacks, multiple robust GNN models have been proposed. Examples include RobustGCN [69], GRAND [70], ProGNN [71], GLNN [72), GAUGM [73]. STABLE [74] and RWL-GNN [75]. In addition, preprocessing-batt based defenders including GNN-SVD [58] and GNNGuard (76) may help to improve GNN robustness.
In this paper, we use different attacks to test GNNs robustness. We compare graph neural flows with different stability settings with the above-mentioned defense methods as robustness benchmarks.
Stable Graph Neural Flow Networks. While traditional GNNs perform message passing on simple, discrete and flat space, graph neural flows model the message passing as a continuous diffusion process that occurs on smooth manifold. GRAND [29] and GRAND++ [30] use heat diffusion to achieve feature information exchange. BLEND [31] exploits the Beltrami diffusion where the nodes' positional information is updated along with their features. GraphCOO [35] adopts the coupled oscillator model that preserves the graph's 's Dirichlet energy over time and thus mitigates the oversmoothing problem. In general, [32] shows that graph neural PDEs are Lyapunov stable and exhibit stronger robustness against graph topology perturbation than traditional GNNs.
While most of the above-mentioned graph neural flows are Lyapunov stable, whether the notion of Lyapunov stability leads to better adversarial robustness is an open question. In this paper, we argue that Lyapunov stability does not necessarily imply adversarial robustness.
Hamiltonian Neural Networks. Hamiltonian equations have been applied to conserve an energy- quantity in (graph) neural networks. The references [26, 27, 77] train a neural network to infer the Hamiltonian dynamics of a physical system, where Hamiltonian equations are solved using neural ODE solvers. In [78], the authors propose to learn Hamiltonian function of the system by neural network to capture the dynamics of physical systems from observed trajectories. They shows that the network performs well noisy and complex systems such spring-chaie system. To forecast dynamics, the work [79] use neural networks that incorporate Hamiltonian dynamics to efficiently learn phase space orbits and demonstrate the effectiveness of Hamiltonian neural networks on several dynamics benchmarks. The paper [80] builds a Hamiltonian- -insiied neural ODE to stabilize the gradients as to avoid gradient vanishing and gradient exploding.
In this paper, inspired by existing Hamiltonian neural networks, we introduce several energy- conservative graph neural flows. We are neither simulating physical system nor forecasting forecast the dynamics for a physical problem. Instead, we combine the Hamiltonian mechanics concept with graph neural networks to develop new robust GNN.
15

---

C More Experiments
C.1 Data and Attackers
The datasets and attack budgets utilized in Table 2 and Table 3 are outlined in Table 5 and Table 6, respectively. These datasets span various domains and scales, thereby providing diverse hase for our study. The adopted attack budget aligns consistently with the specifications set out in the paper [49].
Table 6: Attacks' budgets for GIA. refers to targted GIA.
Table 5:
Dataset Details
Dataset
# Nodes
Edges
# Features
# Classes
Dataset
max # Nodes
max # Edges
Cora
2708
5429
1433
Cora
60
3327
3703
Citeseer
90
PubMed
19717
44338
500
PubMed
200
100
Coauthor
18,333
81,894
6,805
15
Coauthor
300
150
Computers 13,752
245,861
767
10
Ogbn-Arxiv
120
100
Ogba-Arxiv
169343
1166243
128
Computers" 100
150
C.2 Implementation Details
The raw node features are compressed to a fixed dimension, such as 64, using a fully connected (FC) layer to generate the initial features (()) (10). At time p(0) and q()) are initialized identically. For t > 0, both (t) and p(t) undergo updates using a graph ODE. The ODE is solved using the solver from [18]. is observed that different solvers deliver comparable performance in terms of clean accuracy. However, to mitigate computational expense, the Euler solver is employed in our experiments, with an ablation study on different solvers provided for further insight. The integral time 7 acts as hyperparameter in our model. Interestingly, the performance of the model exhibits minimal sensitivity to this time For all datasets, we establish the time as and maintain fixed step size of 1. This setup aligns with a fair comparison to three-layer GNNs.
All the baseline models presented in Table and Table 3 are implemented based on the original work of [49]. The baseline model results in Table are directly extracted from the paper [55]. This is done as we employ the same clean and perturbed graph datasets provided in their research [55].
Our experiment code is developed based on the following repositories:
http:://gttaab.com////aaaaaLLLL
C.3 White-box Attack
In our main study, we utilized black-box attacks. Now, we extend our experiments to incorporate white-box, injection, and evasion attacks. In the context of white-box attacks, the adversaries have full access to the target model, enabling them to directly attack the target model to generate a perturbed graph. This represents significantly more potent form of attack than the black-box variant. Moreover, we execute inductive learning tasks the same as Table 2, with the corresponding results reported in Table 7, We observe that, under white-box attack conditions, all other baseline models exhibit severely reduced performance, essentially collapsing across all datasets. The classification accuracy of the HANG model experiences a slight decline the Cora, Citeseer, and Pubmed datasets. However, its performance remains substantially superior to other graph neural flow models or GNN models. Intriguingly, our HANG-quad model remains virtually unaffected by the white-box attacks, maintaining a performance level similar to that observed under black-box attacks. Such robustness of HANG-quad underscores the pivotal contribution of Lyapunov stability when combined with our Hamiltonian-drive design.
16

---

Table 7: Node classification accuracy (%) on graph injection, evasion, non-targeted, white-box attack in inductive learning. The best and the second-best result for each criterion are highlighted in red and blue respectively.
Attick
HANT
HANG-qua
GraphCONN
GruphBel
GRAND
CAAT
GraphSAGiF
GCN
eleas
87.13+0036
72,68:20.22
86.7720.51
86.1340051
87.53
67.58:20.64
16.65
88.51+0045
Cora
TDGIA
PGD
64. 44.544230 3.05
07.09443384
78.04+0991
77. 6763
42.09::1.74
19:01
37.16+11.99
1544 1.98
36002444009
14.72::9977
30.95+88.22
1.11
29.19-73666
17:83:662
13.05 1.00
35.83--0,71
cleas
74.11+0062
7188:20.48
74.8420.499
69.6220556
67787249.77
63.22499114 14
12.63+1-114
Chesser
TDGIA
PGD
17:54+1-52
13.15
72.21-0.71
70.62 0.096
42.78 1.54
33.55
16.26 10266:1:00
12.24
38.57+1954
30.11::1442
25.87-+6.09
17.46 3.334
2966-444.11
30.09+22.33
21.10
clean
96. 16:10.09
95.27:10.12
95.10:10.12
93:911048
9508-10.11
92.1110411
93.01:0.39
93.33110.37
Counthor'S
TDGIA
PGD
93.40---211
93.18.10..77
94.12:100:3
90.25-+1.00 L02
1.35 1.61
1.10-11118
13.21+44.21
10.38 1.07
4.53-1:1.33
11.96+7.110
1.25 1..55
10.73--6.84
2.89::1.65
11,0245.04
3,61 1:177
clear
89 27
88 0.:10.333
08.78:1.000 66
06.9711..377
88.41:00334 34
1.741:11:73
88:711:::37
88, 38.46::0020
PGD
68 63:8222
87 0164200.3 79
36, 16.8622..65
39. 0.77
59 51.1
58.04229991
58 76466.455 58
39 39.03:20000 10
Pubmed
TXXIA
69 09.56:23116
87.91:20.66
31,49%11.5 1.87
30. 1..15
36 79:1997
34.43:44110
38 38.8920077 76
42.641::441
C.4 Attack Strength
We assess the robustness of the HANG model and its variant under varying attack strengths, with the node classification results displayed in Table 8. Our analysis reveals that the HANG model demonstrates superior robustness as the attack budget escalates. It is noteworthy, however, that under larger attack budgets, HANG-quad may relinquish its robustness attribute in the face of PGD and Meta injection attacks. This result implies that the amalgamation of Lyapunov stability and conservative Hamiltonian system facilitates enhanced robustness only under minor graph perturbations. On the contrary, the HANG model, which solely incorporates the conservative Hamiltonian design, exhibits consistently pegfoperformnnce level regardless of the increasing attack strength.
Table 8: Node classification accuracy (%) on graph injection, evasion, non-targeted, black-box attack in inductive learning under various attack strength.
Daizeet
Astack
A nodeledges injected
HANG
HANG quad
GraphCON
GraphBel
GRAND
GAT
Gra
GCN
PGD
SIN4
75.78-3344
+0.41
33.60:22.622
33.74=2884
1.04
32
26
31.31+00.66
PGD 10065
74.25
76 08.7411048
62 00.69
33.65_2.50
12.72:1477 1447
21
25 75.44
31.23::0000
PGD
120050
71.01 2.60
70
33 46 1.95
34.88= 3.66
52.21 +0.3
01.28-0.155
7
31.23-20.0
PGD
140/100
69. 00.72:22..44 2.94
33.55
34.02
55.31
25.11-27.66
31,23-10.0
PGD
160/120
68. 2.05
45.06-2:777
2.334:2.27
33.07
01
74
31.23-0000
PGD
1801140
66.51
46.6912:002
33. 15.19
45.11=4.76
15.1221164
00.41
74.79---500
31.23-20.00
PGD
200/160
66 66.01:4.100 10
29 39.58:15:11
32, 32.99 2.449
50.51
52.18:10..9
31.47-0.70 70
24 74.74:7996
31.23-1000
TIGHA
78.4810055
1.51
18.70-1.18
22.34
25
26.51
TDGIA
10060
78.73 1.20
75.74-20994
4.16
36.772227
27.10+2004
30.3
30.68 01.8-:0330
TDGIA
120880
78 48 1.91
78:20110778
29. 22:12 1.91
26.4322558
29.06.12224
20.6711.34
30.37:1:441
31.28810.87
Com
TXGIA
140/100
79 70.40-22.20 3.20
77.22:0711
23 23.54:211 1.49
25.54 25.54=4.66 4.26
27.042325
26.996005
28 28.55
11
TIGIA
160V120
99.37 1.30
18.05
34000
2.37
2492
19.744683
29.78-2:666
30.094:0080
TDGIA
180/140
78 49 2.34
06:00:05557
31.20:11133
6.367=6.0
31,44:332
10
31,23:1000
31.23-1000
TDGHA
201V160
78.85 2.222
76 76.70111000 OZ
24 14.9421.11
34.32-1.11
19.37
24641698
$1 $1.07
11. 11.1420.33
MelaGI
KQ/00
2.74
18:10:10071
3..3
33.58 2.74
33.43 20
32 1.77
29 58 58.19.00
31.28110.10
MenGIA
10060
73.23-1..66
33.01 53
30:00.0000
28
11.22:1::04
MeuGIA
120090
73.68=11000
18.21 41
32.91:1:447
47.
22.79
31
24 76
31.23-20000
MeaaGI
140/100
72005:2.87
59.771:2771
1.:97
38.77 5.09
33. 13.12:1:1 1.05
$1.65:1 1.22
26.46.1.550
31 31.23:100.0
MesGIA
160/120
2.20
17 717887788 89
33. 2335
57.77 3.18
12
51.42100000
24 8.:0
31 31.23-2000
MeuGiA
18V110
4.447.44.44
30.31 20033
1.00
47,48-476
25.1512226
25.42.2..15
31.23-1000
MeaaIA
200/160
68. 2.77
16.69:16.29
68:1 1.38
64.38 1.73
22.52:1:00
28.79:17.09
24.69.18.00
31.23:1:000
PGD
110/50
79
08-00-0000 70
3.441
41.09
3.54
1..49:1.044
20.55::4444
18.63-2002
PGD
150550
71.61
71 0.22-7722
22 42.90:4.773
43.05= 1360
26.4266166
18.3921.755
19 20.16:22.0
17.85
PGD
150770
00.68
70.33 0:99
24 28 4.68
38,89 13.86
50.63 4
18
18.73+22.19
17.42+0051
PGD
170550
0.67
7.770
21 21
19
17.51
10.65
PGD
1901110
71. 0.55
12.26:1:1551
38:1301
19.95 2.00
17.314440
17.4 34
17.85
PGD
210/130
71.13
50.15 1.888
24 3.500
31.26-4.511
2006 + 1:23
17:33+3440
17. 17.99 33
17.57
PGD
250V150
71. 13 0.85
16.01-2.113
38 162
35.71 4884
21 2.23
17311140
17 17.58.201
17.75
TIGIA
110/50
48 0.66
12.30:::0.8
1.73
19.96
20.24-1:97
28
TDGIA
130050
72. 02.26:1..77
27.34:12.22
2..0
23
18
21.16:1266
20. 08
TDGHA
150070
72. 15 0.55
72.41 1:0 86
41
17.39-1.225
22.31 1
10
19 29
20 00.3912.10 15
Citssee
TDGIA
17050
72.41 0.66
72.06-20:98
21 63 1.18
25.25=1 1.94
20.05
18 24
19.09-22224
19.57
TDGIA
190/110
72.
180411..55 59
1.:11:1::72
21.53:1:2999
TXGIA
210/150
72. 0.44
3.51
14.
11
28.49-2.111
16
TIGHA
250V150
0.771
71.76 71.76:50.81
29.01:22.74
28.85-166
19.28 1.772
25.742265
20.76-400
18.7
MenGIA
T10/30
72. 72.22 1.38
72.41:10.67
22. 2.28
20.95 2.29
50.29-14.58
21.77-1291
22. 18:12:22
18.72-10022
MeuGiA
190550
72.18:11.1
71:96:10.66 86
144
25.01
20.4412.63
20.67
18.162:0.1
MenGIA
150770
0.99
71.05:1:0911
24 24.69:14559
1.872_1882
26.7
19.1112885
20.24:13.00
18.19:10007
MeaGIA
170900
71 71.76:1:222 1.22
68.81 68.81110077
22. 2.22
20.80=197
29
186041268 68
19:00:1:.75
18.2710000
MeuGIA
191V110
71. 0.66
64 44.44-10777
23. 29224.28
22.912999
26.28 2.54
18.992201
18. 18.70-2:2344
18.28:00.03
MeaGIA
210/130
71.39:10.72
$6.44122.42
22.56.13.09
2.59
26.411441
18.7112..4
1:20
18.27:0000
MesGIA
230V150
71.52:20.65
13.45-13.49
34.1333008
48.37=2855
2677
18.58:1:.53
18.9 29.50:23
18:37:0000
17

---

C,5 Nettack
We further evaluate the robustness of our model under the targeted poisoning attack, Nettack [9]. Adhering to the settings outlined in [55], we select nodes in the test set with degree greater than 10 to be the target nodes. We then vary the number of perturbations applied to each targeted node from to 5, incrementing in steps of 1. It is important to note that Nettack only involves feature perturbations. The test accuracy in Table refer to the classification accuracy on the targeted nodes. As demonstrated in Table 9, our HANG-quad model exhibits exceptional resistance to Nettack, thereby underlining its superior robustness. This suggests that the combination of Lyapunov stability and conservative Hamiltonian design significantly enhances robustness in the face of graph poisoning attacks. On its own, Lyapunov stability has been recognized to defend against only slight feature perturbations the input graph [32]. Furthermore, our HANG model also displays superior resilience compared to other graph neural flows, reinforcing the fact that the Hamiltonian principle contributes to its robustness.
Table 9: Node classification accuracy (%) under modification, poisoning targeted attack (Nettack) in transductive learning. The best and the second-best result for each criterion are highlighted in red and blue respectively.
Dutisett
Pbb
HANG
HANG--uud
GraphCON
GraphBel
GRAND
GAT
GCN
RGCN
GCN-SVD
Pro-GNN
78.54.2.110
36.9943.16
7..25=2.91
63.75
10.12:111
26.04:20008
75.06:1..00 1.02
76.75
77:25:
1181111666
3..73
72.611260
67.83::3.0
22.41
76277:1.79
70.21:::443
10.60:1.10
70.96:1.116 1.14
72.53-1:160
75.90:11.43
68,4322423
13.13423:5
68.10:2.10
61.20
30.48:1.744
6..5411.34
67.95=1.72
16.51
66.75:1:10
70.12:::999 999
66.02-2.211
22.5142:14
57.5022.54
2.77
65:30:2.40
00:00.00
61.57=1447
59.28-2668
60.77:1
65.66:2.335
3..12:33.63
68.50:1:2.55
53.30:1:..7
2..93:2.77
$7.95
58.31:1:003
55.54 1.66
1.66
S1.71 1.82
64.34.11.22
36.03:33551
10.06::::3 38
76.03:2.44
68 2..89
00001 1.05
$1.07:::130 1.18
78.41:21:0 1.62
78,25
80 2,0
81.75:10779
2
2.500
77 17944222200 29
68. 6.662
67.62 11
74 74.2477777
T7 474414489
74.92=3.55
75.40:20044
79.543:0.73
81 81
Citeszer
74.76:22.18
77.
60.47:5224
601.6323.88
57.1441922 28
60855:2999
63.97 3.69
1.19
77.144
79.084
73.49=3.00 3.02
78.41
6.255
53.17
59
61 59:1:4 64
55 15.40:22.60 2.60
15,49:17755
69 50.57:13.11
77.78:22.4
72.06 1,56
73.49 48
51.75=2.77
48. 48.75
48.41 1
66.666662 28
47 5.17
47.44
69.21 6::21
71.27::4999
17.06:20.66
07337
17.07=1.35
38.17:3.255
96.41 2:0.7
97.22::0.25
06.83
97.00-0.077
97 07.56:::0.00
96883+00066
96.39 1.15
96.19-00.16
82.92=1.55
65.48--2.85
92.53 4-421
96.11=0.655
05.61=0.20
95.87-00.25
91.12-+-009
97.17-4-0.12
Polblogs
06.02:0.95
95655-0111
80.7620.04
62.59
91.96:2-222
01.81:0055 56
96.4120:11
05.59-0.27
96.610031
96.32-0.12
71.46=155
0.40
94:0--0666
94.24-0.224
94.37-0.26
96.17:4-0119
9639---0116
91.65--4.67
95.91 1:33
73.30-22.71
39.02-33119
83.09 5-388
93.28 13.28-11.44
93.00-20.48
93,20-00.45
95.13-4-0225
96.13-4-0.25
C.6
ODE solvers
The results from various ODE solvers depicted in Table 10. We consider fixed-step Euler and RK4, along with adaptive-step Dopri5, from [18], and Symplectic-Eur from [35]. The Symplectic- Euler method, being inherently energy-conserving. is particularly suited for preserving the dynamic properties of Hamiltonian systems over long times. Our observations suggest that while the choice of solver slightly influences the clean accuracy for some models, their performance under attack conditions remains fairly consistent. Consequently, there was no specific optimization for solver selection during our experiments. For computational efficiency, we opted for the Euler ODE solver in all experiments presented in the main paper.
Table 10: Node classification accuracy (%) on graph injection, evasion, non-targeted, black-box attack in inductive learning of Citeseer dataset.
Atback
Solver
HANG
HANG-qual
GraphCON
GraphBal
GRAND
74.11:00.62
713540.4
7..84-0.49
69.62:00556
74.984:0.45
cleun
symplettc culer
rk4
73.71::1.58
71.35=11.91
72.61:0:5
72.002004
7..43=0.22
0..80
69 69.87-4778
74.461:0666
75.32:10778
75.70::0.71
depris
75.204:0:95
76
14.65-10.70
75.14400.66
Ealer
12.31+-1.16
71.0740441
40.56-0.36
55.67-4-533
36.08---1005
PGD
sympleetic enler
70.57:11.74
71.85:2.04
72.30-1052
41.26=0:89
40.87-266
41.51
49.09--738
11.51-1:.57
1.52
daprz5
13.59-50.38
10.20-2.21
40.07 4-387
40.53-11.14
Euler
12.12:1:0.2
01:69:1::40
36.67 1.25
34.17:14.68
36.67:1:255
1..63:2:1.4
72.85:1776 78
36.19 2.03
1..90111700
14.211:163
TDGIA
symplectic culer
71.79--2200
73.3140338
35.32-1.78
1.622-1162
depriS
72.14:1:0.71
38.04 1.71
10.63:1:1.0
14.99111199
Euler
72.92-10.66
71 11:60:14448 48
48. 28.36=2.12
45.6011:331
4623312001
r44
70.25:11.45
72. 11190061
12.77_1.09
49.58100.88
41.011:.02
MetalaiA
symplectic euler
71.56:11.00
72.86:10.22
12,57=1.09
16.72:16.28
41.01.011
depri5
71.83:11.26
42.61=0.57
42.95-1:0.1
41.74:1:.00
18

---

C.7 Computation Time
The average inference time and model size for different models used in our study outlined in Table 11. This analysis is performed using the Cora dataset, with all graph PDE models employing the Euler Solver, integration time of 3, and step size of 1. Additionally, for fair comparison, all baseline models are configured with layers. Upon examination, it is observed that our HANG and HANG-quad models necessitate more inference time compared other baseline models. This is primarily due to the requirement in these models to initially calculate the derivative. However, when compared other defense models, such as GCNGUAD, models are still more efficient, thus validating their practical utility.
Table 11: Average inference time and model size
Model
[ANG
HAAGG-qua
CnibCCON
CruphBlel
GRAND
GruphSAGE
CONGUARD
RGCN
Inforence (ms)
17.16
14.41
2.60
6.18
2.25
95.00
2.50
Model Size(R)
0,895
0.9%
0,752
0.734
0.732
0,380 0.380
0.735
C.8
Combination with Adversarial Training(AT)
Adversarial training (AT) [60] is technique employed improve the robustness of machine learning models by exposing them to adversarial examples during the training phase. These adversarial examples maliciously crafted inputs designed to mislead the model, thereby helping it to learn more robust and generalized features.
In this context, conducted additional experiments applying PGD-AT [60] to both GAT and HANG models, evaluating their robust accuracy under PGD, TDGIA, and METAGIA attacks in both white- box and black-box scenarios. As illustrated in Table 12 and Table 13, while PGD-AT enhances the robustness all models, its effect on GAT's resilience to the unique white-box METAGIA attack is notably restrained, highlighting the limitations of PGD-AT's efficacy, particularly when faced with non-AT tactics. Conversely, HANG consistently displays remarkable performance across a variety of attacks, showing significant improvement when combined with PGD-AT. Both HANG-AT and HANG HANG-quad-AT exhibit substantial resistance to these attacks, underscoring HANG's inherent robustness.
Furthermore, we explored adversarial training from a graph spectral perspective, incorporating Spectral Adversarial Training (SAT) [81] into our HANG framework, thus creating the HANG- SAT variant. The integration of SAT, as depicted in Table 12 and Table 13, significantly boosts HANG's robustness against various attacks, with minimal reduction in post-attack accuracy, further exemplifying HANG's s intrinsic resilience.
Table 12: Node classification accuracy (%) on graph injection, evasion, non-targeted, black-box attack in inductive learning.
Ditaset
Attack
HANG
AT
GAT
GAT-AT
HANG-SAT
cleon
89 30.03:1..227
89.4510:39 33
87.4121.73
85.88:20.47
88 08:38820.48
Pubesed
TDGIA
PGD
81.81:21:44
86.02:::.05
87.164:339 39
89:01:0 19
12.94
47.5633.111
77.78-8.99
79.5625.76
82.55:22.11
87.6521177
MaaGTA
87.58:10075
880110010
44.55:3.53
81.86:21.44
88:22:20.22
Table 13: Node classification accuracy (%) on graph injection, evasion, non-targeted, white-box attack in inductive learning.
Attack:
HANG
HANG-AT
GAT-AT
HANG-SAT
clem
89.91-1:0.77
29442-1-222
87.412:..33
85 85.90:20:44
88.38=0.48
PGD
TDGIA
69.56:1.3.16
88 88.96.1:0200
24.43-14.10
12.16:14.110
88.28-0.34
MatGGA
84.64-2:1.20
88.61:1:030
40.02:1:334 34
44.3726.48
85.76_24122
19

---

C.9 Combination with GNN defense mechanisms
We acknowledge the potential advantages of integrating defense mechanisms that operate through diverse strategies. In this context, we combine HANG with GNNGuard [61], preprocessing baasdd based defense renowned for its effectiveness in enhancing GNN robustness, and GARNET [82]. another noteworthy defense mechanism.
We conduct experiments assess the performance of HANG when amalgamated with GNNGuard and GARNET. The results in Table 14 and Table 15 clearly indicate that such integration effectively enhances the model's robustness. This outcome highlights the adaptability of our methodology with current defense strategies and sheds light on the potential for cooperative improvements in model defense.
Table 14: Node classification accuracy (%) on graph injection, evasion, non-targeted attack in inductive learning.
Attack
HANG
HANG-GUARD
HANG-qud
HANG-qual-GUARR
87.13:00.66
96.54:20.57
79.68-2066
81.23=0..00
Cora
TDGIA
PGD
18.37::184
79.76:20099
85 85.56.20..3
86.3:10.55
79 00:54::066
79:05::044
81.11
00.91:0067
MentilA
77.48 1.02
86.0=20.0
78 78.28-0:056
80.10=10..5
clean
74.11+00.22
75.95=20.66
71.85 +:0.48
73.15=00.61
PCT)
12.11:1.11
05.38200.83
71.07:0:44
TDGIA
0.654=0069
71,62:00.44
MetaliAA
72.922-0.06
75.22:
711 71.00::00.44 48
73.11 9:045
clean
89.93-1-0.27
39.96:30.225
88. 88.10:1:0.33
88.93-10.115
Pubmed
PGD
$181=1994
87.72=20.89
87.49-50055 0.57
88.99=00.11
TDGIA
88.86-20.440
87.55
88.80+00.12
MetaGlA
88.23:30.93
87.40:1:0.6
88.78
Table 15: Node classification accuracy (%) on graph Nettack targeted attack in transductive learning.
Datesel
Pib-rate
HANG
HANG-GARNET
HANG-qun
HANG-quiI-GARNTT
GCN
GCN-GARNET
75.54=2.10 10
82.41
T6 36.99:3.16
83.01+0084
70.06=00.81
79.75-22.335
33.73:23.64
1.584:::557
76651-22600
79.38-0055
68.60=1181
19.50-1:150
Com
4.243
80.884:169
79.76::007
65.04=3.511
74.47:22.06
66.0222.221
70021 1.77
72.51:2.144
75.90:0.554
61.6921.448
69 69.60422.67
60:12:3.63
67.83-1:.87
68 28.30:2:2.55
69 19.28-11.34
15.66
67.94-2:.05
D
More Insights about Model Robustness
For a clearer insight into the energy concept within HANG, consider the "time" in graph neural flows as analogous to the "layers" in standard GNNs (note that ODE solvers basically discretize the "time", which indeed approximately turns the model into layered one). Here, the feature vector qtt) and the momentum vector ptt) evolve with time, bound tightly by equation (10). Given that (t)) mirrors the node features at layer f, ptt) can be understood as the variation in node features over time- essentially, the evolution of node features between successive layers. Thus, our defined energy interweaves both the node feature and its rate change across adjacent layers. The constant Hnet implies inherent constraints Heet processes the whole graph data and yields a scalar, it serves as a constraint on the global graph feature and its variation, which we opine to be crucial in countering adversarial attacks.
the node features and their alteration pace over layers. Because
From an adversarial perspective, the attacker modifies either the node features or the underlying graph topology. These modifications are propagated through multiple aggregation steps, such as layers in conventional GNNs or integrals in graph ODEs. While the Hamilronian considers the energy of the entire graph, adversarial attacks often target localized regions of the graph. The inherent global energy perspective of the Hamiltonian system makes resilient to such localized attacks, as local perturbations often get "absorbed" or "mitigated" when viewed from the perspective of the entire system. When adversarial perturbations are introduced, they might indeed tweak the instantaneous features of certain nodes. However, the challenge lies in modifying the trajectory (or evolution) of these nodes (positions p(t) and the variations (()) the phase space in a manner that is aligned with the rest of the graph, all while upholding the energy conservation constraints. This feat is arduous,
20

---

if not impossible, without creating detectable inconsistencies elsewhere. This property ensures that the energy of each node feature preserved over time and multiple aggregation steps. As a result, the distances between features of different nodes are preserved if their norms differ initially before the adversarial attack. The robustness of HANG against topology perturbation may stem from the fact that adversarial topology perturbation has small influence on the node feature energy. This can be seen from Fig. 1. When using HANG, the node feature before and after adversarial attack are relatively stable and well separated between nodes from different classes.
E
Proof of Theorem 1
Recall that
dxx(t) dt
(x((())X(() -X)
(15)
Without loss of generality, we assume X E IR v|x1 since the results be generalized X R on a component-wise basis.
[
Proof of 1).
Given our assumptions where Ag(X(t)) is either column- or row-stochastic, the spectral radius of Ac (x))) is 1, as established by [42][Theorem 8.1.22]. This implies that the modulus of its eigenvalues most 1.
We define a Lyapunov function as ((X())) = (()) = (x where || is the la euclidean
norm. Take the derivative of V with respect time, we have
(I) = (1)()((( + (1) (1) (A)(X()) + (x())) X(()
(16)
We next prove that v((x((1) < 0 when (x()) is a doubly stochastic attention matrix under the GRAND-nl setting.
v(x(t)) xx'
(X)())=XXX(((((((())) (Ao(X(t)) X X(t)
(17)
-XX(0AA(XX()X(( (xx
Note that
X (X)X(( S (XAXA/XXXXXXXXXXXXXXXXXXXXXXXXXX
(18)
where I|-II22 is the spectral matrix norm induced by & vector norm. The given inequality
direct result of the Cauchy-Schwar inequality and the properties of the induced norm. Drawing from [42][[hheorem 5.6.9], we can currently establish that Ag (X(t)ll2 intend to assert that this inequality is indeed an equality when Ax(X(t) is doubly stochastic, i.e.,
Nevertheless, we
A| (X(1) = 1. To see this, from Schur's bounds on the largest singular value [83][eq.(I)) we have
(X())l) S [[)
where I|l is the induced & norm and || is the induced tor norm. Given that (x(((t) is doubly stochastic, we observe that ||| (x((()))).. (x < 1, the equality is consequently confirmed. It follows from (18) that
(X(((l 1. It now follows that
X ((())((()) s |X|| (T)(((())
(19)
From (17) and (19), it follows that
v((()) < 2(11
(20)
when a 2 1. The observation that V ((((t) does not increase over time ensures that X(t) stays bounded. In effect, also have proved Lyapunov stability concerning the equilibrium point
To prove asymptotic stability for the equilibrium point when a > 1, we need to show that the system not only remains bounded but also approaches 0 as t + This is indicated by the fact that
21

---

when a > 1, (t) < O unless ( O. This signifies that (t)( strictly declines over time unless (t) 0. thereby showing that the system will converge as oo. Notably, we can infer global asymptotic stability since V is radially unbounded [84].
The proof for the GRAND-nl case complete.
Proof of
We next prove the claims under GRAND-I setting.
When Ag is either column- stochasa,,,ic, the spectral radius of (x(((t)) is established by [he]rm 8. 8.1.22]. The modulus of its eigenvalues is as most 1. Consider the Jordan canonical form of A represented as SJS where J stands as the Jordan form.
Given that our equation system now represents a linear time-invariant ODE, the solution to (15) can be expressed as:
X(t) = Aat X(0)
-Sc"
(21)
where J = aI. Citing [42][3.2.2. page 177, 4-th equation], for > 1, BIBO and Lyapunov stabilities are attained. Notably, for >> 1, the system achieves global asymptotic stability around the equilibrium point under any disturbances.
Further, by considering strong connectedness and referencing the Perron-Frobenius theorem [4e] orem 8,4.4], it is deduced that serves as the simple eigenvalue for Ag. In this case, when a we still have BIBO stability and Lyapunov stability according to [42][3.2.2, page 177, 4-th equation].
Proof of 3).
We subsequently demonstrate that when Ag selected be column-stochastic and 1, GRAND conserves a certain quantity interpretable as energy. Specifically, this "energy" refers to the sum of the elements of X(),, denoted as (1)(t), with representing an all-ones vector. From (15), this quantity remains conserved if
0 d1) (X) dt
(22)
Given that Ag oolumn-stoocastte it follows directly that 1 Ae = OT, validating (22).
Finally, we aim to prove that GRAND is asymptotic stable concerning a specified equilibrium vector when a 1 and the graph is aperiodic and strongly connected. Based [42][Theorem 3.2.5.2.,11 22,522.TThe The orem 8.5.3.] and [8e][[h 5.6.6,, we observe that limg- (Ag)* = limg-xoc S.J = SAS-1 where A is diagonal matrix with the first element as 1 and all the others as 0:
)
A
Since limg-+50 (A maintains its column stochasticity and the rank of SAS-1 is 1, it follows that each column is the same. Due to the zeros in A, the limit the outer product of the first column of S and the first row of S-1. Since we have the same column for the outer product, we deduce that the first row of S-1 is a1T with being scalar and all-ones vector. We have Ag has eigenvalue of O, with the rest of the eigenvalues having strictly negative real parts. According to [42][3.2.2], follows that
lim X(t) = lim Se J t-0
(23)
t-+
If 1'x(0) remains constant for any perturbed X(0), have that lim4-s0 X(t) where s is the first column of S. We thus conclude that if the perturbation on X(0) maintains the column summations, i.e. the "energy", unchanged, the asymptotic convergence the equilibrium vector sas remains unaffected.
The proof is now complete.
22

---

F Proof of Theorem 2
The energy conservation of our system, represented Huet in (9), remains invariant over time. To understand this better, consider the following
dHnet dt
He
Hent
Σ
q:
Op: pi
Dg:
i=1
H HHuc
H
(-
HHnet
+
(- 0q:
Oq.
Op:
Op:
0.
where the last equality follows from (10).
BIBO stability is inferred since, with an unbounded output, the energy conservation expressed by Het would be violated.
G Complete Algorithm Summary
We present the complete algorithm of HANG in Algorithm 1, which unfortunately had been delayed its inclusion within the main paper due to space constraints.
Limitations
While work graph neural flows presents promising advancements in enhancing adversarial robustness of GNNs using Hamiltonian-inspiee neural ODEs, it is not without limitations. As we demonstrated in the paper, the notions of stability borrowed from dynamical systems, such BIBO stability and Lyapunov stability, do not always guarantee adversarial robustness. Our finding that energy- cerrgyyconservvtive Hamiltonian graph flows improve robustness is only one facet of the broader landscape of potential stability measures. It is possible that other notions of stability, not covered in this work, could yield additional insights into adversarial robustness. Our current Hamiltonian graph neural flows do not explicitly account for quasi-periodic motions in the graph dynamics. (KAM) theory. foundational theory in Hamiltonian dynamics, is renowned for its analysis of persistence of quasi-periodic motions under small perturbations in Hamiltonian dynamical systems. While the energy-conserving nature of our Hamiltonian-insppl model inherently offers some level of robustness to perturbations, an explicit incorporation of KAM theory could potentially further improve the robustness, particularly the face of quasi-pent periodic adversarial attacks. However, this is complex task due to the high dimensionality of typical graph datasets and the intricacies involved in approximating quasi-periodit dynamies.
The wwwpoogorroo-ool----oooooo
Broader Impact
This research, centered enhancing adversarial robustness in graph neural networks (GNNs), carries implications for various sectors, such as social media networks, sensor networks, and chemistry. By improving the resilience of GNNs, greater efficiency, productivity, and effeectfffti The shift towards automation may displace certain jobs, raising ethical concerns about income disparity and job security. Moreover, while models enhance robustness, potential system failures can still occur, with impacts varying based on the application. Lastly, the robustness conferred might be exploited maliciously. Our work underscores the importance of diligent oversight, equitable technology implementation, and continuous innovation in the development of AI technologies.
boost the reliability of Al-driven systems, contributing to
23

---

Algorithm 1:
Graph Node Embedding Learning with HANG
Initialization: Initialize the network modules including Hamiltonian function network Huet, the raw node features compressor network FC, and the final classifier.
2 Training:
for Epoch 1 to .N do
1) Perform the following to obtain the embedding qu (T') for each node k:
Input: g (V,E) with node features
Apply FC to compress raw features for each node and get (0),(0), Pa
Here, we
divide the 2r dimensions into two equal segments. The first half acts as "position" vectors (0) = (4 gh. while the second half acts as "momentum" vectors (0) - (p,,..., pk) that guide system evolution. For simplification, in code implementation. we may the output feature dimension of the compressor FC as and designate pæ (0) = qu(0). (0).
2) To better express the evolution dynamics mathematically, concatenate and relabel the node features as:
(0)) (O),
(q: - q/ q (0))
p(0) ...
(0)()))...PP)(((()))
(24)
Note that in the actual code implementation, the concatenation of node features is not necessary as Het is realized as cither (12) or (13). The concatenate operation is only for mathematical formulation.
3) The trajectory of feature evolution is modelled as per the following canonical Hamilton's equations:
HH
HH
@(t) = dy
p(t)
(25)
0g
with the initial features (q(0), p(0) 22
Various ODE solvers as provided by [18] and the symplectic-euler solver from [35] can be employed to solve (25). We refer readers Appendix C.6 for more details.
10
4) Acquire the evolved features at time as (T)), which then decompressed into
individual node features qu (T) for further utilization.
11
5) Utilize backpropagation to minimize the cross-entrope loss for node classification.
12
6) Perform validation over the validation split.
7) Save the model parameters.
14 II. Testing:
Load the model from the best validation epoch and perform Step I.1-4), to obtain the final feature embedding over the test split. Perform node classification.
24


Reviews:
[{"review_id": "awyT5NCIzk", "reviewer": "Reviewer_bDPL", "summary": "Since neural ordinary differential equation networks can show inherent robustness, in this work the authors try to perform an extensive study on different graph neural flows along with their stability on different stability notions like BIBO stability, Lyapunov stability, Structural stability and Conservative stability. The authors find that the graph neural flows using hamiltonian energy functions can achieve improved empirical adversarial robustness on using black box attacks like PGD and TDGIA. The authors show that designing graph neural flows which can ensure conservation stability along with Lyapunov stability can help in achieving improved adversarial robustness. As common in literature the authors generate a white box attack from a surrogate model and then transfer it to the black box model to testify the robustness of the black box model. Evaluation is done on node injection attacks as well as graph manipulation attacks, and significantly improved robustness is seen in both cases.", "questions": "t would be great if the authors can address the raised concerns in the weakness section. ", "limitations": "Yes the authors seem to have addressed the limitations. I would suggest the authors to kindly look at the weakness section for suggestions and comments.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* I think using graph neural flows to understand if we can achieve inherent robustness without the need for adversarial training is interesting.\n* The way the authors have related different stability criteria with robustness is quite interesting.\n* The results show significant improvements.\n", "weaknesses": "* There have been many instances where defences claiming to be robust have been later evaded using adaptive attacks [1]. Adversarial training has been the most successful defence strategy and the current successful defences use adversarial training to achieve robustness. Therefore, it is necessary that the robustness of the proposed method is evaluated properly. I think that the surrogate model-based black box attacks are not strong enough to get the worst-case robustness. Therefore, it is important that the evaluation on strong white-box attacks like PGD, PGD with max-margin loss / Carlini and Wagner attack [2] is used to testify robustness, and this evaluation should be carried out in a white-box setting. Further, it is important that the authors share a robust accuracy (both white box and black box) vs constraint threat model plot for the proposed approach, GAT and GAT trained using PGD-AT [3]. \n\n* Comparison is done only with models trained using standard training. It is important to include adversarially trained models as baselines. For instance, the authors should include a PGD-AT trained model as a baseline in all the tables.\n\n* In the case of GNNs it is important to ensure that the attack graph remains imperceptible and the attacked nodes in the case of node injection attacks cannot be pruned off. Therefore other than robust accuracy imperceptibility of the graph for a given threat model is also a very important metric to be considered. Similar to [], I request the authors to also include a comparison on the imperceptibility of the attack on the proposed Graph neural flow network and the baselines like GAT. It is important to ensure that the imperceptiblity of the proposed defence is either lower or equal to the baselines against the PGD and TDGIA attacks.\n\n* The authors should also look at these works [4,5] and try to compare their methods with them .\n\n[1] Athalye, Anish et al. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.” International Conference on Machine Learning (2018).\n[2] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\"\n[3] Madry, Aleksander et al. “Towards Deep Learning Models Resistant to Adversarial Attacks.” \n[4] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.”\n[5] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# White Box Attack: Weakness 1\n\nWe are grateful for your attention to the robustness evaluation. We **do include the results of the white-box attack in Table S3 of our supplementary material.** The results clearly demonstrate that both HANG and HANG-quad exhibit superior robustness compared to the other baseline models. We thank you for highlighting this, and we will ensure that these results are appropriately emphasized in the main paper as well. \n\n# Adversarial Training (AT) and New AT Baselines: Weakness 2, 3 and 5\n\n\nThank you for your suggestion. We agree that comparing our model with adversarially trained models would provide a more comprehensive evaluation of its performance. However, we would like to emphasize the core objective of our paper is to elucidate the inherent stability and robustness of graph-ODE-based models.\n It is also noteworthy that our HANG model can be combined with adversarial training techniques, such as PGD-AT, to further enhance its robustness.\n\n\nTo address the reviewer's concern, we carried out more experiments applying **PGD-AT** to both the GAT and HANG models and assessed their robust accuracy under PGD, TDGIA, and METAGIA attacks, encompassing both white-box and black-box scenarios. \nAs depicted in ``Tables R1 and R2``, PGD-AT does provide an uptick in the robustness of all models. Yet, ``Table R2`` interestingly reveals that the effect of PGD-AT on the GAT model's resilience to the **distinct white-box METAGIA attack** is somewhat limited. This observation underscores the boundaries of PGD-AT's effectiveness, especially when attackers adopt tactics not synonymous with AT techniques.\n\nOn the other hand, HANG consistently achieves commendable performance across the spectrum of attacks, with a marked improvement when integrated with PGD-AT. It's noteworthy that  HANG-AT  largely withstands the onslaught of these attacks, a testament to HANG's inherent robustness.\n\nWe would also like to point out that the new baseline [4,5], referencing the same publication, delves into adversarial training from a graph spectral viewpoint. We incorporated the Spectral Adversarial Training (SAT) from [4,5] into our HANG paradigm, giving rise to the HANG-SAT variant. As shown in ``Tables R1 and R2``, the fusion of SAT notably elevates the robustness of HANG against diverse attacks, with minimal post-attack accuracy reduction, further demonstrating HANG's innate resilience. \n\nFurthermore, we have taken the initiative to augment our experimentation by including comparisons with recently proposed defense methods. For further details, please refer to ``Table R6, Table R7, and Table R8.``\n\n\n# Imperceptibility: Weakness 4\n\nThank you for your insightful comments regarding the imperceptibility of attacks on GNNs.  We understand the importance of this facet in adversarial studies and value the chance to elaborate on its role within our HANG framework.\n\n1. Firstly, it is important to emphasize that the nature of graph imperceptibility largely stems from the attack methodology, as highlighted in references [P1], [P2], and [P3], rather than the targeted model itself. Be it under white-box or black-box conditions, the subtlety of the graph structure post-attack remains consistent, regardless of the model in focus. Were there a new measure of imperceptibility, the research community would likely craft an attack to ensure its adherence.\nIn our work, the TDGIA, Metattack, and Nettack techniques utilized have built-in measures to ensure imperceptibility to the graph. These measures strictly adhere to the prescribed attack budgets. For a deeper dive into this aspect, we direct your attention to Table S4 in the supplementary material, which showcases experimental results across different attack intensities.\nFurthermore, it's pertinent to mention that the design philosophy behind HANG is distinct from defenses that either detect or make alterations to the graph structure.\n\n2. Secondly, we initiated an examination of unnoticeability by conducting a degree distribution test under various threat models. The results, presented in ``Figure R1`` of the attached document, unveil a high degree of structural similarity between the original and modified graph topologies, affirming the imperceptibility we aimed to achieve. \n\n\nAdditionally, reference [P1] sheds light on another advanced imperceptibility metric termed the homophily distribution. Here, homophily denotes the propensity of nodes to establish connections with peers sharing similar attributes or labels (Equation (6) in [P1]).\nTo provide more clarity, ``Figure R3`` in the uploaded pdf presents a comparison of the homophily distributions before and after the attacks on both HANG and the baseline models. \nTo rigorously evaluate our approach's resilience, we also incorporated the homophily constraint into the attacks using Harmonious Adversarial Objective (HAO)[P1]. This adaptation has resulted in a notably consistent homophily distribution, evident in the second row of ``Figure R3``. The results in ``Table R3``  demonstrate the enhanced performance of our proposed model over the baselines, emphasizing its robustness even with increased imperceptibility constraints.\n\n[P1] Yongqiang Chen, Han Yang, Yonggang Zhang, Kaili Ma, Tongliang Liu, Bo Han, James Cheng, \"Understanding and Improving Graph Injection Attack by Promoting Unnoticeability,\" ICLR, 2022.\n\n[P2] Fang J, Wen H, Wu J, et al. GANI: Global Attacks on Graph Neural Networks via Imperceptible Node Injections[J]. arXiv preprint arXiv:2210.12598, 2022.\n\n[P3] Liu Z, Wang G, Luo Y, et al. What Does the Gradient Tell When Attacking the Graph Structure[J]. arXiv preprint arXiv:2208.12815, 2022.\n\n\n## _Table R1, Table R2, and Table R3 are provided in the top global rebuttal window._"}, {"review_id": "7PRWNrRySc", "reviewer": "Reviewer_Yk3z", "summary": "This paper explores the robustness of Graph Neural Networks (GNNs) against adversarial attacks. Drawing inspiration from principles in physics, the authors propose a novel model called Hamiltonian Neural Flows for constructing GNN models. The effectiveness of the proposed method is evaluated on various benchmark datasets.\n\n", "questions": "see cons above", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem addressed in this paper is both significant and intriguing, and the proposed method is well-supported by principles from physics.", "weaknesses": "I am a non-expert in physics-based methods, my evaluation is based on my understanding of GNN methods and educated guesses. I suggest AC to ignore my evaluation if there are expert reviewers in the field.\n\nHere are a couple of suggestions to improve the paper:\n\nIn my opinion, the comparison with existing baselines seems inadequate. The TDGIA paper was published in 2021, while MetaGIA was published in early 2022 or late 2021. It would be beneficial if the authors could include more recent graph attack papers and compare their proposed method against them.\n\nAdditionally, it would be interesting to see a comparison between the proposed method and the Graph Isomorphism Network (GIN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Clarify and New Attacks:\n\nThank you for your valuable feedback and suggestions. We truly value the time and effort you've dedicated to reviewing our paper.  \nTo provide some clarity, our work offers a fresh perspective on GNNs' adversarial robustness by probing the stability of graph ODE-based GNNs when viewed as dynamical systems. This insight underlines the critical role energy conservation plays in enhancing robustness. Drawing inspiration from Hamiltonian mechanics, our proposed HANG model paves a novel path to fortify GNNs against adversarial attacks. Importantly, our model can cooperate with other defense methods, such as graph preprocessing or adversarial training, to amplify its robustness. We are optimistic that this approach will spark further investigations in this field.\n\nRegarding your suggestion to include more recent graph attack papers for comparison, we agree that it would be beneficial to our study. New experimental results using the GANI(L) attack from a recent paper [W1] are provided in ``Table R4``. However, this attack appears less potent than the attack strategies we have explored in our paper. As evidenced in ``Table R4``, **HANG consistently exhibits better robustness,** even when faced with this newer attack approach.\n\nAdditionally, ``Tables R1 and R2`` in our response to ``Reviewer bDLP`` and ``Tables R6 and R7`` in our feedback to ``Reviewer ehTk`` provide insights into our model's efficacy when combined with adversarial training and other pre-processing defense strategies, respectively. \n\n\n[W1]. Fang J, Wen H, Wu J, et al. GANI: Global Attacks on Graph Neural Networks via Imperceptible Node Injections[J]. arXiv preprint arXiv:2210.12598, 2022.\n\n| Dataset | Attack   | HANG         | HANG-quad    | GCN   | SGC   |\n|---------|----------|--------------|--------------|-------|-------|\n| Cora    | GANI(L)  | **77.48±0.92**   | 77.37±1.17   | 74.92 | 75.39 |\n\n**Table R4.** Node classification accuracy (%) on graph **injection, posioning, non-targeted GANI** attack in **transductive** learning.\n\n---\n\n# Comparison with GIN:\n\nThank you for your suggestion regarding the comparison with the Graph Isomorphism Network (GIN). We've taken your suggestion on board and incorporated GIN results into our evaluations, as shown in ``Table R5``. Notably, GIN, in line with other non-ODE-based GNNs, struggles under adversarial attacks. This result again highlights the unique resilience offered by HANG's energy conservation properties.\n\n\n\n\n\n\n\n\n| Dataset  | Attack   | HANG         | HANG-quad    | GIN         |\n|----------|----------|--------------|--------------|-------------|\n| Cora     | *clean*  | **87.13±0.86**   | 79.68±0.62   | 82.63±1.02  |\n|          | PGD      | 78.37±1.84   | **79.05±0.42**   | 32.86±0.23  |\n|          | TDGIA    | **79.76±0.99**   | 79.54±0.65   | 31.36±0.46  |\n|          | MetaGIA  | 77.48±1.02   | **78.28±0.56**   | 32.68±0.21  |\n| Citeseer | *clean*  | **74.11±0.62**   | 71.85±0.48   | 72.48±1.0   |\n|          | PGD      | **72.31±1.16**   | 71.07±0.41   | 28.20±3.37  |\n|          | TDGIA    | **72.12±0.52**   | 71.69±0.40   | 19.98±1.43  |\n|          | MetaGIA  | **72.92±0.66**   | 71.60±0.48   | 32.36±0.72  |\n| Pubmed   | *clean*  | **89.93±0.27**   | 88.10±0.33   | 86.46±0.24  |\n|          | PGD      | 81.81±1.94   | **87.69±0.57**   | 39.01±0.17  |\n|          | TDGIA    | 86.62±1.05   | **87.55±0.60**   | 40.35±1.13  |\n|          | MetaGIA  | **87.58±0.75**   | 87.40±0.62   | 40.47±0.41  |\n\n**Table R5.** Node classification accuracy (%) on graph **injection, evasion, non-targeted** attack in **inductive** learning.\n"}, {"review_id": "MsQsbTh15j", "reviewer": "Reviewer_ehTk", "summary": "This paper proposes a robust GNN model by leveraging the notion of Hamiltonian Energy Conservation. Specifically, authors first analyze the stabilities and limitations of several neural ODE-based GNNs, which motivate the proposed model HANG that is inspired by Hamiltonian classical mechanics. Experimental results indicate that HANG outperforms prior ODE-based GNNs against various adversarial attacks on several realistic graph datasets.", "questions": "- Is it possible to further improve GNN robustness by combining HANG with some prior defense methods (e.g. graph preprocessing-based methods), as they improve robustness in orthogonal ways?", "limitations": "Please refer to the Weaknesses section above for limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written. All technical steps are easy to follow.\n- Authors propose a novel approach to improve GNN robustness.\n- Authors clearly motivate the proposed approach by analyzing the stability limitations of prior ODE-based GNNs.\n- The proposed model has been evaluated against various adversarial attacks.", "weaknesses": "- The major issue of this work is that it only compares with a limited number of (and relatively weak) defense baselines. There are some recent methods such as [1, 2] showing better results than the baselines chosen in this paper. It would further improve the paper if authors could compare HANG with those methods.\n- Figure 1 only shows the norm of node features become closer as time goes, which does not support authors' claim that node features gradually become similar (lines 182-183). Authors may want to show the norm of relative difference of node features is approaching to zero instead.\n  \n[1]: Geisler et al., “Robustness of Graph Neural Networks at Scale”, NeurIPS'21. \\\n[2]: Deng et al., “GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks”, LoG'22.\n\nI'm willing to raise the score if my major concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "|Dataset|Attack|HANG|HANG-GUARD|HANG-quad|HANG-quad-GUARD|\n|-|-|-|-|-|-|\n|Cora|clean|87.13±0.86|86.54±0.57|79.68±0.62|81.23±0.70|\n||PGD|78.37±1.84|86.23±0.55|79.05±0.42|80.91±0.67|\n||TDGIA|79.76±0.99|85.56±0.34|79.54±0.65|81.11±0.76|\n||MetaGIA|77.48±1.02|86.0±0.60|78.28±0.56|80.10±0.53|\n|Citeseer|clean|74.11±0.62|75.95±0.66|71.85±0.48|73.15±0.61|\n||PGD|72.31±1.16|75.38±0.82|71.07±0.41|73.07±0.63|\n||TDGIA|72.12±0.52|74.54±0.69|71.69±0.40|73.04±0.52|\n||MetaGIA|72.92±0.66|75.22±0.66|71.60±0.48|73.11±0.45|\n|Pubmed|clean|89.93±0.27|89.96±0.25|88.10±0.33|88.93±0.18|\n||PGD|81.81±1.94|87.72±0.89|87.69±0.57|88.99±0.11|\n||TDGIA|86.62±1.05|88.86±0.40|87.55±0.60|88.80±0.12|\n||MetaGIA|87.58±0.75|88.23±0.93|87.40±0.62|88.78±0.16|\n\n**Table R6.** Node classification accuracy (%) on graph **injection, evasion, non-targeted** attack in **inductive** learning.\n\n----\n\n# New Baselines[1][2]: Weakness 1\n\nWe appreciate the suggestion to broaden our comparison. The updated experiments are detailed subsequently.\n\nHowever, we wish to emphasize that our paper's primary focus is not solely on defense comparisons. Instead, we're keen on delving into the relationship between stability concepts and robustness **in the realm of graph neural ODE flows.** These stability concepts, intrinsic to graph neural flows, highlight the unique properties of such models. Therefore, while there are newer defense techniques like [1] and [2] with promising performance, our exploration into stability in graph neural flows presents a different facet, distinct from conventional defense methods.\n\nIn response to the feedback, we've incorporated additional experiments in ``Table R7``, which include evaluations against defense techniques from [1] and [2].\n\nIn our supplementary material Table S5,  we present the performance of our model when subject to Nettack. This result can be directly compared with the results from [2], as shown in ``Table R7``. \nMoreover, we also added experiments to demonstrate that our method is orthogonal to other defense methods. For example, the method presented in [2], GARNET, is a preprocessing defense mechanism that can be integrated with any GNN model. In our experiments, we employed the HANG model as the backbone and incorporated the GARNET defense. As depicted in ``Table R7``, the robust accuracy of HANG significantly benefits from the integration of GARNET. Furthermore, our combined approach, HANG-GARNET, outperforms the GCN-GARNET combination, underscoring the robustness and efficacy of our proposed model.\n\nThe research by [1] introduced a robust aggregation GCN named Soft-Median-GCN. We evaluated this under two specific graph modification attacks, namely Nettack and Mettattack. The outcomes of these evaluations are in ``Table R7`` and ``Table R8``. It can be seen that our proposed models, HANG and HANG-quad, surpass Soft-Median-GCN under similar attack settings. This reaffirms the robustness of our models against such graph modification attacks.\n\n----\n|Dataset|Ptb-rate|HANG|HANG-GARNET|HANG-quad|HANG-quad-GARNET|GCN|GCN-GATNET[2]|Soft-Median-GCN[1]|\n|-|-|-|-|-|-|-|-|-|\n|Cora|1|75.54±3.10|82.41±0.80|76.99±3.16|83.01±0.84|70.06±0.81|79.75±2.35|78.67±2.29|\n||2|73.73±3.64|80.84±1.57|76.51±2.60|79.88±0.55|68.60±1.81|79.60±1.50|73.98±1.72|\n||3|68.43±4.23|80.48±1.69|73.13±2.85|79.76±0.72|65.04±3.31|74.42±2.06|73.01±1.54|\n||4|66.02±2.21|70.0±1.47|72.53±2.14|75.90±0.54|61.69±1.48|69.60±2.67|68.91±1.77|\n||5|60.12±3.63|67.83±1.87|68.80±2.55|69.28±1.34|55.66±1.95|67.04±2.05|66.51±1.05|\n\n**Table R7.** Node classification accuracy (%) on graph **Nettack targeted** attack in **transductive** learning.\n\n----\n\n# Visualization: Weakness 2\n\nThank you for the valuable suggestion. In response to your suggestion, we have revised Figure 1 to provide a more accurate representation of the gradual similarity of node features. The new plot ``Figure R2`` in the uploaded pdf now displays the norm of the difference of node features, which indeed approaches zero as time progresses. This modification aligns with our claim and better conveys the intended message.\n\n# Combining HANG with Other Defenses: Question 1.\n\n We would like to express our gratitude for your insightful suggestion. We recognize the potential benefits of combining defense mechanisms that operate through orthogonal strategies. To explore this, we integrated HANG with the GNNGuard preprocessing-based defense, known for its effectiveness in enhancing GNN robustness [P2] and GARNET, another preprocessing defense mechanism.\n        \nWe have conducted experiments to evaluate the combined performance of HANG with GNNGuard and GARNET, respectively. The results, as detailed in  ``Table R6 and Table R7``, demonstrate that this combination indeed improves the robustness accuracy of our model. This outcome underscores the compatibility of our approach with existing defense strategies and validates the potential for synergistic enhancements.\n\nFinally, it is also noteworthy that HANG can be combined with adversarial training techniques, such as PGD-AT, to further enhance its robustness. We refer the reviewer to the response **Adversarial Training (AT) and New AT Baselines** to **``Reviewer bDPL``** and ``Tables R1 and R2`` for more details.\n\n[P2] GNNGuard: Defending Graph Neural Networks against Adversarial Attacks, NeurIPS,2020.\n\n\n|Dataset|Ptb-rate(%)|HANG|HANG-quad|Soft-Median-GCN[1]|\n|-|-|-|-|-|\n|Polblogs|0|94.77±1.07|94.63±1.06|94.69±0.35|\n||5|80.19±2.52|94.38±0.82|75.46±0.66|\n||10|74.92±4.32|92.46±1.56|75.63±1.34|\n||15|71.65±1.34|90.85±2.43|71.60±0.74|\n||20|66.27±3.39|89.19±3.72|65.53±0.72|\n||25|65.80±2.33|86.89±8.90|64.72±2.30|\n|Pubmed|0|85.08±0.20|85.23±0.14|85.22±0.11|\n||5|85.08±0.18|85.12±0.18|84.17±0.08|\n||10|85.17±0.23|85.05±0.19|81.90±0.21|\n||15|85.0±0.22|85.15±0.17|79.05±0.51|\n||20|85.20±0.19|85.03±0.19|75.74±0.23|\n||25|85.06±0.17|84.99±0.16|72.75±0.39|\n\n**Table R8.** Node classification accuracy (%) on graph **Metattack non-targeted** attack in **transductive** learning."}, {"review_id": "nQa0m5rJ5n", "reviewer": "Reviewer_Xukb", "summary": "Drawing inspiration from physics principles, the paper proposes the use of conservative Hamiltonian neural flows to construct GNNs that are robust against adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically evaluated on several benchmark datasets, considering a variety of adversarial attacks.", "questions": "Refer to Weaknesses", "limitations": "Refer to Weaknesses", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This paper investigates the vulnerability of graph neural networks (GNNs) to adversarial perturbations that affect both node features and graph topology. The study focuses on GNNs derived from various neural flows and explores their connection to different stability notions.", "weaknesses": "The followings are four weaknesses.\n\n1. Paper's writing quality is poor due to excessive amount of equations, formulas, and its explanation in a view of physics. The authors should use the adversarial research language to bring out the main insight for why this work is important for adversarial research.\n\n2. In addition, numerous words need to penetrate the core idea of equations rather than to revolve around technical terms.\n\n3. The reviewer did not see the connection bridging the gap between Hamiltonian mechanics and energy conservation.\n\n4. Is there any other simpler option that possess the notion of conservation other than Hamiltonian mechanics at all?  In addition, why is it important to conserve energy in a view of topological structure?\n\nOverall, the reviewer recommends the authors to answer these questions using insightful language at least to the reviewer, not using mathematical or physical language. The reviewer does not want a theoretical proof, only want really contributional insight; Why is it important to consider energy or conservation and why are \"energy and topology\" bridged? Why is the hamiltonian important? Please answer using adversarial language\n\n---\n\nThe authors' rebuttal has fully explained the connection between the energy conservation in the realm of the Hamiltonian and the adversarial robustness of GNNs using the adversarial research language. Thank the authors for their thorough commentary on the reviewer's point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Improve Paper Writing: Weakness 1 and 2\n\nWe greatly value your feedback. Our paper delves into the relationship between the stability of graph ODE models and adversarial robustness. This connection is elaborated upon in segments such as Remark 2, lines 116-125, 165-180, and 280-285, where we correlate system stability with adversarial resistance.\n\nTo enhance clarity:\n\n1. Appendix Inclusion: Some secondary equations will be moved to an appendix, ensuring the main content remains focused.\n\n2. Restrained Physics Emphasis: We will succinctly present physics-related sections, especially Section 4.1, while retaining essential insights. In-depth physics discussions will be directed to the appendix.\n\n3. Highlighting Adversarial Context: We will provide more intuitive explanations for the choice of Hamiltonian mechanics and how it contributes to combatting adversarial attacks. Some example answers have been provided in our following responses, which will be further expanded in our revision.\n\n\n\n#  Hamiltonian Mechanics, Energy Conservation and Robustness: Weakness 3, 4 and 5\n\nThank you for raising this point, and we are grateful for the opportunity to provide further clarity.\n\nHamiltonian mechanics is not just a theoretical underpinning in our work but a core mechanism to underscore conservation principles. Specifically, the energy conservation of our system, represented as $H_{n e t}$ in equation (8), remains invariant over time, notably in the context of graph ODE models. To shed more light on this:\n\\begin{align*}\n\\frac{\\mathrm{d} H\\_{net}}{\\mathrm{~d} t} & =\\sum\\_{i=1}\\^n \\frac{\\partial H\\_{net}}{\\partial q\\_i} \\dot{q}\\_i+\\sum\\_{i=1}\\^n \\frac{\\partial H_{net}}{\\partial p\\_i} \\dot{p}\\_i\\\\\\\\\n& =\\sum\\_{i=1}\\^n \\frac{\\partial H\\_{net}}{\\partial q\\_i} \\frac{\\partial H\\_{net}}{\\partial p\\_i}+\\sum\\_{i=1}\\^n \\frac{\\partial H\\_{net}}{\\partial p\\_i}\\left(-\\frac{\\partial H\\_{net}}{\\partial q\\_i}\\right)\\\\\\\\\n& =0.\n\\end{align*}\n\n\nWhat this means is that $H_{net}$ retains its value as $t$ changes, as stated by Theorem 2. Now, in the realm of physics, $H_{n e t}$ is interpreted differently across varied contexts, say as mechanical energy in a pendulum or as energy in an electron's charged field. In our framework, we model $H_{n e t}$ using a neural network model that ingests the full graph data and yields a scalar. This scalar, by design, remains invariant over time, echoing the conservation principle rooted in Hamiltonian mechanics.\n\n\nFor a clearer insight into the energy concept within HANG, consider the \"time\" in graph neural flows as analogous to the \"layers\" in standard GNNs (note that ODE solvers basically discretize the \"time\", which indeed approximately turns the model into a layered one). Here, the feature vector $\\mathbf{q}(t)$ and the momentum vector $\\mathbf{p}(t)$ evolve with time, bound tightly by equation (9). Given that $\\mathbf{q}(t)$ mirrors the node features at layer $t, \\mathbf{p}(t)$ can be understood as the variation in node features over time - essentially, the evolution of node features between successive layers. Thus, our defined energy interweaves both the node feature and its rate of change across adjacent layers. The constant $H_{n e t}$ implies inherent constraints on the node features and their alteration pace over layers. \nBecause $H_{net}$ processes the whole graph data and yields a scalar, it serves as a constraint on the global graph feature and its variation, which we opine to be crucial in countering adversarial attacks. \n\nFurthermore, by rewriting equation (9) in the following form, \n\\begin{align}\n\\left[\\begin{array}{c}\n\\dot{q}(t) \\\\\\\\\n\\dot{p}(t)\n\\end{array}\\right]\n&=\n\\mathbf{M}\n\\left[\\begin{array}{c}\n \\frac{\\partial H_{\\mathrm{net}}}{\\partial p}\\\\\\\\\n\\frac{\\partial H_{\\mathrm{net}}}{\\partial q}\n\\end{array}\\right],\n\\end{align}\nwhere $\\mathbf{M}=\\left[\\begin{array}{cc}\n\\mathbf{0} & \\mathbf{I}\\\\\\\\\n-\\mathbf{I} & \\mathbf{0}\n\\end{array}\\right]$ is a real antisymmetric matrix and its eigenvalues (real parts) are all zeros, \nwe can see that the antisymmetric matrix $\\mathbf{M}$ leads to a rotation in feature space $\\left[\\begin{array}{c}\nq(t) \\\\\np(t)\n\\end{array}\\right]$. This implies that the norm of a node remains constant over time while the phase of the feature may change. This property preserves distances between features of different nodes when their norms differ at the initial point. This, in turn, ensures a well-posed and robust forward propagation and learning process. Figure 1 in the main paper illustrates this property.\n\nFrom an adversarial perspective, the attacker modifies either the node features or the underlying graph topology. These modifications are propagated  through multiple aggregation steps, such as layers in conventional GNNs or integrals in graph ODEs. \nHere is where the Hamiltonian energy conservation property (as mentioned in our previous response) plays a crucial role. While the Hamiltonian considers the energy of the entire graph, adversarial attacks often target localized regions of the graph. The inherent global energy perspective of the Hamiltonian system makes it resilient to such localized attacks, as local perturbations often get \"absorbed\" or \"mitigated\" when viewed from the perspective of the entire system. When adversarial perturbations are introduced, they might indeed tweak the instantaneous features of certain nodes. However, the challenge lies in modifying the trajectory (or evolution) of these nodes (positions $p(t)$ and the variations $q(t)$) in the phase space in a manner that's aligned with the rest of the graph, all while upholding the energy conservation constraints. This feat is arduous, if not impossible, without creating detectable inconsistencies elsewhere.\nThis property ensures that the energy of each node feature is preserved over time and multiple aggregation steps. As a result, the distances between features of different nodes are preserved if their norms differ initially before the adversarial attack."}]

Author rebuttal:
In response to the reviewers' feedback, we have undertaken several substantial efforts to enhance the quality and comprehensibility of our paper. Here is a summary of the key actions we have taken:

1. **New Experiments**: We have executed a series of new experiments to bolster our findings and comparisons:
    - Conducted adversarial training on our model, presenting the results in ``Table R1`` and ``Table R2``.
    - Introduced new baselines and defense methods, capturing the outcomes in ``Tables R5, R7 and R8``.
    - Explored the implications of a novel graph attack method, reflected in ``Table R4``.
    - Explored the synergy of preprocessing defence methods combined with our model, showcased in ``Tables R6 and R7``.
    - Implemented more imperceptible attacks ``Table R3`` and visualized their imperceptibility through ``Figures R1 and R3``.
2. **Energy Conservation Visualization**: To underline the energy conservation property inherent to our approach, we introduced a visualization in ``Figure R2`` depicting the norm of feature differences versus time. This illustration effectively highlights the consistent energy preservation in our model.
3. **Intuitive Explanation of Hamiltonian and Energy Conservation**: Recognizing the need for clarity, we have provided a more intuitive explanation regarding the significance of Hamiltonian mechanics and how energy conservation intricately links to adversarial robustness. These insights aim to enhance the reader's understanding of our approach.
4. **Core Contribution Clarification**: We have further clarified and succinctly articulated the core contribution of our paper. This ensures that the reader grasps the pivotal insights and implications of our research.



### New experimental results: **Tables R1 - R3**:

|Dataset|Attack|HANG|HANG-AT|GAT|GAT-AT|HANG-SAT[4][5]|
|-|-|-|-|-|-|-|
|Pubmed|*clean*|**89.93±0.27**|89.45±0.33|87.41±1.73|85.88±0.47|88.38±0.48|
||PGD|81.81±1.94|**89.03±0.19**|48.94±12.99|77.78±8.39|87.65±1.77|
||TDGIA|86.62±1.05|87.16±3.89|47.56±3.11|79.56±5.76|**87.55±2.11**|
||MetaGIA|87.58±0.75|**88.93±0.18**|44.75±2.53|81.86±3.44|88.22±0.22|



**Table R1.** Node classification accuracy (%) on graph **injection, evasion, non-targeted, black-box** attack in **inductive** learning.

----


|Dataset|Attack|HANG|HANG-AT|GAT|GAT-AT|HANG-SAT[4][5]|
|-|-|-|-|-|-|-|
|Pubmed|*clean*|**89.93±0.27**|89.42±0.22|87.41±1.73|85.90±0.44|88.38±0.48|
||PGD|68.62±2.82|**88.86±0.15**|38.04±4.91|80.85±4.51|88.19±0.28|
||TDGIA|69.56±3.16|**88.98±0.20**|24.43±4.10|82.16±4.10|88.28±0.34|
||MetaGIA|84.64±1.20|**88.61±0.20**|40.02±1.34|44.37±6.48|85.76±4.12|


**Table R2.** Node classification accuracy (%) on graph **injection, evasion, non-targeted, white-box** attack in **inductive** learning.

----

| Dataset | Attack | HANG | HANG-quad | GRAND | GAT | GCN |
|---|---|---|---|---|---|---|
| Cora | *clean* | 86.32±0.53 | 80.56±0.65 | 87.62±0.75 | 87.47±0.51 | **87.99±0.45** |
| | TDGIA+HAO | **82.81±0.96** | 80.60±0.65 | 42.21±4.21 | 50.04±15.04 | 35.19±0.96 |
| | MetaGIA+HAO | **81.30±0.74** | 80.05±0.87 | 44.33±1.70 | 41.58±2.83 | 39.95±0.41 |
| Citeseer |*clean*| 74.78±0.71 | 73.22±0.74 | **76.31±0.86** | 72.45±0.89 | 73.22±0.65 |
| | TDGIA+HAO | **73.38±0.35** | 72.38±0.48 | 36.79±2.98 | 31.18±8.62 | 24.41±2.32 |
| | MetaGIA+HAO | 70.05±0.56 | **72.37±0.83** | 20.57±3.59 | 18.11±1.82 | 18.41±0.33 |
| Pubmed |*clean*| **90.04±0.24** | 89.02±0.20 | 88.66±0.17 | 87.44±0.19 | 88.24±0.23 |
| | TDGIA+HAO | 86.19±1.45 | **88.89±0.17** | 42.45±1.20 | 44.35±3.11 | 43.56±1.03 |
| | MetaGIA+HAO | 85.89±0.87 | **88.87±0.79** | 57.49±0.99 | 39.57±0.55 | 39.58±0.05 |



**Table R3.** Node classification accuracy (%) on graph **injection, evasion, non-targeted, black-box** attack in **inductive** learning.

Meta-review: The reviewers consistently think the paper studies an important research problem on the robustness of GNN. I therefore recommend acceptance of the paper. Please revise the paper according to the reviews in the camera-ready version.
Decision: Accept (spotlight)