Title: Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial robustness

graph neural networks
Abstract: Graph neural networks (GNNs) are vulnerable to adversarial perturbations, including those that affect both node features and graph topology. This paper investigates GNNs derived from diverse neural flows, concentrating on their connection to various stability notions such as BIBO stability, Lyapunov stability, structural stability, and conservative stability. We argue that Lyapunov stability, despite its common use, does not necessarily ensure adversarial robustness. Inspired by physics principles, we advocate for the use of conservative Hamiltonian neural flows to construct GNNs that are robust to adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically compared on several benchmark datasets under a variety of adversarial attacks. Extensive numerical experiments demonstrate that GNNs leveraging conservative Hamiltonian flows with Lyapunov stability substantially improve robustness against adversarial perturbations. The implementation code of experiments  is available at \url{https://github.com/zknus/NeurIPS-2023-HANG-Robustness}.

Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach

Abstract

Graph neural networks (GNNs) are vulnerable to adversarial perturbations, including those that affect both node features and graph topology. This paper investigates GNNs derived from diverse neural flows, concentrating on their connection to various stability notions such as BIBO stability, Lyapunov stability, structural stability, and conservative stability. We argue that Lyapunov stability, despite its common use, does not necessarily ensure adversarial robustness. Inspired by physics principles, we advocate for the use of conservative Hamiltonian neural flows to construct GNNs that are robust to adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically compared on several benchmark datasets under a variety of adversarial attacks. Extensive numerical experiments demonstrate that GNNs leveraging conservative Hamiltonian flows with Lyapunov stability substantially improve robustness against adversarial perturbations. The implementation code of experiments is available at https://github.com/zknus/NeurIPS-2023-HANG-Robustness .

1 Introduction

Graph neural networks (GNNs) [1-8] have achieved great success in inference tasks involving graphstructured data, including applications from social media networks, molecular chemistry, and mobility networks. However, GNNs are known to be vulnerable to adversarial attacks [9]. To fool a trained GNN, adversaries can either add new nodes to the graph during the inference phase or remove/add edges from/to the graph. The former is called an injection attack [10-13], and the latter is called a modification attack [14-16]. In some works [9,17], node feature perturbations are also considered to enable stronger modification attacks.

Neural ordinary differential equation (ODE) networks [18] have recently gained popularity due to their inherent robustness [19-24]. Neural ODEs can be considered as a continuous analog of ResNet [25]. Many neural ODE networks have since been proposed, including but not limited to [19,20,26-28]. Using neural ODEs, we can constrain the input and output of a neural network to follow certain physics laws. Injecting physics constraints to black-box neural networks improves neural networks' explainability. More recently, neural ODEs have also been successfully applied to GNNs by modeling

the way nodes exchange information given the adjacency structure of the underlying graph. We call them graph neural flows which enables the interpretation of GNNs as evolutionary dynamical systems. These system equations can be learned by instantiating them using neural ODEs [29-34]. For instance, [29, 30] models the message-passing process, i.e., feature exchanges between nodes, as the heat diffusion, while [31, 32] model the message passing process as the Beltrami diffusion. The reference [35] models the graph nodes as coupled oscillators with a coupled oscillating ODE guiding the message-passing process.

Although adversarial robustness of GNNs has been investigated in various works, including [9,13,17], robustness study of graph neural flows is still in its fancy. To the best of our knowledge, only the recent paper [32] has started to formulate theoretical insights into why graph neural diffusion is generally more robust against topology perturbation than conventional GNNs. The concept of Lyapunov stability was used in [20,21]. However, there are many different notions of stability in the dynamic system literature [36,37]. In this paper, we focus on the study of different notions of stability for graph neural flows and investigate which notion is most strongly connected to adversarial robustness. We impose an energy conservation constraint on graph neural flows that lead to a Hamiltonian graph neural flow. We find that energy-conservative graph Hamiltonian flows endowed with Lyapunov stability improve the robustness the most as compared to other existing stable graph neural flows.

Main contributions. This research is centered on examining various stability notions within the realm of graph neural flows, especially as they relate to adversarial robustness. Our main contributions are summarized as follows:

We revisit the definitions of stability from the perspective of dynamical systems as applicable to graph neural flows. We argue that vanilla Lyapunov stability does not necessarily confer adversarial robustness and provide a rationale for this observation.

We propose Hamiltonian-inspired graph neural ODEs, noted for their energy-conservative nature. We perform comprehensive numerical experiments to verify their performance on standard benchmark datasets. Crucially, our results demonstrate that Hamiltonian flow GNNs present enhanced robustness against various adversarial perturbations. Moreover, it is found that the effectiveness of Lyapunov stability becomes pronounced when layered on top of Hamiltonian flow GNNs, thereby fortifying their adversarial robustness.

The rest of this paper is organized as follows. We introduce various stability notions from a dynamical system perspective in Section 2. A review of existing graph neural flows is presented in Section 3 with links to the stability notions defined in Section 2. We present a new type of graph neural flow inspired by the Hamiltonian system with energy conservation in Section 4. Two different variants of this model are proposed in Section 5. Section 6 details our extensive experimental outcomes. The supplementary section provides an overview of related studies, an exhaustive outline of the algorithm, further insights into model robustness, supplementary experimental data, and the proofs for the theoretical propositions made throughout the paper.

2 Stability in Dynamical Systems

It is well known that a small perturbation at the input of an unstable dynamical system will result in a large distortion in the system's output. In this section, we first introduce various types of stability in dynamical physical systems and then relate them to graph neural flows. We consider the evolution of a dynamical system that is described as the following autonomous nonlinear differential equation:

where f θ : R n → R n denotes the system dynamics, which may be non-linear in general, θ denotes the system parameters, and z : [0 , ∞ ) → R n represents the n -dimensional system state.

We first introduce the stability notions from a dynamical systems perspective that is related to the GNN robustness against node feature perturbation .

Definition 1 (BIBO stability) . The system is called BIBO (bounded input bounded output) stable if for any bounded input, there exists a constant M s.t. the output ∥ z ( t ) ∥ < M, ∀ t ≥ 0 .

Suppose f has an equilibrium at z e so that f θ ( z e ) = 0 . We can define the stability notion for z e .

Definition 2 (Lyapunov stability and asymptotically stable [38]) . The equilibrium z e is Lyapunov stable if for every ϵ > 0 , there exists a δ > 0 such that, if ∥ z (0) -z e ∥ < δ , then for every t ≥ 0 we have ∥ z ( t ) -z e ∥ < ϵ . Furthermore, the equilibrium point z e is said to be asymptotically stable if it is Lyapunov stable and there exists a δ ′ > 0 such that if ∥ z (0) -z e ∥ < δ ′ , then lim t →∞ ∥ z ( t ) -z e ∥ = 0 .

Remark 1. Lyapunov stability indicates that the solutions whose initial points are near an equilibrium point z e stay near z e forever. For the special linear time-invariant system d z ( t ) / d t = Az ( t ) with a constant matrix A , it is Lyapunov stable if and only if all eigenvalues of A have non-positive real parts and those with zero real parts are the simple roots of the minimal polynomial of A [39, 40]. Asymptotically stable means that not only do trajectories stay near z e for all time (Lyapunov stability), but trajectories also converge to z e as time goes to infinity (asymptotic stability).

We next introduce the concept of structural stability from dynamical systems theory, which is related to the robustness of GNNs against graph topological perturbation. It describes the sensitivity of the qualitative features of a solution to changes in parameters θ . The definition of structural stability requires the introduction of a topology on the space of z in (1), which we do not however present here rigorously due to space constraints and not to distract the reader with too much mathematical details. Instead, we provide a qualitative description of structural stability to elucidate how it can indicate the robustness of a graph neural flow against topology perturbations.

Definition 3 (Structural stability) . Unlike Lyapunov stability, which considers perturbations of initial conditions for a fixed f θ , structural stability deals with perturbations of the dynamic function f θ by perturbing the parameter θ . The qualitative behavior of the solution is unaffected by small perturbations of f θ in the sense that there is a homeomorphism that globally maps the original solution to the solution under perturbation.

Remark 2. In the graph neural flows to be detailed in Section 3 and Section 4, the parameter θ includes the graph topology (i.e., the adjacency matrix) and learnable neural network weights. Unlike adversarial attacks on other deep learning neural networks where the attacker targets only the input z , it is worth noting that adversaries for GNNs can also attack the graph topology, which forms part of θ . If there are different Lyapunov stable equilibrium points, one for each class of nodes, one intuitive example of breaking structural stability in graph neural flows is by perturbing the graph topology in such a way that there are strictly fewer equilibrium points than the number of classes.

In this study, we will propose GNNs drawing inspiration from Hamiltonian mechanics. In a Hamiltonian system, z = ( q, p ) ∈ R 2 n refers to the generalized coordinates, with q and p corresponding to the generalized position and momentum, respectively. The dynamical system is characterized by the following nonlinear differential equation:

where ∇ H ( z ) is the gradient of a scalar function H at z and J = ( 0 I -I 0 ) is the 2 n × 2 n skew-symmetric matrix with I being the n × n identity matrix.

We now turn our attention to the notion of conservative stability in dynamical systems. It is worth noting that a general dynamical system, as characterized in (1), might not consistently resonate with traditional perspectives on energy and conservation, especially when compared to physics-inspired neural networks, like (2).

Definition 4 (Conservative stability) . In a dynamical system inspired by physical principles, such as (2) , a conserved quantity might be present. This quantity, which frequently embodies the notion of the system's energy, remains invariant along the system's evolution trajectory z ( t ) .

Our focus in this work is on graph neural flows that can be described by either (1) or (2). Chamberlain et al. [29] postulate that many GNN architectures such as GAT can be construed as discrete versions of (1) via different choices of the function f θ and discretization schemes. Therefore, the stability definitions provided above can offer additional insights into many popular GNNs. Most existing graph neural flows only scrutinize the BIBO/Lyapunov stability of their system. For instance, GRAND [29] proposes BIBO/Lyapunov stability against node feature perturbation over z . However, the more fundamental structural stability in graph neural flows, which is related to the robustness against graph topological changes, remains largely unexplored. Some models, such as GraphCON [29,35], exhibit conservative stability under certain conditions. We direct the reader to Section 3 and Table 1 for a comprehensive discussion of the stability properties of each model.

3 Existing Graph Neural Flows and Stability

Consider an undirected, weighted graph G = ( V , E ) where V is a finite set of vertices and E ⊂ V × V denotes the set of edges. The adjacency matrix of the graph is denoted as ( W [ u, v ]) = W ([ v, u ]) for all [ u, v ] ∈ E . Let X ( t ) ∈ R |V|× r represent the features associated with the vertices at time t . The feature vector for the i -th node in V at time t can be represented as the i -th row of X ( t ) , indicated by x ⊺ i ( t ) . In this section, we introduce several graph neural flows on G , categorizing them according to the stability concepts outlined in Section 2.

GRAND: Inspired by the heat diffusion equation, GRAND [29] employs the following dynamical system:

with the initial condition X (0) . Within this model, A G ( X ( t )) is either a time-invariant static matrix, represented as GRAND-l, or a trainable time-variant attention matrix ( a G ( x i ( t ) , x j ( t ))) , labeled as GRAND-nl, reflecting the graph's evolutionary features. The function a G ( · ) calculates similarity for pairs of vertices, and I is an identity matrix with dimensions that fit the context. In [29], α is set to be 1. Let D be the diagonal node degree matrix where D [ u, u ] = ∑ v ∈V W [ u, v ] .

Theorem 1. We can prove the following stability:

1). For GRAND-nl, if the attention matrix A G ( X ( t )) is set as a doubly stochastic attention [41], we have BIBO stability and Lyapunov stability for any α ≥ 1 . When α > 1 , it reaches global asymptotic stability under any perturbation.

2). Within the GRAND-l setting, if A G is set as a constant column- or row-stochastic matrix, such as the normalized adjacency matrices WD -1 or D -1 W , global asymptotic stability is achieved for α > 1 under any perturbation. If the graph is additionally assumed to be strongly connected [42][Sec.6.3], BIBO and Lyapunov stability are realized for α = 1 .

3). Furthermore, when A G is specifically a constant column-stochastic matrix like WD -1 and α = 1 , GRAND conserves a quantity that can be interpreted as energy. Furthermore, in this settting, asymptotic stability is attained when the graph is aperiodic and strongly connected and the perturbations on X (0) ensure unaltered column summations.

BLEND: In comparison to GRAND, BLEND [31] introduces the use of positional encodings. Following a similar line of reasoning to that used for GRAND, BLEND also exhibits BIBO/Lyapunov stability as stated in Theorem 1. Moreover, it is noteworthy that if positional features U ( t ) are eliminated, for instance by setting them as a constant, BLEND simplifies to the GRAND model.

GraphCON: Inspired by oscillator dynamical systems, GraphCON is a graph neural flow proposed in [35] and defined as

where F θ ( · ) is a learnable 1 -neighborhood coupling function, Y ( t ) is an auxiliary velocity variable, σ denotes an activation function, and γ and α are tunable parameters.

As described in [35, Proposition 3.1], under specific settings where σ is the identity function and F θ ( X ( t ) , t ) = AX ( t ) with A being a constant matrix, GraphCON conserves Dirichlet energy (11) , thereby demonstrating conservative stability.

GraphBel: Generalizing the Beltrami flow, mean curvature flow and heat flow, a stable graph neural flow [32] is designed as

where ⊙ is the element-wise multiplication. A S ( · ) and B S ( · ) are learnable attention function and normalized vector map, respectively. Ψ ( X ( t )) is a diagonal matrix in which Ψ( x i , x i ) = ∑ x j ( A ⊙ B )( x i , x j ) .

Analogous to BLEND, under certain conditions with Ψ( X ( t )) = B S ( X ( t )) = I , GraphBel simplifies to the GRAND model. Consequently, it exhibits BIBO/Lyapunov stability in certain scenarios.

𝑨

𝑨 + 𝜺

misclassification

Class 1

Class 2

(a)

Example 1.

0

20

40

60

80

100

Time step

0

5

10

15

20

Feature norm

Node label 0 of GRAND

Node label 1 of GRAND

Node label 0 of HANG

Node label 1 of HANG

(b) Before GIA attack

0

20

40

60

80

100

Time step

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Feature norm

Node label 0 of GRAND

Node label 1 of GRAND

Node label 0 of HANG

Node label 1 of HANG

(c) After GIA attack

The incorporation of ODEs via graph neural flows may enhance the stability of graph feature representations. A summarized relationship between model stability and these graph neural flows can be found in Table 1.

3.1 Lyapunov Stability vs. Node Classification Robustness:

At first glance, Lyapunov stability has a strong correlation with node classification robustness against feature perturbations. However, before diving into experimental evidence, we point out an important conclusion: Lyapunov stability by itself does not necessarily imply adversarial robustness. Consider a scenario where a graph neural flow has only one equilibrium point z e , while the node features are derived from more than one class. In a Lyapunov asymptotically stable graph neural flow, such as GRAND (as shown in Theorem 1), all node features across different classes would inevitably converge to a single point z e due to global contraction. We note that this is why the model in [20] requires a diversity-promoting layer to ensure that different classes converge to different Lyapunov-stable equilibrium points.

Example 1. We provide an example to demonstrate our claim. Consider the following Lyapunov stable ODE

with initial condition x (0) = [ x 1 (0) , x 2 (0)] ⊺ . The solution to this ODE is given by x ( t ) = x 1 (0) e -t [1 , 0] ⊺ + x 2 (0) e -5 t [0 , 1] ⊺ . For all initial points in R 2 , we have x ( t ) → 0 as t → ∞ . Furthermore, as t → ∞ , the trajectory x ( t ) for any initial point is approximately parallel to the x-axis. We draw the phase plane in Fig. 1a.

Assume that the points on the upper half y-axis belongs to class 1 while we have a linear classifier that seperates class 1 and class 2 as shown in Fig. 1a. We observe that for the initial point A belonging to class 1, the solution from a small perturbed initial point A + ϵ is misclassified as class 2 for a large enough t for any linear classifier. We see from this example that Lyapunov stability iteself does not imply adversarial robustness in graph neural flow models.

This example indicates that Lyapunov stability does not guarantee node classification robustness. Additionally, for a system exhibiting global contraction to a single equilibrium point, structural stability may also be ensured. For instance, in the case of GRAND, even if the edges are perturbed, the system maintains the same number of equilibrium points with global contraction. We conclude that even an amalgamation of both Lyapunov stability and structural stability may not help the graph's adversarial robustness for node classification.

In the example shown in Fig. 1, we observe that in the case of GRAND when α > 1 , the node features from different classes tend to become closer to each other as time progresses. This phenomenon can potentially create more vulnerability to adversarial attacks.

̇ 𝑞

!

𝑡 =

𝜕𝐻

"#$

𝜕𝑝

!

̇ 𝑝

!

𝑡 =

𝜕𝐻

"#$

𝜕𝑞

!

𝐩

|"|

𝐪

|"|

𝐩

#

𝐪

#

𝐩

$

𝐪

$

𝐩

%

𝐪

%

𝐩

&

𝐪

&

𝐩

!

(0)

𝐪

!

(0)

𝐩

"

(0)

𝐪

"

(0)

𝐩

#

(0)

𝐪

#

(0)

𝐩

$

(0)

𝐪

$

(0)

𝐩

%

(0) 𝐪

%

(0)

𝐻

%&'

𝐩

!

(𝑇)

𝐪

!

(𝑇)

𝐩

"

(𝑇)

𝐪

"

(𝑇)

𝐩

#

(𝑇)

𝐪

#

(𝑇)

𝐩

$

(𝑇)

𝐪

$

(𝑇)

𝐩

%

(𝑇) 𝐪

%

(𝑇)

Hamiltonian Information Propagation

𝑝(0)

𝑞(0)

𝑝(𝑇)

𝑞(𝑇)

4 Hamiltonian-Inspired Graph Neural Flow

Drawing inspiration from the principles of Hamiltonian classical mechanics, we introduce a novel graph neural flow paradigm, namely HamiltoniAN Graph diffusion (HANG). In Hamiltonian mechanics, the notation ( q, p ) is traditionally used to represent the properties of nodes (position and momentum). Hence, in this section, we adopt this notation instead of X (as used in Section 3) to denote node features.

4.1 Physics Philosophy Behind HANG

In Hamiltonian mechanics, the state evolution of a multi-object physical system, such as an electromagnetic field, double pendulum, or spring network [43,44], adheres to well-established physical laws. For instance, in a system of charged particles, each particle generates an electromagnetic field that influences other particles. The Hamiltonian for such a system includes terms representing the kinetic and potential energy of each particle, along with terms representing the interactions between particles via their electromagnetic fields. In this paper, we propose a novel concept of information propagation between graph nodes, where interactions follow a similar Hamiltonian style.

In a Hamiltonian system, the position q and momentum p together constitute the phase space ( q, p ) , which comprehensively characterizes the system's evolution. In our HANG model, we process the raw node features of the graph using a linear input layer, yielding 2 r -dimensional vectors. Following the methodologies introduced in the GNN work [31,35], we split the 2 r dimensions into two equal halves, with the first half serving as the feature (position) vector and the second half as 'momentum' vector that guides the system evolution. Concretely, each node k is associated with an r -dimensional feature vector q k (0) = ( q 1 k , . . . , q r k ) and an r -dimensional momentum vector p k (0) = ( p 1 k , . . . , p r k ) . Subsequently, q k ( t ) and p k ( t ) will evolve along with the propagation of information between graph nodes, with q k (0) and p k (0) serving as the initial conditions.

Following the modeling conventions in physical systems, we concatenate the feature positions of all |V| vertices into a single vector, treating it as the system's generalized coordinate within a r |V| -dimensional manifold, a process that involves index relabeling 3 .

This r |V| -dimensional coordinate representation at each time instance provides a snapshot of the state of the graph system. Similarly, we concatenate all the 'momentum' vectors at time t to construct an r |V| -dimensional vector:

which can be interpreted as a generalized momentum vector for the entire graph system.

In physics, the system evolves in accordance with fundamental physical laws, and a conserved quantity function H ( q, p ) remains constant along the system's evolution trajectory. This conserved quantity is typically interpreted as the 'system energy'. In our HANG model, instead of defining an explicit energy function H ( p, q ) from a fixed physical law, we utilize a learnable energy function H net : G → R + parameterized by a neural network, referred to as the Hamiltonian energy function :

We allow the graph features to evolve according to a learnable Hamiltonian law analogous to basic physical laws. More specifically, we model the feature evolution trajectory as the following canonical Hamilton's equations, which is a restatement of (2):

with the initial features ( q (0) , p (0)) ∈ R 2 r |V| at time t = 0 being the vectors after the raw node features transformation.

The neural ODE given by (10) can be trained and solved through integration to obtain the trajectory ( q ( t ) , p ( t )) . At the terminal time point t = T , the system's solution is represented as ( q ( T ) , p ( T )) . We then apply the canonical projection map π to extract the nodes' concatenated feature vector q ( T ) as follows: π (( q ( T ) , p ( T ))) = q ( T ) . This concatenated feature vector q ( T ) is subsequently decompressed into individual node features for utilization in downstream tasks. For this study, we employ backpropagation to minimize the cross-entropy in node classification tasks. The complete model architecture is depicted in Fig. 2, while a comprehensive summary of the full algorithm can be found in the Appendix G.

4.2 Hamiltonian Energy Conservation

Referring to [45], it is known that the total Hamiltonian energy H net remains constant along the trajectory of its induced Hamiltonian flow. This principle is recognized as the law of energy conservation in a Hamiltonian system.

Theorem 2. If the graph system evolves in accordance with (10) , the total energy H net ( q ( t ) , p ( t )) of the system remains constant. BIBO stability is achieved if H net remains bounded for all bounded inputs and, as ( q, p ) →∞ , H net ( q, p ) →∞ .

In light of Theorem 2, if our system evolves following (10), it adheres to the law of energy conservation. As a result, our model guarantees conservative stability. Compared to GraphCON, which conserves Dirichlet energy over time t under specific conditions, the notion of Hamiltonian energy conservation is broader in scope. Under the settings delineated in Section 3, GraphCON can be considered as a particular instance of HANG when H net is set to represent Dirichlet energy E ( q ) :

Definition 5 (Dirichlet energy [35]) . The Dirichlet energy is defined on node features q ( t ) at time t of an undirected graph G as

where N ( i ) = { j : [ i, j ] ∈ E} is the set of neighbors adjacent to node i in the graph.

5 Different Hamiltonian Energy Functions

In physical systems, the system is often depicted as a graph where two neighboring vertices with mass are connected by a spring of given stiffness and length [44]. The system's energy is thus related to the graph's topology. Similarly, in our graph system, the energy function H net involves interactions between neighboring nodes, signifying the importance of the graph's topology. There exist multiple ways to learn the energy function, and we present two examples below.

5.1 Vanilla HANG

We define H net as a composition of two graph convolutional layers:

where g gcn 1 : R 2 r ×|V| → R d ×|V| and g gcn 2 : R d ×|V| → R |V| are two GCN [3] layers with different hidden dimensions. A tanh activation function is applied between the two GCN layers, and ∥ · ∥ 2 denotes the ℓ 2 norm. We concatenate q k and p k for each node k at the input of the above composite function, resulting in an input dimension of 2 r per node. From Theorem 2, it follows that HANG exhibits BIBO stability. If ( q ( t ) , p ( t )) were unbounded, the value of H net would also become unbounded, contradicting the energy conservation principle. In subsequent discussions, this invariant is referred to as HANG.

5.2 Quadratic HANG (HANG-quad)

The general vanilla HANG with conservative stability does not possess Lyapunov stability. Other additional conditions may be required for Lyapunov stability. The following Lagrange-Dirichlet Theorem provides a sufficient condition.

Theorem 3 (Lagrange-Dirichlet Theorem [46]) . Let z e be a locally quadratic equilibrium of the natural Hamiltonian (2) with

where T is a positive definite, quadratic function of p . Then z e is Lyapunov stable if the position of it is a strict local minimum of U ( q ) .

Theorem 3 implies that we can design an energy function H net such that the induced graph neural flow is both Lyapunov stable and energy conservative. For instance, we can define T as

where h i = ∑ j ∈N ( i ) a G ( q i , q j ) p j is the aggregation of p j through the attention mechanism a G ( q i , q j ) calculated based on q (or we directly use adjacency matrix with a G ( q i , q j ) ≡ W [ i, j ] ). The term λ is a small positive number included to ensure the positive definiteness. If U ( q ) has only a single global minimum, such as U ( q ) = ∥ q ∥ with an ℓ 2 norm, this stability may not ensure adversarial robustness as discussed in Section 3.1. An alternative is setting U ( q ) = ∥ sin( q ) ∥ , which fosters Lyapunov stability across multiple local equilibrium points. However, this choice considerably restricts the form that U can take and may consequently limit the model's capacity. In the implementation, we set the function U ( q ) in H to be a single GAT [6] layer g gat : R r ×|V| → R r ×V with a sin activation function followed by an ℓ 2 norm, i.e., U ( q ) = ∥ g gat ( q ) ∥ 2 . The stability of HANG and HANG-quad is summarized in Table 1.

6 Experiments

In this section, we conduct a comprehensive evaluation of our theoretical findings and assess the robustness of two conservative stable models: HANG and HANG-Quad. We compare their performance against various benchmark GNN models, including GAT [47], GraphSAGE [48], GCN [3] and other prevalent graph neural flows. We incorporate different types of graph adversarial attacks as described in Section 6.1 and Section 6.3. These attacks are conducted in a black-box setting, where a surrogate model is trained to generate perturbed graphs or features. For more experiments, we direct readers to Appendix C.

6.1 Graph Injection Attacks (GIA)

We implement various GIA algorithms following the methodology in [49]. This framework consists of node injection and feature update procedures. Node injection involves generating new edges for injected nodes using gradient information or heuristics. We use the PGD-GIA method [49] to randomly inject nodes and determine their features with the PGD algorithm [50]. TDGIA [12] identifies topological vulnerabilities to guide edge generation and optimize a smooth loss function for feature generation. MetaGIA [49] performes iterative updates of the adjacency matrix and node features using gradient information. Our datasets include citation networks (Cora, Citeseer, Pubmed) [51], the Coauthor academic network [52], an Amazon co-purchase network (Computers) [52], and the Ogbn-Arxiv dataset [53]. For inductive learning, we follow the data splitting method in the GRB framework [54], with 60% for training, 10% for validation, and 20% for testing. Details on data statistics and attack budgets can be found in Appendix C.1. Targeted attacks are applied to the OgbnArxiv and Computers datasets [49]. Additional results for various attack strengths and white-box attacks can be found in Appendix C.4 and Appendix C.3, respectively.

6.2 Performance Results Under GIAs

Upon examining the results in Table 2 and Table 3 pertaining to experiments conducted under GIA conditions, the robustness of our proposed HANG and HANG-quad, is notably prominent across different GIA scenarios. Interestingly, GRAND, despite its Lyapunov stability as analyzed in Theorem 1, does not significantly outperform GAT under certain attacks. In contrast, HANG consistently displays robustness against attacks. Notably, HANG-quad exhibits superior performance to HANG on the Pubmed dataset under GIA perturbations, underscoring the effectiveness of integrating both Lyapunov stability and Hamiltonian mechanics to boost robustness. Although other graph neural flows might demonstrate a range of improved performance compared to conventional GNN models under GIA, the degree of improvement is not consistently distinct. Despite the pronounced association between conservative stability in Hamiltonian systems and adversarial robustness, a clear relationship between adversarial robustness with the other stability of the graph neural flows outlined in Table 1 is not immediately discernible. The performance differential between HANG variants and other graph neural flows further underscores the potential of our proposed Hamiltonian models in enhancing robustness against GIA attacks.

6.3 Graph Modification Attacks

To evaluate the robustness of our proposed conservative models, we conducted graph modification adversarial attacks using the Metattack method [17]. We followed the attack setting described in the Pro-GNN [55] and utilized the perturbed graph provided by the library [56] to ensure a fair comparison. The perturbation rate, which indicates the proportion of altered edges, was incrementally varied in 5% increments from 0% to 25%. For comparison, we also considered other defense models for GNNs, namely Pro-GNN [55], RGCN [57], and GCN-SVD [58]. We report the results of baseline models from [55].

6.4 Performance Results Under Modification/Poisoning/Transductive Attacks

In the case of the Polblogs dataset [59], as shown in Table 4, our proposed HANG-quad model demonstrates superior performance compared to other methods, including existing defense models. This result indicates that incorporating Lyapunov stability indeed enhances HANG's robustness against graph modification and poisoning attacks. For the Pubmed dataset, we note that the impact of Meta-attacks of varying strengths on all graph neural flows, including our proposed ones, is negligible. Conversely, traditional GNN models such as GAT, GCN, and RGCN are marginally affected as the attack strength escalates. This observation underlines the robustness of graph neural flows, including our proposed models, against Meta-attacks on this dataset.

6.5 Combination with other defense mechanisms

It merits noting that our models, HANG and HANG-quad, can be readily integrated with additional defense mechanisms against adversarial attacks. These include Adversarial Training (AT) [60] and other preprocessing methods such as GNNGUARD [61]. This integration can further bolster the robustness of the HANG model. To validate this enhancement, extensive experiments are conducted, with results detailed in Appendix C.8 and Appendix C.9.

7 Conclusion

In this paper, we conducted a comprehensive study on stability notions in the context of graph neural flows and made significant findings. While Lyapunov stability is frequently employed, it alone may not suffice in guaranteeing robustness against adversarial attacks. With a grounding in foundational physics principles, we proposed a shift towards conservative Hamiltonian neural flows for crafting GNNs resilient against adversarial attacks. Our empirical comparisons across diverse neural flow GNNs, as tested on multiple benchmark datasets subjected to a range of adversarial attacks, have further corroborated this proposition. Notably, GNNs that amalgamate conservative Hamiltonian flows with Lyapunov stability exhibited marked enhancement in their robustness metrics. We are optimistic that our work will inspire further research into marrying physics principles with machine learning paradigms for enhanced security.

8 Acknowledgments and Disclosure of Funding

This research is supported by the Singapore Ministry of Education Academic Research Fund Tier 2 grant MOE-T2EP20220-0002, and the National Research Foundation, Singapore and Infocomm Media Development Authority under its Future Communications Research and Development Programme. To improve the readability, parts of this paper have been grammatically revised using ChatGPT [62].

References

X. Yue, Z. Wang, J. Huang, S. Parthasarathy, S. Moosavinasab, Y. Huang, S. M. Lin, W. Zhang, P. Zhang, and H. Sun, 'Graph embedding on biomedical networks: methods, applications and evaluations,' Bioinformatics , vol. 36, no. 4, pp. 1241-1251, 2019.

H. Ashoor, X. Chen, W. Rosikiewicz, J. Wang, A. Cheng, P. Wang, Y. Ruan, and S. Li, 'Graph embedding and unsupervised learning predict genomic sub-compartments from hic chromatin interaction data,' Nat. Commun. , vol. 11, 2020.

T. N. Kipf and M. Welling, 'Semi-supervised classification with graph convolutional networks,' in Proc. Int. Conf. Learn. Representations , 2017.

Z. Zhang, P. Cui, and W. Zhu, 'Deep learning on graphs: A survey,' IEEE Trans. Knowl. Data Eng. , vol. 34, no. 1, pp. 249-270, Jan 2022.

Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, 'A comprehensive survey on graph neural networks,' IEEE Trans. Neural Netw. Learn. Syst. , vol. 32, no. 1, pp. 4-24, 2021.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, 'Graph attention networks,' in Proc. Int. Conf. Learn. Representations , 2018, pp. 1-12.

F. Ji, S. H. Lee, H. Meng, K. Zhao, J. Yang, and W. P. Tay, 'Leveraging label non-uniformity for node classification in graph neural networks,' in Proc. Int. Conf. Mach. Learn. , vol. 202, Jul. 2023, pp. 14 869-14 885.

S. H. Lee, F. Ji, and W. P. Tay, 'SGAT: Simplicial graph attention network,' in Proc. Inter. Joint Conf. Artificial Intell. , Jul. 2022.

D. Zügner, A. Akbarnejad, and S. Günnemann, 'Adversarial attacks on neural networks for graph data,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2018.

J. Wang, M. Luo, F. Suya, J. Li, Z. Yang, and Q. Zheng, 'Scalable attack on graph data by injecting vicious nodes,' Data Mining Knowl. Discovery , pp. 1 - 27, 2020.

Q. Zheng, Y. Fei, Y. Li, Q. Liu, M. Hu, and Q. Sun. Kdd cup 2020 ml track 2 adversarial attacks and defense on academic graph 1st place solution. Accessed: May 1, 2022. [Online]. Available: https://github.com/Stanislas0/KDD_CUP_2020_MLTrack2_SPEIT

X. Zou, Q. Zheng, Y. Dong, X. Guan, E. Kharlamov, J. Lu, and J. Tang, 'Tdgia: Effective injection attacks on graph neural networks,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2021, p. 2461-2471.

H. Hussain, M. Cao, S. Sikdar, D. Helic, E. Lex, M. Strohmaier, and R. Kern, 'Adversarial inter-group link injection degrades the fairness of graph neural networks,' in Prof. Int. Conf. Data Mining , 2022, pp. 975-980.

J. Chen, Y. Wu, X. Xu, Y. Chen, H. Zheng, and Q. Xuan, 'Fast gradient attack on network embedding,' arXiv preprint arXiv:1809.02797 , 2018.

M. Waniek, T. P. Michalak, M. J. Wooldridge, and T. Rahwan, 'Hiding individuals and communities in a social network,' Nature Human Behaviour , vol. 2, no. 1, pp. 139-147, 2018.

J. Du, S. Zhang, G. Wu, J. M. F. Moura, and S. Kar, 'Topology adaptive graph convolutional networks,' ArXiv , vol. abs/1710.10370, 2017.

D. Zügner and S. Günnemann, 'Adversarial attacks on graph neural networks via meta learning,' in Proc. Int. Conf. Learn. Representations , 2019.

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, 'Neural ordinary differential equations,' in Advances Neural Inf. Process. Syst. , 2018.

H. Yan, J. Du, V. Y. Tan, and J. Feng, 'On robustness of neural ordinary differential equations,' in Advances Neural Inf. Process. Syst. , 2018, pp. 1-13.

Q. Kang, Y. Song, Q. Ding, and W. P. Tay, 'Stable neural ODE with Lyapunov-stable equilibrium points for defending against adversarial attacks,' in Advances Neural Inf. Process. Syst. , 2021.

I. D. J. Rodriguez, A. Ames, and Y. Yue, 'Lyanet: A lyapunov framework for training neural odes,' in Proc. Int. Conf. Mach. Learn. , 2022, pp. 18 687-18 703.

R. She, Q. Kang, S. Wang, Y.-R. Yáng, K. Zhao, Y. Song, and W. P. Tay, 'Robustmat: Neural diffusion for street landmark patch matching under challenging environments,' IEEE Trans. Image Process. , 2023.

S. Wang, Q. Kang, R. She, W. P. Tay, A. Hartmannsgruber, and D. N. Navarro, 'RobustLoc: Robust camera pose regression in challenging driving environments,' in Proc. AAAI Conference on Artificial Intelligence , Feb. 2023.

R. She, Q. Kang, S. Wang, W. P. Tay, Y. L. Guan, D. N. Navarro, and A. Hartmannsgruber, 'Image patch-matching with graph-based learning in street scenes,' IEEE Trans. Image Process. , vol. 32, pp. 3465-3480, 2023.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proc. IEEE Int. Conf. Comput. Vision , 2016, pp. 770-778.

Y. D. Zhong, B. Dey, and A. Chakraborty, 'Symplectic ode-net: Learning hamiltonian dynamics with control,' in Proc. Int. Conf. Learn. Representations , 2020.

Y. Chen, T. Matsubara, and T. Yaguchi, 'Neural symplectic form: Learning hamiltonian equations on general coordinate systems,' in Advances Neural Inf. Process. Syst. , 2021.

Y. Huang, Y. Yu, H. Zhang, Y. Ma, and Y. Yao, 'Adversarial robustness of stabilized neural ode might be from obfuscated gradients,' in Proc. Math. Sci. Mach. Learn. Conf. , J. Bruna, J. Hesthaven, and L. Zdeborova, Eds., 2022, pp. 497-515.

B. P. Chamberlain, J. Rowbottom, M. Goronova, S. Webb, E. Rossi, and M. M. Bronstein, 'Grand: Graph neural diffusion,' in Proc. Int. Conf. Mach. Learn. , 2021.

M. Thorpe, T. M. Nguyen, H. Xia, T. Strohmer, A. Bertozzi, S. Osher, and B. Wang, 'Grand++: Graph neural diffusion with a source term,' in Proc. Int. Conf. Learn. Representations , 2021.

B. P. Chamberlain, J. Rowbottom, D. Eynard, F. Di Giovanni, D. Xiaowen, and M. M. Bronstein, 'Beltrami flow and neural diffusion on graphs,' in Advances Neural Inf. Process. Syst. , 2021.

Y. Song, Q. Kang, S. Wang, K. Zhao, and W. P. Tay, 'On the robustness of graph neural diffusion to topology perturbations,' in Advances Neural Inf. Process. Syst. , New Orleans, USA, Nov. 2022.

K. Zhao, Q. Kang, Y. Song, R. She, S. Wang, and W. P. Tay, 'Graph neural convection-diffusion with heterophily,' in Proc. Inter. Joint Conf. Artificial Intell. , Aug. 2023.

Q. Kang, K. Zhao, Y. Song, S. Wang, and W. P. Tay, 'Node embedding from neural Hamiltonian orbits in graph neural networks,' in Proc. Int. Conf. Mach. Learn. , vol. 202, Jul. 2023, pp. 15 786-15 808.

T. K. Rusch, B. P. Chamberlain, J. Rowbottom, S. Mishra, and M. M. Bronstein, 'Graph-coupled oscillator networks,' in Proc. Int. Conf. Mach. Learn. , 2022.

W. E. Boyce and R. C. a. DiPrima, Elementary Differential Equations and Boundary Value Problems . Wiley, New York, 1997, vol. 3.

M. W. Hirsch and S. Smale, Differential Equations, Dynamical Systems and an Introduction to Chaos . Academic Press/Elsevier, San Diego, 2004.

A. Katok and B. Hasselblatt, Introduction to the modern theory of dynamical systems . Cambridge University Press, 1995, no. 54.

C.-T. Chen and B. Shafai, Linear system theory and design . New York: Oxford university press New York, 1999.

D. Arrowsmith and C. M. Place, Dynamical systems: differential equations, maps, and chaotic behaviour . London: CRC Press, 1992.

M. E. Sander, P. Ablin, M. Blondel, and G. Peyré, 'Sinkformers: Transformers with doubly stochastic attention,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2022, pp. 3515-3530.

R. A. Horn and C. R. Johnson, Matrix analysis . New York: Cambridge university press, 2012.

M. De León and P. R. Rodrigues, Generalized Classical Mechanics and Field Theory: a geometrical approach of Lagrangian and Hamiltonian formalisms involving higher order derivatives . Elsevier, 2011.

W. Curtin and H. Scher, 'Mechanics modeling using a spring network,' J. Mater. Research , vol. 5, no. 3, pp. 554-562, 1990.

A. C. Da Silva and A. C. Da Salva, Lectures on symplectic geometry . Springer, 2008, vol. 3575.

V. I. Arnold, V. V. Kozlov, and A. I. Neishtadt, Mathematical aspects of classical and celestial mechanics . Springer Science & Business Media, 2007, vol. 3.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, 'Graph attention networks,' in Proc. Int. Conf. Learn. Representations , 2018, pp. 1-12.

W. L. Hamilton, R. Ying, and J. Leskovec, 'Inductive representation learning on large graphs,' in Advances Neural Inf. Process. Syst. , 2017.

Y. Chen, H. Yang, Y. Zhang, K. Ma, T. Liu, B. Han, and J. Cheng, 'Understanding and improving graph injection attack by promoting unnoticeability,' in Proc. Int. Conf. Learn. Representations , 2022.

A. M ˛ adry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, 'Towards deep learning models resistant to adversarial attacks,' in Proc. Int. Conf. Learn. Representations , 2018.

Z. Yang, W. Cohen, and R. Salakhudinov, 'Revisiting semi-supervised learning with graph embeddings,' in Proc. Int. Conf. Mach. Learn. , 2016, pp. 40-48.

O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, 'Pitfalls of graph neural network evaluation,' Relational Representation Learning Workshop, Advances Neural Inf. Process. Syst. , 2018.

W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, 'Open graph benchmark: Datasets for machine learning on graphs,' Advances Neural Inf. Process. Syst. , vol. 33, pp. 22 118-22 133, 2020.

Q. Zheng, X. Zou, Y. Dong, Y. Cen, D. Yin, J. Xu, Y. Yang, and J. Tang, 'Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning,' Advances Neural Inf. Process. Syst. Track Datasets Benchmarks , 2021.

W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang, 'Graph structure learning for robust graph neural networks,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2020, pp. 66-74.

Y. Li, W. Jin, H. Xu, and J. Tang, 'Deeprobust: A pytorch library for adversarial attacks and defenses,' arXiv preprint arXiv:2005.06149 , 2020.

D. Zhu, Z. Zhang, P. Cui, and W. Zhu, 'Robust graph convolutional networks against adversarial attacks,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2019, pp. 1399-1407.

N. Entezari, S. A. Al-Sayouri, A. Darvishzadeh, and E. E. Papalexakis, 'All you need is low (rank): Defending against adversarial attacks on graphs,' in Proc. Int. Conf. Web Search Data Mining , 2020, p. 169-177.

L. A. Adamic and N. Glance, 'The political blogosphere and the 2004 us election: divided they blog,' in Proc. Int. Workshop Link Discovery , 2005, pp. 36-43.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, 'Towards deep learning models resistant to adversarial attacks,' arXiv preprint arXiv:1706.06083 , 2017.

X. Zhang and M. Zitnik, 'Gnnguard: Defending graph neural networks against adversarial attacks,' Advances Neural Inf. Process. Syst. , vol. 33, pp. 9263-9275, 2020.

OpenAI, 'Chatgpt-4,' 2022, available at: https://www.openai.com (Accessed: 26 September 2023).

Y. Ma, S. Wang, T. Derr, L. Wu, and J. Tang, 'Graph adversarial attack via rewiring,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2021, p. 1161-1169.

Y. Sun, S. Wang, X. Tang, T.-Y. Hsieh, and V. Honavar, 'Adversarial attacks on graph neural networks via node injections: A hierarchical reinforcement learning approach,' in Proc. Web Conf. , 2020, p. 673-683.

X. Wan, H. Kenlay, B. Ru, A. Blaas, M. A. Osborne, and X. Dong, 'Adversarial attacks on graph classification via bayesian optimisation,' arXiv preprint arXiv:2111.02842 , 2021.

S. Geisler, T. Schmidt, H. ¸ Sirin, D. Zügner, A. Bojchevski, and S. Günnemann, 'Robustness of graph neural networks at scale,' Advances Neural Inf. Process. Syst. , vol. 34, pp. 7637-7649, 2021.

J. Ma, J. Deng, and Q. Mei, 'Adversarial attack on graph neural networks as an influence maximization problem,' in Prof. Int. Conf. Web Search Data Mining , 2022, pp. 675-685.

B. Finkelshtein, C. Baskin, E. Zheltonozhskii, and U. Alon, 'Single-node attacks for fooling graph neural networks,' Neurocomputing , vol. 513, pp. 1-12, 2022.

D. Zhu, Z. Zhang, P. Cui, and W. Zhu, 'Robust graph convolutional networks against adversarial attacks,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2019, p. 1399-1407.

W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, 'Graph random neural networks for semi-supervised learning on graphs,' in Proc. Advances Neural Inf. Process. Syst. , 2020.

W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang, 'Graph structure learning for robust graph neural networks,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2020, p. 66-74.

X. Gao, W. Hu, and Z. Guo, 'Exploring structure-adaptive graph learning for robust semisupervised classification,' in 2020 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 2020, pp. 1-6.

T. Zhao, Y. Liu, L. Neves, O. Woodford, M. Jiang, and N. Shah, 'Data augmentation for graph neural networks,' in Proc. AAAI Conference on Artificial Intelligence , vol. 35, no. 12, 2021, pp. 11 015-11 023.

K. Li, Y. Liu, X. Ao, J. Chi, J. Feng, H. Yang, and Q. He, 'Reliable representations make a stronger defender: Unsupervised structure refinement for robust gnn,' in Proc. Int. Conf. Knowl. Discovery Data Mining , 2022, pp. 925-935.

B. Runwal, S. Kumar et al. , 'Robust graph neural networks using weighted graph laplacian,' arXiv preprint arXiv:2208.01853 , 2022.

X. Zhang and M. Zitnik, 'Gnnguard: Defending graph neural networks against adversarial attacks,' in Proc. Advances Neural Inf. Process. Syst. , 2020.

S. Greydanus, M. Dzamba, and J. Yosinski, 'Hamiltonian neural networks,' in Advances Neural Inf. Process. Syst. , 2019.

Z. Chen, J. Zhang, M. Arjovsky, and L. Bottou, 'Symplectic recurrent neural networks,' in Proc. Int. Conf. Learn. Representations , 2020.

A. Choudhary, J. F. Lindner, E. G. Holliday, S. T. Miller, S. Sinha, and W. L. Ditto, 'Physicsenhanced neural networks learn order and chaos,' Physical Review E , vol. 101, no. 6, p. 062207, 2020.

E. Haber and L. Ruthotto, 'Stable architectures for deep neural networks,' Inverse Problems , vol. 34, no. 1, pp. 1-23, Dec. 2017.

J. Li, J. Peng, L. Chen, Z. Zheng, T. Liang, and Q. Ling, 'Spectral adversarial training for robust graph neural network,' IEEE Transactions on Knowledge and Data Engineering , 2022.

C. Deng, X. Li, Z. Feng, and Z. Zhang, 'Garnet: Reduced-rank topology learning for robust and scalable graph neural networks,' in Learning on Graphs Conference . PMLR, 2022, pp. 3-1.

V. Nikiforov, 'Revisiting schur's bound on the largest singular value,' arXiv preprint math/0702722 , 2007.

H. K. Khalil, 'Nonlinear control,' 2015, available at: https://www.egr.msu.edu/~khalil/ NonlinearControl/Slides-Short/Lecture_2.pdf (Accessed: 26 September 2023).

R. Durrett, Probability: theory and examples . Cambridge university press, 2019, vol. 49.

A Summary of Supplement

In this supplement, we expand on material from the main paper, addressing each point as follows:

An in-depth comparison with related work is covered in Appendix B.

Appendix C presents the datasets and details of the attackers, and includes additional experimental results, inference time, and model size to reinforce our model's effectiveness.

Theoretical proofs supporting assertions made in the main paper are presented in Appendix E.

Further discussions on the robustness insights of our model are covered in Appendix D.

The complete summary of the algorithm can be found in Appendix G.

Lastly, we address the limitations of our work and discuss its broader impact.

B Related Work

In what follows, we briefly review a few concepts closely related to our work.

Graph Adversarial Attacks and Defenses. In modification attacks , adversaries can perturb a graph's topology by adding or removing edges [9, 14-17, 63-66]. To improve the modification attack performance, adversaries are also permitted to perturb node attributes [9, 17, 63, 64, 67, 68]. In injection attacks , adversaries can only inject malicious nodes into the original graph [10-13] while the edges and nodes inside the original graph are not allowed to be perturbed. For defense methods against adversarial attacks, multiple robust GNN models have been proposed. Examples include RobustGCN [69], GRAND [70], ProGNN [71], GLNN [72], GAUGM [73], STABLE [74] and RWL-GNN [75]. In addition, preprocessing-based defenders including GNN-SVD [58] and GNNGuard [76] may help to improve GNN robustness.

In this paper, we use different attacks to test GNNs robustness. We compare graph neural flows with different stability settings with the above-mentioned defense methods as robustness benchmarks.

Stable Graph Neural Flow Networks. While traditional GNNs perform message passing on a simple, discrete and flat space, graph neural flows model the message passing as a continuous diffusion process that occurs on a smooth manifold. GRAND [29] and GRAND++ [30] use heat diffusion to achieve feature information exchange. BLEND [31] exploits the Beltrami diffusion where the nodes' positional information is updated along with their features. GraphCON [35] adopts the coupled oscillator model that preserves the graph's Dirichlet energy over time and thus mitigates the oversmoothing problem. In general, [32] shows that graph neural PDEs are Lyapunov stable and exhibit stronger robustness against graph topology perturbation than traditional GNNs.

While most of the above-mentioned graph neural flows are Lyapunov stable, whether the notion of Lyapunov stability leads to better adversarial robustness is an open question. In this paper, we argue that Lyapunov stability does not necessarily imply adversarial robustness.

Hamiltonian Neural Networks. Hamiltonian equations have been applied to conserve an energy-like quantity in (graph) neural networks. The references [26, 27, 77] train a neural network to infer the Hamiltonian dynamics of a physical system, where Hamiltonian equations are solved using neural ODE solvers. In [78], the authors propose to learn a Hamiltonian function of the system by a neural network to capture the dynamics of physical systems from observed trajectories. They shows that the network performs well on noisy and complex systems such as a spring-chain system. To forecast dynamics, the work [79] use neural networks that incorporate Hamiltonian dynamics to efficiently learn phase space orbits and demonstrate the effectiveness of Hamiltonian neural networks on several dynamics benchmarks. The paper [80] builds a Hamiltonian-inspired neural ODE to stabilize the gradients so as to avoid gradient vanishing and gradient exploding.

In this paper, inspired by existing Hamiltonian neural networks, we introduce several energyconservative graph neural flows. We are neither simulating a physical system nor forecasting a forecast the dynamics for a physical problem. Instead, we combine the Hamiltonian mechanics concept with graph neural networks to develop a new robust GNN.

C More Experiments

C.1 Data and Attackers

The datasets and attack budgets utilized in Table 2 and Table 3 are outlined in Table 5 and Table 6, respectively. These datasets span various domains and scales, thereby providing a diverse base for our study. The adopted attack budget aligns consistently with the specifications set out in the paper [49].

C.2 Implementation Details

The raw node features are compressed to a fixed dimension, such as 64, using a fully connected (FC) layer to generate the initial features q (0) in (10). At time t = 0 , p (0) and q (0) are initialized identically. For t > 0 , both q ( t ) and p ( t ) undergo updates using a graph ODE. The ODE is solved using the solver from [18]. It is observed that different solvers deliver comparable performance in terms of clean accuracy. However, to mitigate computational expense, the Euler solver is employed in our experiments, with an ablation study on different solvers provided for further insight. The integral time T acts as a hyperparameter in our model. Interestingly, the performance of the model exhibits minimal sensitivity to this time T . For all datasets, we establish the time T as 3 and maintain a fixed step size of 1. This setup aligns with a fair comparison to three-layer GNNs.

All the baseline models presented in Table 2 and Table 3 are implemented based on the original work of [49]. The baseline model results in Table 4 are directly extracted from the paper [55]. This is done as we employ the same clean and perturbed graph datasets provided in their research [55].

Our experiment code is developed based on the following repositories:

https://github.com/tk-rusch/GraphCON

https://github.com/twitter-research/graph-neural-pde

https://github.com/LFhase/GIA-HAO

https://github.com/ChandlerBang/Pro-GNN

C.3 White-box Attack

In our main study, we utilized black-box attacks. Now, we extend our experiments to incorporate white-box, injection, and evasion attacks. In the context of white-box attacks, the adversaries have full access to the target model, enabling them to directly attack the target model to generate a perturbed graph. This represents a significantly more potent form of attack than the black-box variant. Moreover, we execute inductive learning tasks the same as Table 2, with the corresponding results reported in Table 7. We observe that, under white-box attack conditions, all other baseline models exhibit severely reduced performance, essentially collapsing across all datasets. The classification accuracy of the HANG model experiences a slight decline on the Cora, Citeseer, and Pubmed datasets. However, its performance remains substantially superior to other graph neural flow models or GNN models. Intriguingly, our HANG-quad model remains virtually unaffected by the white-box attacks, maintaining a performance level similar to that observed under black-box attacks. Such robustness of HANG-quad underscores the pivotal contribution of Lyapunov stability when combined with our Hamiltonian-driven design.

C.4 Attack Strength

We assess the robustness of the HANG model and its variant under varying attack strengths, with the node classification results displayed in Table 8. Our analysis reveals that the HANG model demonstrates superior robustness as the attack budget escalates. It is noteworthy, however, that under larger attack budgets, HANG-quad may relinquish its robustness attribute in the face of PGD and Meta injection attacks. This result implies that the amalgamation of Lyapunov stability and conservative Hamiltonian system facilitates enhanced robustness only under minor graph perturbations. On the contrary, the HANG model, which solely incorporates the conservative Hamiltonian design, exhibits a consistently high-performance level regardless of the increasing attack strength.

C.5 Nettack

We further evaluate the robustness of our model under the targeted poisoning attack, Nettack [9]. Adhering to the settings outlined in [55], we select nodes in the test set with a degree greater than 10 to be the target nodes. We then vary the number of perturbations applied to each targeted node from 1 to 5, incrementing in steps of 1. It is important to note that Nettack only involves feature perturbations. The test accuracy in Table 9 refer to the classification accuracy on the targeted nodes. As demonstrated in Table 9, our HANG-quad model exhibits exceptional resistance to Nettack, thereby underlining its superior robustness. This suggests that the combination of Lyapunov stability and conservative Hamiltonian design significantly enhances robustness in the face of graph poisoning attacks. On its own, Lyapunov stability has been recognized to defend against only slight feature perturbations in the input graph [32]. Furthermore, our HANG model also displays superior resilience compared to other graph neural flows, reinforcing the fact that the Hamiltonian principle contributes to its robustness.

C.6 ODE solvers

The results from various ODE solvers are depicted in Table 10. We consider fixed-step Euler and RK4, along with adaptive-step Dopri5, from [18], and Symplectic-Euler from [35]. The SymplecticEuler method, being inherently energy-conserving, is particularly suited for preserving the dynamic properties of Hamiltonian systems over long times. Our observations suggest that while the choice of solver slightly influences the clean accuracy for some models, their performance under attack conditions remains fairly consistent. Consequently, there was no specific optimization for solver selection during our experiments. For computational efficiency, we opted for the Euler ODE solver in all experiments presented in the main paper.

C.7 Computation Time

The average inference time and model size for different models used in our study are outlined in Table 11. This analysis is performed using the Cora dataset, with all graph PDE models employing the Euler Solver, an integration time of 3, and a step size of 1. Additionally, for fair comparison, all baseline models are configured with 3 layers. Upon examination, it is observed that our HANG and HANG-quad models necessitate more inference time compared to other baseline models. This is primarily due to the requirement in these models to initially calculate the derivative. However, when compared to other defense models, such as GCNGUAD, our models are still more efficient, thus validating their practical utility.

C.8 Combination with Adversarial Training(AT)

Adversarial training (AT) [60] is a technique employed to improve the robustness of machine learning models by exposing them to adversarial examples during the training phase. These adversarial examples are maliciously crafted inputs designed to mislead the model, thereby helping it to learn more robust and generalized features.

In this context, we conducted additional experiments applying PGD-AT [60] to both GAT and HANG models, evaluating their robust accuracy under PGD, TDGIA, and METAGIA attacks in both whitebox and black-box scenarios. As illustrated in Table 12 and Table 13, while PGD-AT enhances the robustness of all models, its effect on GAT's resilience to the unique white-box METAGIA attack is notably restrained, highlighting the limitations of PGD-AT's efficacy, particularly when faced with non-AT tactics. Conversely, HANG consistently displays remarkable performance across a variety of attacks, showing significant improvement when combined with PGD-AT. Both HANG-AT and HANG-quad-AT exhibit substantial resistance to these attacks, underscoring HANG's inherent robustness.

Furthermore, we explored adversarial training from a graph spectral perspective, incorporating Spectral Adversarial Training (SAT) [81] into our HANG framework, thus creating the HANGSAT variant. The integration of SAT, as depicted in Table 12 and Table 13, significantly boosts HANG's robustness against various attacks, with minimal reduction in post-attack accuracy, further exemplifying HANG's intrinsic resilience.

C.9 Combination with GNN defense mechanisms

We acknowledge the potential advantages of integrating defense mechanisms that operate through diverse strategies. In this context, we combine HANG with GNNGuard [61], a preprocessing-based defense renowned for its effectiveness in enhancing GNN robustness, and GARNET [82], another noteworthy defense mechanism.

We conduct experiments to assess the performance of HANG when amalgamated with GNNGuard and GARNET. The results in Table 14 and Table 15 clearly indicate that such integration effectively enhances the model's robustness. This outcome highlights the adaptability of our methodology with current defense strategies and sheds light on the potential for cooperative improvements in model defense.

D More Insights about Model Robustness

For a clearer insight into the energy concept within HANG, consider the 'time' in graph neural flows as analogous to the 'layers' in standard GNNs (note that ODE solvers basically discretize the 'time', which indeed approximately turns the model into a layered one). Here, the feature vector q ( t ) and the momentum vector p ( t ) evolve with time, bound tightly by equation (10). Given that q ( t ) mirrors the node features at layer t, p ( t ) can be understood as the variation in node features over time - essentially, the evolution of node features between successive layers. Thus, our defined energy interweaves both the node feature and its rate of change across adjacent layers. The constant H net implies inherent constraints on the node features and their alteration pace over layers. Because H net processes the whole graph data and yields a scalar, it serves as a constraint on the global graph feature and its variation, which we opine to be crucial in countering adversarial attacks.

From an adversarial perspective, the attacker modifies either the node features or the underlying graph topology. These modifications are propagated through multiple aggregation steps, such as layers in conventional GNNs or integrals in graph ODEs. While the Hamiltonian considers the energy of the entire graph, adversarial attacks often target localized regions of the graph. The inherent global energy perspective of the Hamiltonian system makes it resilient to such localized attacks, as local perturbations often get 'absorbed' or 'mitigated' when viewed from the perspective of the entire system. When adversarial perturbations are introduced, they might indeed tweak the instantaneous features of certain nodes. However, the challenge lies in modifying the trajectory (or evolution) of these nodes (positions p ( t ) and the variations q ( t ) ) in the phase space in a manner that is aligned with the rest of the graph, all while upholding the energy conservation constraints. This feat is arduous,

if not impossible, without creating detectable inconsistencies elsewhere. This property ensures that the energy of each node feature is preserved over time and multiple aggregation steps. As a result, the distances between features of different nodes are preserved if their norms differ initially before the adversarial attack. The robustness of HANG against topology perturbation may stem from the fact that adversarial topology perturbation has small influence on the node feature energy. This can be seen from Fig. 1. When using HANG, the node feature before and after adversarial attack are relatively stable and well separated between nodes from different classes.

E Proof of Theorem 1

Recall that

Without loss of generality, we assume X ∈ R |V|× 1 since the results can be generalized to X ∈ R |V|× r on a component-wise basis.

Proof of 1).

Given our assumptions where A G ( X ( t )) is either column- or row-stochastic, the spectral radius of A G ( X ( t )) is 1, as established by [42][Theorem 8.1.22]. This implies that the modulus of its eigenvalues is as most 1.

We define a Lyapunov function as V ( X ( t )) = X ( t ) ⊺ X ( t ) = ∥ X ( t ) ∥ 2 2 where ∥·∥ 2 is the ℓ 2 euclidean norm. Take the derivative of V with respect to time, we have

We next prove that ˙ V ( X ( t )) ≤ 0 when A G ( X ( t )) is a doubly stochastic attention matrix under the GRAND-nl setting.

where ~ · ~ 2 is the spectral matrix norm induced by ℓ 2 vector norm. The given inequality is a direct result of the Cauchy-Schwarz inequality and the properties of the induced norm. Drawing from [42][Theorem 5.6.9], we can currently establish that ~ A G ( X ( t ) ~ 2 ≥ 1 . Nevertheless, we intend to assert that this inequality is indeed an equality when A G ( X ( t )) is doubly stochastic, i.e., ~ A G ( X ( t ) ~ 2 = 1 . To see this, from Schur's bounds on the largest singular value [83][eq.(1)], we have

where ~ · ~ 1 is the induced ℓ 1 norm and ~ · ~ ∞ is the induced ℓ ∞ norm. Given that A G ( X ( t ) is doubly stochastic, we observe that ~ A G ( X ( t ) ~ ∞ = ~ A G ( X ( t ) ~ 1 = 1 . It now follows that ~ A G ( X ( t ) ~ 2 ≤ 1 , the equality is consequently confirmed. It follows from (18) that

From (17) and (19), it follows that

when α ≥ 1 . The observation that V ( X ( t )) does not increase over time ensures that X ( t ) stays bounded. In effect, we also have proved Lyapunov stability concerning the equilibrium point 0 .

To prove asymptotic stability for the equilibrium point 0 when α > 1 , we need to show that the system not only remains bounded but also approaches 0 as t →∞ . This is indicated by the fact that

Note that

when α > 1 , ˙ V ( X ( t )) < 0 unless X ( t ) = 0 . This signifies that V ( X ( t )) strictly declines over time unless X ( t ) = 0 , thereby showing that the system will converge to 0 as t →∞ . Notably, we can infer global asymptotic stability since V is radially unbounded [84].

The proof for the GRAND-nl case is complete.

Proof of 2).

We next prove the claims under GRAND-l setting.

When A G is either column- or row-stochastic, the spectral radius of A G ( X ( t )) is 1, as established by [42][Theorem 8.1.22]. The modulus of its eigenvalues is as most 1. Consider the Jordan canonical form of A G represented as SJS -1 where J stands as the Jordan form.

Given that our equation system now represents a linear time-invariant ODE, the solution to (15) can be expressed as:

where ¯ J = J -α I . Citing [42][3.2.2. page 177, 4-th equation], for α > 1 , BIBO and Lyapunov stabilities are attained. Notably, for α > 1 , the system achieves global asymptotic stability around the equilibrium point 0 under any disturbances.

Further, by considering strong connectedness and referencing the Perron-Frobenius theorem [42][Theorem 8.4.4], it is deduced that 1 serves as the simple eigenvalue for A G . In this case, when α = 1 , we still have BIBO stability and Lyapunov stability according to [42][3.2.2. page 177, 4-th equation].

Proof of 3).

We subsequently demonstrate that when A G is selected to be column-stochastic and α = 1 , GRAND conserves a certain quantity interpretable as energy. Specifically, this 'energy' refers to the sum of the elements of X ( t ) , denoted as 1 ⊺ X ( t ) , with 1 representing an all-ones vector. From (15), this quantity remains conserved if

Given that A G is column-stochastic, it follows directly that 1 ⊺ A G = 0 ⊺ , validating (22).

Finally, we aim to prove that GRAND is asymptotic stable concerning a specified equilibrium vector when α = 1 and the graph is aperiodic and strongly connected. Based on [42][Theorem 3.2.5.2.,Theorem 8.5.3.] and [85][Theorem 5.6.6], we observe that lim k →∞ ( A G ) k = lim k →∞ SJ k S -1 = SΛS -1 , where Λ is a diagonal matrix with the first element as 1 and all the others as 0 :

Since lim k →∞ ( A G ) k maintains its column stochasticity and the rank of SΛS -1 is 1, it follows that each column is the same . Due to the zeros in Λ , the limit is the outer product of the first column of S and the first row of S -1 . Since we have the same column for the outer product, we deduce that the first row of S -1 is a 1 ⊺ with a being a scalar and 1 an all-ones vector. We have A G has an eigenvalue of 0, with the rest of the eigenvalues having strictly negative real parts . According to [42][3.2.2], it follows that

If s := 1 ⊺ X (0) remains constant for any perturbed X (0) , we have that lim t → 0 X ( t ) = sa s where s is the first column of S . We thus conclude that if the perturbation on X (0) maintains the column summations, i.e. the 'energy', unchanged, the asymptotic convergence to the equilibrium vector sa s remains unaffected.

The proof is now complete.

F Proof of Theorem 2

The energy conservation of our system, represented as H net in (9), remains invariant over time. To understand this better, consider the following

where the last equality follows from (10).

BIBO stability is inferred since, with an unbounded output, the energy conservation expressed by H net would be violated.

G Complete Algorithm Summary

We present the complete algorithm of HANG in Algorithm 1, which unfortunately had been delayed in its inclusion within the main paper due to space constraints.

Limitations

While our work on graph neural flows presents promising advancements in enhancing adversarial robustness of GNNs using Hamiltonian-inspired neural ODEs, it is not without limitations. As we demonstrated in the paper, the notions of stability borrowed from dynamical systems, such as BIBO stability and Lyapunov stability, do not always guarantee adversarial robustness. Our finding that energy-conservative Hamiltonian graph flows improve robustness is only one facet of the broader landscape of potential stability measures. It is possible that other notions of stability, not covered in this work, could yield additional insights into adversarial robustness. Our current Hamiltonian graph neural flows do not explicitly account for quasi-periodic motions in the graph dynamics. The Kolmogorov-Arnold-Moser (KAM) theory, a foundational theory in Hamiltonian dynamics, is renowned for its analysis of persistence of quasi-periodic motions under small perturbations in Hamiltonian dynamical systems. While the energy-conserving nature of our Hamiltonian-inspired model inherently offers some level of robustness to perturbations, an explicit incorporation of KAM theory could potentially further improve the robustness, particularly in the face of quasi-periodic adversarial attacks. However, this is a complex task due to the high dimensionality of typical graph datasets and the intricacies involved in approximating quasi-periodic dynamics.

Broader Impact

This research, centered on enhancing adversarial robustness in graph neural networks (GNNs), carries implications for various sectors, such as social media networks, sensor networks, and chemistry. By improving the resilience of GNNs, we can boost the reliability of AI-driven systems, contributing to greater efficiency, productivity, and cost-effectiveness. The shift towards automation may displace certain jobs, raising ethical concerns about income disparity and job security. Moreover, while our models enhance robustness, potential system failures can still occur, with impacts varying based on the application. Lastly, the robustness conferred might be exploited maliciously. Our work underscores the importance of diligent oversight, equitable technology implementation, and continuous innovation in the development of AI technologies.

Algorithm 1: Graph Node Embedding Learning with HANG

1 Initialization: Initialize the network modules including Hamiltonian function network H net , the raw node features compressor network FC, and the final classifier.

2 I. Training:

3 for Epoch 1 to N do

4 1) Perform the following to obtain the embedding q k ( T ) for each node k :

5 Input: G = ( V , E ) with raw node features

6 Apply FC to compress raw features for each node and get { ( q k (0) , p k (0)) } |V| k =1 . Here, we divide the 2 r dimensions into two equal segments. The first half acts as 'position' vectors q k (0) = ( q 1 , . . . , q r ) , while the second half acts as 'momentum' vectors

k k p k (0) = ( p 1 k , . . . , p r k ) that guide system evolution. For simplification, in code implementation, we may set the output feature dimension of the compressor FC as r and designate p k (0) = q k (0) .

7

8

9

10

11

12

To better express the evolution dynamics mathematically, concatenate and relabel the node features as:

Note that in the actual code implementation, the concatenation of node features is not necessary as H net is realized as either (12) or (13). The concatenate operation is only for mathematical formulation.

The trajectory of feature evolution is modelled as per the following canonical Hamilton's equations:

with the initial features ( q (0) , p (0)) ∈ R 2 r |V| .

Various ODE solvers as provided by [18] and the symplectic-euler solver from [35] can be employed to solve (25). We refer readers to Appendix C.6 for more details.

Acquire the evolved features at time T as q ( T ) , which is then decompressed into q ( T ) |V|

Utilize backpropagation to minimize the cross-entropy loss for node classification.

individual node features k k =1 for further utilization.

Perform validation over the validation split.

13

Save the model parameters.

14 II. Testing:

15 Load the model from the best validation epoch and perform Step I.1-4). to obtain the final feature embedding over the test split. Perform node classification.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDrawing inspiration from physics principles, the paper proposes the use of conservative Hamiltonian neural flows to construct GNNs that are robust against adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically evaluated on several benchmark datasets, considering a variety of adversarial attacks.\n\nSTRENGTHS:\nThis paper investigates the vulnerability of graph neural networks (GNNs) to adversarial perturbations that affect both node features and graph topology. The study focuses on GNNs derived from various neural flows and explores their connection to different stability notions.\n\nWEAKNESSES:\nThe followings are four weaknesses.\n\n1. Paper's writing quality is poor due to excessive amount of equations, formulas, and its explanation in a view of physics. The authors should use the adversarial research language to bring out the main insight for why this work is important for adversarial research.\n\n2. In addition, numerous words need to penetrate the core idea of equations rather than to revolve around technical terms.\n\n3. The reviewer did not see the connection bridging the gap between Hamiltonian mechanics and energy conservation.\n\n4. Is there any other simpler option that possess the notion of conservation other than Hamiltonian mechanics at all?  In addition, why is it important to conserve energy in a view of topological structure?\n\nOverall, the reviewer recommends the authors to answer these questions using insightful language at least to the reviewer, not using mathematical or physical language. The reviewer does not want a theoretical proof, only want really contributional insight; Why is it important to consider energy or conservation and why are \"energy and topology\" bridged? Why is the hamiltonian important? Please answer using adversarial language\n\n---\n\nThe authors' rebuttal has fully explained the connection between the energy conservation in the realm of the Hamiltonian and the adversarial robustness of GNNs using the adversarial research language. Thank the authors for their thorough commentary on the reviewer's point.\n\nQUESTIONS:\nRefer to Weaknesses\n\nLIMITATIONS:\nRefer to Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a robust GNN model by leveraging the notion of Hamiltonian Energy Conservation. Specifically, authors first analyze the stabilities and limitations of several neural ODE-based GNNs, which motivate the proposed model HANG that is inspired by Hamiltonian classical mechanics. Experimental results indicate that HANG outperforms prior ODE-based GNNs against various adversarial attacks on several realistic graph datasets.\n\nSTRENGTHS:\n- The paper is well written. All technical steps are easy to follow.\n- Authors propose a novel approach to improve GNN robustness.\n- Authors clearly motivate the proposed approach by analyzing the stability limitations of prior ODE-based GNNs.\n- The proposed model has been evaluated against various adversarial attacks.\n\nWEAKNESSES:\n- The major issue of this work is that it only compares with a limited number of (and relatively weak) defense baselines. There are some recent methods such as [1, 2] showing better results than the baselines chosen in this paper. It would further improve the paper if authors could compare HANG with those methods.\n- Figure 1 only shows the norm of node features become closer as time goes, which does not support authors' claim that node features gradually become similar (lines 182-183). Authors may want to show the norm of relative difference of node features is approaching to zero instead.\n  \n[1]: Geisler et al., “Robustness of Graph Neural Networks at Scale”, NeurIPS'21. \\\n[2]: Deng et al., “GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks”, LoG'22.\n\nI'm willing to raise the score if my major concerns are addressed.\n\nQUESTIONS:\n- Is it possible to further improve GNN robustness by combining HANG with some prior defense methods (e.g. graph preprocessing-based methods), as they improve robustness in orthogonal ways?\n\nLIMITATIONS:\nPlease refer to the Weaknesses section above for limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper explores the robustness of Graph Neural Networks (GNNs) against adversarial attacks. Drawing inspiration from principles in physics, the authors propose a novel model called Hamiltonian Neural Flows for constructing GNN models. The effectiveness of the proposed method is evaluated on various benchmark datasets.

STRENGTHS:
The problem addressed in this paper is both significant and intriguing, and the proposed method is well-supported by principles from physics.

WEAKNESSES:
I am a non-expert in physics-based methods, my evaluation is based on my understanding of GNN methods and educated guesses. I suggest AC to ignore my evaluation if there are expert reviewers in the field.

Here are a couple of suggestions to improve the paper:

In my opinion, the comparison with existing baselines seems inadequate. The TDGIA paper was published in 2021, while MetaGIA was published in early 2022 or late 2021. It would be beneficial if the authors could include more recent graph attack papers and compare their proposed method against them.

Additionally, it would be interesting to see a comparison between the proposed method and the Graph Isomorphism Network (GIN).

QUESTIONS:
see cons above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSince neural ordinary differential equation networks can show inherent robustness, in this work the authors try to perform an extensive study on different graph neural flows along with their stability on different stability notions like BIBO stability, Lyapunov stability, Structural stability and Conservative stability. The authors find that the graph neural flows using hamiltonian energy functions can achieve improved empirical adversarial robustness on using black box attacks like PGD and TDGIA. The authors show that designing graph neural flows which can ensure conservation stability along with Lyapunov stability can help in achieving improved adversarial robustness. As common in literature the authors generate a white box attack from a surrogate model and then transfer it to the black box model to testify the robustness of the black box model. Evaluation is done on node injection attacks as well as graph manipulation attacks, and significantly improved robustness is seen in both cases.\n\nSTRENGTHS:\n* I think using graph neural flows to understand if we can achieve inherent robustness without the need for adversarial training is interesting.\n* The way the authors have related different stability criteria with robustness is quite interesting.\n* The results show significant improvements.\n\nWEAKNESSES:\n* There have been many instances where defences claiming to be robust have been later evaded using adaptive attacks [1]. Adversarial training has been the most successful defence strategy and the current successful defences use adversarial training to achieve robustness. Therefore, it is necessary that the robustness of the proposed method is evaluated properly. I think that the surrogate model-based black box attacks are not strong enough to get the worst-case robustness. Therefore, it is important that the evaluation on strong white-box attacks like PGD, PGD with max-margin loss / Carlini and Wagner attack [2] is used to testify robustness, and this evaluation should be carried out in a white-box setting. Further, it is important that the authors share a robust accuracy (both white box and black box) vs constraint threat model plot for the proposed approach, GAT and GAT trained using PGD-AT [3]. \n\n* Comparison is done only with models trained using standard training. It is important to include adversarially trained models as baselines. For instance, the authors should include a PGD-AT trained model as a baseline in all the tables.\n\n* In the case of GNNs it is important to ensure that the attack graph remains imperceptible and the attacked nodes in the case of node injection attacks cannot be pruned off. Therefore other than robust accuracy imperceptibility of the graph for a given threat model is also a very important metric to be considered. Similar to [], I request the authors to also include a comparison on the imperceptibility of the attack on the proposed Graph neural flow network and the baselines like GAT. It is important to ensure that the imperceptiblity of the proposed defence is either lower or equal to the baselines against the PGD and TDGIA attacks.\n\n* The authors should also look at these works [4,5] and try to compare their methods with them .\n\n[1] Athalye, Anish et al. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.” International Conference on Machine Learning (2018).\n[2] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\"\n[3] Madry, Aleksander et al. “Towards Deep Learning Models Resistant to Adversarial Attacks.” \n[4] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.”\n[5] Li, Jintang et al. “Spectral Adversarial Training for Robust Graph Neural Network.”\n\nQUESTIONS:\nt would be great if the authors can address the raised concerns in the weakness section.\n\nLIMITATIONS:\nYes the authors seem to have addressed the limitations. I would suggest the authors to kindly look at the weakness section for suggestions and comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}