SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We subsequently demonstrate that when  $\mathbf{A}_G$  is selected to be column-stochastic and  $\alpha = 1$ , GRAND conserves a certain quantity interpretable as energy. Specifically, this "energy" refers to the sum of the elements of  $\mathbf{X}(t)$ , denoted as  $\mathbf{1}^{\mathrm{T}}\mathbf{X}(t)$ , with  $\mathbf{1}$  representing an all-ones vector. From (15), this quantity remains conserved if
$$
0 = \frac {\mathrm {d} \mathbf {1} ^ {\intercal} \mathbf {X} (t)}{\mathrm {d} t} = \mathbf {1} ^ {\intercal} \overline {{\mathbf {A}}} _ {G} (\mathbf {X} (t)) \mathbf {X} (t) \tag {22}
$$
Given that  $\mathbf{A}_G$  is column-stochastic, it follows directly that  $\mathbf{1}^{\intercal}\overline{\mathbf{A}}_G = \mathbf{0}^{\intercal}$ , validating (22).
Finally, we aim to prove that GRAND is asymptotic stable concerning a specified equilibrium vector when  $\alpha = 1$  and the graph is aperiodic and strongly connected. Based on [42][Theorem 3.2.5.2., Theorem 8.5.3.] and [85][Theorem 5.6.6], we observe that  $\lim_{k\to \infty}(\mathbf{A}_G)^k = \lim_{k\to \infty}\mathbf{S}\mathbf{J}^k\mathbf{S}^{-1} = \mathbf{S}\boldsymbol{\Lambda}\mathbf{S}^{-1}$ , where  $\boldsymbol{\Lambda}$  is a diagonal matrix with the first element as 1 and all the others as 0:
$$
\boldsymbol {\Lambda} = \left( \begin{array}{c c c c} 1 & & & \\ & 0 & & \\ & & \ddots & \\ & & & 0 \end{array} \right)
$$
Since  $\lim_{k\to \infty}(\mathbf{A}_G)^k$  maintains its column stochasticity and the rank of  $\mathbf{S}\pmb{\Lambda}\mathbf{S}^{-1}$  is 1, it follows that each column is the same. Due to the zeros in  $\pmb{\Lambda}$ , the limit is the outer product of the first column of  $\mathbf{S}$  and the first row of  $\mathbf{S}^{-1}$ . Since we have the same column for the outer product, we deduce that the first row of  $\mathbf{S}^{-1}$  is  $a\mathbf{1}^{\intercal}$  with  $a$  being a scalar and  $\mathbf{1}$  an all-ones vector. We have  $\overline{\mathbf{A}}_G$  has an eigenvalue of 0, with the rest of the eigenvalues having strictly negative real parts. According to [42][3.2.2], it follows that
$$
\lim  _ {t \rightarrow 0} \mathbf {X} (t) = \lim  _ {t \rightarrow 0} \mathbf {S} e ^ {\bar {\mathbf {J}} t} \mathbf {S} ^ {- 1} \mathbf {X} (0) = \mathbf {S} \boldsymbol {\Lambda} \mathbf {S} ^ {- 1} \mathbf {X} (0) \tag {23}
$$
If  $s \coloneqq \mathbf{1}^{\mathsf{T}}\mathbf{X}(0)$  remains constant for any perturbed  $\mathbf{X}(0)$ , we have that  $\lim_{t \to 0} \mathbf{X}(t) = s\mathbf{a}$  where  $\mathbf{s}$  is the first column of  $\mathbf{S}$ . We thus conclude that if the perturbation on  $\mathbf{X}(0)$  maintains the column summations, i.e. the "energy", unchanged, the asymptotic convergence to the equilibrium vector  $s\mathbf{a}$  remains unaffected.
The proof is now complete.
F Proof of Theorem 2
The energy conservation of our system, represented as  $H_{\mathrm{net}}$  in (9), remains invariant over time. To understand this better, consider the following
$$
\begin{array}{l} \frac {\mathrm {d} H _ {\text {n e t}}}{\mathrm {d} t} = \sum_ {i = 1} ^ {n} \frac {\partial H _ {\text {n e t}}}{\partial q _ {i}} \dot {q} _ {i} + \sum_ {i = 1} ^ {n} \frac {\partial H _ {\text {n e t}}}{\partial p _ {i}} \dot {p} _ {i} \\ = \sum_ {i = 1} ^ {n} \frac {\partial H _ {\mathrm {n e t}}}{\partial q _ {i}} \frac {\partial H _ {\mathrm {n e t}}}{\partial p _ {i}} + \sum_ {i = 1} ^ {n} \frac {\partial H _ {\mathrm {n e t}}}{\partial p _ {i}} \left(- \frac {\partial H _ {\mathrm {n e t}}}{\partial q _ {i}}\right) \\ = 0. \\ \end{array}
$$
where the last equality follows from (10).
BIBO stability is inferred since, with an unbounded output, the energy conservation expressed by  $H_{\mathrm{net}}$  would be violated.
G Complete Algorithm Summary
We present the complete algorithm of HANG in Algorithm 1, which unfortunately had been delayed in its inclusion within the main paper due to space constraints.
While our work on graph neural flows presents promising advancements in enhancing adversarial robustness of GNNs using Hamiltonian-inspired neural ODEs, it is not without limitations. As we demonstrated in the paper, the notions of stability borrowed from dynamical systems, such as BIBO stability and Lyapunov stability, do not always guarantee adversarial robustness. Our finding that energy-conservative Hamiltonian graph flows improve robustness is only one facet of the broader landscape of potential stability measures. It is possible that other notions of stability, not covered in this work, could yield additional insights into adversarial robustness. Our current Hamiltonian graph neural flows do not explicitly account for quasi-periodic motions in the graph dynamics. The Kolmogorov-Arnold-Moser (KAM) theory, a foundational theory in Hamiltonian dynamics, is renowned for its analysis of persistence of quasi-periodic motions under small perturbations in Hamiltonian dynamical systems. While the energy-conserving nature of our Hamiltonian-inspired model inherently offers some level of robustness to perturbations, an explicit incorporation of KAM theory could potentially further improve the robustness, particularly in the face of quasi-periodic adversarial attacks. However, this is a complex task due to the high dimensionality of typical graph datasets and the intricacies involved in approximating quasi-periodic dynamics.
This research, centered on enhancing adversarial robustness in graph neural networks (GNNs), carries implications for various sectors, such as social media networks, sensor networks, and chemistry. By improving the resilience of GNNs, we can boost the reliability of AI-driven systems, contributing to greater efficiency, productivity, and cost-effectiveness. The shift towards automation may displace certain jobs, raising ethical concerns about income disparity and job security. Moreover, while our models enhance robustness, potential system failures can still occur, with impacts varying based on the application. Lastly, the robustness conferred might be exploited maliciously. Our work underscores the importance of diligent oversight, equitable technology implementation, and continuous innovation in the development of AI technologies.
Algorithm 1: Graph Node Embedding Learning with HANG
1 Initialization: Initialize the network modules including Hamiltonian function network  $H_{\mathrm{net}}$ , the raw node features compressor network FC, and the final classifier.
2 I. Training:
3 for Epoch 1 to  $N$  do
4 | 1) Perform the following to obtain the embedding  $\mathbf{q}_k(T)$  for each node  $k$ :
5 | Input:  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  with raw node features
Apply FC to compress raw features for each node and get  $\{(\mathbf{q}_k(0),\mathbf{p}_k(0))\}_{k = 1}^{|\mathcal{V}|}$ . Here, we divide the  $2r$  dimensions into two equal segments. The first half acts as "position" vectors  $\mathbf{q}_k(0) = (q_k^1,\dots,q_k^r)$ , while the second half acts as "momentum" vectors  $\mathbf{p}_k(0) = (p_k^1,\dots,p_k^r)$  that guide system evolution. For simplification, in code implementation, we may set the output feature dimension of the compressor FC as  $r$  and designate  $\mathbf{p}_k(0) = \mathbf{q}_k(0)$ .
7 2) To better express the evolution dynamics mathematically, concatenate and relabel the node features as:
8
9
10
11
12
13
14 II. Testing:
15 Load the model from the best validation epoch and perform Step I.1-4). to obtain the final feature embedding over the test split. Perform node classification.