NEURAL HAMILTONIAN FLOWS IN GRAPH NEURAL NETWORKS
Authors: Qiyu Kang
Kai Zhao
Yang Song
Sijie Wang
Wee Peng Tay
Venue: Submitted to ICLR 2023 (2023)

Abstract: Graph neural networks (GNNs) suffer from oversmoothing and oversquashing problems when node features are updated over too many layers. Embedding spaces can also vary significantly for different data types, leading to the need for different GNN model types. In this paper, we model the embedding of a node feature as a Hamiltonian flow over time. As in physics where Hamiltonian flow conserves the energy over time, its induced GNNs enable a more stable feature updating mechanism. Moreover, since the Hamiltonian flows are defined on a general symplectic manifold, this approach allows us to learn the underlying manifold of the graph in training, in contrast to most of the existing literature that assumes a fixed graph embedding manifold. We test Hamiltonian flows of different forms and demonstrate empirically that our approach achieves better node classification accuracy than popular state-of-the-art GNNs.

NODE EMBEDDING FROM NEURAL HAMILTONIAN OR- BITS IN GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In the graph node embedding problem, embedding spaces can vary significantly for different data types, leading to the need for different GNN model types. In this paper, we model the embedding update of node feature as Hamiltonian orbit over time. Since the Hamiltonian orbits generalize the hyperbolic exponential maps, this approach allows us learn the underlying manifold of the graph in training, in contrast to most of the existing literature that assumes a fixed graph embedding manifold. Our proposed node embedding strategy can automatically learn, without extensive tuning, the underlying geometry of any given graph dataset even if it has diverse geometries. We test Hamiltonian functions of different forms and verify the performance of our approach on two graph node embedding downstream tasks: node classification and link prediction. Numerical experiments demonstrate that our approach adapts better to different types of graph datasets than popular state-of-the-ar graph node embedding GNNs.
INTRODUCTION
Graph neural networks (GNNs) (Yue et al., 2019; Ashoor et al., 2020; Kipf & Welling, 2017b; Zhang al., 2022; Wu et al., 2021) have achieved good inference performance graph-structured data such as social media networks, citation networks, and molecular graphs in chemistry. Most existing GNNs embed graph nodes in Euclidean spaces without further consideration of the dataset graph geometry. For some graph structures like the tree-like graphs (Liu et al., 2019), the Euclidean space may not be a proper choice for the node embedding. Recently, hyperbolic GNNs (Chami al., 2019; Liu et al., 2019) propose to embed nodes into hyperbolic space instead of the conventional Euclidean space. has been shown that tree-like graphs can inferred more accurately by hyperbolic GNNs. Furthermore, works like Zhu et al. (2020b) have attempted to embed graph nodes in a mixture of the Fuclidean and hyperbolic spaces, where the intrinsic graph local geometry is attained from the mixing weight.
Embedding nodes a hyperbolic space is achieved through the exponential map (Chami et al., 2019), which is essentially a geodesic curve on the hyperbolic manifold as the projected curve of the cogeodesic orbits on the manifold's 's cotangent bundle (Lee, 2013; Klingenberg, 2011). 1). In our work, we propose to embed the nodes, via more general Hamiltonian orbits, into a general manifold, which generalizes the hyperbolic embedding space, i.e., a strongly constrained Riemannian manifold of constant sectional curvature equal to 11
From the physics perspective, the cotangent bundles are the natural phase spaces in classical mechan- ies (De León & Rodrigues, 201 where the physical system evolves according the basic laws of physics modeled as differential equations on the phase spaces. In this paper, we propose new GNN paradigm based on Hamiltonian mechanics (Goldstein al., 2001) with flexible Hamiltonian functions. Our objective is to design new node embedding strategy that can automatically learn, without extensive tuning, the underlying geometry of any given graph dataset even if it has diverse geometries. We enable the node features to evolve on the manifold under the influence of neighbors. The learnable Hamiltonian function on the manifold guides the node embedding evolution to follow learnable law analogous basic physical laws.
Main contributions. Our main contributions are summarized as follows: 1. We take the graph discretization of underlying manifold and enable node embedding through a learnable Hamiltonian orbit associated with the Hamiltonian scalar function on its cotangent bundle.

---

2. Our node embedding strategy automatically learn, without extensive tuning, the underlying geometry of any given graph dataset even if it has diverse geometries. We empirically demonstrate its ability by testing on two graph node embedding downstream tasks: node classification and link prediction.
3. From empirical experiments, we observe that the oversmoothing problem of GNNs can be mitigated if the node features evolve through Hamiltonian orbits. By the conservative nature of the Hamiltonian equations, our model enables stable training and inference process while updating the node features over time and layers.
2 RELATED WORK
While our paper related to Hamiltonian neural networks in the literature, we are the first, to our best knowledge, to model graph-structured data with Hamiltonian equations. In what follows, we briefly review Hamiltonian neural networks, Riemannian manifold GNNs, and physics-inspired GNNs. Hamiltonian neural networks. Among these physics-inspirdd deep learning approaches, Hamilto- nian equations have been applied to conserve an energy-like quantity when training neural networks. The papers Greydanus et al. (2019); Zhong et al. (2020); Chen et al. (2021) train neural network to infer the Hamiltonian dynamics of a physical system, where the Hamiltonian equations are solved using neural ODE solvers. The work Haber & Ruthotto (2017) builds Hamiltonian-ins inspiredd neural ODE to stabilize the gradients so as to avoid vanishing and exploding gradients. The paper Huang al. (2022) further studies the adversarial robustness of Hamiltonian ODE, In this paper, we focus on applying Hamiltonian equations to graph neural networks, which has not been investigated in the mention works.
Riemannian manifold GNNs. Most GNNs in the literature Yue et al. (2019); Ashoor et al. (2020); Kipf & Welling (2017b); Zhang et al. (2022); Wu et al. (2021) embed graph nodes in Euclidean spaces. In what follows, we simply call them (vanilla) GNNs. They perform well on some datasets like the Cora dataset McCallum et al. (2004) whose o-hyperbolicity Chami et al. (2019) is high. When dealing with the datasets whose 6-hyperbolicity are low (hence embedding should more appropriately be in a hyperbolic space) such as the Disease Chami et al. (2019) and Airport Chami et al. (2019) 9) datasets, those GNNs suffer from improper node embedding. To better handle hierarchical graph data, (Liu et al., 2019; Chami et al., 2019; Zhang et al., b021 Zhu et al., 2020b) propose to embed nodes into a hyperbolic space, thus yielding hyperbolic GNNs. Moreover, Zhu et al. (2020b) proposes mixture of embeddings from Euclidean and hyperbolic spaces. This mixing operation relaxes the strong space assumption of using only one type of space for a dataset. In this paper. we embed nodes into a general learnable manifold via the Hamiltonian orbit on its symplectic cotangent bundle. This allows our model to flexibly adapt to the inherent geometry of the dataset.
Graph Neural diffusion: Neural Partial Differential Equations (PDEs) have been applied to graph- structured data (Chamberlain et al., 20222022b22 b;a; Song et al., 2022), where different diffusion schemes are assumed when performing message passing on graphs. To be more specific, the heat diffusion model is assumed in (Chamberlain et al.. 2021b) and the Beltrami diffusion model is assumed in (Chamberlain et al., 2021a; Song et al., 2022). (Rusch al., 2022) models the nodes in the graph as coupled oscillators, i.e., a second-order ODE. While above mentioned graph neural diffusion schemes and our model all use ODEs, there is a fundamental difference between our model and graph neural flows. In the graph PDEs, they wrap the message passing function, e.g., constant aggregation function like the one in GCN, and attention- -base aggregation function like the one in GAT, into an ODE function. In contrast, our model treats the node embedding process and node aggregation process as two independent processes, where we use the ODE function only to learn a suitable node embedding space which is then followed by a node aggregation step. To sum up, our ODE is actually a node embedding layer taking node features as the input whereas graph PDEs are node aggregation layers taking node features as well as graph udjacency matrix as the input.
Notations:
We use the Einstein summation convention (Lee, 2013) for expressions with tensor indices.
MOTIVATIONS AND PRELIMINARIES
In this section, we briefly review the concepts of the geodesic curve on a Riemannian manifold from the principle of stationary action in the form of Lagrange's equations. We then further generalize the geodesic curve to the Hamiltonian orbit associated with an energy function H, which is a conserved quantity along the orbit. We first summarize the motivation of our work follows.
Motivation I: from the hyperbolic exponential map to Riemannian geodesic. The geodesic curve gives rise to the exponential map that maps points from the tangent space to the manifold and has been
2

---

utilized in (Chami et al., 2019) enable graph node embedding in special Riemannian manifold known as the hyperbolic space. From this perspective, by using the geodesic curve, we generalize the graph node embedding to an arbitrary (pseudo- Minudd-nRimanniaan manifold with learnable local geometry using Lagrange's 's equations.
Motivation I:: from Riemannian geodesic to Hamiltonian orbit. Despite the above conceptual generalization for node embedding using geodesic curves, the specific curve formulation involving minimization of curve length may result loss of generality for node feature evolution along the curve, We thus further generalize the geodesic curve to the Hamiltonian orbit associated with an energy function H that conserved along the orbit. In Section 4, we propose graph node embedding without an explicit metric by using Hamiltonian orbits with learnable energy functions H.
3.1 MANIFOLD AND RIEMANNIAN METRIC
Manifold and local chart representation. On a d-dimensional manifold M, for each point on M, there exists a triple {g. U. }), called a chart, such that U is an open neighborhood of the point in V is homeomorphism, which gives us a coordinate M, V is an open subset of Rdd, and q U - representation for local area in M.
Tangent and cotangent vector spaces. For any point on M (we identify each point covered by a local chart on M by its representation q). may assign two vector spaces named the tangent vector space T.M and cotangent vector space T M. The vectors from the tangent and cotangent spaces can be interpreted as representing a velocity and generalized momentum of motion in classical mechanics, respectively.
Riemannian metric. A Riemannian manifold is manifold M equipped with Riemannian metric where we assign to any point E M and pair of vectors u, v E M an inner product (u, u) 9(g)- This assignment is assumed be smooth with respect to the base point q M. The length of tangent vector E T T.M is then defined as
| == (u, u) 1/2 g(g))
Local coordinates representation: In local coordinates with q (q'....,,4') (²)
M, u
(u')...,,'') ) E TyM and = (o (u'...., vd) T,M, the Riemannian metric g = g(9) is
real symmetric positive definite matrix and the inner product above given by
(u, g)1) == 9.y (d)
Secudo-Riemannin metric: We may generalize the Riemannian metric to metric tensor that only requires non-degenerate condition (Lee, 2018) instead of the stringent positive definiteness condition in the inner product. One example of Sheudo-Riemanee manifold is the Lorentzian manifold, which is important applications of general relativity.
3.2 GRODESIC CURVES AND EXPONENTIAL MAPS
Length and energy of curve. Let : [a, b] M be smooth curve. We define the following:
((q) = ["" (1(()) dt.
length of the curve:
1 E(g) = 2 / I1((0)) dt.
energy of the curve:
Geodesic curves. On Riemannian manifold, geodesic curves are defined as curves that have minimal length as given by (3) and with two fixed endpoints q(a) and q(b). However, computations based on minimizing the length obtain the curves are difficult. It turns out that the minimizers of E(q) also minimize E(g) (Malham, 2016). Consequently, the geodesic curve formulation may be obtained by minimizing the energy of smooth curve on M.
Principle of stationary action and Laler-Lagrange equation. The Lagrangian function L(q(I), (()) minimizes the following functional (in physies, the functional is known as an action)
S(g) = (la(t).((t)de)
We abuse notations in denoting the chart coordinate map as and the curve as g(t). It will be clear from the context which is heing referred
3

---

with two fixed endpoints at t and t only if the following Euler Lagrange equation is satisfied:
OL 0q' ((()) di aij
d AL (g(1), (()) 0.
(6)
Geodesic equation for geodesic curves. The Euler Lagrange equation derived from minimizing the energy (4) with local coordinates representation,
2 gh "
g")
expressed as the following ordinary differential equations called the geodesic equation:
G' + + O,
(8)
for all I,..... d, where the Christoffel symbols T 2 denotes the inverse matrix of the matrix [gg] The solutions to the geodesic equation (8) give us the geodesic curves.
0gr 80g
+
Baus dy
0gik Oae
and [g]]
Exponential map. Given the geodesic curves, at each point I E M, for velocity vector v T, the exponential map is defined to obtain the point on M reached by the unique geodesic that passes through x with velocity at time 1 (Lee, 20.8). Formally, we have
(x) = (11)
where (() is the curve given by the geodesic equation (8) with initial conditions q(0) and ((0) = vt.
With regards Motivation I, note that (Chami et al., 2019) considers graph node embedding over a homogeneous negative-curvature Riemannian manifold called hyperboloid manifold. In contrast, we generalize the embedding of nodes to an arbitrary pseudo-Riemande manifold through the geodesic equation (8) with learnable metric g that derives the local graph geometry from the nodes and their neighbors.
3.3 FROM GEODESICS TO GENERAL HAMILTONIAN ORBITS
The geodesic curves and the derived exponential map essentially come from (5) with L in (7) specified from the curve energy (4). However, the curves derived from this specific action may potentially sacrifice efficacy for the graph node embedding task since we do not know what is reasonable action formulation that guides the evolution of the node feature in this task. Therefore, we follow the principle of stationary action but consider learnable action that is more flexible than the length or energy of the curve. To better model the conserved quantity during the feature evolution, we reformulate the Lagrange equation to the Hamilton equation. This is our Motivation II. Hamiltonian function and equation. The Hamiltonian orbit (g(t), ((t)) is given by the following Hamiltonian equation with Hamiltonian function H:
0H
OH tq
p::
(10)
Op:"
where q is the local chart coordinate on the manifold while can be interpreted as a vector of generalized momenta in the cotangent vector space. In classical mechanics, the 2d-dimensional pair (p, 9) is called phase space coordinates that fully specify the state of dynamic system. Later, we consider the node feature evolution following the trajectory specified by the phase space coordinates. Hamiltonian function vs. Lagrangian function. The Hamiltonian function can be taken as the Legendre transform of the Lagrangian function:
0L
H(g. p) = p.i' - I(g. i) with ş = i(p) such that
(11)
oq If H is restricted to strictly convex functions, the Hamiltonian formalism is equivalent to Lagrangian formalism (De León & Rodrigues, 2011).
Geodesic equation reformulated as Hamiltonian function. If H is set as
H(g.v) 1 (1)pppp
(12)
4

---

where [g denotes the inverse matrix of the matrix [ghy], have the following Hamiltonian function:
='= g P: 1==-20.9h PjPu.
(13)
The Hamiltonian orbit (p(1), (()), as the solution of (13), gives us again the geodesic curves p(t) on the manifold M if we only look the first d-dimensional coordinates.
Theorem (Conservation of energy (Da Silva & Da Salva, 2008)). ((p(f),((t) is constant along the Hamiltonian orbil as solutions of (10).
In physics, H typically represents the total energy of the system, and Theorem indicates the time-evolution the system follows the law of conservation of energy.
4 PROPOSED FRAMEWORK
We consider an undirected graph G - (V,E) consisting of a finite set V of vertices, together with subset & C Vx x V of edges. Since the input node features most datasets are sparse, fully connected (FC) layer is first applied to compress the raw input node features. Let "q be the d-dimensional compressed node feature for node n after the FC layer. However, empirical experiments (see "MLP"" results in Section 5.1) indicate that for the graph node embedding task, such simple raw compressing without any consideration of the graph topology does not render good embedding. Further graph neural network architecture is thus required update the node embedding.
consider the node features n"g)nev to be located local chart an embedding manifold M and take the node features as the chart coordinate representations for points on the manifold. In Motivations and II in Section 3, we have provided the rationale for generalizing graph node embedding from the hyperbolic exponential map to the Riemannian geodesic, and further to the Hamiltonian orbit. To enforce the graph node feature update on the manifold with well-adapted leamable local geometry, make use of the concepts from Section 3.
Hamiltonian orbit evolution
concatenate
Hamiltonian equation eq(15)
p
("q Pp)
4
Hamiltonian Layer
Hamiltonian orbit evolution
Aggregation
FC
Hamiltonian Layer
Hamiltonian Layer
Hamiltonian Layer 1.
Figure 1: HamGNN architecture: in each layer, each node is assigned a learnable "momentum" vector "p (cf. (14)) at time t = 0, which initializes the evolution of the node feature. The node features evolve on manifold following (15) to ("'(T") "p(Z") the time f == T'. We only take "q(T") the embedding and input to the next layer. After L layers, we take (1) (T') as the final node embedding.
4.1 MODEL ARCHITECTURE
Node feature evolution along Hamiltonian orbits a Hamiltonian layer. As introduced in Section 3.3, the 2d-dimensional phase space coordinates (p. q) fully specify system's 's state. Conse- quently, for the node feature as a point on the manifold M, we associate to each point "q a learnable momentum vector "p external force to enable the node feature evolve along Hamiltonian orbits on the manifold. More specifically, we set
"p = Quet ("
(14)
We put the node index the left of the variable distinguish from the manifold dimension index.

---

where Quet is instantiated by an FC layer. We consider learnable Hamiltonian function Hnet 3 that specifies the node feature evolution trajectory in the phase space by the Hamiltonian equation
net
pi
(15)
0p:
Oq'
with learnable Hamiltonian energy function
Huet: (g.p) + R.
(16)
The node features are updated along the Hamiltonian orbits which are curves starting from each node ("q, "p) at f = 0. In other words, they are the solution of (15) with the initial conditions (")(0), (p)) = ("q, "p) at t = 0. The solution of (15) on the phase space for each node n E V at time T given by the differential equation solver (Chen et al., 2018a), and denoted by ("q(T)." " p(T)). The canonical projection x (")(()), (TT))(""(TT))
"T(T) is taken obtaín the node feature positions on the manifold at time T. The aforementioned operations are performed within one layer, and we call it the Hamiltonian layer. Neighborhood Aggregation. After the node features update along the Hamiltonian orbits, we perform neighborhood aggregation on the features (T) where indicates the E-th layer. Let (n)) {m : (n,, m)) E E denote the set of neighbors of node n E V. We only perform simple yet efficient aggregation for node n as follows:
ng(ett)= "T((0(T) + [W())
1
Σ
(T(O(T) (T).
(17)
mEN(m)
Layer stacking for local geometry learning, We stack up multiple Hamiltonian layers with neigh- borhood aggregation in between them. We first give intuitive explanation for the case Het is set as (12) where a learnable metric Jnet for the manifold is involved (see Section 4.2.1 for more details) and the features are evolved following the geodesic curves with minimal length (see Section 3). Within each Hamiltonian layer, the metric Jnet that is instantiated by smooth FC layer only depends on the local node position on a pseudo Riemannian pseudo-Riemannian manifold that varies from point to point. Note that with layer stacking, these features contain information aggregated from their neighbors. The metric Snet. therefore, learns from the graph topology, and each node is embedded with a local geometry that depends its neighbors. In contrast, (Chami et al., 2019) considers graph node embedding using geodesic curves over a homogeneous negative-curvature curvature hyperboloid manifold without adjustment of the local geometry.
At the beginning of Section 4, we have assumed the node features ("4), to be located in a local chart of preliminary embedding manifold M. The basic philosophy is that the embedding manifold evolves with metric structure that adapts successively with neighborhood aggregation along multiple layers, whereas each node's features evolve the most appropriate position as the embedding on the manifold along the curves. For general learnable Hnet, the Hamiltonian orbit that starts from one node has aggregated information from its neighbors, which guides the learning of the curve that the node will be evolved along on the manifold. Therefore, each node is embedded into a manifold with good adaptation to the underlying geometry of any given graph dataset even if has diverse geometries.
Conservation of Hnec. From Theorem 1, the feature updating through the orbit indicates that the Hnet is conserved along the curve.
Model summary. Our model is called HamGNN as we use Hamiltonian orbits for node feature updating on the manifold. We summarize the HamGNN model architecture in Fig. 1. The forms of the Hamiltonian function Hner are given in Section 4.2.
4.2 DIFFERENT HAMILTONIAN ORBITS
We next propose different forms for Hnet from which the corresponding Hamiltonian orbit and its variations are obtained in our GNN model. The node features are updated along the Hamiltonian orbits, which are curves starting from each node ("q, "p) at =
4.2.1 LEARNABLE METRIC Gnet
In this subsection, consider node embedding onto a where a learnable metric fnet for the manifold involved. Within each Hamiltonian layer, the
and set Hnet as (12)
'The subseript net of function Fres indicates that the function F is parameterized by neural network.

---

metric Inet instantiated by smooth FC layer only depends the local node position on the pseudo- Riemannian manifold that varies from point to point. The output of Juet at position q represents the inverse metric local representation [g However, from (12), the space complexity is order d due to the partial derivative of g's output being a d x matrix. We therefore only consider diagonal metrics to mitigate the space complexity. More specifically, now define
gnet (q) = diag(
1.....1 @ Nnet (q))
(18)
where uuet Rd Rd consists of non-linear trainable layers and @ denotes element- multi- plication. To ensure non-degeneracy of the metrie, the output of net is set to be away from 0 with the final activation function of it being strictly positive. The vector [-l,..., -1, 1, .,....,, ..1] controls the signature (r, s) of the metric 9 with + 8 d, where and respectively. The signature of the metric is set to be a hyperparameter. According to (13), we have
the number of -1s and 1s,
1 p' = Snee, P4 2 -.
(19)
Intuitively, the node features evolve through the "shortest" curves on the manifold. The exponential map used in hyperbolic GNNs (Chami et al., 2019) is essentially the geodesic curve on a hyperbolic manifold with an explicit formulation due to the hyperbolic assumption. We do not enforce any assumption here and let the model learn the embedding geometry. 4.2.2 LEARNABLE Het
Different from Section 4.2.1 where H/ is set as (12) with SecudoRRiemanniar metric, we choose a more flexible H instantiated by an FC layer and consider the Hamiltonian function:
He
HH
Gº
ps = 0q
(20)
Op:
4.2.3 LEARNABLE CONVEX Hnet
As discussed in Section 3.3, if Hnet is restricted to strictly convex functions, the Hamiltonian formalism can degenerate to a Lagrangian formalism through the Legendre transformation (11). We take the following restricted Hamiltonian equation
Hent Op:
HH
P
S. Heet is convex,
(21)
0q'
where a stationary action in (5) is achieved. To guarantee that Hnet is convex, we follow the work in Amos et al. (2017) to set non-negative layer weights from the second layer in Heets and all activation functions in Het to be convex and non-decreasing. This network design is shown to be able to approximate any convex functions in Chen al. (2018b).
4.2.4 LEARNABLE Het WITH RELAXATION
Different from Section 4.2.2, enforce additional system biases along the curve follows:
HH
ÖH
Op.- D =
+ fnee(g). (q).
(22)
dq²
Instead of keeping the energy during the feature update along the Hamiltonian orbit, we now also include an additional energy term during the node feature update.
4.2.5 LEARNABLE Hee WITH A FLEXIBLE SYMPLECTIC FORM
Hamiltonian equations have coordinate- free representation using the symplectic 2-form. We present a self-contained review of differential geometry that is related to symplectic 2-form and Hamiltonian equations in Appendix A. The chart coordinate representation (q.p) may not be the Darboux coordi- nate system for the symplectic 2-rom.. Even if our learnable Hnet may be able to learn the energy representation under the chosen chart coordinate system, we consider a learnable symplectic 2-form to act in concert with Huet. More specifically, following (Chen et al., 2021), we have g = d'''
where fnet M + IRd is the output's i-th component of the neural network parameterized function from which the Hamiltonian orbit is given by (Chen et al., 2021) as follows:
(g'.p")
W- (q. Wieee
(23)
where the skew semmetymmttric 2d x 2d matrix W, whose elements are written in terms of j ni is given in (50) due to space constraints.
7

---

5 EXPERIMENTS
In this section, we implement the proposed HamGNNs with different settings as shown in Section 4.2 and Appendix B. We select datasets with various geometry including the three citation networks Cora (McCallum et al., 2004), Citeseer (Sen et al., 2008). Pubmed (Namata et al., 2012), and two low hyperbolicity datasets (Chami et al., 201 9), named Disease and Airport as the benchmark datasets. To fairly compare the performance of the proposed HamGNN, we sclect several popular GNN models as the baseline, including the Euclidean GNNs: GCN (Kipf & Welling, 2017a), GAT (Veličković et al., 2018), SAGE (Hamilton et al., 2017), and SGC (Wu et al., 2019); the hyperbolic GNNs (Chami et al., 2019; Liu et al., 2019): HGNN, HGCN, and HGAT, and also GIL. (Zhu et al., 2020b) which learns a weighting between Euclidean and hyperbolic space embeddings; the Graph Neural diffusion GNNs; GRAND (Chamberlain et al., 2021b), GraphCON Rusch et al. (2022) and LGCN (Zhang et al., 2021b). The MLP baseline does not utilize the graph topology information. To further demonstrate the advantage of HamGNN, we also include one vanilla ODE system, whose formulation is given in Appendix D.I without the crucial Hamiltonian layer. Due to space constraints, we refer the readers to Appendix C for the datasets and implementation details. We stress at the outset that we do not aim to outperform all the baselines or other general-purpose GNN models on specific datasets. Instead, our objective is to design a new node embedding strategy that can automatically learn, without extensive tuning, the underlying geometry of any given graph dataset even if it has diverse geometries. To demonstrate that HamGNN can automatically adapt well, we include two graph node embedding downstream tasks, including node classification and link prediction. Due to space constraints, we refer the readers Appendix C for the datasets and implementation details.
Method
Disease
Airport
Pubased
Citeseer
Cora
MLP
50.00=00000
1.77
1
18.10:11.87
1..15:::.15
HNNN (Ganca al., 2018)
36,40=6.32
80.49:1.54
71.00++0.47
55.13++-004
58.03-+0.35
GCN (Kipf & Welling, 20176)
x1.10=1.33
82.25 56
77.33-00777
71.78::0:34
80.29:2..09
GAT (Nelikkowi et al.. 2018)
87.01 2.77 2.77
02.991:0.83
77.58:10.81
68.07:11.311
80.33:10:61
SAGE (Hamilion al.,
81.60=7.66
81.97:0
77.6310115
65.90:12::22
74, 74.50:20.8
SGC (G' et al., 2019)
82.78:0.95 0.93
81,40::2:11
76.83:11.11
70.88:11.322
81.98:11.1 1.71
HGNN (Liu et 20191
90.51 5.770
84.54-0772
76666+1138
69.43--0099
79.53-0095
HGCN (Chomi et al., 2019)
80.87=1.13
05.351065
76. 382:088
65.80:2 25.80:2::044
78 70:20 96
HGAT (Zhang al., 20121a
3.36
87,50
78.00400.50
60.20:
80,88=0.755
GZhu 20100)
90,78:0045
91.521174
77.66-0.57
71.10
12,10:11.12
Vanilla ODE (53)
18.85
90.34:00.77
73200433:31
1.226
68.38 1,18
GRAND (Chamberhin et al. 2021b)
3.35
60.02:::555
79.30:10.51
71.76:10.79
0,.80:10.92
GraphCON Rusch (2022)
87.50=24.06
6861
78 78:00100.97
71.33:083
82, 122-99:2.00 08
L,GCN (7hing al., 2021b)
88.47=1.8
88.2220000
77.35
68.08:1 1.98
80.60:20.22
HamGNN
91.26 1.40
95. 05.50:::.44
78:08-00:44
70. 11:2::0886
82, 0.:2::::880
HamGNN (20)
88.74=1.17
95:11:1:.:0
18.18:10..4
71.48:1..41
1.27
IhmONN (21)
$4.57=5.76
93.28:1: 10
78.83:10046
72.00 72.00:1:073
81, 31.84:20.888
HamONN (22)
87.48 87.48_5.990 5.90
95.461:0.68
78. 30:00:0000 34
72.38:10.85
81. 5.92
HamGNN (23)
$8.35 1.51
93.66
78 78.60:1:0322
71.52:1.1.41
81. $1.24:10.599
ThmGNN
91.18=0.99
05.80
77.90:1:.49
69.18:1:1.63
80 30.90:10.35
HumONN (57)
21.50=2.007
05.19:00113
78 78.062:064
60.10:21.09
80. 60.10:21.56
Table 1: Node classification accuracy($)) The best and the second-best result for each criterion are highlighted in bold and underlined and italic respectively.
5.1 PERFORMANCE RESULTS AND ABLATION STUDIES
Node classification. The node classification performance the benchmark datasets using the base- line models and the proposed HamGNNs with different HnetS is shown in Table 1. We observe that HamGNN adapts well to all datasets with various geometry. As an illustration, HamGNN (19) achieves the third best performance on the Cora dataset, which is Euclidean in nature. According to (Chami et al., 2019 Liu et al., 2019), the Airport dataset has tree-like structure that cannot be properly embedded into Euclidean spaces. We observe that HamGNN (19) has a well-adapted performance that beats all the baselines on this dataset due to the learnable embedding manifold without many constraints. As discussed in Section 4.1, HamGNN with the learnable metric struc- ture Jnet or other general Hnet learns the graph local geometry successively with neighborhood aggregation. The superior performance of HamGNNs on all the datasets over the other baselines has demonstrated the advantage of including Hamiltonian orbits on the manifold to embed the node features. Furthermore, the comparison between HamGNNs with the GNN with vanilla ODE in (53) without any Hamiltonian mechanism indicates the indispensability of the Hamiltonian layer.
Link prediction. In Table 2, we report the averaged ROC for the link prediction task. We observe
8

---

HamGNN adapts well to all datasets and is the best performer the Airport, Citeseer, and Cora.
Methad
Disease
Aisport
Pubased
Cloeseer
Cora
MLP
83.37:55.04 81.37:88778
87.04:1:..6
88.69-11.599
59.65:1:1000
9.00710.556
INNN (Ganea et 2018)
86,06:2:.08
94.69:1:0.25
59.83:1:0399
92.83:10.66
GCN Walling. 2017a)
60.38:22.51
90:97::0.65
91.37:1:009
93:20:1::388
92. 07.89:20.77
GAT (Velilković al.. 2018)
62.03:t:.55
91,05 1.083
91.03:1:0.67
91.33:1:005
90.36:20.50
SACE (Tamilton al., 2017)
68. 0::02:20:43
91 5s
93:61::0:26
91.37:1:088
92,96:20.0 40
SGC (Wa al. 2019)
59.83-24.01
89.72::0.2
92. 02.16:0::13
94.78::0.77
93.1520.22
HGNN (1.n at., 2019)
60, 10.200:11114
02.46:20.20
93.09:0000
90.35:00.57
92.05=0.335
IIGN (Chumi al. 2019)
28.09:22.79
94:281::20
06.29::101
91:60:1::14
94,10:00005
HGAT (Zhang et al, 202
76.3723.441
94.6420.51
96.86:0003
93.45:0.25
94 0439
(Zlu et 201000
99.97:00.08
07.9242-644
91.224:325
05.99
27.78+2.31
HamLiNN
99.73-0026
99.99-00011
92.1540300
99.99-10.00
98.20-
Table 2:
Link prediction (OC(%) The best and the second-best result for each criterion are high- lighted in bold and underlined and italic respectively.
Comparison between different H We compare HamGNNs with different H as elaborated in Section 4.2 on the node classification task. In Section 3.3, we argue that the geodesic curves derived from the action of curve length (3) may potentially sacrifice efficacy for the graph node embedding task since we do not know what is a reasonable action formulation that guides the evolution of the node feature in this task. We therefore also include a more flexible Het in (20) with other variations, e.g., (21) to (23). From Table 1, we observe that the basic (20) has good adaptations for node embedding of various datasets. Flexibility (20) helps the node classification task slightly for the Citeseer dataset, but the improvement is not significant. Other variants of Hnet also present good adaptations for all the datasets. This observation indicates that good geometry adaptations may come from the HamGNN model architecture with the Hamiltonian orbit evolution. This is further verified by the node classification performance from vanilla ODE in (53), which is designed without the philosophy of Hamiltonian mechanics. Vanilla ODE (53) does not have well-adapted node embedding performance for datasets with various geometries.
We then compare the HamGNN using (2 and (23). The difference between those two settings is that the symplectic form 2² in (20) is set to he the special Poincaré 2-form while in (23), the symplectic form is learnable one. We however observe that the two HamGNNs achieve similar performance and the more flexible symplectic form does not improve the model performance. This may be because of the fundamental Darboux theorem Theorem in symplectic geometry which states that we can always find Darboux coordinate system to give any symplectic form the Poincaré 2-form. The feature compressing FC may have the network capacity to approximate the Darboux coordinate map while the flexible learnable Hnet also has the network capacity to get the energy representation under the chosen chart coordinate system.
Dataset
Models
Layers
5 layers
10 layers
20 layers
GCN
80,29
69,87 L.12
26.50
4..42
Cora
HGCN
0.99
38.13=6.20
31 31.50-00000
2623--977
HamKiNN (21)
$1.84 0.88
81.08=0.16
81.40-10.44
$0.58-10300
Table 3: Node classification accuracy %) when increasing the number of layers on the Cora dataset.
5.2 OBSERVATION OF RESILIENCE TO -MO-TSMOOTHNNG
As a side benefit of HamGNN, we observe from Table 3 that more Hamiltonian layers are stacked, HamGNN still retains its node classification ability, while the classification accuracies of other GNNs decrease significantly. This is known as the over-smoothing problem (Chen et al., 2020a) in GNNs. One reason for HamGNN's stability against over-smoothing is that the node feature energy indicated by the Hamiltonian Huet is fixed during the feature update along the Hamiltonian orbit. More A Ablation Studies and Experiments. See Appendix D where we include experimental results on heterophilic graph datasets, more empirical analysis of oversmoothness, etc.
6 CONCLUSION
In this paper, we have designed new node embedding strategy from Hamiltonian orbits that can automatically learn, without extensive tuning, the underlying geometry of any given graph dataset even if it has diverse geometries. We demonstrate empirically that our approach adapts better than popular state-of-the-art graph node embedding GNNs to different types of graph datasets via two graph node embedding downstream tasks, including node classification and link prediction. From experiments, we observe that the over-shoothing problem can be mitigated if the node features evolve through the Hamiltonian orbits.

---

REFERENCES
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, pp. 146-155, PMLR, 2017.
Haitham Ashoor, Xiaowen Chen, Wojciech Rosikiewicz, Jiahui Wang, Albert Cheng, Ping Wang. Yijun Ruan, and Sheng Li. Graph embedding and unsupervised learning predict genomic sub- compartments from hic chromatin interaction data. Nat. Commun.. 11, 2020.
Wendong Bi. Lun Du, Qiang Fu, Yanlin Wang. Shi Han, and Dongmei Zhang. Make heterophily graphs better fit gnn: A graph rewiring approach. arXiv preprint arXiv:2209.08264 2022.
Benjamin Paul Chamberlain, James Rowbottom, Davide Eynard, Francesco Di Giovanni, Dong Xiaowen, and Michael M Bronstein. Beltrami flow and neural diffusion on graphs. Proc. Advances Neural Inf. Process. Syst., 2021a.
Benjamin Paul Chamberlain, James Rowbottom, Maria Goronova, Stefan Webb, Emanuele Rossi, and Michael Bronstein. Grand: Graph neural diffusion. In Proc. Int. Conf. Mach. Learn., 2021b.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Proc. Advances Neural Inf. Process. Syst., 2019.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3438-3445, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pp. 1735-1735. PMLR, 2020b.
Ricky T. Q. Chen. torchdiffeq.
torchdiffeq, 2018.
URL htttps://ggttuu..co//rtte
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proc. Advances Neural Inf. Process. Syst., 2018a.
Yize Chen, Yuanyuan Shi, and Baosen Zhang. Optimal control via neural networks: A convex approach. arXiv preprint arXiv: 11835 2018b.
Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Neural symplectic form: Learning hamil-
tonian equations on general coordinate systems. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint XXiv:2006.07988. 2020.
Ana Cannas Da Silva and A Cannas Da Salva. Lectures on symplectic geometry, volume 3575. Springer, 2008.
Manuel De León and Paulo R Rodrigues. Generalized Classical Mechanics and Field Theory: a geo- metrical approach of Lagrangian and Hamiltonian formalisms involving higher order derivatives. Elsevier, 2011.
Francesco Di Giovanni, James Rowbottom, Benjamin Chamberlain, Thomas Markovich, and Michael M Bronstein. Graph neural networks as gradient flows. arXiv preprint 100991, 2022.
Marián Fecko. Differential geometry and Lie groups for physicists. Cambridge university press, 2006.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Proc. Advances Neural Inf. Process. Syst., 2018.
Herbert Goldstein, Charles Poole, and John Safko. Classical Mechanics. Addison Wesley, edition, 2001.
10

---

Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Proc. Advances Neural Inf. Process. Syst. 2019.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (I):1-23, December 2017.
William Hamilton, Rex Ying. and Jure Leskovee. Inductive representation learning on large graphs. In Proc. Advances Neural Inf. Process. Syst., 2017.
Philip Hartman. Ordinary differential equations. SIAM, 2002.
Yifei Huang, Yaodong Yu, Hongyang Zhang, Yi Ma, and Yuan Yao. Adversarial robustness of stabilized neural ode might be from obfuscated gradients. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova (eds.), Proc. Mathematical and Scientific Machine Learning Conference, pp. 497-515, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proc. Int. Conf. Learn. Representations, pp. 1-14, 2017a.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proc. Int. Conf. Learn. Representations, 2017b.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In Proc. Int. Conf. Learning Representations, 2019.
Wilhelm PA Klingenberg. Riemannian geometry, volume 1. Walter de Gruyter, 2011.
John M Lee. Introduction to smooth manifolds. Springer, London, 2013.
John M Lee. Introduction to Riemannian manifolds. Springer, London, 2018.
Qi Liu, Maximilian Nickel, and Douwe Kicla. Hyperbolic graph neural networks. Proc. Advances Neural Inf. Process. Syst., 2019.
Simon JA Malham. An introduction to lagrangian and hamiltonian mechanics, 2016.
Andrew McCallum, Kamal Nigam, Jason D. M. Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163, 2004.
Galileo Mark Namata, Ben London, Lise Getoor, and Bert Huang. Query-driven active surveying for collective classification. In Workshop Mining and Learn. with Graphs, 2012.
Honghin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05281 2020.
Walter Rudin al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1976.
Konstantin Rusch, Benjamin Paul Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael M. Bronstein. Graph-coupled oscillator networks. In Proc. Int. Conf. Mach. Learn., 2022.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Rhiasi-RRad Collective classification network data. AI Magazine, 29(3):93, Sep. 2008
Yang Song, Qiyu Kang, Sijie Wang, Kai Zhao, and Wee Peng Tay. On the robustness graph neural diffusion topology perturbations. In Advances in Neural Information Processing Systems (NeurIPS), New Orleans, USA, Nov. 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proc. Int. Conf. Learn. Representations, pp. 1-12, 2018
Felix Wu. Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplify- ing graph convolutional networks. In Proc. Int. Conf. Mach. Learn., 2019.
11

---

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. 32(1):4 24, 2021.
Xiang Yue, Zhen Wang, Jingong Huang, Srinivasan Parthasarathy, Soheil Moosavinasab, Yungui Huang, Simon M Lin, Wen Zhang, Ping Zhang, and Huan Sun. Graph embedding on biomedical networks: methods, applications and evaluations. Bioinformatics, (((4):224111551) 2019.
Yiding Zhang, Xiao Wang, Chuan Shi, Xungiang Jiang, and Yanfang Fanny Ye. Hyperbolic graph attention network. IEEE Tran. Big Data, 63(1), 2021a.
Yiding Zhang, Xiao Wang, Chuan Shi, Nian Liu, and Guojie Song. Lorentzian graph convolutional networks. In Proc. Web Conf., 2021b.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning graphs: A survey. 34(1):249-270, Jan 2022.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In Proc. Int. Conf. Learn. Representations, 2020.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-78044 2020a.
Shichao Zhu, Shirui Pan, Chuan Zhou, Jia Wu, Yanan Cao, and Bin Wang. Graph geometry interaction learning. In Proc. Advances Neural Inf. Process. Syst., 2020b.
A DIFFERENTIAL GEOMETRY AND HAMILTONIAN SYSTEM
In this supplementary material, we review some concepts from differential geometry perspective. We hope that this overview makes the paper more accessible for readers from the graph learning community.
A.1 MANIFOLD, BUNDLES, AND FIELDS
Roughly speaking, a manifold is a topological space that locally looks like Euclidean space. More strictly speaking, a topological space (M, O), where O is the collection of open sets on space M, is called d-dimensional manifold if for every point æ E M, we can find an open neighborhood U E O for x and a coordinate map
q U - (U) C R
that is homeomorphism, where Rd is the d-dimensional Euclidean space with the standard topology. (U. q) called chart of the manifold, which gives us a numerical representation for a local area in M. In this work, only consider smooth manifolds (Lee, 2013) that any two overlapped charts are smoothly compatible. The set all smooth functions from M R is denoted as C (M). On top of the smooth manifolds, and the more general tensor bundles, From bundles, we can define the vector or covector fields and the more general tensor fields. In this work, mainly consider the manifold with the 2-forms that are special (0, 2) smooth tensor fields with antisymmetric constraints. More specifically, we mainly consider the symplectic form (Lee, 2013) with some other light shed on the metric tensor which is another type of (0, 2) smooth tensor field.
define other related manifolds like the tangent cotangent bundles
Definition (tangent vector and tangent space). Let I + M be a smooth curve through x E s.t. (0) = and is interval neighborhood of 0. The tangent vector is directional derivative operator at x along that is the linear map
Vayya C (M)
=R
f (f
We also call the directional derivative operator as "velocity" of n at 0 and denote it as (().. (More generally, the 'velocity" of y at is denoted at a((1) which is tangent vector point 71t) M from
12

---

T-M M
(0)
X TrM
M
Figure 2: Visualization of 1-dimensional manifold and its tangent bundle.
the curve reparametrization trick (Fecko, 2006)). Correspondingly the tangent space M at x is the vector space over R. with the underlying set
TzM = (Uy, y is smooth curve and (0) =x)
(24)
Note for f e C (M), using the chart (U, q) with U, have the local representation
0y (f)
o 7)(
(ʃ o oq00'00))
(q' (7) (s oo "
gtm))
where q' is the i-th component of q, is the function composition and (.) g(t) means evaluating (-) at q(x).. Therefore for local chart around n, we have basis of T M as
Oq
Ogd
and we call it the chart induced basis, where
9
0.
))((2))
Definition (tangent bundle). Given a smooth manifold M, the tangent bundle of M is the disjoint union of all the tangent spaces to M, i.e. we have
TM = II T.M,
(25)
zEM
equipped with the canonical projection map
#: TM + M
X TX),
where 7XX) sends each vector in T,M M to the point æ at which tangent, and I[ is the disjoint
union. Furthermore, the tangent bundle is 2d-dimensional manifold. If X *-)(U) C TM with
a local chart (U, q) on s.t. x E U, then X E Taxx;M from the definition. Since ñt (X) E U, X can be writlen in terms of the chart induced basis:
② Oq' +(X)
X ('x)
(26)
where p) .'.....p' are smooth sealar functions. We can then define the following map as a local chart for the manifold TM induced from the chart (U, q) M:
:: #= + q(U) Rd C Rdd x (q'
and the topological structure TM is derived from the initial topology ensure continuity.
IIn this paper, we may just use the word "bundle" to indicate the total space in the bundle.
13

---

Definition 3 (vector field). A vector field on M is a smooth section (Lee, 2013) of the tangent bundle, i.e. a smooth map a M + TM such that स a = idd,, where id is the identity map on M.
TM
(27)
M
We denote the set of all vector fields on M by (TMM):
T(TM) {a M - TM smooth and ā idd3.
(28)
Definition 4 (cotangent vector, cotangent bundle, dual basis, and covector field). For the vector space Ty M, a continuous linear functional from T,M to R is called cotangent vector The set of all such linear maps is denoted as T M which is the dual vector space Tf For f E C (M), at each point x, we define the following linear operator Tg M
(df)x T.M - R
X₂ (df) Xx(f)-
Given chart (U, with a E U and its chart induced basis, the dual basis for the dual space T M the set
....))...... (dg")))))
where we have (dg")z )2 ((o))) 04 (0¹) 5% with 8% = 1 = and 69 = 0 otherwise. We call the chart induced dual basis Analogous to the above definition of the vector field, we can define the cotangent bundle of M
T II T. #EM
(29)
which is again 2d-dimensional manifold equipped with the canonical projection map
π T - M
(30)
x(W),
where x (w) sends euch vector in T M to the point at which it is cotangent. Ifw - C M with local chart (U.g) 8.1. x E U, then T M from the definition. Since (w) E U, w can be written in terms of the chart induced dual basis:
xlw)
(d(w) In(w)>
(31)
where pv..-- Pa are smooth scalar functions. We can then define the following map as a local chart for manifold T M induced from the chart (U, 4) M:
E: π (T) - q(U) x 1 C Rdd (q' (T(W)).-.. (
(p(W))..,,,(()),,
and the topological structure on M is derived from the initial topology to ensure continuity. The covector fields are smooth sections of T M. The set of all covector fields denoted I(T" M).
Definition 5 (tensor field and 2-form). The tensor field can be defined using the smooth sections on tensor bundles analogously to the vector fields or the covector fields. We refer readers to (Lee, 2013) for more details. Here, instead of rigorous definition, show some basic properties of the tensor fields. For a (r, 8) tensor field T, at x. E M, it a multilinear map
M ... x T M x T,M ... M.M - RR
copies
sccopees
The differential k-form w is the (0, k) tensor field that admits alternating (Lee, 2013). Specifically, for the 2-form w, at each point x, W2 is a antisymmetric (0, 2) tensor
We T.M T T.M- R s.t. W. (X1, -. (X2, X₂ T.M
(32)
14

---

which in other words, w satisfies
w.: T(TM) x T(TM) + C C (M) s.t. -X, T(TM)
(33)
In local chart representation, we have that every k-forrm w can be expressed locally on U as
w = War dx A A... dx
(34)
where War & (U),, ai ... On S dim M are increasing sequences and dx A dx is the wedge product (Lee, 2013). Here we could abstractly view the set {dx A... dr fai....o.. with enumerated from 1 to d. abstractly as basis without more illustrations of the wedge product (We refer readers (Lee. 2013) for more details).
A
Definition (integral curve). Given a vector field X M, an integral curve of X is a differentiable curve 7 - M, where C R is an interval, whose velocity at each point is equal to the value of X that point:
4(t) = Xx()) for all I.
(35)
O J, the point (0) is called the starting point of From Picard's theorem (Hartman, 2002), know that locally we always have an interval I on which the solution exists and is necessarily unique. Definition 7 (exterior derivative and closed form). The exterior derivative a linear operator that maps k-forms to k: + 1-forms. In local chart representation, we have that if is k-form M with the local representation as
Wai ma dx A ... "x
(36)
Then, we have the exterior derivative
dw dw A ax A... dx
Oo dæ A dx A... ax
(37)
A form w is called closed if dw = O. Theorem 2. From (Rudin et al., 1976; Lee, 2013), we know that
d d 0.
(38)
which is dual statement that the boundary of the boundary of manifold is empty from Stokes' theo- rem.
A.2 HAMILTONIAN VECTOR FIELDS ON SYMPLECTIC COTANGENT BUNDLE
Definition 8 (symplectic vs. Riemannian). Let M be smooth manifold.
1. symplectic form: A 2-form (so it is antisymmetric) w is said to be symplectic form on M if closed, i.e,
dw = 0,
and it is non-degenerate, i.e,
Y
(xx ((Xx)))))((X)))>))))) 0
(39)
2. metric tensor: (0 tensor field g said be Riemannian metric M if it is non-degenerate and symmetric at each x, i.e.,
9x T.M x T.M + s.t.
s.t. 9e (X1, gz (x,, VX,, T.M
(40)
Remark 1. A manifold equipped with symplectic form is called symplectic manifold, while manifold equipped with a metric tensor is called a pseudo-Riemannian Riemannian manifold. The Riemannian metric g is (0, -eesoo field measuring the norms of tangent vectors and the angles between them. To some extent, the " "shape structure" " of the manifold M only available if we equipped M with metric g.
15

---

From the above definition, know the symplectic form and the metric tensor are both nondegenerate bilinear (0.2) tensor fields. One difference is the symplectic form is anti- symmetric while the metric tensor is symmetric. At each point M, if we use the local chart coondinate representation, the (0, 2) tensor can be represented as the following matrix multiplication
p W.
where x d matrix is symmetric for metric tensor and is skew-symmetric for symplectic form. p is the transpose of the velocity representation (which is numerical vector) in a local chart.
Because of non-degenerate, on symplectic manifold M, can define an isomorphism between ((TM) and [(T* by mapping vector field X E I(TM) to a I-form ny e M()*M), where
(xx+))= w
(41)
Similarly, on a pseudo- Riemannian manifold, we can define an isomorphism between ((TM) a (T M) by mapping a vector field X E ETTM) to I-form E (TTTM), M), where
0g TTM- - T.
(42)
In local chart coordinate representation, 0gg 9rj and its inverse ogy ged with Σ}= 9.g9 = 8 and 8 iff = k and 0 otherwise. Note the components of the metric and the inverse metric are all taken
given chart without explicitly mentioning them. Definition 9 (Hamiltonian flow and orbit). For general symplectic manifold M with symplectic form w² if we have H E - (M), then d/H is a differential 1-form on M. We define the vector field called the Hamiltonian flow Xи associated the Hamiltonian H, which satisfies that
Tx (-) dH()).
The integral curves of are called Hamiltonian orbits of H:
((t = (XH)r(t) for all t E I.
(43)
where (XH))(e) is the tangent vector at (t) M. Definition 10 (Poincaré 1-form and 2-form). On the cotangent bundle ('M of manifold M, we have a natural symplectic form, called the Poincaré 1-form
P p,dq'
(44)
Therefore, by the exterior derivative, we have the Poincaré 2-form
Poincaré = dd d(p:dg') = Σ dp, A dg
(45)
which is closed from from (38). Therefore Poincaré 2-form a symplectic on the cotangent bundle and cotangent bundles the natural phase spaces of classical mechanics (De León & Rodrigues, 2011).
For more general symplectic forms on the cotangent bundle, we can again use (38) to construct the closed 2-form from a 1-form which is potentially symplectic:
Corollary ((Chen et al., 2021)). According to (38), on the cotangent bundle T M of a manifold M, from a 1-form,
0 -xdg''
derive closed 2-form using the exterior derivative (37), and its local representation is given by the following
w² - -0- d(fidg')
Σ (arf) - 0,ft) dp: dq²
16

---

Remark 2. Note, strictly speaking, only get the necessary "closed" condition for w² to be potentially symplectic. However, it is enough for our use in our proposed framework.
We now state one of the most fundamental results in symplectic geometry that links the general symplectic form to the special Poincaré 2-form.
Theorem 3 (Darboux (Lee, 2013)). Let M be 2d-dimensional symplectic manifold. For any
......'' ..) centered at which w² has the M. there are smooth coordinates (Q'.....''' ........ coordinate representation
Σ dq A dp'.
(46)
as a Poincaré 2-form, any coordinates satisfying (46) are called Darboux or symplectic coordinates.
Remark 3. Therefore, for any symplectic form w² on the above cotangent bundle T M of M, we can always find a symplectic coordinate with the Poincaré 2-form.
Corollary 2 ((Da Silva Da Salva, 2008)). According to (33).from the antisymmetric the 2-forms, we have
w (X XH) =
(47)
which implies that hamiltonian vector fields preserve their hamiltonian functions H.
Remark 4. In physics, hamiltonian functions are typically energy functions for a physical system. Corollary indicates if the system updates its status according to the hamiltonian vector fields, the time-evolution of the system follows the law of conservation of energy.
Definition 11 (Hamiltonian orbit generated from Poincaré 2-form (Lee, 2013)). The Hamiltonian orbit generated by the Hamiltonian flow Xи on the cotangent bundle equipped with the Poincaré 2-form given in local coordinates (g. P) is given by
0H
(48)
Op:
Oq
Definition 12 (cogeodesic orbits), If additionally M is equipped with a metric tensor g, i.e, if M is manifold with metric g, and if we set the hamiltonian H on T M as
H(g v) = - (a)p.p,
the Hamiltonian orbil generuted from Poincaré 2-form is given by
aH
OH
g' Pj.
0igg* PyPk-
(49)
Op.
Oq
It is called the cogeodesic orbits of (M, 9). The canonical projection of cogeodesic orbits under (30) is called geodesic on the base manifold M which generalizes the notion of "straight or shortest line" manifold where the length is measured by the metric tensor.
B SOME FORMULATIONS AND MORE HAMILTONIAN ORBITS
In section, we first present the formulations which shown in detail in the main paper due space constraints. More Hamilonian-related flows are also presented, which however do not strictly follow the Hamiltonian orbits on the cotangent bundle T M.
B. IN SECTION 4.2.5
f2. - -02fi.net 01 - fg.f,
0g D.ei. Og f2.
0
0zʃ3,net
0zf2,net
W
0g.fimmet
Og fs.
0g O2fs.mes
0
(50)
17

---

B.2 POINCARÉ 2-FORM w Poincssaré AND LEARNABLE METRIC Gnet WITH RELAXATION
Similar to Section 4.2.4, we now impose additional system biases along the curve compared to the cogeodesic orbits Section 4.2.1,
1 p: =
q'
4J Pj
DiguertPPP + fnee(
(51)
Therefore, the projection of the curve from (51) now no longer follows the geodesic curve along the base manifold equipped with metric Jnet-
B.3 HAMILTONIAN RELAXATION FLOW WITH HIGHIRR DIMENSIONAL "MOMENTUM"
In the paper main context, present new type of mamiltonnian-elltee flow, which does not strictly follow the Hamiltonian equations. Inspired from the work (Haber & Ruthotto, 2017), we now associate each node y E I& an additional a learnable momentum vector p E R which however is not strictly cotangent vector of the manifold if d o k. We update the node features using the following equations
(hnot (p) pq)
(52)
أ = (huee (q) - pp)
where Net and hhat are neural networks with d-dimensional output and k-dimensional output respectively, e is non-linear activation function and is a scalar hyper-parameter.
C MAIN PAPER EXPERIMENTS SETTING
We select the citation networks Cora(MeCallum et al., 2004), Citeseer(Sen et al., 2008), and Pubmed(Namata et al., 2012), and the wow-hyperbolicity (Chami et al., 2019) Disease, Airport as the benchmark datasets. The citation dataset is widely used graph representation learning tasks. We randomly generate 20 samples per class for training. 500 samples for validation, and 1000 samples for testing in each dataset as the same setting in Kipf & Welling (2017a). The low-hyperbolicity datasets Disease and Airport proposed in Chami al. (2019), where the Euclidean GNN models cannot learn the node embeddings effectively. We follow the same data splitting and pre-processing in Chami al. (2019) for Disease and Airport datasets. For the link prediction tasks, we report the best results from different versions of HamGNN. HamGNN (19) for the Disease dataset and (21) for other datasets. This because we get asynchronous CUDA kernel errors on Disease on our A5000 GPU in this rebuttal period if using (21). We will update all to (21) in the final version after this bug resolved.
We adjust the model parameters in HamGNN based on the results from the validation data. We use the ADAM optimizer (Kingma & Ba, 2014) with the weight decay as 0.001. We set the learning rate as 0.01 for citation networks and 0.001 for Disease and Airport datasets. The results presented in Table are under the layers HamGNN setting. We report the results by running the experiments over 10 times with different initial random seeds.
HamGNN first compresses the dimension of input features to the fixed hidden dimension (c.g. 64) through fully connected (FC) layer. Then the obtained hidden features are input to the stacked Hee ODE layers and aggregation layers. The q in Hamiltonian flow is initialized by the node embeddings after the FC layer. Table 7 shows the implementation details of layer in HamGNN.
C. ODE SOLVER FOR HAMILTONIAN EQUATIONS
We employ the ODE solver (Chen, 2018) in the implementation of HamGNN. For computation efficiency and performance effectiveness, the fixed-step explicit Euler solver (Chen et al., 201 Sa) is used in HamGNN, We also compare the influence of ODE solvers and report the results in Table 4. One drawback of the ODE solvers provided in (Chen, 2018) is that they are not guaranteed to have the amergy-preserving property in solving the Hamiltonian equations. However, this flaw does not significantly deteriorate our model performance regarding the embedding adaptation to datasets with various structures. Our extensive experiments on the node classification and link prediction tasks
18

---

have demonstrated that the solvers provided in (Chen, 2018) are sufficient for our use. We leave the use of Hamiltonian equation solvers for future work to investigate whether solvers with the energy-preserving property can better help graph node embedding mitigate the over-smoothing problem.
ODE solver
Fuler
Fuler
Implicit Adams
Implicit Adams
Dopri5
Step size
0.1
0.5
0.1
0.5
Cora
81.10+1.13
81.52+1..27
81.62:+0.58
81.40++0.77
81.62
Table 4: Node classification accuracy(%) %) under different ODE solvers in HamGNN (20).
D MORE ABLATION STUDIES AND EXPERIMENTS
D.1 VANILLA ODE
To demonstrate the advantage of HamGNN's design, we also conduct more experiments that replace the Hamiltonian layer in HamGNN with vanilla ODE as follows:
((1) Jnet (4(1))
(53)
where the Fuet (q())) is composed of two FC layers and non-linear activation function. Compared to the Hamiltonian orbits in Section 4.2, the equation (53) does not include the learnable "momentum" " vector for each node and does not follow the Hamiltonian orbits on the cotangent bundle T M.
D.2 NODE CLASSIFICATION HETEROPHILIC DATASET
Method
Texas
Wisconsin
Cornell
GCN
55.56 3.21
61.96 + 1.27
52.35 + 7.07
GAT
56.22 + 6.02
60.36 4 5.55
49.61 + 6.20
SAGE
80.08 2.96
32.03:22
81.36 3.91
APPNP
16.76:4555
55.10 6.23
54.59 + 6.13
GCNII
6.37
52.75 4.23
GPRGNN
72.78 + 6.05
69.37 1.27
76.08 + 5.86
H2GCN
79.06 6.36
80.27 5.41
80.20 4.51
GRAND
75.68 + 7.25
79.41
3.64
82.16 7.09
GraphCON
80.00 + 3.66
84.90 + 2.64
75.14 + 4.95
HamGNN 81.62 6.22
83.92 4.87
76.49 5.10
Table 5: Node classification accuracy(%)) on heterophilic datasets. Due to insufficient time during the rebuttal period, the results for GRAND are reported from (Di Giovanni et al., 2022).
To further demonstrate that HamGNN can automatically learn the underlying geometry for datasets with different structures, we include more experiments on the node classification task using het- crophilic graph datasets. We select the heterophilic graph datasets Cornell, Texas and Wisconsin from the CMU WebKB project where randomly generated splits of data are provided by Pei et al. (2020). The edges in these graphs represent the hyperlinks between webpages nodes. The labels are manually selected into five classes, student, project, course, staff, and faculty. The features on node are the -oodds-words of the web pages.
For the heterophilic graph datasets, we include the baselines GCN, GAT, SAGE, APPNP (Klicpera et al., 2019), GCNII (Chen et al., 2020b), GPRGNN (Chien et al., 2020), and H2GCN (Zhu et al., 2020a) which are the common baselines for heterophilic graph datasets (Bi et al., 2022). Additionally, we also include GraphCON (Rusch et al., 2022) and GraphCON (Chamberlain al., 2021b), for
http::/ www.. cmv. wdw.afs.//////e/eeeeeeee.../ae cmu wdwpprjjecc/tteeeeeeeeee 11 /wwp/wwwb/
19

---

Dataset
Models
3 layers
layers
10 layers
20 layers
GCN
80.29++2.29
69.87+1.12
26.50++4.68
23.97++5.42
HGCN
78.70:0..9
38.1346.20
31.90+0.00
26.23+9.87
Cora
HamGNN (20)
81.52::1.27
81.58-10.33
79.0012.17
76.20:10.13
HamGNN (21)
81.84:00.88
81.08+0.116
81.40:10.44
80.58+00.0
HamGNN (21) type 2
0..0
81.10+0.110
81.06+1.49
79.26+11007
GCN
77.83=00777
76.00=0.87
77.53+1.06
56.50+12.79
HGCN
76.88
77.20+
65.90-19.44
42.16:1-2.54
Pubmed
HamGNN (20)
78.18+0.54
77.86+1.45
77.73+1.115
0.80
HamGNN (21)
78.83+00.46
78.50+0061 78.46
78.43++0225
77.20+00.69 78.53
HamGNN (21) type 2
79.0310.58
77.5010.44
Table 6: Node classification accuracy(%) when increasing the number of layers on the Pubmed dataset.
comparisons. We report the results by running the experiments over 10 times with different initial random seeds for GraphCON and HamGNN, while for the other baselines on the heterophilic graph datasets, we use the results reported in the paper (Bi et al., 2022) (since we use the same experimental setting) due to the time limitation of the rebuttal period.
D.3 OVER-SMOOTHING
We continue from Section 5.2 to conduct more experiments on the Cora and Pubmed datasets to demonstrate the resilience of HamGNN against over- werrsmoothiing From Table 6, we observe that if the Haet in (21) is convex, HamGNN can even retain its classification ability better than vanilla Hnet in (20). One possible reason has been indicated in Section 4.2.3 since now the Hamiltonian formalism degenerates to a I. Lagrangian formalism with possible minimization of the dual energy functional (5). In physics, lower energy in most cases indicates more stable system equilibrium. Moreover, to further show that with the convex Hee in (21) HamGNN can perform better than the vanilla Hnet in (20) against over-smoothing, we now include more choices of convex network Hues with different layer sizes and different activation functions (as along as the layer weights from the second layer in Het are non-negative, and all activation functions Hnet are convex and non-decreasing). The network details of Het are given in Table 7. The experiment results are shown in Table 6. We clearly observe that for different convex functions and on different datasets, HamGNN with convex Hntt nearly keeps the full node classification ability even though have stacked 20 Hamiltonian layers.
Fig. shows how the node features evolve over 10 layers. We sample node from the test set of PubMed dataset and input to three neural ODE-based GNN models, which are 1) HamGNN, 2) an ODE using a positive-defintt linear layer as the ODE function, and 3) an ODE using a negative- definite linear layer the ODE function. Each of these three GNNs contains 10 layers and compute the node feature magnitude, which is defined to be the Ln-norm of the feature vector, and the node feature phase, which defined to be the cosine similarity between the output feature the current layer and the input feature to the first layer. We can observe from Fig. that HamGNN has its learned node features change steadily and slowly, while the node features learned by GIL and GCN change abruptly, especially for the feature phases. The features from two nodes of different classes, learned by GIL and GCN, are converging to each other much faster than HamGNN.
D.4 STABILITY
It is well known that a Hamiltonian system is an energy-conservative system. Our HamGNN inherits this conservative or stability property. The results are shown Fig. 4, where the experimental setting is the same as those in Fig. 3. The trends are very obvious, where the feature magnitude (you may think of it as feature energy) learned by HamGNN conserved over layers, while the feature magnitude learned by the positive-definit explodes after few layers, and the feature magnitude learned by the negative-definite ODE very close to zero. Regarding the feature phase, the feature phases from two nodes steadily and slowly approaching each other when using HamGNN, while
20

---

piee Mala
map Soad
loe Tab
Fecture phase (rodaad)) mapittal
Faature plane (radiant)
phase (radlnn))
Feamuee migninste
Layer
Layer
Figure 3: Two nodes from different classes and the evolution of their feature vectors over layers. Left:: HamGNN, middle: GIL, right: GCN.
the feature phases learned by the other neural ODEs change abruptly and the difference between two nodes using negative-definite ODE is negligibly small.
Fotur phaoe (rallans)
Festure phane (callnd))
3
Testre magniaas
Layor
Layer
Layer
Figure 4: Two nodes from different classes and the evolution of their feature vectors over layers. Left: HamGNN, middle: neural ODE using a positive-definte linear layer, right: neural ODE using negative-definite linear layer.
21

---

Table 7: Neural network modules and the parameters, where x(xx) 0.5x if a exp(x) 1f x < 0
0 and
Network
Modules
Activation
Output Channels
Linear
(20)
tanh
tanh
Linear
ReHUUAmos et al., 2017)
Linear
ReHU
(21)
Linear
ReHU
ReHU
Linear
ReHU
Linear
(21) type 2
Linear
ReHU
Linear
Linear
actl
(52)
act
576
Linear
act1
Linear
(22)
tanh
tanh
Linear
Linear
tanh
tanh
(51)
64
Linear
Sigmoid
Sigmoid
Linear
sin
H:1 W:128
(23)
sin
Linear
sin
Quet
Linear
identity identity
64
Raw feature compressing FC
Linear
64
Linear
ReLU
MLP in Section 5
Linear
ReLU
Number of classes
ReLU
Linear
22


Reviews:
[{"review_id": "zR8OsDKpPq", "reviewer": "Reviewer_X8ht", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Extending recent works on Hamiltonian Neural Networks (and Riemannian Manifold GNNs), the paper considers the graph as a discretization of an underlying manifold, and construct a Hamiltonian neural network architecture (HamGNN) for graph learning problems. Building on work  in [Chen et al, 2021],  The Hamiltonian neural network architecture learns the symplectic form defining  the Hamilton flow associated with the Hamiltonian scalar function on its cotangent bundle. The authors investigate several symplectic forms and empirically demonstrate that the proposed approach achieves better node classification accuracy than popular state-of-the-art GNNs.", "summary_of_the_review": "Building on work  in [Chen et al, 2021], the paper considers the graph as a discretization of an underlying manifold, and construct a Hamiltonian neural network architecture (HamGNN) for graph learning problems.  The authors investigate several symplectic forms and empirically demonstrate that the proposed approach achieves better node classification accuracy than popular state-of-the-art GNNs. The authors could have done a better job in justifying why the proposed framework (that originates from solving physics problems) is a general fit for other non-physics graph learning problems. ", "strength_and_weaknesses": "Strengths\n\n   + While based on the \"Neural symplectic form\" first developed in  [Chen et al, 2021], the paper appears to generalize to graph learning problems.\n   + The paper provides right amount of background and mathematical preliminaries to make the paper accessible.\n   + The terms and notations are clearly defined.\n\n\nWeaknesses:\n   - The model architecture in section 4.3 could be better explained. Reading the paper and supplementary material only do not provide the reviewer a good idea how the proposed model works. One has to go back to  [Chen et al, 2021] to learn how simpletic forms are actually learned.\n   -  While the evaluation results show the proposed HmmGNN outperforms some of the state-of-the-art, I would have liked to better explain why Hamilton flows make sense for the node classifications using Citeseer and Cora datasets. For example, what does  Poincaré 2-form intuitively capture here?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nKK30fS8eS7", "reviewer": "Reviewer_XpN8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to address the oversmoothing and oversquashing problem in graph neural networks by introducing the Hamiltionian flow. \n\nThe main contributions are: \n1. Introduction of Hamlitonian flow from differential geometry perspective\n2. Propose to use Hamiltonian Neural Network in graph node embedding and classification problem\n3. By the claim of the paper, the proposed method achieves state-of-the-art performance in node classification task \n", "summary_of_the_review": "In summary, i would not recommend this paper for ICLR. There are several issues that the author need to rework:\n\n1. Rephrase the problem and stress the challenge in the introduction section, instead of focussing on the geometry. Introduction is supposed to motivate the reader to follow the paper not to confuse them.\n\n2. Emphasize the intuition over the abstraction. Differential geometry focus on invariance under diffeomorphism and it is important to stress what is invariant under this new formulation.\n\n3. Please push more geometrical concepts into the appendices and swap them with the experiment details. It is the experiments that the readers are more interested in. \n\n4. Providing more details on your implementation which also helps to make abstract concepts concrete. This entire paper is floating in the air now since no concrete formulation is provided. \n\n5. Consider publication in the applied math domains. It is likely that the geometry instead of the graph node classification is the main interest of this paper. It is thus recommended for applied math related domains. \n", "strength_and_weaknesses": "The strength of this paper lies in its novel idea of using the Hamiltonian dynamic system in replace of the traditional bipartite graph-based neural network architecture in each layer. By leveraging the energy conservation property of the Hamiltonian, it stabilizes the node embedding process to overcome the over-smoothing issue due to deep architecture of the neural network.   \n\nThe major weakness of this paper are as below:\n1. The paper structure is highly imbalanced. The main contribution of this paper is a novel graph neural architecture in node classification, which is introduced within one paragraph. On the other hands, 3 pages of paper are dedicated to “preliminaries” that neither sufficient nor necessary. It is not sufficient since it requires the readers to finish at least one year worth of study in smooth manifold theory to comprehend all the concepts introduced in the section. It is unnecessary since when it reaches to the implementation of neural network, it comes back to the classical Hamiltonian mechanics which can be introduced without using differential geometry. \n\nMoreover, what is the benefit to view the problem from differential geometrical perspective is not clear to the reader. As stated above, the node embedding itself is certainly not invariant under change of coordination or diffeomorphism, and the goal for this system to be introduced is to learn the stable/optimal node embedding for classification. It is also unclear, after the introduction of concepts on symplectic vs Riemanian, how these two types of manifolds generate different graphs after discretization. The reader is left wondering why we need to introduce a new type of manifold when Riemanian manifold is naturally available. \n\n2. The imbalance of the paper is also reflected in the introduction section. The abstract discussed that the challenge of graph neural network is the oversmoothing and oversquashing. These challenges are not discussed in the introduction section. Instead, the authors discuss differential geometric perspective in node embedding models. It feels that the author is motivated by the geometry, not the problem. \n\n3. The details of the experiment sections are not enough in the main page and many important results are left in the appendix which itself is not clear. For instance, the implementation of Hamiltonian flow part is left with no explanation. The reader is expected to guess what type of ODE solver is involved. It is also not mentioning that normal ODE solver is not guaranteed to have the energy preserving property thus it is not suitable for this experiment. Thus lack of details in implementation makes it impossible to reproduce the result unless to copy the code of the author (which is very uninformative and unclear)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n6lXwSCJJi", "reviewer": "Reviewer_skVS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a Hamiltonian based feature learning in GNNs to overcome the oversmoothing and oversquashing phenomena that are evident in many existing GNNs. \nA rigorous theoretical discussion is provided and several experiments are conducted. ", "summary_of_the_review": "The paper sheds more light and analyzes the Hamiltonian flows approach in GNNs. Several experiments are conducted, but they are far from current model accuracy and performance, and proper discussion of existing and relevant model is missing, including a quantitative comparison with recent GNNs.", "strength_and_weaknesses": "Strengths:\n\n- The authors provide a rigorous theoretical discussion.\n- The authors clearly explain why Hamiltonian based GNNs can be useful.\n\nWeaknesses:\n\n- I am not convinced that the proposed method is the first to be based on Hamiltonian (hyperbolic) flows in GNNs. For example:\n\"PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations\"\n\n\"Graph-Coupled Oscillator Networks\"\n\nalso utilize such flows but are not discussed in the paper.\n\n- The experimental section lacks comparison with recent and state-of-the-art methods, which are stronger by a significant margin. For example:\nGRAND: Graph Neural Diffusion\n\nSimple and Deep Graph Convolutional Networks\n\nDirichlet Energy Constrained Learning for Deep Graph Neural Networks\n\n- The experimental section's scope is quite narrow. I think that more experiments on additional datasets (e.g., heterophilic datasets like Cornell and Wisconsin, larger datasets like ogbn-arxiv) are required to measure the benefit of such an approach. \n\n- The authors focus only on transductive datasets (this is stated in text), but it is not clear why. Is it a limitation of the method?\n\n-The paper is hard to follow and in my opinion can be better organized.\n\n- The authors discuss the oversmoothing problem but in the experimental section only report the results with up to 20 layers. I can not draw conclusions based on 20 layers and would recommend the authors to include results with a larger number of layers(e.g, 64 layers).\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "i6m3djJyxi", "reviewer": "Reviewer_FBem", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a novel application of differential geometry and Hamiltonian flows to graph neural networks and demonstrate its applicability to the common GNN problem of node classification. The theory gives rise to a GNN model which is then shown to perform relatively well on standard datasets and somewhat to resist over-smoothing as the depth of the network increases.\n", "summary_of_the_review": "I find the paper very interesting, however the authors do not present compelling motivation or outstanding experimental results for this reviewer to consider using the methods. They do not discuss the methods and experimental setups in sufficient detail, and do not provide comparisons to state-of-the-art methods.\n\nThe paper is well presented, though could benefit from more editing work to fix typos, clarify notations, and add necessary details to motivate its inclusion into scientific archives.\n", "strength_and_weaknesses": "The authors propose an interesting application of Hamiltonian flows and differential geometry to graph neural networks. However, most of the models they compare with in the experiments section are not the newest and despite the paper’s claims, no longer represent the state of the art. I recommend the authors compare to more modern graph neural networks including GCNII (2020), LGCN (which is cited in the related works section but not compared to), GRAND (Chamberlain et al.), PDE-GCN (Eliasof et al. 2021), GraphCon (Rusch et al. 2021), EGNN (Zhou et al 2021) or newer. According to the results listed, the proposed method does not achieve SOTA performance compared to the works cited above.\n\nIn the absence of SOTA performance, the paper does not otherwise substantiate the use of the mathematical theory. The contribution of the proposed methods is not discussed in terms of any other metric, such as reduced running times or the number of parameters, etc. Furthermore, the authors explore different network depths, but the point of obtaining deeper networks is lost when accuracy is not comparable to other models of similar depth.\n\nThe paper is difficult to follow, and the method of Hamiltonian flows is not clearly motivated for the task of node classification. See the next section for details.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject