Curvature-aware Graph Attention for PDEs on Manifolds
Authors: Yunfeng Liao
Jiawen Guan
Xiucheng Li
Venue: ICML 2025 poster (2025)
Keywords: Graph neural network
Partial differential equations
Curvature-aware attention
Parallel transport
Differential geometry
Primary area: deep_learning->attention_mechanisms

Abstract: Deep models have recently achieved remarkable performances in solving partial differential equations (PDEs). The previous methods are mostly focused on PDEs arising in Euclidean spaces with less emphasis on the general manifolds with rich geometry. Several proposals attempt to account for the geometry by exploiting the spatial coordinates but overlook the underlying intrinsic geometry of manifolds. In this paper, we propose a Curvature-aware Graph Attention for PDEs on manifolds by exploring the important intrinsic geometric quantities such as curvature and discrete gradient operator. It is realized via parallel transport and tensor field on manifolds. To accelerate computation, we present three curvature-oriented graph embedding approaches and derive closed-form parallel transport equations, and a subtree partition method is also developed to promote parameter-sharing. Our proposed curvature-aware attention can be used as a replacement for vanilla attention, and experiments show that it significantly improves the performance of the existing methods for solving PDEs on manifolds. Our code is available at https://github.com/Supradax/CurvGT.

Curvature-wware Graph Attention for PDEs on Manifolds
Yunfeng Liao'
Jiawen Guan
Xiucheng Li'
Abstract
solve PDBs by learning the mapping between input-output pairs in an end-to-end manner. Thus they outperform nu- merical solvers in time-sensitive scenarios (e.g.. real-time rendering in gaming engines) while slight inaccuracy is tolerable. However, most of them are dedicated to PDEs in Euclidean space. To generalize the methods to discrete manifolds, the work Geo-FNO (Li et al., 2023a) and Tran- solver (Wu et al., 2024) are proposed.
Deep models have recently achieved remarkable performances in solving partial differential equa- tions (PDEs). The previous methods are mostly focused on PDEs arising in Euclidean spaces with less emphasis on the general manifolds with rich geometry. Several proposals attempt to ac- count for the geometry by exploiting the spa- tial coordinates but overlook the underlying in- trinsic geometry of manifolds. In this paper, we propose a Curvature-aware Graph Attention for PDEs on manifolds by exploring the impor- tant intrinsic geometric quantities such as curva- ture and discrete gradient operator, is realized via parallel transport and tensor field on mani- folds. To accelerate computation, we present three curvature-oriented graph embedding approaches and derive closed-form parallel transport equa- tions, and a subtree partition method is also de- veloped to promote parameter-sharing Our pro- posed curvature-
Geo-FNO (Li et al., 2023a) attempts to learn universal mapping that transforms curved surface into regularly spaced grids and then solves the PDEs in Euclidean space with Fourier neural operators. However, the reduction to Euclidean space might fail since the existence of such a universal parametrization is not always guaranteed. By contrast, Transolver (Wu al., 2024) directly handles the ambient space in which the surface resides. By establishing global coordinate system, it encodes relative positions into node features so as to distinguish different parts of the surface. Nevertheless, relying solely on spatial coordinates leaves the intrinsic geometry of the manifold unexplored. In particular, it a well-established theory that the curvature of underlying manifolds dominates the physical laws of observable phenomena (Naber, 2011). Hence, the curvature is essential for developing accurate solver for PDEs on the manifolds. However, neither Geo-FNO nor Transolver explicitly considers it in their model design.
attention can be used as a replacement for vanilla attention, and exper- iments show that it significantly improves the performance of the existing methods for solv- ing PDEs on manifolds, Our code is available at https://g thub. com/ Supradax CurrGT.
On the other hand, the discrete manifolds are often repre- sented by graphs (with node coordinates) and Graph Trans- formers (Ying al., 2021; Zhang al., 2023) have achieved promising results on a wide spectrum of graph-related tasks, whose successes can largely be attributed to the inductive bias injected into the self-attention to capture the underly- ing topology and geometry. However, as one of the most important geometry quantities manifolds, the curvature is still under-explored. To further explore the intrinsic ge- ometry of the underlying manifold, we propose a Curvature- aware Graph Attention for dependepn PDEs discrete manifolds, which naturally encodes the curvature into the self-attention mechanism. However, simply treating curva- tures as additional node features has limited impact because it only reveals the local curvature at points, and it is the curvature of the path connecting points that dominates the physics laws. In addition, in contrast to message-passing in Euclidean space, the tangent vectors of different points
1. Introduction
Partial differential equations (PDEs) serve fundamental tools for describing wide range of scientific phenomena. However, deriving closed-form solutions for general PDEs often infeasible. The numerical methods such as finite difference methods and finite element methods, discretizing PDE domains into grids, offer a viable alternative but are often costly. Recently, data-driven deep models (Lu al., 2021; Li et al., 2023d; 2020a; Kovachki et al., 2023) have emerged as powerful tools for PDE-governed tasks. They
'School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), China. Correspondence to: Xiucheng Li <lixiucheng @ edd...d..cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Curvature-aware Graph Attention for PDEs on Manifolds
on manifold residing in different tangent spaces own dis- tinct local coordinates, which prevents us from aggregating information with shared linear mapping.
Background Manifold
Message Aggregation
Node Update
Paralli Trancprt?
I.M
To address these challenges, we propose to generalize the vanilla attention to manifolds via parallel transport. As a well-developed mathematical tool in differential geometry, parallel transport can naturally carry the path curvature from one point another and align the local coordinate systems of different tangent spaces. Moreover, we propose to replace the linear mapping with the tensor field (multi-linear map- ping) on manifolds to generalize the message aggregation. This is illustrated in Figure 1. To accelerate the computation of parallel transport, we present three aurvature-oriente graph embedding methods and derive closed-forn parallel transport equations accordingly; we also develop a subtree partition to promote parameter-shariin for the tensor field implementation. Lastly, a multi-head curvature-aware at- tention is presented to enhance the representation learning capability. Our proposed curvature-awar attention can be used as a replacement for vanilla attention in existing meth- ods such as Graph Transformers, Graph Attention Networks, etc. The experimental results show that it able improve the performances of the existing methods significantly for PDEs general manifolds. To summarize, our contribu- tions are as follows.
n
Figure I. By interpreting node features as tangent vectors on the manifold, the message passing on a manifold is naturally defined as the parallel transport onward and node update can be generalized to multi-linear mapping in the tangent vector space.
given by PDEs (Lu et al., 2021; Jin et al., 2022; Li et al., 2023d). As for neural operators on graphs, spatial methods like multi-scale techniques implemented by pooling (Gao & Ji, 2019) or matrix decomposition (Li et al., 2020b) and spee- tral methods from Fourier operator (Li et al., 2020a; 2023a) to wavelet decomposition (Gupta al., 2021; Tripura & Chakraborty, 2023; Xiao et al., 2023) are respectively pro- posed to enhance the graph-model performance. Spatial and spectral methods have recently achieved seamless in- tegration. U-FNO (Wen et al., 2022) and U-NO (Rahman et al., 2023) utilize U-Net and FNO to capture multi-scale features in spatial and spectral domains, respectively. Since spectral methods are easier to handle in regular spaces, Geo- FNO (Li et al., 2023a) maps an irregular spatial domain to regular grid followed by spectral blocks while LSM (Wu et al., 2023) first hierarchically projects a high-dimension spatial domain to latent physical tokens.
We propose novel Curvature-aware Graph Attention for PDEs on manifolds, which naturally incorporates the curvature and discrete gradient operator as the in- ductive bias.
The proposed attention is realized by a combination of parallel transport and tensor fields on manifolds; we present three curvature- methods and derive closed-fore parallel transport equa- tions to accelerate computation, and a subtree partition method is also proposed to promote parameter-sharing.
The paradigm of operator learning has been recently shifted to Transformer architectures with a focus on finding PDE- compatible attention mechanisms. HT-Net (Liu et al., 2023) adopts the canonical V-cycle (Gao & Ji, 2019; Li et al., 2020b) to obtain hierarchical attention on the PDE domain; GNOT (Hao et al., 2023) gains insight from Mixture of Experts(MoE) in which boundary shapes and edge features get involved in cross- aoossnntt:n Transolver (Wu et al., 2024) decomposes the non-Fuclidean PDE domain into slices and establishes slice-wise attention onward. To mitigate the quadratic attention problem in transformers, two types of softmax- attention (Cao, 2021) are proposed to enhance the efficiency, which is further extended in OFormer (Li et al., 2023c) and FactFormer (Li et al., 2023e).
graph embedding
Our proposed curvature- attention can be used as direct replacement for self- relf-attentonn It significantly improves the performances of the existing methods for solving PDEs on manifolds.
2. Related Work
Physics-Iffrrmed Neural Networks (PINN) are originally proposed to solve PDEs with exact formulas in continuous domains (Raissi, 2018), applied to fluid simulation in graphics (Jain et al., 2024), which is, however, intractable on a manifold. PINN based on graphs can better adapt to manifolds in the form of point clouds and structured meshes (Brandstetter et al., 2022). Different from PINNs, neural operators map a function to another by leveraging data-driven loss instead of the physical loss
Neural Operator.
Geometric Graph Embedding. It is usually assumed that data features concentrate on a low-dimension manifold in a hidden feature space (Ghojogh et al., 2023). Among those non-trivial manifolds, hyperbolic space is a popular choice as the embedded space for hierarchical graphs, which is realized by bringing in gyrovector spaces (Peng et al., 2022) or classical hyperbolic models (Nickel & Kiela, 2017; 2018). In our task, the graph has natural geometry inherited from
2

---

Curvature-aware Graph Attention for PDEs on Manifolds
11
The illustration of geometric quantities n, V Piy can be found in Figure 2.
D123
NOTATION
IMPLICATION
PLL
D151
SPATIAL COORDINATES OF 2FH VERTEN
U2
4/
Dije gF
VJ NORMAL TRIANGLE Y,Y
4. Method
DIREDRAL ANGLE arvoom(nigs - (iae))
Problem setup. In this work, we consider the time-
U3
dependent PDEs on a manifold M of the form
Figure 2. Notations of Geometric Quantities
du(x, t) = R(u)
(2)
the manifold M by discretization. The work that utilizes spatial coordinates as geometric information has been stud- ied (Wu et al., 2024) whereas the intrinsic geometry of surfaces has not yet been explored.
dt
where x M, u(x, 1) is a function of and time R is linear or nonlinear differential operator. For instance, R(u) = Au(x, t) + f(x t) for a heat equation, u(x, t) and fxx, represent the observed temperature field and heat- source distribution, respectively. In practice, M is often given as a discrete manifold, i.e., a graph G with node set V and edge E. Given a discrete manifold G and collec- tion of m functions u1 (x, t), u2(xx, t), -.... Un (x, t) on G at time t, the neural operator F aims produce the function "(x, l + 1) on G at / + 1. As most of the PDEs of inter- est in practice are established on 2-dimensional manifolds, we will primarily focus on these manifolds in this study. (xxx, t) will also be denoted by u!") for short.
3. Preliminaries
Riemannian Manifold. A Riemannian manifold M is a topological space that locally resembles Euclidean space and is equipped with a positive metric tensor field g. Vp E M, there an associated tangent space T.-M with inner product glp- The cotangent vector space TjM is the dual space of TpMM consisting of linear functionals that map u E T.M to a scalar via v"( := (v, ") A tensor is multi- linear mapping comprised of tangent vectors and cotangent vectors. For example, a (1,1)-tensor u @ v" acting on a tangent vector w gives a tangent vector (v, w) u while a (0,2)-tensor u² @ v² acting on two tangent vectors W1, W2 in order gives scalar (u, w1)gi, (v, W2)
4.1. Design Motivation and Challenges
Design Motivation. Self-attention mechanisms (Vaswani et al., 2017) have become the de-facto choice for founda- tion model designs across various domains. In particular, Graph Transformers (Ying et al., 2021; Zhang et al., 2023) have extended their successes to graph structure data and achieved state-of-the-art performance on a wide range of graph-related tasks. The success of Graph Transformer can be largely credited to the inductive bias injected into the self- attention to capture the underlying local structures, which are oblivious to the naive implementation. For example, Graphormer (Ying et al., 2021) introduces the node central- ity and shortest path distance (SPD) to correct the attention score, thereby making it aware of the node degree and pair- wise node relationship: Graphormer- (Zhang et al., 2023) further explores the Resistance Distance (RD) enhance its expressiveness. In nutshell, the core idea behind this success is to introduce bias to help the attention mechanism be aware of the local topology (node degree) and geometry (pairwise node distance) of the underlying graph.
Parallel Transport. Though Euclidean vectors can be par- alleled in any manner without changing their inner products. However, to preserve inner products on M, their compo- nents must vary in accordance to and the specific path y between p. 9 E M. Formally, let 7(0) = p. (s) = q. and the tangent vector E Tp.M is parallel transported along 7, resulting in the vector (()) E TyM. The par- allel transport of cotangent vector v can be defined in consistency with the tmner-rroduct-peservvin nature by: [())ōv((u) = (T())Vv, gi.
Curvature. Curvature is an essential intrinsic geometry property of manifolds and is central to many physics theo- ries such as general relativity (Stephani. 2004), gauge the- ory (Naber, 2011), and string theory (Zwiehach, 2009). The sectional curvature K (u,, V)lp measures how much the ge- ometry of a 2-dimensional sectional of manifold spanned by Y E T.MM deviates from being flat. For the 2- dimensional manifolds, the sectional curvature is identical to the Gaussian curvature K that characterizes the degree and nature of local bending (Tu, 2017). The Gaussian cur- vature K vv point v, can be estimated with the aid of the normal bundle theory (Yau& & Gu, 2008) in a robust manner:
Inspired by the above observation, it is appealing to gen- eralize such success the discrete manifold, represented as a graph with rich geometry. As mentioned in Section I, one of the critically important geometry quantities on the manifold that dominate the underlying physics law is the cur- vature. In this paper, we aim to propose a curvature- self- self-attention mechanism to solve the PDEs on a manifold. Instead of adopting the Graph Transformer architecture, we still opt for the message-passingg framework to avoid the
Klv, 1 VVe. ||
Σ
VJ= V,
Pij
(L)
Vil12
3

---

Curvature-aware Graph Attention for PDEs on Manifolds
Fast Parallel Transport
Parameter Sharing
g
R2
PS
h
Prrule
(N)
h
T( b.
h(y)..
h
((1)
Figure 3. A node is locally embedded into one of the constant Gaussian curvature surfaces, enabling us to perform fast parallel transport. Parameter sharing are implemented by broadcasting source node parameters via parallel transport.
quadratic complexity O(V
the representations from different tangent spaces T.M with v E (u)) to produce m For the same reason, Eq. (4) and Eq. (5) have similar issue.
Challenges of Encoding Curvature. To illustrate our method, we first recall the general graph message-passingg steps by focusing on one single-layer transformation,
To address the above two challenges, we propose align dif- ferent tangent spaces manifold via parallel transport and generalize the matrix multiplication (linear mapping) with the tensor field (multi-linear mapping). By integrating the two techniques in systematic way, we develop a curvature- aware graph attention mechanism that can more effectively solve the PDEs on manifold, the details are presented in Section 4.2. Besides, to further accelerate the computation of parallel transport, we propose three curvature-oriented graph embedding methods and derive closed-for parallel transport equations in Section 4.3, and in Section 4.4 we also present a subtree partition approach to promote the parameter-sharing for the tensor field implementation. The framework is presented in Figure
m' =
Σ
Que Wh,
vEN(u)
h2 = a Whh, + m')
(4)
Oue = softmax,(a [h. (ho))
(5)
in which h, RD h4, RD are node representations for node E V, m', is the aggregated message for node from its neighbors whereas W, W E DDD'D E 1 are learnable parameters, and o() is the activation function. incorporate curvature information, one may attempt to mimic the Graphormer by either computing the local curvature of node u, v denoted by Cu, Cy and concate- nating to h h, adding a bias term oue (e.g., SPD or RD) to correct the attention score, i.e., replacing Eq. (5) with
4,2. Curvature-aware Graph Attention
Oue = softmax,, (a [h., cc [,,l,,,]) + duv-
The parallel transport serves as a foundation tool for our proposed curvature-aware graph attention. To compute m each neighboring node representation h, is moved from T.M [(>) E T.M via parallel transport. The benefits twofold: 1) the parallel transport is well- developed tool in differential geometry and it can reveal the curvature of the space (more concretely, the curve along which moves), and thus we can encode the curvature of the path y into the transported vector [(7))[h.]; 2) the transported [(>) [b,] becomes tangent vector in T.M, which enables us to generalize the matrix multiplication with the tensor field calculate m
However, such straightforward adaptation will work for reasons. (i) It is the curvature of the path between u, that dominates the physics law, simply concatenating c, to h, will not work as c, can only reveal the local cur- vature at point v; unlike the distance (SPD or RD), there no closed-form formula to calculate the curvature of the path between two points u, u on a manifold. (ii) To acquire the aggregated message m in Eq. (3), we transform h, from _ ig with linear mapping h, + Wh,, This is feasible as both h, and m', live in the Euclidean space. However, on manifold M, h, and m', will reside in two different tangent spaces, T.M and M,M, respectively, and makes no sense to transform vectors via linear mapping between two tangent spaces since each of them owns a dis- tinct local coordinate, Hence, unclear how to aggregate
To motivate our design, we first rewrite the matrix multipli- cation with its singular value decomposition form:

---

Curvature-aware Graph Attention for PDEs on Manifolds
Wh = Σ Appaq = Σ (p)
1/3
II2
y
+1
Ex(e 8 (()
(6)
1/2
1
-74
13
2/2
1/
U1
1/2
U4
V3
(a) Sphere
(b) Psuedosphere
in which Ps, qi are left- and right-singular vectors of W. The key observation is that the inner product (q, h) times Pa can be viewed as (1, 1)-tensor or multi-linear mapping Pa & qf acting on a tangent vector h, which can be general- ized to manifolds naturally. In view of tensor fields, we can rewrite the first two ressage-passing steps in Eq. (3) and Eq. (4) in the following forms.
Figure 4. Subgraphs are embedded into constant curvature surfaces. It is common to handle the pseudosphere PS² with the Poincaré's half-plane IH? via isometry yp PS + H.?.
Remark 2. In our proposed attention, message passing is achieved by using tangent vector fields. However, in many PDEs, both input and output are scalar fields. The conver- sion from a vector field to a scalar field can be achieved by adding learnable cotangent vector field. Conversely, the most natural vector field related scalar field is its gradi- ent, obtained by the discrete gradient operator G (Jacobson, 2013), which is also an important geometry quantity.
m = Σ
Bun(w, & (T) VEN((u)
hn = a ((W1 & w.5) + m())
in which W1, W2, W1. We E RS are parameters to learn. We first move h, E T.M via parallel transport to obtain a tangent vector [()) [h]] E T... Then we apply a (L, 1)-tensor wi @ W2 [(r)ā[be] to yield a transformed tangent vector in M, and consequently, m', will also be tangent vector in TuM, which can be used to generate hyg In a similar spirit, Eq. (5) can be generalized with (0, 2)-eesorr a 2 a to produce scalar as follows:
4.3. Fast Parallel Transport
To support our proposed curvature-aware graph attention, we need fast and accurate implementation of parallel trans- port on manifold M. Since M is often represented by its discrete version G = (V, E), it suffices to consider the de- sign of parallel transport along directed edge (u, u) E E. To this end, we propose to embed an edge (æ, u) into three types of constant Gaussian curvature surfaces, namely, the sphere $2, the plane R.2 and the pseudosphere PS?, which owns positive, zero, and negative curvature, respectively, The idea is to approximate the local geometry of (r, u) with the three constant curvature surfaces depending on the value of curvature Kl, at u as follows:
Ohur = softmax, ((aj @ (h.. (())ā(h,))
(9)
where , a2 E 3 are learnable parameters.
Implementation of Tensor Field. Our proposed curvature- aware graph attention involves (1, -ttnnso w1 @ w and a (2, O)-tensor aj @ a. operation. which is defined by inner product on the manifold with associated metric tensor field g as follows (assuming both tensors are assigned to point u and h, h' M).
S2
if Klu > E,
R2
if - S K la Se
(12)
(Wi & (h) wi (W2, h) glu.
(10)
PS2 if Klu < <C
(aj @ (h, (a1, h) (ag, gl)
in which threshold e > 0 given. Therefore, we only need to consider the parallel transport along the three constant Gaussian curvature surfaces. To offer fast implementation, we will derive a closed-form formula for each of them in the remainder of this section. First, the parallel transport on IR? is simply an identity map. Next, we consider $2 and pg?
By Jacobi field theory (Jost, 2017), a trivial Jacobi field along radial geodesic enables a Taylor expansion to esti-
mate the metric tensor field by curvature tensor Ruay:
(x)(x) = 6ij 3 R xt + +((())))
(11)
in which Rasy is tractable with Taubin tensor (Taubin, 1995). Notably, for a surface embedded in R3, there is only one independent component R1212, which simplifies computation dramatically.
Parallel Transport on $?, The embedding of edge (v, u) into $ is simple. We can just treat the spatial coordinates u, R of u, as vectors on $2 and calculate their rotation angle 0, which is closely related to the path curva- ture from to on $ Theorem 4. states that the parallel transport on the sphere is uniquely determined by
Remark 1. As side benefit, generalizing the matrix multi- plication in Euclidean space to the tensor field on manifold M ensures that the obtained h', in Eq. (8) still resides in T,M, which permits us to stack up multiple curvature- aware graph attention layers to learn more expressive repre- sentations.
Theorem 4.1 (Parallel transport on S²). Let u, v S C
R f == arccos v) and the orthonormal basis be (el = u, v/ [|u x vllg. e3 = e1 x e2 Suppose the geodesic between v and u is (t) = cos fcj + sin Ac3.
5

---

Curvature-aware Graph Attention for PDEs on Manifolds
TH AM
Ta
H
11
v(w)
h,
=(v]
Initial State
Parameter Sharing Scheme
v(u)
Figure 6. The graph is partitioned into two subtrees with depth d 3, highlighted in different colors.
Figure 5. The computation diagram commutes.
Theorem 4.2 (Parallel transport on HP³), Let u, H s.t.
then h. 6C3 + bc2 E T.S² parallel transported to [())gh, a 0eg a sin 0ei be2 E T,S
Tu = Iy. Parallel transport along the segment from v to a is given by [(r)ā,, = Bu/ bu/ueh, E T.H?.
Lemma 4.3. The special Möbius group M(C) == {2 + (cz + d)/(az b) a, c, E C, ad - be = 1] is an isometric transformation group of H². Inversion 2 + 1/2 is a special case of M MCC which maps circles that pass through the origin a line parallel the y-axis and vice versa. Besides, the translations Ta R) along x-axis in H² also form an isometric automorphism group.
The proof of the theorem is given in Appendix A. In short, Theorem 4.1 offers us a very cheap manner to transport h, from the tangent space T.S² to T.2? on the sphere.
Parallel Transport on PS?. To derive the parallel transport in PS?, we propose map an edge (v, u) to Poincare's half- half-plane HH? with an isometry op PS? - H2, because the two spaces are isometric and H² is well-studied hy- perbolic space that enjoys appealing properties. To achieve the transformation, it suffices to design the pushforward pp : TS. + T2 of the isometry by making it pre- serve the direction and distance between points PS?.
Lemma 4.4. If (M,9) + (N, h) is isometric, that is, (u, v)2 = (p-,, 90. V) holds for all u, v E T,MM, then the pushforward yr and parallel transport commute:
g. [())ā = = ((((())) p.
(13)
To this end, we note that the two principal directions at point exactly the two axes of HHI2 as illustrated in Figure 4-(b). Hence, ye. can be designed to map to the coordinate (0, 1) in IH2 and v to the location along the direc- tion projn.! (v - u) with a distance dga (u, v), where proj the projection operator and n denotes the orthogonal complement of normal vector (the tangent plane of PS² at point u). In the remainder of this section, we will denote the coordinate of E H² by (Iv. Vo) E IR² or represented by the complex number 2, = Fy + igo-
Furthermore, if = id then we also have
[(7) = p. ((p())) o p..
(14)
Theorem 4.5 (Parallel transport on HP)), If on a semi- circle then by the commutativity between the pushforward of isometric transformation and parallel transport we have
op, r()),,, (7
(15)
Moreover, ler ( (-a, a, 0) be the left intersection between 7y and z-axis, and 2 == (p()) = (1(22aa), then Eq. (15) becomes
Now we are ready to derive the parallel transport H² First, if y happens to be a line segment parallel to y-axis, T(1)6 is then simple scaling transformation stated by Theorem 4.2. However, in general, 7 is more likely to be section of semi-circle centered on zzaxis and Theorem 4.2 will fail. Fortunately, we can turn the semi-circle into a seg- ment by inversion 2 + 1/2 and translation Ta + z + a, which are isometric transformations in IHI (as justified by Lemma 4.3). Meanwhile, Lemma 4.4 states that parallel transport and the pushforward of isometric transformation commutative (the lemma is illustrated in Figure 5), abling to transform the semi-circle into a segment parallel y-axis and then apply Theorem 4.2 to derive the foosed-form parallel transport on H for the general cases. The result is summarized in Theorem 4.5, the computation only involves x 2-matrix multiplication, which incurs little overhead.
Mu J J| 1lt E T.
T(7)ā
(16)
(x2 - y² -2ry x²
2xy
where J |2|
y²
In our implementation, h, is represented by 3D-vector (for the convenience of computation) of dimension two (in its local coordinate). Hence, it can be transformed by a x matrix locally.
4.4. Subtree Partition to Facilitate Parameter Sharing
When generalizing the matrix multiplication from the Eu- clidean space to the tensor field on manifolds, the learnable
6

---

Curvature-aware Graph Attention for PDEs on Manifolds
parameters @ (w, W,, a (i 1, in Eq. (7-9) will all fall in the tangent space T.MM In contrast the Eu- clidean space IR", each TrM is local coordinate and it is geometrically unmeaningful to share parameters across distinct coordinates. As a consequence, we have to assign a group of different parameters to each node u E and the number of parameters will grow with [V], which hinders the statistical strength sharing across nodes and renders the learning infeasible.
Attcation Head I
Attention Head II
Atteation Head III
Base GNN
To control the number of parameters and promote statistical strength sharing, we further develop a subtree partition ap- proach. We start by noting that the tensor field is eventually instantiated with the inner product on manifolds in Eq. (10) and the parallel transport preserves inner products, i.e.,
Figure 7. Multi Multi-head attention via diverse subtree partitions.
4.5. Generalized Geometric Multi-Head Attention
Multi-head attention expects the model to learn from differ- ent represent subspaces (Vaswani et al., 2017). Following the same spirit and noting that the subtree partition is not unique, we present a multi-head attention mechanism on manifolds by different partitions, as shown in Figure 7. The realization is simple since it only involves an extra concate- nation operation. Specifically, the curvature-aware graph attention layer with C' heads is shown as follows:
(()) = (Vi, gl
(17)
This enlightens us to share parameters across nodes with parallel transport. Specifically, we can move the parameters @ from a node them share one set of parameters. Then the curvature-aware attention becomes
node via parallel transport and let
m m- [nn),wi, @ w2,1 (T(1)6(b...)
m'. = Σ vEN(w)
One In)llwi & wa](T()6((b))
m =
Oewe m
VEN (ww)
(18)
h = a (T(w)lw) @ (h)( + m'm) Awe = softmax, (TmMaat @ hh (())((h,))
h/. -[n)6 @ Wa (hm,a)
h = (11 hin + m'r
in which is the piecewise geodesic from u to w (to trans- port parameters 9) and y is the geodesic from v to w (to transport representation h.))
O = softmax, T(mbia @0 ag (1)))))..)))
(19)
where h is the feature tangent vector in M of the i-th head and mar h E R3C
To reduce the average transporting distance, we start with randomly selected node V (as the source node) and run the BFS (breadth first search) with a maximum depth d to form subtree T of G. Next, we randomly select another source node u' from the unvisited to repeat the above procedure until all nodes in G are visited. In such a manner, we partition G into collection of subtrees. We only assign parameters θ source nodes and the other nodes in subtree share the same set of parameters with the source nodes via parallel transport. Figure 6 shows partition with two subtrees.
5. Experiments
Experiment Settings. A dataset a collection of tuples (ul ..... Um ((-1)) and the associated neural operator is F: I.M x x... x (.2 - I.2(M) Following the paradigm in (Li et al., 2020a), we assume the input functions
I following a certain distribution a and define the loss by:
C= El (F(I), u ((+1)
(20)
Computation Complexity. The complexity of a typical single-head GAT layer is [[[VI+++E]] (Veličković et al., 2018) where D is the number of features. Note that a set of edge-wise parameters support parallel transport only includes 2-order transform matrix, 2-order metric matrix, and local coordinates while the parallel transport along each edge can be done within O(). A subtree with maximum depth d requires parallel transport for (d-1) times to ensure the parameters are broadcast to all nodes. Therefore, the complexity of a single-head layer is O(V)Dd + D).
where é is a function that measures the difference between the output F(I) and the ground truth ((+1)) is usually chosen be L norm when solving PDEs in Euclidean spaces. Here we adopt L norm || ||2 and Hilbert H² norm || (Im since they can be naturally extended to general manifold M:
||u(x) | = [ (((dd,r,
(21)
JEEM
[[u())) = 1|x(x))113 ([
(22)
7

---

Curvature-aware Graph Attention for PDEs on Manifolds
Table 1. Performances of different methods on the discrete wrinkle (see Figure 10) governed by isotropic di diffusion, p-I Laplacian diffusion and wave equation (see Appendix B.I), respectively. The relative errors of L and H are reported.
Model
Isotropic Diffusion
p- p-Laplacian Diffusion H' H'(%)
Wave Equation
(2))
(%)
(2)%
L (26%)
(%)
ResNet
2.008 187
7.470
29.362:6..42
54.117
4.218+3.917
4.484=t4.22
GCN
7.947=0.805
44,587 11.203
5.445+2.885
12.923+8.527
21.783=1.580
24.901=1.985
DeepONet
2.296=0.004 046
8.977=0.534
37.855+0.530
48.302+2.133 134
25.773=1.051
27.536#1.399
Transfrmmer
0.061
0.221+0.056
9.695+4.236
14.443+4.780
11.848=4.495
13.679+5.499
Foee
0.086=0.0277
0.327+0.1 114
14.04444.405
20.313=6. 184
15.824+44.552
17.530=5,95
GalerkinType
0.077=0.028
0.291
13.456+2.711
16.928+2.938
14.204=5.655
15.748=7.033
Graph U-Net
7.043+0.282
41.98946.144 142
10.971
19.765432.059
16.18420.697
19.116#2.217
MKGN
0.640#0.699 693
222
7,418+0.731
21.0445..351
2.966#1.599
4.10542.324
GNOT
0.040#0.025 023
0.160#0.061
2,466+
1.39
0..5140.111
0.457=0.274
GINO
0.320=0.011
1.415=0.051
42.819+1.144 149
52.579+1.011
2.043=0.555
2.136#0.343
Geo-FNO
0.075=0.029
0,249+0.055
1.936+0.389
1.177
2.394=0.245
2.19600.306
Transolver
0.681=0.188
1.687=0.344
1,935+03318
3.905+0.550
1.646#0.493
1.855=0:497
Curv-GT (Ours)
0.025=0.008
0.101=0.033
1.577=0.839
3.212+1.432
0.035+0.014
0.039=0.017
To calculate the above integrals, first compute the mass matrix M with finite element methods. M is a diagonal matrix whose diagonal element m; is the surface area that vertex U2 takes up. Therefore, we have []u(2)]1 x u Mu. Likewise, one can adopt a discrete gradient operator G induced by a Witney basis to take the place of V (Jacobson, 2013), which is indeed matrix:
Transformer (Yun al., 2019), MKGN (Li et al., 2020c), Galerkin-type Attention (Cao, 2021), GNOT (Hao et al., 2023), GINO (Li et al., 2023b), Geo-FNO (Li et al., 2023a) and Transolver (Wu et al., 2024). The implementation de- tails are provided in Appendix C.
Main Results. We first study the performance of different methods for PDEs on the wrinkle manifold-a complex manifold containing both positive, constant, and negative curvatures (shown in Figure 10). To make it more intuitive, the results are shown in the form of relative loss C/ Bese where Lase denotes the loss obtained by simply taking the observation (1)) as the prediction at time t + 1, As shown in Table 1, our proposed Curv-GT consistently achieves the best results over three wrinkle manifold benchmarks. The non-graph-bast based models like ResNet and DeepONet struggle with the p-Laplacian Diffusion equation as they pre- dict with node features and global 3D coordinates, failing to capture features from neighbors and in addition, solely relying on 3D coordinates limits solving the PDEs in R&3 instead of M, which lose the geometry completely. Tradi- tional graph models like GT with various different types of attention are not able to perceive the geometric structure since these attentions are all based on node fea- tures. Spectral methods like GCN and GINO can yield large errors as they neglect the local spatial structures. Geo- FNO and Transolver give rise to relative small errors among the baseline methods, indicating the usefulness of geome- try information. However, the performance gaps between Geo-FNO, Transolver and Curv-GT also show that directly learning the mapping from parameter space to the manifold or merely leveraging the extrinsic geometry is not sufficient for solving PDEs on manifolds. The experiments on more manifolds are available in Appendix D.
[Vu( my (f(G)), glu
(23)
where /' maps the vector field on faces to that on vertices
and the inner product in each tangent space at is are defined as those in constant curvature spaces.
Datasets. The previous methods (Li al., 2023a) select function u(x, t) on a geometry object and compute its source terms (a, t) by the corresponding equations to ob-
tain a data tuple (u(*), f();; 1(.+ Such an approach is out of the following considerations: (i) Popular datasets (Li et al., 2023a) are limited to Euclidean spaces, which does not match the task we focus on; (ii) Convergence and ac- curacy of numerical methods on general manifold cannot be guaranteed. Hence, we generate the dataset by selecting collection of functions on certain parametric surface in advance, and closed-fomn computations on parametric man- ifolds make the dataset more reliable. In this paper, various time-dependent PDEs on different manifolds are studied. More details are available in Appendix B.
Baselines and Implementation Details. To showcase the efficacy of our proposed Curvature-awee aware Attention, we equip it with the Graph Transformer and denote the result- ing method as Curv-GT. We evaluate Curv-GT against the following neural PDE solvers including, GCN (Xu et al., 2023), GAT (Veličković et al., 2018). DeepONet (Lu et al., 2021; Jin et al., 2022), Graph U-Net (Gao & Ji, 2019), Graph
Ablation on Curvature Geometry. The performances of different methods with and without our proposed curvature-
8

---

Curvature-aware Graph Attention for PDEs on Manifolds
Table Performances of different methods with/without curvature-
aware attention for p-Laplacian diffusion on the wrinkle manifold.
Model
Configuration
(76(
GAT
Non-Curvature-Awan
8.47+2.82
Sqare the
GAT
Direct Concatenation
6.92
GAT
Linear Mapping
4.57=0.70
6x
GAT
Curvature- Aware
3.41=0.59
GT
-on-CervaatureAAaare
9.70=4.24
4x30
GT
Curvature-Awary
1.58=0.84
GNOT
Non-Curvature-Aware
1.39+1.92
3x300
haids-
3
GNOT
Curvature- Aware
0.9220.31
heads-5
2x300
Maximum Depth
Table Repeated experiments on a torus with different subtree partition schemes.
Figure 8. Performance change against the number of heads and the maximum depth of subtrees d. the shaded areas around the line indicate standard deviations.
Subtree amount
33.51#1.61
Subtree scale
Mean
30.6341.44 147
Standard error
1.9240.18
20.0
GAI
_ loss (%)
1.6942.85
Aveeage Thiining Time par Epoch Sreasld
17.5
Transofver
4 loss (%)
1.93+2.82
GNOT
15.0
CINO
Curv GAT
aware attention are reported in Table 2, including GAT, GT, and GNOT. It shows that our proposed curvature-aware attention can outperform their non-curvature- coun- terparts by large margins. Besides, Table 2 also presents the performances of GAT by directly concatenating (Direct Concatenation) the point curvatures to node features and aggregating information with shared linear mapping (Lin- ear Mapping). It can be observed that the results are much worse than our proposed curvature-aware attention.
12.5
10.0
100
400
900
1600
2500
Groph
Ablation on Multi-Head Attention. In this experiment, we aim to verify the efficacy of the proposed muli-head curvature-aware attention. Figure 8 presents the perfor- mance change against the number of heads C and the maxi- mum depth of subtrees d. It demonstrates that 1) the increase in the number of heads can enhance model accuracy and stability, and limits the model learning capability if its value is too small; 2) the performance drops if is too large because it cannot capture the local features well with large depth. In conclusion, a proper depth should be associated with the specific structure of discrete manifold.
Figure Single-epoch training time of different models.
6. Conclusion and Future Work
In this paper. curvature-awar attention is proposed to act as an effective and efficient injected bias for solving PDEs manifolds. It is achieved via a combination of parallel transport and tensor fields on manifolds. Our proposed curvature-awarr attention can be used as a replacement for vanilla attention, and the experimental results show that it can significantly improve the performance of existing methods for solving PDEs manifolds. In the future, would like to explore how to extend our curvature-aware attention to address more general tasks on manifolds.
Training Time Comparison. In this experiment, run dif- ferent neural PDE solvers on elliptic paraboloids in different resolutions. Figure presents the average one-epoch train- ing times of different methods varying against the graph size (VV). It can be observed that our proposed Curv-GAT is still faster than GINO. In particular, the one-epoch training time over graph with size of 2500 is around 15 seconds, which can satisfy the practical requirements.
Acknowledgments
This work is supported by the National Natural Sci- ence Foundation of China under Grant No. 62206074 and 62472125, Guangdong Basic and Applied Basic Re- search Foundation under Grant No. 2025A1515012933 Shenzhen College Stability Support Plan under Grant No. GXWD202208 1100001111323000111 Technology Program No. JCYI20241120212350000 and ZDSYSS2023062601100000
Stability of Subtree Partitioning. The subtree partitioning may be very imbalanced due to random selections. Our 100 trials on a 1.024-nod torus show stable results, as shown in Table
Shenzhen Science and

---

Curvature-aware Graph Attention for PDEs on Manifolds
Impact Statement
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 2023.
We study the neural PDE-solvers by exploring the intrinsic geometry of manifolds. Our work aims to advance scientific computing with machine learning techniques and will not have a negative impact on the community.
Li, Z., Kovachki, N., Azizzadenesheli,, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neu- ral operator for parametric partial differential equations. In International Conference on Learning Representations (ICLR), 2020a.
References
Anderson, J. W. Hyperbolic Geometry. Springer, 2nd edi- tion, 2005.
Brandstetter, J., Worrall, D. E.,, and Welling, M. Message passing neural PDE solvers. In International Conference on Learning Representations (ICLR), 2022.
Li, Z., Kovachki, N., Azizzadenesheli, K.. Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Multi- pole graph neural operator for parametric partial differen- tial equations. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurlPS), 2020b.
Cao, S. Choose a transformer: Fourier or galerkin. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Li, Z., Kovachki, N., Arizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In Advances in Neural Information Processing Systems (NeurIPS), pp.
Ciani, S., Henriques, E., and Skrypnik, I. Fine boundary continuity for degenerate double-phase diffusion, 2024.
Gao, H. and Ji, S. Graph u-nets. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
2020c.
Li, Z.,, Huang, D. Z.,, Liu, B.,, and Anandkumar, A. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Re- search, 2023a.
Ghojogh, B., Crowley, M., Karray, F., and Ghodsi, A. Ele- ments of Dimensionality Reduction and Manifold Learn- ing. Springer Cham, 2023.
Li, Z.. Kovachki, N. B., Choy, C., Li, B.,, Kossaifi, J., Otta, S. P.. Nabian, M. A., Stadler, M., Hundt, C., Azizzade- nesheli, K., and Anandkumar, A. Geometry-informer neural operator for large-scale 3d PDEs. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023b.
Gupta, G., Xiao, X., and Bogdan, P. Multiwavelet-basse based
operator learning for differential equations. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., Cheng, Z., Song, J., and Zhu, J. GNOT: A general neural operator transformer for operator learning. In Proceed- ings of the 40th International Conference on Machine Learning (ICML), 2023.
Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations' operator learning. Transac- tions on Machine Learning Research, 2023c.
Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations' operator learning. Transac- tions on Machine Learning Research, 2023d.
Jacobson, A. Algorithms and Interfaces for Real-Time De- formation 2D and 3D Shapes. PhD thesis, ETH Zurich, 2013.
Li, Z., Shu, D., and Farimani, A. B. Scalable transformer for PDE surrogate modeling. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023e.
Jain, P., Qu. Z., Chen, P. Y., and Stein, O. Neural monte carlo fluid simulation. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPH '24. Association for Computing Ma- chinery, 2024.
Liu, X., Xu, B., and Zhang, L. HT-net: Hierarchical transformer-ban operator learning model for multiscale PDEs, 2023.
Jin, P., Meng, S., and Lu, L. Mionet: Learning multiple- input operators via tensor product. SIAM Journal on Scientific Computing, 2022.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nat. Mach. Intell., 2021.
Jost, J. Riemannian Geometry and Geometric Analysis. Springer International Publishing, 7th edition, 2017.
10

---

Curvature-aware Graph Attention for PDEs on Manifolds
Naber, G. L. Topology, Geometry, and Gauge Fields. Springer, 2nd edition, 2011.
Wu, H., Hu, T., Luo, H., Wang, J., and Long, M. Solv- ing high-dimensional pdes with latent spectral models. In Proceedings of the 40th International Conference Machine Learning (ICML). 2023.
Nickel, M. and Kiela, D. Poincaré embeddings for learn- ing hierarchical representations. In Advances in Neural Information Processing Systems, 2017.
Wu, H., Luo, H., Wang, H., Wang, J., and Long, M. Tran- solver: A fast transformer solver for PDEs general geometries. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
Nickel, M. and Kiela, D. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In Proceed- ings of the 35th International Conference on Machine Learning (ICML), pp. 1779-3788, 2018.
Xiao, X., Cao, D., Yang, R., Gupta, G., Liu, G., Yin, C.,
Balan, R., and Bogdan, P. Coupled multiwavelet operator learning for coupled differential equations. In Interna- tional Conference Learning Representations, 2023.
Peng, W., Varanka, T., Mostafa, A., Shi, H., and Zhao, G. Hyperbolic deep neural networks: A survey. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2022.
Xu, X., Zhao, X., Wei, M., and Li, Z. comprehensive review of graph convolutional networks: Approaches and applications. Electronic Research Archive, 2023.
Rahman, M. A., Ross, Z. E., and Azizzadenesheli, K. U- NO: U-shaped neural operators. Transactions on Machine Learning Research, 2023.
Yau, S.-T. and Gu, D. X. Computational Conformal Geom- etry. Higher Education Press, 1st edition, 2008
Raissi, M. Deep hidden physics models: Deep learning of nonlinear partial differential equations. Journal of Machine Learning Research, 2018.
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Stephani, H. Relativity: An Introduction to Special and General Relativity. Cambridge University Press, 3rd edition, 2004.
Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. Graph transformer networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Taubin, G. Estimating the tensor of curvature of a surface from a polyhedral approximation. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 1995.
Zhang, B., Luo, S., Wang, I.., and He, D. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Tripura, and Chakraborty, S. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 2023.
Zwiebach, B. A First Course in String Theory, Cambridge University Press, 2nd edition, 2009.
Tu, L. W. Differential Geometry: Connections, Curvature, and Characteristic Classes. Springer, 1st edition, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin. I. Atten- tion is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Veličković. P., Cucurull, G.. Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. In International Conference on Learning Representations, 2018
Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., and Benson, S. M. U-fno an enhanced fourier neural operator-basee deep-leaming model for multiphase flow. Advances in Water Resources, 2022.

---

Curvature-aware Graph Attention for PDEs on Manifolds
A. Formulas on parallel transport
A.1. Proof of Theorem 4.1
Note that the tangent space at a point on the sphere is isometric to that at any other point. Combined with the fact that parallel transport preserves the inner product. Thus eg keeps orthogonal to the tangent vector the geodesic v and its length keeps fixed due the isometric property. Likewise, ey must rotate along with 7(1) and the orthogonal property makes it be ÿ.
To make it self-contained, we offer complete proof for Theorem 4.2 and Lemma 4.4 whereas the proofs for Lemma 4.3 can be found in (Anderson, 2005),
A.2. Proof of Theorem 4.2
The metric of 2 is g Ky² dr & dx + y-axis is a geodesic as said in (Anderson, 2005). One shall compute the Christoffel symbols via Eq. (38) to establish the differential equations of parallel transport.
Kya dy & dy, namely, 1 k Any segment parallel to
0 - )
m
y
r
(24)
1
One can verify that U y cos 00, + y sin sin (0 is constant) is indeed parallel transported vector field by
Va, U =0 0
(25)
A.3. Proof of Lemma 4.3
The isometry property of Möbius transform group and Poincaré half-plane can be found in (Anderson, 2005), whereas the isometry of translation holds by its definition.
A.4. Proof of Lemma 4.4
Let {x²), { denote the coordinate systems pushforward sp. TM TN gives pp(
N and ip M - / be the isomorphic mapping. The corresponding as
8
dz DF!
The isometric condition means Vp M, YU, T,M, V) "(
v. =( (V)loo)) and equivalently,
0
0
96
g
dr
Ox
(26)
x
(27)
Ox
Ori
0mm
0
3
0
(28)
Or dim
m Or
0
0
9
(29)
Ox Ozá
0
m. n
Ozi Oz) Imn
(30)
Due to the uniqueness of gā we can show the identity
m 3x
mn
Ozi Dāi g
(31)
12

---

Curvature-aware Graph Attention for PDEs on Manifolds
via direct verification by showing gmm 9nl
1x Ox
Or dzg
gmnn 9n!
dis Oās 96)
Or Ox! 9ya
(32)
Ox" 05
Oz Ox! 99 Amn
(33)
Ois 0xm
2xm 0āg
Oze
9d 9pq
(34)
d5i Ox
2xm 8i9
(35)
O5 dx!
2xm 0i9
8%
(36)
Oi Ox!
87
(37)
Recall the definition of Christoffel symbol
T in
29
(grm,i + Dim,r (immr)
(38)
and note that
S 0
(
1 0r
Oi Ox Jap.j
1 O dx dxm
O5
a
O
O5
g
Ori 9rm
Ox
die
Or
dxm 900
+
Oxi
Ia.
Oej
Ox
2xm
(39)
Substitute ghT, irmm Dim.rs Dirmm with the metric tensor in W and we yield the Christoffel symbol transform formula:
Oz dr
a
82 x
i&k
r% 0,8
(40)
Oz
O55
0i
We first show the identity holds for any isometric mapping so.
6x p (VxY)= (Vx))=ppx(ppp))
(41)
Let X x' Dzd and Y y' e the left-hand-side becomes:
6 (VxY)= (x (y' ②
(42)
pa Oyy ②
+ T is aze
(43)
Ox Oxs
Oy'
Or s.
+ 3/ T
1- + +a'r'
(44)
Ord
Fx
Oy' t5
①
m θ
(45)
dr dxt 0am
Dx 7am
_x¹ (Oy' Oyg' -50 Oxt Ozs
mam
mrm θ
+ 1
(46)
Oxk mmm
13

---

Curvature-aware Graph Attention for PDEs on Manifolds
And the right-hand-side is:
a dos
xx.x(p.)) = V (ot b
(47)
dik θ
(48)
ezt
Oas 0itk
Oi
Oit
(49)
Oxt
D21
8
Oas 0ā*
0z
Oy
dz 0
05
+ v
Ii
(50)
Fz
Oi Ozd
0.
Or'O.
Ozi
0z
Oyd O 0
Oath
x²
+ v
I'm Uk a..m
(51)
Dxt
aid Oad Oāk
dr
x² 0z
Dy a
ka
y
I
(52)
0x
O Ou3
Ox
0mm
It suffices to show that
Oy 0.
8mm
Oā!
Oyp 8mm
05
T
+ y
I'm
(53)
Ox Ozá
Or
Ox.
Oi! Oxl
Oxs
First, note that
By oz' OF d2
0yy dg
and it is thus reduced to show
1 drm
a 0i Im
(54)
1) Oz
O. Ozi
And the identity holds since
oz 0
oi a 0.m
(Tr, Ox dx
0xx7
Im
(55)
Oz dzl
Or Oxd Or
dz 0g
0im
Oi Oxa
@ak
2x
kā dam
Ox 0ā! )
135 0
1xx
r7
+
(56)
Ox
Ox 0i
Oxi 05%
Oas Oxy
05%
d
ki d
067
Ox7 17 07 63
+
(57)
dzi
dr
Oxn 17
(58)
r d..
(59)
Ork
Whence for any parallel transported vector field (s) along geodesic ? parametrized by are length E [0, 7 M, we
must have V O. Note that the isomorphic mapping &o maintains are length and we can that 0 y.. ( (V;U) p. (.(o))) (Q.U), Ys E [o, T). Therefore UpU is also a parallel transported vector field along (p()). Note that
U()) op (z:
(60)
It is then proved that the pushforward and the parallel transport commute.
A.5, Proof of Theorem 4.5
Consider the isometric mapping, ip o Ta x. + fj ト - = f + gi. And we have:
-=
z2 + 2 9 x² + y2
(61)
14

---

Curvature-aware Graph Attention for PDEs on Manifolds
(a) Sphere
(b) Elliptic
(c) Hyperbolic
(d) Torus
(c) Wrinkles
Figure 10. The manifolds studied in the datasets.
Table 4. Various surfaces are used as background manifolds and PDEs listed in Section B. are generated on all of them.
Surface
Sphere
Elliptic Paraboloid Hyperbolic Paraboloid
Torus
Wrinkles
Vertex amount
162
2,500
2,500
512
1,225
Edge amount
960
14,406
14,406
3,072
6,936
Face amount
320
4.802
4,802
1,024
2,312
Train set size
396
396
396
396
396
Test set size
99
99
99
99
99
Then 7.H? - TzH is:
27
2
(62)
8y
y²²
2xy
(63)
2
22
-.ry
y²
B7
x
y
2xy
|
2
(64)
2xry
(65)
05
Then the pushforward of T-a o sp. ((x(7))) o sp. To is given by id oJ a, o ds id oJ 1zuta o id.
B. Benchmark Details
B.1. T'me-dependent Equations
Consider the Riemannian manifold (M, and we have the following common useful operators: the gradient operator V. the divergence operator div and the Laplacian-Beltrami Beltrami operator A.
Vu = g Ou
(66)
Ixi Oxi
div Y det - à
(ldetglby))
(67)
Ord
Ou (Idetg)
Au = det gl
(68)
Or
Ord
where X = =y d
Heat Equation. The governing equation of a temperature field on the manifold M is:
du
Øt (I,
Au(z, t) = f(x,t)
(69)
15

---

Curvature-aware Graph Attention for PDEs on Manifolds
where u is the temperature of E M at time I, A is the Kaplacian-Beltrami operator associated with M and f is the source term. The neural operator is required to predict the temperature field time t + with observations of u(x, t) and [(x t) time t.
Wave Equation. Wave equations can be used to depict the behaviors of electric field intensity E(z, t) and magnetic induction (xx, 1) etc in electro dynamics and Schrödinger Equation in quantum theory. The wave equation adopted in the dataset is of the simplified form:
S2u 062 (a. Au(z,t) fxxtt)
(70)
Compared with Eq. (69), it only differs in the time-dependent term. Note that the second-order derivative offers an extra degree of freedom and thus the neural operator is required to predict the field at time t + 1 given two observations of u(x, t) and (a, !) at time I and t - 1.
Non-Isotropic Diffusion Equations. The diffusion term in Eq. (69) is linear and isotropic, however, this may fail to reflect the phenomenon in some heterogeneous materials since the diffusion velocity in different directions may differ. The adopted governing PDE is given by:
0²u
0r2 (z, t) div(a(x)Vu(z(
f(x,t)
(71)
Therefore, the neural operator takes observations of uxx f), f t) at time t as inputs and outputs a prediction for u(x,& () at time / ++1 1. In some scenarios, a(x) is modeled to be Vuip-2 and the diffusion term is termed as p-Laplacian A
div( Viv(
(72)
The associated PDE turns to be:
Ou Øt (z. 1) - A, = S (xx 1)
(73)
In a more complicated scenario, for example, a generalized non-Newtonian fluid can be modeled after an anisotropic dissipative potential restricted to two power laws (Ciani et al., 2024), which is:
Ou (z,t) div ((Vuu)p-2 + a(x, t)) = t(x
(74)
àt
C. Implementation Details
The experiments were conducted on Ubuntu 20.04 LST equipped with 4 NVIDIA RTX A6000 GPUs, each with 48 GB of GPU memory. Our method is implemented with Pytorch 1.13 and Python 32 The initial learning rate y of each model is selected from the set (i x 10-3 i E j (1, 2)3 to optimize its performance. Besides, the Adam optimizer is used with a decay rate 81 = 0.9. Datasets are partitioned into train-set and test-set by ratio 0.8 randomly. A validation set is then created with ratio of 0.1 from the train-set. The batch size is fixed at either 10 or 20 up to the graph scale. Specifically, for experiments on the sphere, torus and wrinkle manifold, the batch size is set to 20 while 10 is adopted for the others. All models are trained for 300 epochs. The one behaving best on the validation set is selected to participate in the comparison. The curvature threshold is set to 10 3, in order to both avoid numerical instability and capture the curvature information of most vertices. All curvature-avee aware models use a 3-head curvature attention mechanism. resulting the attention head amount D= 3. Guided by Figure 8, proper subtree partition maximum depth d is to For instance, in that kind of partition, the wrinkle is then decomposed into 27 subtrees.
D. Experiments on More Manifolds
The performance of different methods for solving PDEs on more manifolds, including the Sphere, Elliptic Paraboloid, Hyperbolic Paraboloid, and Torus. The results are reported in Table 5 and visualized in Figure 12-14. Our proposed Curv-GT can achieve the best results in most cases.
We further conduct experiment on heat equation the canonical Stanford Bunny (a complex manifold beyond toy examples), in which the dataset is obtained by finite difference method. The result is shown in Table 6 and visualized in Figure 15.
16

---

Curvature-aware Graph Attention for PDEs on Manifolds
Frekiee
-5
Prokeke. 6 "l
-23
Figure 1I. Isotropic diffusion on the wrinkle manifold
uifes
Prodidoo 2014 Teachadii Pasliiaad
Obravad Saato
Boor
Rdlaiie Brer
Figure 12. p-Laplacian hyperbolic paraboloid
Greand Trabs fo Oe Bliak Paabekid
Obsavee Tempenaun
Cooca (ir
Tradiato 2
Obemri Stams
time
Figure 13. Heterogeneous diffusion an elliptic paraboloid
17

---

Curvature-aware Graph Attention for PDEs on Manifolds
Greed Tr
(heend Way e
Greed Tusta
Prwtlae (11A De ToeK
Gleeved St
Bddiri Bour
Figure 14. Wave equation on a torus
Table S. Performance of different methods on more manifolds.
Isotropic Diffusion on Elliptic Paraboloid
Wave Equation
Isotropic Diffusion on Sphere
Model
on Torus
(%)%
H (%)
(²3%)
(*)
L2(%)
H H'(%)
DeepONet
19.456+2.168
34.41843.577
0.005#0.002
0.007=0.002
15.929#1.529
24. 24.119442 19#3.319
FourierType
2.399:0.479
9.301+1.960
0.018+0.004
0.025=0,006
0,875+0.115
1.605+0.151
GalerkinType
3.082+1.107
11.49914.600
0.027=0.013
0.043+0.021
0.932+40.376
1.72740.562
MKGN
5.25240.623
27.15242.998
0.120#0.019
0.19440.030
5.571
10.13540.521
GNOT
0.18040.00 0664
0.345+0.229
0.00740.003
0.009#0.003
0.06010.025
0.094
GINO
19.823+2.622
29.597+2.509
0.31340.035
0.341+0.037
4.898+0.432
6.450#0.265
Transolver
0.099+0.041
0.367+0.188
0.135+0.043
0.269+0.042
3.002+00.63
4.986*
Curv-GT (Ours)
0.073+0.013
0.343+0.048
0.00240.001
0.002+0.001
0.080+0.011
0.131
p-Laplacian Diffusion on Hyperbolic Paraboloid (2(%)
Heterogeneous Diffusion on Elliptic Paraboloid
p-Laplacian Diffusion
Model
on Torus
H (%)
L3(%)
H' (%)
L3(%)
H (')
DeepONet
43.768+1.261
53.701+1.683 1.633
11.6231..022
10. 1.443373
64.574+1.699
40.440+6.05
FourierType
15.22141.761
21.972+1.968
16.117#1.142
16.575#1.943
34.360+7.227
70.818+1.910
GalerkinType
16.850+2.951
23.58743.820
17.428+1.46
16.595+2.513
68.499+19.88
75.289+17.174
MKGN
10.649#0.855
39.236+2.943
3.260+0.628
2.658+0.447
48.342+8.06
58.26648.042
GNOT
0.843=0.135
1.838+0.341
2.22840.071
0.777+0.036
4,917
5.445+1.559
GINO
49.120:1..77
46.68340.918
7.078+1.00 .068
6.115+1 1.230
58.005+6.218
62.090+6.777
Transolver
1.137+0.192
2.19220.392
2.10040.070
0.653+0.067
2.903+1.444
1.356
Curv-GT (Ours)
0.79920.065
1.38340. 108
1.973+0.149
167740.002
0.73640.
1.050+0.130
Table 6. Different models are compared on the Stanford bunny heat equation dataset.
Model
Curv-GAT (ours)
GAT
GNOT
Transolver
1,2 loss (%)
0.0058+0.0008
0.01 16+0.0022
0.0106#0.0003
0.0102+0.0006
18

---

Curvature-aware Graph Attention for PDEs on Manifolds
Ground Truth
Model Prediction
Relative Error
Figure 15. Heat equation Stanford bunny.
19


Reviews:
[{"review_id": "u5imgvVPG3", "reviewer": "Reviewer_y7yT", "summary": "The paper proposes a curvature-aware graph attention architecture and applies it to produce a supervised neural time-stepper for PDEs on surfaces embedded in $\\mathbb{R}^3$. This architecture leverages the concept of parallel transport on surfaces, and proposes embedding an edge into a constant curvature surface to perform this parallel transport. It builds off of existing frameworks for Graph transformers. \n\nThey test their method against many other graph-based network models on the same task, across 4 model geometries and 3 PDEs and show superior performance. Ablation studies also seem to show that the method improves other architectures as well.\n\n## Update after rebuttal:\n\nI appreciated th clarifications from the authors, but I still find the use case for such a method to be relatively niche, and some of the design decisions to be a bit strange. Hence I will keep my score as is (weak reject). If there is sufficient enthusiasm from the other reviewers, I would not stand in the way of ultimate acceptance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response To Reviewer y7yT\nThank you for your time and constructive feedback. We appreciate your thorough evaluation and valuable suggestions, which have helped improve our work. Below is our point-by-point response.\n\n>**Q1.**  Why use this GNN for surface PDEs? It is less accurate than FEM. It requires training with precomputed numerical solutions and retraining for any geometry changes due to its surface-specific nature. \n\n**A:** Reviewer DBfv raised similar concerns. Due to space limitations, we kindly refer you to **Q2 in our response to Reviewer DBfv**. We will revise the manuscript to incorporate these examples, further highlighting the motivation behind our proposed method.\n\n> **Q2.** Can it accomodate node features that cannot be interpreted as elements of the tangent space above a node?\n\n**A:** We would like to clarify that our method **can actually** solve PDEs where variables are not tangent vectors. In fact, the PDEs in our experiments involve scalar fields—for instance, in the heat equation, the node feature $u(\\mathbf{x})$ represents temperature. Since parallel transport acts on vectors rather than scalars, we **bridge this gap by constructing a natural tangent vector field associated with the function and manifold, namely, the discrete gradient field of temperature (as noted in Remark 2, line 256)**. Furthermore, since the gradient of tensor products can be defined, our approach can theoretically extend to PDEs with tensor variables.  \n\nWe appreciate this insightful question and will update the manuscript to make it clearer.\n\n> **Q3.** The embeddings themselves seemed a bit unclearly defined to me. For example, in the spherical case, you consider the two edge endpoints as vectors in $S^2$, but it was not clear how you normalize these and with respect to what center. This would seem to be crucial. (On a related note: it's surprising that one did not just scale the sphere or pseudosphere to account for fractional curvature, as would naturally arise for $K|_u$)\n\n**A:** We are sorry for the confusion caused. It would be better to first clarify our embedding methods before resolving your concerns in \"Claims and evidence I\".  \n\nFor a node $u$ on a mesh $\\mathcal M$, we first compute its Gaussian curvature $K|_u$ at $u$. If $K|_u>ε>0$, then we embed the tangent space $T_u\\mathcal M$ at $u$ into a sphere $S$ with curvature $K|_u$ which is tangent to $\\mathcal M$ at $u$. Therefore, the edge connecting $u$ and $v$ is embedded into $S$ such that the Euclidean distance between $u,v$ equals the spherical distance (**embedding in isometric sense**). In this way, $\\mathcal M$ is locally approximated by $S$ at $u$. Therefore, **we do not need normalizations and we do scale the spheres based on the curvature estimate**. It is the same for pseudospheres.\n\nWe will revise the manuscript to further clarify these in our next version.\n\n> **Q4.** It was based off of a curvature estimate at a vertex, but one was philisophically aimed at embedding the edge. Can you explain your reasoning here?\n\n**A:** Sure. It seems embedding at a node $u$ cannot reflect the curvature of an edge $(u,v)$ at first sight. However, in the context of attention mechanism, what matters is how to distinguish the neighbors in a neighborhood. Each $(u,v)$ has a different length and thus the vector will rotate differently with $K|_u>0$. **A feature tangent vector becomes different along different edges due to $K|_u$.** In this sense, the net can discern edge differences based on $K|_u$.  \n\nMoreover, under a mild mesh assumption, the geodesic between $u$ and $v$ can be approximated by $K|_u$. This is because, the Jacobi field $J(t)$ has an expansion $|J(t)|^2=t^2-{1\\over 3}\\braket{R(X,Y)X,Y}t^4+o(t^4)$ where $R$ is the curvature tensor at $u$ and $X,Y\\in T_u\\mathcal M$. Thus, the edge curvature can be reflected by $K|_u$ if $(u,v)$ is short enough. \n\n> **Q5.**  If given an explicit triangulation, why not leverage prior works that come up with an explicit notion of discrete connection?\n\n**A:** We have read the *Globally Optimal Direction Fields* you suggest and find it less suitable in our model. Discrete connections require geodesic estimation (Eq.(2)), but **as no free lunch in discretization, discrete geodesics must lose some smooth-case properties**. In our implementation, the geodesic we estimate **is obedient to its Euclidean distance**. Besides, **triangulation is not a must in our framework**. So it is just as you say, there is no right or wrong. \n\n> **Q6**. It might be nice to include references to the many recent papers that use a neurally-parameterized space of functions to solve PDEs in an unsupervised fashion.\n\n**A:** Thanks for pointing out the useful references. We will include them in the *Related Work* section in the next version.\n\nThanks again for your time!", "claims_and_evidence": "The method takes a well-motivated tack overall, but I did have some more detailed technical questions on the approach, which caused me to question the specifics of the design. \n\n1. I found the embedding into constant-curvature surfaces to be rather coarse. Moreover, it was based off of a curvature estimate at a vertex, but one was philisophically aimed at embedding the edge. Can you explain your reasoning here? If given an explicit triangulation, why not leverage prior works that come up with an explicit notion of discrete connection, e.g., \"Globally Optimal Direction Fields\" by Knoppel et al.\n\n2. The embeddings themselves seemed a bit unclearly defined to me. For example, in the spherical case, you consider the two edge endpoints as vectors in $\\mathbb{S}^2$, but it was not clear how you normalize these and with respect to what center. This would seem to be crucial. (On a related note: it's surprising that one did not just scale the sphere or pseudosphere to account for fractional curvature, as would naturally arise for $K|_u$). \n\nUltimately, there is no right or wrong on the design choices above, so I should say that on the empirical evidence, I felt that the evaluation and experiments were sufficient for comparison to other graph-based models, and did show significant improved performance.\n\nAs for the technical arguments of 4.3, I did not read them carefully, but they seemed correct. I would also like to note that these are not novel in any way, and could have merely referenced existing texts on Riemannian geometry.", "methods_and_evaluation_criteria": "See claims and evidence above.", "theoretical_claims": "See claims and evidence above.", "experimental_designs_or_analyses": "See claims and evidence above.", "relation_to_broader_scientific_literature": "The paper is one of several methods aimed at graph-based methods for learning of PDEs and I did not see any glaring omissions amongst the references.\n\nIt might be nice to include references to the many recent papers that use a neurally-parameterized space of functions to solve PDEs in an unsupervised fashion, e.g., \"Neural Monte Carlo Fluid Simulation\" by Jain et al. & \"Model reduction for the material point method via an implicit neural representation of the deformation map\" by Chen et al.", "essential_references_not_discussed": "See above on relation to broader scientific literature.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "Explanation of rating: I gave the paper a borderline reject proposal, due to my uncertainty on the utility of such an approach (see below), and concerns on the technical motivations for the approach (see questions in \"Claims and Evidence\"). On the positive side, the experiments seem to show significant improvement in the application domain over other methods in its class.", "questions_for_authors": "1. Why would you use such a network to solve a PDE on a surface? I can understand that a trained network is faster than perhaps a more standard numerical solve (like FEM-based). But one must train the network in the first place, presumably with many computed numerical solves. Moreover, any such method is sure to be less accurate and the network is tied to the specific geometry of the surface and would require retraining for any modified geometry.\n\n2. How could this method accomodate node features that cannot be interpreted as elements of the tangent space above a node? It's unclear to me how this could be done, and whether this restriction is vital to the application at hand. In other words, it seems like a well-motivated network model may well take features that cannot be interpreted as elements of the tangent space, so this capability would seem to be very desirable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HvZ8HeQFif", "reviewer": "Reviewer_nE4H", "summary": "This paper focus on solving pdes on 2-dim manifolds. It generalizes message passing algorithms to manifolds by adding Gaussian curvature in to consideration. It approximate the complex manifold by constant curvature surfaces in Eq. 11. Such approach using parallel transport on constant curvature surfaces is a better approximation than move vectors in Euclidean spaces. I really like this approach, since constant curvature is one order higher than Euclidean space. The experiments (Fig.8) considered both positive and negative curvatures. Manifold with both positive and negative curvature is also studied.\n\nWeakness: Such approach requires constant curvature. However, it cannot be generalized to high-dim manifolds since in line 139, section curvature will be different that Gaussian curvatures.\n\nIn summary, I think considering curvature in solving message passing nn is an interesting task and this work solves this task in 2-dim space. I would rate as accept and I will keep learning this work and update my rating during the discussion period. Looking forward to the reply from the authors!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response To Reviewer nE4H\nWe sincerely thank you for your insightful feedback and recognition of our work. We remain fully open to implementing any additional revisions. Below, we address each of your comments in detail.\n\n> **Q1**.  Such approach requires constant curvature.\n\n**A:** This approach only assumes that the surface at a node has approximately constant curvature. Therefore, **the entire surface is not necessary to have constant curvature**, as we solve PDEs on a more complicated manifold *wrinkle* (in Figure 11, line 915). Due to this locality, we cannot directly parallel transport a vector at a node to those too far away since the numerical errors will accumulate. Instead, we turn to a sub-tree partition strategy to balance the trade-off between the computation accuracy and reflecting the curvature changes in a wider range.  \n\n> **Q2**. It cannot be generalized to high-dim manifolds since in line 139, section curvature will be different that Gaussian curvatures.\n\n**A:** Your observation here is actually a crucial limitation of the proposed framework. This approach cannot be directly extended to manifolds in higher dimensions since in that case, the **sectional curvature** is no longer a real number.\n\n**Nevertheless, a slight modification is sufficient to extend it to high-dim cases**. If the manifold is compact, then we can use finite charts (each with a coordinate frame) to describe the manifold. Note that sectional curvature is the extended version of Gaussian curvature in higher-dimensional manifolds, and it is indeed a map $K:T\\mathcal M\\times T\\mathcal M\\to\\mathbb R,(X,Y)\\mapsto K(X,Y)$. On a 2d-manifold, it gives Gaussian curvature if the two input tangent vectors are linearly independent. \n\n**This observation sheds light on the high-dim cases.** For instance, on a 3d-manifold with a coordinate frame $\\{X_1,X_2,X_3\\}$, there are 3 sectional curvatures, $K(X_1,X_2),K(X_2,X_3),K(X_3,X_1)$. We can treat each of them in the same way as the 2D case and combine them. But how to compute them consistently and efficiently requires further consideration. We leave it for our future work.\n\nWe will update the manuscript to incorporate the above discussion. Thanks again for your time!", "claims_and_evidence": "-", "methods_and_evaluation_criteria": "-", "theoretical_claims": "-", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjCRZlrSHS", "reviewer": "Reviewer_Ao56", "summary": "The authors propose a new PDE-solver based on neural nets for PDE's on manifolds. They claim that taking into account the curvature of the manifold plays a significant role in computing accurately the dynamics of the process to solve. \nThe authors align the tangent spaces on a manifold via parallel transport use tensors instead of matrix multiplication. This yields  a curvature aware graph attention mechanism, which is better suited for solving PDEs on manifold. The reasoning in sensible. A good literature survey is presented. Performance evaluation is done on various (toy) manifolds and 3 type of PDEs, including nonlinear ones. The methos outperfomrms all methods, in terms of accurace (both L2 and H1), sometimes by an order of magnitude. The paper is supported by theoretical justifications and proofs, along with additional experiments in the appendix. It appears a good fit for the conference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response To Reviewer Ao56\nWe are sincerely grateful for the time and effort you have dedicated to reviewing our manuscript. Below, we address each of your comments in detail. Should additional revisions be necessary, we are more than willing to make further adjustments.\n\n> **Q1**. Time considerations in the computations.\n\n**A:** Yes, solving speed is crucial, as it is one of the key advantages of neural solvers over traditional numerical solvers. We find that both the training and inference speeds of our model are acceptable.\n\nWe have compared the training times of five neural PDE solvers in Figure 14 (line 973), where our model remains faster than the GNN-based solver GINO. In practice, training time is closely tied to the maximum depth of subtrees $d$ and the number of attention heads. Notably, reducing the number of attention heads to one brings the training speed close to that of Transolver (ICML 2024) and GNOT.\n\nMoreover, once training is complete, neural PDE solvers are much faster than traditional numerical methods in solving forward problems. When it comes to inference speed, there is no significant difference among neural solvers, even on large meshes with around 2,500 vertices. Specifically, as analyzed (line 365), its inference computational complexity is on par with GAT.\n\n> **Q2**.  Have you tried it on more complex manifolds?\n\n**A:** While our current experiments primarily focus on simple manifolds and more complex ones (*wrinkled surfaces*), our framework is designed to generalize to broader geometric settings. In our experiments, we **primarily aim to verify the effectiveness of curvature-awareness via parallel transport** in GNNs, and the results strongly support our assumptions.  \n\nAlthough these manifolds may be considered toy examples, in practice, **many complex surfaces can be decomposed into simpler ones**. These simple manifolds are able to encompass a wide range of common surfaces encountered in game engine design.\n\nWe are looking forward to your reply!", "claims_and_evidence": "Claim better solutions on manifolds should take into account curvature consideartation.", "methods_and_evaluation_criteria": "Evalutiona is quite robust, although on toy manifolds, but with increasing complexity, such as Wrinkles. 3 different PDEs are tested, comparison is against 11 competing algorithms, including very recent ones such as Transolver of ICML 2024. The performance of the proposed algorithm consistently outperforms other methods.", "theoretical_claims": "The theory appears fine (although checked only briefly).", "experimental_designs_or_analyses": "OK", "relation_to_broader_scientific_literature": "Fine", "essential_references_not_discussed": "Refs are fine", "other_strengths_and_weaknesses": "Writing and illustrations are good.", "other_comments_or_suggestions": "--", "questions_for_authors": "* Time considerations in the computations.\n* Have you tried it on more complex manifolds?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k0C4J4GbdD", "reviewer": "Reviewer_DBfv", "summary": "This paper introduces a Curvature-aware Graph Attention method specifically designed for solving PDEs on manifolds. It addresses the limitations of previous approaches that focused on Euclidean spaces or overlooked the intrinsic geometry of manifolds. The proposed method uses fast parallel transport and tensor product on manifolds to fix the original message passing and aggregation process. The authors also introduce a sub-tree partition method to optimize parameter-sharing and reduce computational complexity. Experimental results show this novel attention mechanism improves the performance on solving PDEs on manifolds.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response To Reviewer DBfv\nWe sincerely appreciate your constructive feedback and meticulous evaluation of our work. Below, we provide responses to each point raised. \n\n> **Q1**. In _Preliminaries_ section, there is a typo: (2,0)-tensor $u^∗⊗v^∗$  should be (0,2)-tensor.\n\n**A:** Thank you for catching this typo. We will fix it in the next version of manuscript. \n\n> **Q2.** The motivation for using attention-based GNN to solve PDEs is unclear.\n\n**A:** Reviewer y7yT raised similar concerns. While accuracy is crucial in many applications, **time-sensitive scenarios prioritize speed over minor inaccuracies**, motivating the development of neural solvers like **Neural Operators** in the ML community (see **Related Work: Neural Operator** in our paper).  \n\nA key example is **gaming engines**, where faster rendering enhances real-time FPS. To achieve realistic illumination based on blackbody radiation, the heat equation is solved to determine surface temperature distribution. In this case, **the underlying geometry remains fixed**, and high accuracy is less critical as long as the rendering appears plausible. **Smooth scene transitions (speed) matter far more than pixel-perfect details (accuracy).**  \n\nBesides, our GNN solver is SE(3)-invariant and can operate without direct coordinate input. Even if retraining is required for a new mesh, **it excels in scenarios where the same mesh appears repeatedly under rotations and translations**, which are common in **material science and gaming engines** (e.g., **triply periodic minimal surfaces** in crystals and block copolymers). In these cases, our method outperforms traditional solvers like FDM and FEM.  \n\nWe will revise our manuscript to incorporate these examples, further reinforcing the motivation for our approach.\n\n> **Q3.** The subtrees partition may be very imbalanced. Will this affect the model performance? Will the random selection for the source nodes affect the results?\n\n**A:** Your concern about stability makes sense. We conducted an extra experiment involving random subtree partition **100 times** on a torus with **1024 nodes**. It shows that a rather imbalanced partition is unlikely and the outcomes is relatively stable:\n- Subtree amount: 33.51±1.61\n- Mean and standard error of Subtree Scale: 30.63±1.47, 1.92±0.18\n- $L^2$ & $H^1$ loss(%): 1.69±2.85, 1.93±2.82\n\nOther settings align with the original paper.\n\n> **Q4.** Eq. (17) contains the parallel transports(PT) of tangent vector and covector, but the paper only gives the closed form of tangent vector parallel transport. \n\n**A:** By musical isomorphism, we can identify a tangent vector $v$ as a 1-form $v^♭$, a.k.a. a cotangent vector, by $v^♭(u):=\\braket{v,u}$. The motivation to introduce PT $Γ(η)$ is to use its **inner-product-preserving nature**. Thus the PT of a 1-form is $Γ(η)v^♭(u):=\\braket{Γ(η)v,u}$ and thus is equivalent to PT a tangent vector in implementation. \n\nThus in your example, the PT of a (0,2)-type tensor is $Γ(η)(\\tilde w_1^\\*⊗\\tilde w_2^\\*)(u,v)=\\braket{Γ(η)\\tilde w_1,u}\\braket{Γ(η)\\tilde w_2,v}$.\n\n > **Q5.** For experiments, the wave equation is important for the theory of relativity, which connects tightly with the Lorentz model. Thus, I think this manifold should be considered.\n\n**A:** Yes, the wave equation is crucial in relativity. It will be valuable if our model can also handle it on the Lorentz model, which is, however, not a Riemannian manifold since its metric is **not positive-definite.** This raises questions about whether the tools adopted currently apply to a *psuedo-Riemannian manifold*. This is a constructive suggestion and we will leave it for future work. \n\nFrom your point of view, I understand that it will be more impactful **if it can handle PDEs in mechanics**. Thus, we also examine the **Navier-Stokes equation** for incompressible and viscous fluid with unit density on a sphere: $∂_t\\mathbf u+(\\mathbf u·∇)\\mathbf u=-∇p+νΔ\\mathbf u$. It is insightful in studying the ocean currents on Earth. The experiment setting and dataset generation align with the presented paper. \n\n| Model | $L^2$ loss (%) |\n| --- | --- |\n| Curv-GAT(ours) |**0.128±0.002** |\n| GAT | 1.381±0.006 |  \n| Transolver | 3.314±0.651  |\n| GNOT | 0.542±0.164 |\n\n> **Q6.** If a neighborhood has large-scale paths, the Poincaré half-plane model may not accurately reflect the parallel transmission characteristics on the original surface. \n\n**A:** Perhaps our illustration on embedding is also not clear enough to you, you may refer to **Q3 and Q4 in the response to Reviewer y7yT if needed**. We **embed the surface at a node locally** into a constant-curvature surface, with parallel transport decorated **edge-wise, not path-wise**. Subtree partition also mitigates this issue, ensuring consistent parallel transmission unless mesh quality is rather poor.\n\nThanks again for your energy!", "claims_and_evidence": "Yes. In the *Experiments* section, the results on solving various PDEs are better than other PDE solvers. In the ablation study, the proposed Curvature-Aware Attention module indeed improves the performance of three attention-based graph neural networks.", "methods_and_evaluation_criteria": "Yes. The evaluation datasets, from three important physical PDEs, are really meaningful.", "theoretical_claims": "Yes. I have checked the derivation for the closed form of parallel transport on the sphere and the Poincaré half plane, I think they are correct.", "experimental_designs_or_analyses": "For dataset generation, this paper first discretizes the parameterized manifold, then selects a function $u(x,t)$ and computes the sources term $f(x,t)$ to obtain a data tuple $(u^{(t)}, f^{(t)},u^{(t+1)})$. I think this process makes sense.\n\nThe main results on Table 1, the authors use 2 criteria $L^2$ and $H^1$, which makes results more convincing and comprehensive. \n\nAlthough authors have conducted a model on five different 2-manifolds, the Lorentz model for hyperbolic geometry is neglected. I point out this view since the Lorentz model is tightly connected to the theory of relativity, in which the wave equation plays a vital role.\n\nIn Appendix H, the authors make visualizations for prediction results, which intuitively support the effectiveness of Curvature-aware Graph Attention.", "relation_to_broader_scientific_literature": "The main contribution of this work is that the proposed Curvature-aware Graph Attention replaces vanilla attention on GT and GAT, and it improves the performance on solving PDEs on manifolds. I think this work will affect the methods on solving manifold PDEs, leading them to focus more on intrinsic geometry. Even for the development of graph neural network, this paper may provide some new ideas about geometric GNN.", "essential_references_not_discussed": "No, there are no essential related works ignored.", "other_strengths_and_weaknesses": "Pros:\n1. Considering the manifold's curvature by parallel transport, instead of explicitly encoding curvature by a neural network.\n2. It generalizes the matrix multiplication with the tensor field, combining with the proposed subtree partition to optimize parameter sharing, reducing parameter sizes and computational complexity.\n3. The extensive experimental results can support the contribution claimed by the authors, from quantitative and visual experimental results.\n\nCons:\n1. The motivation for using attention-based GNN to solve PDEs is unclear.\n2. I think some points need to be discussed further: after running BFS with depth $d$, the scale of subtrees may be very imbalanced, will this situation affect the model performance? Will the random selection for the source nodes affect the results?\n3. Eq. (17) contains the parallel transports of tangent vector and covector, but the paper only gives the closed form of tangent vector parallel transport. Under general settings, the two formulas are not the same.\n4. For experiments, the wave equation is important for the theory of relativity, which connects tightly with the Lorentz model. Thus, I think this manifold should be considered.", "other_comments_or_suggestions": "1. In *Preliminaries* section, I think there is a typo: “(2,0)-tensor $u^∗ \\otimes v^∗$”. (s, t)-tensor $T$ means the multi-linear map  $T$ takes s number of cotangent vectors and t number of tangent vectors as inputs, so this example is about (0,2)-tensor.", "questions_for_authors": "1. The method uses local geometry approximation and Gaussian curvature estimation, replacing the complex surfaces locally with constant curvature surfaces. For curvature zero, the surfaces are isomorphic to the Euclidean 2-plane. For positive curvature, the surface is diffeomorphic to a sphere by the Gauss–Bonnet theorem. But for negative curvature, the surfaces are only conformally equivalent to the Poincaré half plane, if a neighborhood on the graph node $u$ comes to large-scale paths or paths around the entire surface, the Poincaré half-plane model may not accurately reflect the parallel transmission characteristics on the original surface. How do you avoid this situation?\n2. In the second line of Eq. (17), the authors want to share the parameters from node $u$ to $w$ by parallel transport, but how do you guarantee the consistency between $\\Gamma(\\eta)_0^t[\\tilde{w}_1\\otimes \\tilde{w}_2]$ and $\\tilde{w}_1\\otimes \\tilde{w}_2$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: General comment. The authors present a novel approach for operator‑style physical learning by leveraging the geometric properties of the data manifold. They extend self‑attention graph architectures with analytically computed curvature measures based on predefined heuristics. Specifically, they select an appropriate curvature model along the path between nodes u and v and apply a parallel‐transport‑inspired mapping to neighbors’ embeddings during message passing.

The strengths of the work: 
- interesting idea of using parallel transport framework to align the representation of the node features on the manifolds\
- interesting idea to apply parallel transport move to the weights of the GNN
- demonstration of the loss improvement when incorporating existing self-attention methods with the proposed approach

Weaknesses
- poor experimental setup - heat equation as well as wave equation are rarely computed on the surface mesh
- absence of comparison with the other methods which use the curvature information - authors compare their method with direct insertion of curvature to the nodes' embedding but there are much more profound method to do so: e.g. https://www.jmlr.org/papers/volume24/23-0064/23-0064.pdf    https://arxiv.org/pdf/2302.08166. Authors only claim Geo-FNO to fail in proper parametrization of complex surfaces but in fact use pretty simple geometries in their experiments.

The reviews were primarily concerned with the following problems:

- how to classify the node based on the local curvature
- overall simplicity of the experimental setup
- problems with the initial graph partition
- inapplicability to the high-dimensional manifolds
- overall motivation to predict the solution of the PDE instead of using a numerical solver

The authors' responses were consistent. In the course of rebuttal they performed experiments on the complex geometry like stanford bunny, and extended the PDE pool by Navier-Stoks equation. 

Overall, this work introduces a compelling geometric perspective to graph‑based operator learning. I believe this paper will make a contribution. I encourage the authors to integrate the rebuttal experiments into the final draft. Despite my big concerns about the full scale experiments I won't interrupt the prevailing positive feedback of the work.

Decision: Weak accept.
Decision: Accept (poster)