Modeling Dynamics over Meshes with Gauge Equivariant Nonlinear Message Passing
Authors: Jung Yeon Park
Lawson L.S. Wong
Robin Walters
Venue: NeurIPS 2023 poster (2023)
Keywords: message passing
dynamics
mesh
symmetry
equivariance
TL;DR: Nonlinear message passing with gauge equivariance performs better than convolutions or attention mechanisms in modeling complex surface dynamics

Abstract: Data over non-Euclidean manifolds, often discretized as surface meshes, naturally arise in computer graphics and biological and physical systems. In particular, solutions to partial differential equations (PDEs) over manifolds depend critically on the underlying geometry. While graph neural networks have been successfully applied to PDEs, they do not incorporate surface geometry and do not consider local gauge symmetries of the manifold. Alternatively, recent works on gauge equivariant convolutional and attentional architectures on meshes leverage the underlying geometry but underperform in modeling surface PDEs with complex nonlinear dynamics. To address these issues, we introduce a new gauge equivariant architecture using nonlinear message passing. Our novel architecture achieves higher performance than either convolutional or attentional networks on domains with highly complex and nonlinear dynamics. However, similar to the non-mesh case, design trade-offs favor convolutional, attentional, or message passing networks for different tasks; we investigate in which circumstances our message passing method provides the most benefit.

Modeling Dynamies over Meshes with Gauge Equivariant Nonlinear Message Passing
Jung Yeon Park, Lawson L.S. Wong', Robin Walters
Khoury College of Computer Sciences
Northeastern University
Boston, MA 02115
edu, lswdccs neu. edu, (park. walleers@horthhasteenn eduu)
Abstract
Data non-Euclidean manifolds, often discretized surface meshes, naturally arise in computer graphics and biological and physical systems. In particular, solutions partial differential equations (PDEs) over manifolds depend critically on the underlying geometry. While graph neural networks have been successfully applied to PDEs, they do not incorporate surface geometry and do not consider local gauge symmetries of the manifold. Alternatively, recent works on gauge equivariant convolutional and attentional architectures on meshes leverage the underlying geometry but underperform in modeling surface PDEs with complex nonlinear dynamics. To address these issues, we introduce a new gauge equivariant architecture using nonlinear message passing. Our novel architecture achieves higher performance than either convolutional attentional networks on domains with highly complex and nonlinear dynamics. However, similar to the non menh case, design trade-offs favor convolutional, attentional, or message passing net- works for different tasks; we investigate in which circumstances our message passing method provides the most benefit
Introduction
Surfaces embedded in 3D space appear many domains such as computer graphics [1], structural biology [2, 3], neuroscience [4], climate pattern segmentation [5, 6), and fluid dynamics [7]. Such objects are naturally Riemannian manifolds but are often discretized into meshes for computational tractability. Unlike other 3D representations such as point clouds or voxels, meshes encode both the geometry and topology the surface.
One important task that naturally arises over meshes is solving partial differential equations (PDEs) over surfaces and -un-Euciddean manifolds [11-14]. classic approach solving surface PDEs to model the domain as a mesh and use the finite element method and the variational formulation [15, 16]. Recently, there has been increasing success in applying deep learning methods accelerate solving PDEs and building differentiable surrogates for dynamics models [17-20]. While there are many effective approaches for gridded data or point clouds, fewer methods exist for leveraging the geometric structure of meshes. Several approaches simply treat the mesh as graph by approximating patches of the manifold as Euclidean [21-23]. However, these approaches do not exploit the geometric properties of meshes, and Verma et al. [24]. de Haan al. [9] have shown that not retaining this information leads to poor performance. One way to express such information is to specify the local
$ Equal advising.
Project website with code: https://jypark0. httppparrr...gttubb.e.... to/hermes
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2018
(a) Mesh
GemCNN
EMAN
Hermes
Figure I: Mesh example. The neighbors vertex b are projected onto the tangent plane to compute their local orientation (0a: 0.) after choosing reference neighbor (gauge). There are 3 flavors of gauge equivariant message passing, cf. Fig. 17 in Bronstein al. [8]. GemCNN [9] computes linear convolutions using the neighbor node features, EMAN [10] uses linear messages with attention, while Hermes performs nonlinear message passing. Self-interactions are omitted for clarity.
gauge symmetries and explicitly model operations that are equivariant to these transformations [6, 9], by using parallel transport and applying gauge equivariant filter.
Although previous work on gauge equivariance for meshes including convolutional (GemCNN [9]) and attentional (EMAN [10]) methods have shown good performance shape correspondence and mesh classification tasks, we find they are inadequate for modeling complex dynamics on meshes, such solving surface PDEs, which can be highly nonlinear. In this work, introduce new architecture, Hermes, that combines gauge equivariance with nonlinear message passing. As Hermes can express linear convolutions or attention, Hermes is strictly more expressive than GemCNN and EMAN. Hermes completes the 3 flavors of gauge equivariant message passing, analogous to graph neural networks [8]. See Figure I for a comparison.
We evaluate Hermes on several linear and nonlinear partial differential equations, and also on shape correspondence, object interaction system, and cloth dynamics. Our experiments show that Hermes outperforms convolutional or attentional counterparts in most domains, particularly on nonlinear surface PDEs. We find that Hermes is more robust to both the fineness of the mesh and surface roughness compared to both GemCNN and EMAN. We further investigate when the added model complexity of nonlinear message passing outweighs the cost. With this work, we hope to help guide practitioners on when to use nonlinear message passing schemes over linear schemes.
Our contributions are summarized follows: 1) we propose novel gauge equivariant, nonlinear message passing architecture, Hermes, for learning on meshes, 2) when evaluated on complex and nonlinear dynamics such as surface PDEs, Hermes outperforms both convolutional and attentional architectures and can generate realistic prediction rollouts, and 3) we investigate in which situations nonlinear message passing should be preferred over convolutional attentional counterparts.
2 Background
In this work, we focus on learning signals over meshes. Similar to de Haan et al. [9], our goal is to define model in coordinates that are intrinsic the 2D mesh instead of using the extrinsic 3D coordinates of the embedding space. This avoids dependence on the 3D embedding, which is irrelevant. However, in order to encode data over the mesh it is still necessary to make choice of local coordinate frame at each vertex. Since this coordinate frame is arbitrary, it follows that the model should be invariant change of coordinates frame each vertex, i.e. gauge equivariant.
2.1 Data over Meshes
The Mesh Datum A mesh M consists of (V,E,F) where V the set of vertices, E (i, is the set of ordered vertex indices i.j connected by an edge, and F = {(i, j. k)} is the set of ordered vertex indices i, k connected by triangular face. Let Nr denote the set of vertices connected to vertex We assume that the mesh represents a discrete 2 2-dimensional manifold embedded in R (i.e. a manifold mesh). Let rú E IR33 be the coordinates of vertex b. We assign normal vector ni, to each vertex b equal to the normalized average of the surface normals of the faces that contain the vertex. The associated tangent plane T,M vertex b the 2-dimensional affine space through Ty orthogonal to the normal ni.-

---

Local Coordinate Frame We adopt the strategy of [9] by defining the local coordinate frame at each vertex terms of reference neighboring vertex d E N/.. See Figure la for a visualization. The reference neighbor defines a positively oriented orthonormal basis {ei, egj for the tangent space T.M.. The orientation of every neighbor u E N, can be represented with polar angles, where 0, is the angle between v and the reference neighbor after projection to the tangent plane. In Figure 1a, 0a = as is itself the reference neighbor and we show the 0g for the other neighbors with respect to the reference neighbor d. See Appendix A. I for more information.
Change of Gauge A different choice of reference neighbor will result in a different positively oriented orthonormal basis. Any two such bases will be related by an element g E SO(2). Here S0(2) is called the gauge group and g the gauge transformation. Note that such a change is local, and a different change gd can be performed at each vertex E V. For example, if we switch from reference neighbor to e, then we induce gauge transformation (Ø) E SO(2) where @ O. 0₂ is the angle between d and c. The orientations are then updated 0g + 0g Φ Vq E NN
Feature Fields on the Mesh Input, output, and hidden features are encoded data over the vertices (frlev or edges of the mesh {epu (p.))). We consider directed graphs and so for the edge feature epg. p is the source and q is the target node. We situate Epg to be at T,M for simplicity as they are often chosen to be vectors relative to p or the edge distance [25, 18]. A scalar field, such as temperature, is invariant change of gauge, and thus fp E R transforms according to the trivial representation Po of (2).). A vector field over the mesh representing e.g. a flow across the manifold would have fp E R²2 and transform according to the standard rotation representation P1 of (2 by 2 x 2 matrices. In general, feature fields may be any representation P., of the gauge group SO(2).
Parallel Transport Let fp and fg be feature vectors nodes p, q, respectively, where q is adjacent to p. As features at different nodes live in different tangent spaces and have different bases, in order for the anisotropic kernel be applied to fa for each q E NNp, each Sq must be parallel transported to T.MM and be in the same gauge. The parallel transporter 9--p first aligns the tangent plane at to the tangent plane at p by 3D rotation and then transforms the gauge at g to the gauge p by planar rotation. Acting on fy with Ie- transports fg to the gauge at p. For the edge features Epg. no parallel transport is required as the features live in M, For more details, see Appendix A.2 and Appendix A.2 of [9].
2.2 Flavors of Message Passing
Meshes are graphs with additional geometric information. Neural networks designed for meshes may be understood by analogy to graph neural networks (GNNs). As outlined in Bronstein et al. [8], there are largely three flavors of GNNs: convolutional, attentional, and nonlinear message-passing.
The three images on the right of Figure show the three different GNN flavors in the context of gauge equivariant networks over meshes. To date, both convolutional and attentional GNNs have gauge equivariant analogues for learning over meshes. Here, we introduce gauge equivariant mesh network in the nonlinear message-passing flavor. Let p be the source node (b in Figure la), q E N, be the target nodes (one-hop neighborhood), and f the signals over nodes.
Convolutional GemCNN [9] uses convolutions where the kernel Kesigh is applied (amisotropically) to features at the target nodes. It is comparable to graph convolutional networks [26] but with anisotropy and gauge equivariance. As convolutions are linear, the messages passed to the source node are linear, and these messages are aggregated in a permutation invariant way. GemCNN also has a kernel Kytr to model inffinteerccions before updating the feature at the source node. The gauge equivariant convolution (GemCNN layer) is defined
f = Ksettfp + EgEN, (Beiggh(yy) (14--)p)
where K (leigh((g)) is the kernel for q and pin (gg-s)) is the feature vector at & parallel transported to the gauge at p. Input features f, transform according to Pin and output features f according Pout-
The filter Kneigh is anisotropic, i.e., it depends on the orientation 0,. This is strictly more expressive than an isotropic filter (as in GNN) which would simply use the same weights for each neighboring vertex. However, the dependence on orientation means that without constraints, the convolution depends the choice of local coordinate frame. To fix this and impose gauge equivariance, Knagh must satisfy K Powr(g) " (g). We also define the kernel to only depend on the orientation and
3

---

not on the radial distance of neighboring nodes, as including the radius in the parameterization was beneficial [9] and verified our experiments.
Attentional EMAN [10] extends GemCNN by adding an attention mechanism to the messages before the aggregation step, analogous graph attention networks [27]. Gauge equivariant attention with the self-interaction term is defined as
fp
at K Keselt + ΣyEN, Opg K (p))((()))))p
(2)
where app: apy are the attention weights for the self-interaction and neighbor interaction terms (see Equation (12) in Appendix B.2). Multi-headed attention can be used by concatenating the results of the individual heads. The attention weights are combined linearly with the neighboring node features and can be seen as weighted local averaging. Note that due to the separate kernels for the keys, queries, and values, an EMAN layer uses many more parameters than an equivalent GemCNN layer with the same dimensions.
3 Gauge Equivariant Nonlinear Message Passing
We propose gauge equivariant nonlinear message passing for meshes, complementing the convolu- tional and attentional gauge equivariant methods. As is the case for nonlinear message passing GNNs, our network is designed to be better suited for tasks with complex local interactions. Our network, named Hermes, consists of sequence of message passing blocks, each of which contains edge network de, an aggregation (we use sum throughout), and a node network Φn.
The edge network models neighboring interactions and takes inputs the source node features fp. target node features fa where q E Np, and edge features Epg where (p. q) E The target node features are parallel transported to the gauge p p(9g-p)) The network &e consists of N. gauge equivariant convolutions followed by regular nonlinearities [9] which also preserve gauge equivariance. The nonlinear messages myg are then aggregated to my. The node network models self-interactions and takes as input mp + fp. where @ represents the direct sum, and then applics Nr gauge equivariant convolutions with Kaell kernels and regular nonlinearities. Equations (3)-(7) define gauge equivariant nonlinear message passing,
hpg = fp 1 p(9g-1)ffg + Epg:
v(p.g) E
(3)
mpp )p( = a Kniigh (0z) ................................ ... - K Kn (0g)
(4)
mp Σ mmpa
Vp E V
(5)
qENp
hp my + fp
(6)
f/ Φn (lip) = o o K o 0...00 o (hp),
(7)
where hig and ho are inputs to de and d.., is the regular nonlinearity, and o denotes composition.
Figure 2 shows the Hermes architecture for one of the datasets used in experiments (wave PDE, see Section 4.1). For comparison, we also include the GemCNN and EMAN architectures in Appendix B. A residual connection is included at the end of each HermesBlock for more expressivity, see Section 5.5 for ablation results.
Proof of Gauge Equivariance We explicitly show that Hermes is equivariant to local gauge transformations in Proposition
Proposition Hermes is equivariant to local gauge transformations g" for any p E V of a mesh.
The proof, provided in Appendix B.3, uses the fact that all of the kernels used in Hermes satisfy the constraints for gauge equivariance. The nonlinearities are exactly gauge equivariant as the number of intermediate samples N used in the discrete Fourier transform approaches infinity and approximately gauge equivariant for finite (see Section 4 in [9] for more details). Thus Hermes is also gauge equivariant (in the limit).
Hermes combines both nonlinear message passing and gauge equivariance, generalizing GemCNN and EMAN, and completes the full picture with the flavors of message passing (cf. Figure I). An

---

0 (--6)
HermesDoock
Mermen
Edgee Netuork
hid
hid
Nock
hid
12ps 12p
Edge Network
Node Network
GaugeConv
CougeConv
GangeConv
f0
p (9c
Kyelf
81
6ba
K neight (ox)
Kusiga (Bac)
f.
fo
Rogalar Nocilis
Rogular Noulin
Keelf
8
p(
fb
fr
K K (px)
K Kuiigh (0x)
Cbe
m.
f.
fr
fa
(
eld
Monghh (bs))
K K (Bue)
mi
Kill
fd
fa
Figure 2: Hermes network architecture for the Wave PDE dataset. There are two message blocks, each with 2 layers for the edge network Φe and layer for the node network on- To illustrate the computations within the edge and node networks, In the edge network, we show only the computations for node for clarity.
the example mesh from Figure la input.
important point to note is that nonlinear message passing decouples the effect of the number of layers in the network (depth) with the receptive field of the graph. For graph convolutions using 1-hop neighbors, the receptive field of messages is exactly equal to the network depth. In contrast, nonlinear message passing computes messages using features from neighbors at an arbitrary hop distance away from the source node before the message is passed to the source node. We hypothesize that this decoupling is beneficial for tasks with complicated interaction dynamics between node neighbors, and previous works with graph message passing networks [28, 29] have shown good performance for abstract graphs (non-spatial graphs) involving objects. Although meshes are inherently spatial, show that nonlinear message passing is an important tool for solving PDEs surfaces and can outperform either convolutional or attentional mechanisms.
Experiment Design
To evaluate gauge equivariant nonlinear message passing, consider several different tasks. Our primary domains are partial differential equations on meshes, but we also consider shape correspon- dence, object interactions, and cloth simulation. Throughout, we consider triangular meshes as they are the most common. Further details and visualizations of the domains are provided in Appendix C.
4.1 Domains
Partial Differential Equations meshes We consider three linear and nonlinear surface partial differential equations (PDEs), where the dynamics occur on the surface of an object, represented as two dimensional mesh embedded in 3D space. Since solving PDEs meshes, e.g. heat diffusing a surface, naturally depends on the intrinsic mesh geometry, gauge-equivariant nonlinear message passing promising solution. For all three equations, we use example meshes in the Py Vista library [30] and generate 5 trajectories with different initial conditions, see Appendix C for more details.
Heat/Wave equation: The heat and wave equations second-order linear partial differential equa- tions (see Appendix C. Solving the heat equation on a surface mesh can correspond to modeling the dynamics when externl hot object makes contact at certain points thin hollow object. The wave equation describes how acoustic waves propagate on thin hollow surface. The dynamics of both equations highly dependent on the local mesh geometry.
Cahn- Hilliard equation: The Cahn-Hilliard equation [31] describes phase separation in a binary fluid mixture and is often used to model spinodal decomposition. It is fourth-order, nonlinear,

---

time-dependent PDE and is often factored into two coupled second-order equations. The surface equation can model real-world applications such as cell proliferation [32] and two- component vesicles [33]. As a nonlinear PDE, we expect our nonlinear message passing method to exhibit greater performance than other linear message passing flavors.
Shape correspondence As a standard mesh benchmark, we use the same FAUST dataset used in previous work [9, 10]. The dataset consists of 80 train and 20 test high resolution scans of 10 humans in 10 different poses. The task is to determine shape correspondence between different meshes. As the vertices are all registered and represent the same position on the human body, this task is equivalent to classifying the correct label for each vertex.
Object interactions Inspired by interaction systems [28, 29, 34], we consider complex dynamics of interacting objects on a mesh. On a coarse triangular mesh with random hills, each object occupies vertex and is oriented towards neighhoring vertex. An action can either turn the object left (changing its orientation), move the object forward, or turn right. Objects cannot move forward if there another object at the destination vertex, giving rise complex interacting dynamics. Furthermore, we consider the geometry of the mesh such that an object cannot move forward if the height difference between the current vertex and destination vertex is too high, or if the angle between the vertex normals is too large. If an object is able to move forward, we parallel transport its orientation and then choose the nearest neighboring node as its new orientation.
FlagSimple We also include the FlagSimple dataset from [20]. which simulates cloth dynamics of flag with self-collisions. Unlike the other datasets, the mesh is dynamic where the node positions change over time. The dataset was created using AreSim [35] with regular meshing over 400 timesteps. See Appendix A. of [20] for more information.
4.2 Training Details
For the PDE datasets, we report test root mean squared error (RMSE) of the prediction at the next timestep given the previous 5 timesteps. We use three separate test datasets and evaluate generalization to future timesteps (test time), unseen initial conditions (test init), and unseen meshes (test mesh). For test time, we train on timesteps T O..... 149 and test on T 150, ... 200. For test init, we test on trajectories with new initial conditions. For test mesh, we evaluate on completely unseen meshes to see whether a method overfits to specific mesh geometries.
For baselines, our main comparison is with GemCNN and EMAN to gauge how beneficial nonlinear message passing is over convolutional or attentional flavors. We also compare against 1) a SOTA messsage passing, mesh-aware method MeshGraphNet [20], 2) an E(3) ((3)-equivariant, non mesh-aware message passing network EGNN [25], 3) a non-equivariant, mesh-aware baseline SpiralNet++ [36], and 4) standard non-equivariant GNNs; graph convolutional networks (GCN) [26] and message passing networks (MPNN) [37, 38). We tune each architecture to use similar number of trainable parameters for fair comparison, see Table 9 in Appendix D. A more detailed feature comparison of each method is provided in Table 8 and all other training details are relegated to Appendix D.
5 Results
Table 1 shows the results on the PDE datasets and other results are given in Table 10 in Appendix On Heat, we see that EMAN outperforms GemCNN and Hermes outperforms EMAN significantly. This coincides with our expectation as attention mechanisms can express convolutions using constant weights, and Hermes generalizes both linear convolution and attention mechanisms. However, this does not hold for the Wave dataset, where EMAN is noticeably worse than GemCNN while Hermes still performs well. On the nonlinear
dataset, we see that EMAN cannot generalize well to unseen meshes. Overall Hermes achieves an RMSE approximately 3 times lower than that of GemCNN, and between 2 to 8 times lower than that of EMAN
Compared to other non gauge equivariant baselines, Hermes outperforms all methods, except for certain cases with MeshGraphNet. Hermes is worse than MeshGraphNet on Heat, outperforms on Wave, and performs similarly on Cahn-Hilliard. performs substantially better on all test mesh datasets, which may indicate that Hermes can generalize to the true dynamics function, rather than the specific dynamics seen in the training trajectories. In other words, Hermes better adapted to use the underlying geometry while MeshGraphNet overfits to specific geometries.

---

Table RMSE on PDE domains, using 5 runs. All values are expressed in x10 and gray denotes 95% confidence intervals. Hermes generally outperforms baselines, except for some cases with MeshGraphNet (MGN), Hermes outperforms MeshGraphNet on all test mesh datasets.
Hermes
GemCNN
EMAN
GCN
MPNN
MGN
EGNN
SpiralNet++
Test time
1.18
3.88
2.93
152s1.2
2.66
0.93
3.09
2.82
HEAT
Test init
1.16.003
3.85...
2.90
152.0.9
2.63.0.8
0.93
3.07
6.44_..1
Test mesh
1.01 10.3
3.50
2.47.0.7
12742.2
2.36
2.41
8.96
22.0.0.3
Test time
5.43 10.8
12
19.0
162.5.0
9.07
6.26
45.9
8.88-1.2
WAVE
Test init
3.72 11.3
7.28
15.3
158...9
5.24
1.24
12.1
8.47
Test mesh
3.79
8.23
15.8
164:5.1
6.29=1.3
7.01
54.5+18
10.8
Test time
3.48+0.0
13.8+12
8.41
250+7.6
7.25=3.1
4.49
8.36+1.4
11.6-3.8
CAHN-
Test init
4.23
14.9
8.75+40
383
7.52=30
2.64
10.6=1.1
12.9-28
HILLIARD
Test mesh =..1
14.1+12
43.8
391 48.6
7.63
18.7
9.38=1.7
13.4-2.6
Heat
x10-1
x10-
Cahn-Hillird
1.5
0.8
RMMEE
0.6
RMSSE
1.0-
EMAN
0.5-
Hormes
0.0-
200
Figure 3: Errors from long-horizon prediction rollouts on unseen meshes, given only initial conditions. Error bars denote standard error over 5 runs averaged over unseen meshes (2 meshes for heat/wave, 1 for "'ahn-Hilliare GemCNN has exploding errors with increasing t on Heat and Wave. Hermes generally outperforms GemCNN and EMAN, with the exception of EMAN on Wave.
On the FlagSimple dataset (Table 10), Hermes outperforms MeshGraphNet considerably, suggesting that Hermes is not limited static meshes and can handle temporally changing meshes well. Note that MeshGraphNet was modified to have a similar number of parameters as Hermes and so the results are different than reported in Pfaff et al. [20].
5.1 Long-horizon prediction rollouts
Using representative random run, we generate predictions from each model autoregressively given only the initial conditions. We generate predictions on the unseen meshes (test mesh) and roll out the entire trajectory of the PDE (T 200). Figure 3 shows how the errors change with the increasing rollout horizon. For GemCNN, the errors accumulate quickly and the predictions diverge for Heat and Wave. EMAN performs similarly Hermes on Heat but underperforms on methods eventually diverge on Wave; this may be due to the fact that the wave amplitude oscillates multiple times over the longer horizon and may be more difficult to predict the periodic nature.
All
Table 2 shows prediction samples at T + 50 generated autoregressively given only initial conditions. GemCNN fails on the Heat dataset and does not produce the correct spatial patterns for Wave and Cahn-Hilliard. EMAN performs well on Wave, but fails on Cahn-Hilliard. Hermes gives fairly realistic predictions on all datasets. See Appendix E.1 for more samples with varying T.
5.2
Mesh Fineness
Here we investigate how mesh fineness impacts performance with respect to the three flavors of message passing. Our intuition is that as meshes become finer, the features at each node become more similar. Thus the dynamics between nodes would become more linear and convolutional approaches may perform comparably with nonlinear message passing.
For this experiment, we solve the heat and wave equations on single mesh ("armadillo""). To generate different mesh resolutions, we simplify the original mesh (# vertices = 172, to (348, 867, 1731, 3461, 8650) vertices using quadric decimation [39] (1731 vertices were used for the main results in Table 1). See Figure in Appendix E.2 for data visualizations. tories with Taxx 100 and use 5 of the 15 trajectories remaining 10 trajectories for the test time. As we use a single mesh, we do not test generalization to unseen meshes. We use 3 random seeds for each method.
generate 15 trajec- test init and use T = 81, ... 100 of the
7

---

Table 2: Qualitative prediction samples rolled out to the full path using only initial conditions. GemCNN completely fails Heat, while EMAN fails on Wave. Hermes predicts the spatial patterns accurately and outperforms GemCNN and EMAN on all datasets.
Dataset
GemCNN
EMAN
Hermes
Ground Truth
Heat
(T+50)
Wave (T+50)
Cahn-Hilliard (T+50)
x10-2
Test time RMSE
1.00
Test time RMSE
0.75
0.50
EMAN
0.25
0.00
8650
3461
1731
867
348
8650
3461
1731
867
348
No. vertices
No. vertices
Figure 4: Performance for varying mesh fineness for heat the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. The errors for GemCNN and EMAN increase the mesh becomes coarser, while Hermes performs similarly throughout.
Figure shows the results for the heat dataset. RMSE on both the test time and test init datasets increase for GemCNN and EMAN as the mesh becomes coarser (decreasing number of vertices). This coincides with the intuition that coarser meshes have more nonlinear dynamics between nodes and thus contributing to the increase in errors. On the other hand, Hermes still quantitatively outperform GemCNN and EMAN across all mesh resolutions. It is also robust to varying mesh fineness and has similar error values throughout.
On the wave dataset, we find that Hermes still outperforms GemCNN and EMAN at each resolution though the gap is not large, see Figure 10 (Appendix F.2). There is surprising decreasing trend in errors for all methods as the mesh becomes finer. This indicates that mesh fineness may not be the only factor play: it is possible that the type of dynamics (PDE) used and the specific architecture (e.g. graph receptive field) can affect the results. Another possible explanation is that as we use single simplified mesh, there may be some artifacts that affect the wave PDE simulations.
5.3 Surface Roughness
We investigate whether surface roughness affects the performance gap between GemCNN and Hermes. We use the same armadillo mesh with 1,731 vertices from before and extract the vertex normals. The vertex normals are then randomly scaled using Gaussian distribution (0, s²), where E {0.1, 0.5, 33. The vertex coordinates the original coordinates, resulting in different surface roughnesses. See Figure 11 in Appendix E.3 for visualizations. We the same settings in the fineness experiments in Section 5.2.
then modified by adding these scaled normals to
Similarly to the results on mesh fineness, we find that Hermes retains the performance advantage over GemCNN and EMAN across all roughness scales. Its RMSE errors are also nearly constant

---

x10
Test time RMSE
1.00
Test init RMSE
0.75
GemCNN
0.50
EMAN
Hermes
0.25
0.00
Scale
Scale
Figure 5: Performance for varying surface roughness for heat on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. Increasing scale increases the surface roughness the mesh. Hermes outperforms GemCNN and EMAN across different roughness scales and is much more robust.
Table 3: Mean forward computation time in seconds (ms) during inference on test time PDE datasets.
GemCNN
EMAN
Hermes
GCN
MPNN
MeshGraphNert
EGNN
SpiralNet++
HEAT (ms) WAVE (ms)
12.4
19.5
10.8
1.5
1.2
2.2
1.4
0.9
12.5
17.7
10.5
1.4
1.3
2.2
1.3
0.9
MAHIN-HILLIARE (ms)
6.9
9.2
6.2
1.9
1.5
2.1
1.6
1.3
throughout indicating robustness to surface roughness. Surprisingly, GemCNN seems to perform increasingly better as the surface becomes rougher while EMAN seems to perform roughly equal. On the dataset, Hermes still outperforms other baselines most roughness scales, Figure 12 in Appendix E.3. There does not seem to be any correlation in errors with surface roughness.
5.4 Computation Time
As the edge and node networks 6n de consist of multi-layer perceptrons, might think Hermes requires more computation time than its simpler convolutional or attentional counterparts. Table shows that this is not the case; Hermes has lower computation time than both GemCNN and EMAN. As we control for a similar number of parameters in each architecture, there are fewer aggregations in Hermes than in GemCNN or EMAN resulting in lower average computation time in the forward pass. For EMAN in particular, due to its attention mechanism. we find that it uses many more parameters than either GemCNN or Hermes. Thus constraining EMAN have similar number of parameters as GemCNN restricts its expressivity, On the other hand, Hermes is very flexible as it can use different number of layers for the edge and node networks. Even though Hermes uses slightly lower hidden dimension than GemCNN in all domains, the results show that this does not hamper model expressivity and still achieves higher performance. As expected, all three gauge equivariant methods are significantly more computationally expensive than standard graph neural networks.
5.5 Residual connection ablation
As Hermes models the self-influence of nodes, we also test whether residual connection is necessary. Table 12 in Appendix E.4 shows the ablation of the residual connection. Having residual connection Thus
improves performance on Heat, but decreases performance slightly on Wave and the residual connection seems to be dask-dependent and should be considered a hyperparameter. In our experiments, residual connection was used for each message passing block.
6 Related Work
Learning over meshes Several different mesh operators have been studied within computer
graphics, in the context of shape classification [40-42], dense shape correspondence [43, 36], mesh segmentation [44-46]. Due the success of graph neural networks (GNNs) [26, 27, 38]. several approaches use GNNs to process meshes [21, 22, 47, 36] and incorporate geometric information via geodesic convolutions, anisotropic kernels, dual graphs, or spiral operators. Pfaff al. [20] introduce state of the art method for learning simulations with meshes by representing a graph in mesh space and in world space. this work, we introduce a novel architecture that combines nonlinear message passing with explicit equivariance to local gauge transformations.

---

Gauge Symmetry Most works on non-Euclidean manifolds have used convolutions on discretized patches of the manifold, approximating them to be Euclidean. Masci al. [21] define the patch operator using local geodesic coordinate systems. Monti et al. [23] use mixture of parametric Gaussian kernels. All of these works use (linear) convolutions, unlike our work. Boscaini et al. [22] uses anisotropic diffusion kernels, but uses the principal curvature direction as the preferred gauge which may be ill-defined for some shapes. On the other hand, several works have considered equivariance to gauge symmetry. Cohen et al. [6] proposes gauge equivariant convolutions on the icosahedron. Most similar to our work is that of [9. 48, 10], which explicitly incorporate gauge equivariant kernels for meshes. de Haan et al. [9] use convolutions while He et al. [48] and Basu et al. [10] use attention to discrete and continuous gauge transformations, respectively. In this work, we complete the picture and propose general nonlinear message passing with gauge equivariance for meshes. For more in-depth theory and discussion of local gauge equívariance on manifolds, see [49].
Solving PDEs on surfaces With the rise of physics-informed neural networks [50-52] and their increased sample efficiency and performance over classical methods, several recent works have focused on solving complex partial differential equations with deep learning, e.g. fluid dynamics [53-55], thermodynamics [56], structural mechanics [57, 58J, and material science [59, 60]. Some approaches learn the finite-dimensional [51, 53, 54] or nnfinit-ddimensional [61, 17, 62] solution operators. However, there has been less work on applying deep learning to solving PDEs on surfaces embedded in 3D space using meshes, with some exceptions [63-65]. Fang and Zhan [63] solve the Laplace-Beltrami Beltrami equation over a 3D surface with neural networks while Tang et al. [64] propose an extrinsic approach. However, both methods are mesh-free and only use points and their normals, discarding any connectivity information. Li et al. [65] extend Fourier neural operator [66] and learns diffeomorphic deformation between the input space and the computational mesh. While Geo-FNO depends on the embedding space of the mesh (e.g. embedding a rough 2D mesh in 3D), our method works directly on the intrinsic mesh surface. Perhaps most relevant to this paper is [67], which extends GemCNN [9] in several ways to predict hemodynamics on artery walls. In this work, we propose nonlinear message passing architecture for meshes that retains the underlying geometry and demonstrate their effectiveness in predicting variety of surface dynamics.
Discussion
We introduce a novel architecture, Hermes, that performs gauge equivariant, nonlinear message passing for meshes. Hermes complements convolutional GemCNN and attentional EMAN and completes development of the 3 flavors of message passing in the gauge equivariant setting. In the context of meshes, similar to GNNs, there seems to be a tradeoff' between simple linear operations such as convolutions versus nonlinear message passing. Convolutions are computationally efficient and can perform well on simpler tasks such as shape correspondence, see Table 10 in Appendix E. However, when the interactions passing surpass lincar schemes. By decoupling the degree of nonlincarity from the depth of the network and receptive field, Hermes outperforms GemCNN and EMAN significantly on the PDE datasets and produces realistic predictions given only initial conditions.
more complicated such in surface PDEs, nonlinear message
A limitation is that Hermes may use more parameters depending on the architectures of the edge and node networks. In particular, both EMAN and Hermes cannot scale well to meshes with large number of vertices naively, and one may need to consider more sophisticated approaches such as multi-scale or graph expander approaches. Furthermore, the architecture search space for Hermes larger than that of GemCNN or EMAN, as one needs to consider different combinations of numbers of layers in the edge and node networks, along with the number of message passing blocks.
For future work, one direction is to consider different dynamics such as non-stationary or chaotic dynamics, and other PDEs important in real world applications such as Navier-Stokes for climate or blood flow. Another direction is to analyze the design space of gauge equivariant networks. While GNNs have been extensively studied, far less work exists for mesh methods. GNNs are often highly task- and there are many design dimensions (e.g. residual connections, message passing iterations, etc.) to consider [68]. It would be particularly helpful for practitioners to have guidelines on when to use gauge equivariance or message passing over simpler approaches. This work aims to be a first step in this direction by demonstrating Hermes as a good fit for predicting nonlinear dynamics on meshes.
10

---

Acknowledgments and Disclosure of Funding
We sincerely thank Vedanshi Shah for the initial data generation code and Pim de Haan for clarifying parts of GemCNN. This work is supported in part by NSF 2107256 and 2134178. This work was completed in part using the Discovery cluster, supported by Northeastern University's Research Computing team.
References
[1] Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon mesh processing. CRC press, 2010.
[2] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192 2020.
[3] Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15272-15281, 2021.
[4] Franco Dassi, Bree Ettinger, Simona Perotto, and Laura M Sangalli. mesh simplification strategy for a spatial regression analysis over the cortical surface of the brain. Applied Numerical Mathematics, 90:1 131,, 2015.
[5] Mayur Mudigonda, Sookyung Kim, Ankur Mahesh, Samira Kahou, Karthik Kashinath, Dean Williams, Vincen Michalski, Travis O' O'Brien, and Mr Prabhat. Segmenting and tracking extreme climate events using neural networks. Deep Learning for Physical Sciences (DLPS) Workshop, held with NIPS Conference, 2017.
[6] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pages 1321-1330. PMLR, 2019.
[7] Kenneth G Powell, Philip L Roe, and James Quirk. Adaptive-mesh algorithms for computational fluid dynamics. In Algorithmic trends in computational fluid dynamics, pages 303-337. Springer, 1993.
[8] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104, 13478, 2021.
[9] Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions geometric graphs. In International Conference Learning Representations, 2021. URL.
oIZE.
[10] Sourya Basu, Jose Dallego-Posadde Francesco Viganò, James Rowbottom, and Taco Cohen. Equivariant mesh attention networks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https: httppp/:///peeeeiee.e.eo wwt/ ieeiiidd=33aa0 id=3IqgJh2Ycy.
[11] Marcelo Bertalmo, Li-Tien Cheng, Stanley Osher, and Guillermo Sapiro. Variational problems and partial differential equations on implicit surfaces. Journal of Computational Physics, 174 (2):759-788) 780, 2001.
[12] Thomas D Economon, Francisco Palacios, Sean R Copeland, Trent W Lukaczyk, and Juan J Alonso. Su2: An open-source suite for multiphysics simulation and design. Aiaa Journal, 54 (3):828-88) 846, 2016.
[13] Steffen Marburg and Bodo Nolte. Computational acoustics of noise propagation in fluids: finite and boundary element methods, volume 578. Springer, 2008
[14] Amy Novick-Cohen. Chapter 4 the cahn-hilliard equation. volume 4 of Handbook of Differential Equations: Evolutionary Equations, pages 201-228. North-Holland, 2008. doi: https://doi. mmg10000066651114455511000000000 URL www.hhttt/////e sciencedirect. com/ /ci ence/ articiee/pi//1188777777700000

---

[15] Martin Reuter, Franz-Erich Wolter, and Niklas Peinecke. beltrami spectra as "shape- dna' of surfaces and solids. Computer-Aides Design. 38(4):342-666, 2006.
[16] Gerhard Dziuk and Charles M Elliott. Finite element methods for surface pdes. Acta Numerica, 22:289-368 -96,, 2013.
[17] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[18] Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. arXiv preprint arXiv:2202.033776 2022.
[19] Rui Wang. Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint 20Xiv:2002.03061 2020.
[20] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gorie Gonzalez, and Peter Battaglia. Learning mesh- based simulation with graph networks. In International Conference Learning Representations, 2021.
[21] Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pages 37-45, 2015.
[22] Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. Learning shape correspondence with anisotropic convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[23] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5115-5124, 2017.
[24] Nitika Verma, Edmond Boyer, and Jakob Verbeek. Feastnet: Feature-steered graph convolutions for 3d shape analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018
[25] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. International conference on machine learning, pages 9323-9332. PMLR, 2021.
[26] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
[27] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
(28) Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez. Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[29] Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International conference machine learning. pages 2688-2697. PMLR, 2018.
[30] Bane Sullivan and Alexander Kaszynski. PyVista: 3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK). Journal of Open Source Software, (37):1450, May 2019. doi: 10.21105/joss.014555 014550 URL satpps///ool org/10. 21105/josss 01450.
[31] John W Cahn and John E Hilliard. Free energy of a nonuniform system. i. interfacial free energy. The Journal of chemical physics, 28(2):258-267, 1958.
[32] Exgeniy Khain and Leonard M Sander. Generalized cahn-hilliard equation for biological applications. Physical review E, 77(5):05112)) 1229, 2008.
12

---

[33] Andreas Rätz. A benchmark for the surface aahn-hilliard equation. Applied Mathematics Letters, 56:65-71, 2016.
[34] Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[35] Rahul Narain, Armin Samii, and James O'brien. Adaptive anioorropic remeshing for cloth simulation. ACM transactions graphics (TOG), 31(6):1-10, 2012.
[36] Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. Spiralnet++: A fast and highly efficient mesh convolution operator. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0-0, 2019.
[37] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272 PMLR, 2017.
[38] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-GGozzllez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1rXiv:1806.01200 012661 2018.
[39] Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics, In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 209-216 1997.
[40] Iyy Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. Surface networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2540-25440 2018.
[41] Yutong Feng. Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8279-8286, 2019.
[42] Alon Lahav and Ayellet Tal. Meshwalker: Deep mesh understanding by random walks, ACM Transactions on Graphics (TOG), 3913) 2020.
[43] Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt. simple approach to intrinsic correspondence learning unstructured 3d meshes. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018
[44] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yr Zhou. Tangent convolutions for dense prediction 3d. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3887-3896, 2018
[45] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshcnn: a network with an edge. ACM Transactions on Graphics (TOG), 1-14 2019.
[46] Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, and I Leonidas J Guibas. Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4440-4449, 2019.
[47] Francesco Milano, Antonio Loquercio, Antoni Rosinol, Davide Scaramuzza. and Luca Carlone. Primal-dual mesh convolutional neural networks. Advances in Neural Information Processing Systems, 33:952-9633 963, 2020.
[48] Lingshen He, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. Advances in Neural Information Processing Systems, 34:27331-27343, 2021.
[49] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020 2021.
13

---

[50] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. informee neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[51] Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1): 955, 2018
[52] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data, In International conference on machine learning, pages 3208-3216, PMLR, 2018.
[53] Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 481 1990 2016.
[54] Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64:525-545, 2019.
[55] Rui Wang. Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
[56] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis. inysics-informed neural networks for heat transfer problems. Journal of Heat Transfer, 143(6), 2021.
[57] Chengping Rao, Hao Sun, and Yang Liu. Physics-informed deep learning for computational elastodynamics without labeled data. Journal of Engineering Mechanics, 144(1)304221043, 2021.
[58] Marco Maurizi, Chao Gao, and Filippo Berto. Predicting stress, strain and deformation fields in materials and structures with graph neural networks. Scientific Reports, 12(1):21834, 2022.
[59] Ruijin Cang, Hechao Li, Hope Yao, Yang Jiao, and Yi Ren. Improving direct physical properties prediction of heterogeneous materials from imaging data via convolutional neural network and a morphology-awaar aware generative model. Computational Materials Science, 150:212- 221, 2018.
[60] Dehao Liu and Yan Wang. Multi-fidelity physics- constri-const neural network and its application materials modeling. Journal of Mechanical Design, 141(12), 2019.
[61] Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshell, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485. 2020.
[62] Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121-157, 2021.
[63] Zhiwei Fang and Justin Zhan. A physics-informed neural network framework for pdes 3d surfaces: Time independent problems. IEEE Access, 8:26328-266 -26335, 2019.
[64] Zhuochao Tang, Zhuojia Fu, and Sergiy Reutskiy. An extrinsic approach based on physics- informed neural networks for pdes surfaces. Mathematics, 10(16):2861, 2022.
[65] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural oper- ator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209. 2022.
[66] Zongyi LI, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.0889) 2020.
[67] Julian Suk, Pim de Haan, Phillip Lippe, Christoph Brune, and Jelmer M Wolterink. Mesh neural networks for se (3) e3uvvirirrt hemodynamics estimation on the artery wall. arXiv preprint arXiv:2212.05023: 2022.
14

---

[68] Jiaxuan You, Zhitao Ying, and Jure Leskovec, Design space for graph neural networks. Advances in Neural Information Processing Systems, 17021, 2020.
[69] O. Tange. Gnu parallel the command-line power tool. :login: The USENIX Magazine, 36(1): 42-47, Feb 2011.
[70] Martin Reuter, Silvia Biasotti, Daniela Giorgi, Giuseppe Patanè, and Michela Spagnuolo. Dis- crete laplace-belle beltrami operators for shape analysis and segmentation. Computers & Graphics, 33(3):381-390, 2009.
[71] Will Schroeder, Kenneth M Martin, and William E Lorensen. The visualization toolkit an object-oriented approach to 3D graphics. Prentice-Hall, Inc., 1998.
[72] A. Logg and G. N. Wells. DOLFIN: automated finite element computing. ACM Transactions on Mathematical Software, 37, 2010. doi: 10. 145/17310 1731030.
[73] Carl Kelley, Solving nonlinear equations with Newton's method. SIAM, 2003.
[74] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Piscataway, NJ, USA, June 2014. IEEE.
[75] Ronald J Williams and David Zipser. Gradient-based learning algorithms for recurrent. Back- propagation: Theory, architectures, and applications, 433:17, 1995.
15

---

A Background
A.1 Local Coordinate Frame
As discussed in the main text, the reference neighbor d defines the basis {et, eb} for the tangent space T, M. (see Figure la). Denote by log, = Projt,M (2ia) 26 the vector from 26 (the coordinates of b) the orthogonal projection of x E I& (the coordinates of onto the tangent plane T,M. Let et = log,(za)/ (=a))/1 loga (za)ll and e n& x ef. Then the orientation of every neighbor v E No can be represented with polar angles, where
0, atan2 cc² logı(e). of logo(v)
(8)
the angle between and the reference neighbor after projection to the tangent plane.
A.2 Parallel Transport
Let fp and fg be feature vectors at nodes q, respectively, where q is adjacent to p. As features at different nodes live in different tangent spaces and have different bases, in order for the anisotropic kernel to be applied to fq for each E Np, each fa must be parallel transported to F and be in the same gauge.
Let fp and fg be feature vectors at nodes p, q, respectively, where q adjacent to p. As mentioned in the main text, the parallel transporter 94->p first aligns the tangent plane Tp TpM to the tangent plane Tg M by a 3D rotation and then transforms the gauge at q to the gauge at p by planar rotation.
be the 3D rotation required to align T, and Let Rp, Mg be the normal at and q, and Ryp E SO(3) Tp M. Then Sg-op" [0, 2xz) is given by
Je-p atan2 e cr (Rype Tey).
where eg) and (2) are the frames at node q and p. respectively.
B Gauge-Equivariant Message Passing Methods
In this section, we provide the architectures GemCNN and EMAN for the wave dataset compare with Hermes (see Figure 2), and prove that Hermes is equivariant to local gauge transformations.
B.1
GemCNN
The architecture for GemCNN is given in Figure 6. Note that while the equations in GemCNN (see Equations 1) allow for self-interactions, the implemented architecture does not actually use self-interaction kernels, possibly due to residual connections.
GangeConvLayer
GangeConn
fa
plge a)
K (9.)
GFMRaalloc
Sapaki Nedlis
Regyter Noalls
GFM
hid
Claspe
fc
p(g-as)
K (0)
plas a)
K (9c)
bis 14p 18p 19 18ps
Figure GemCNN network architecture for the Wave PDE dataset. There are three message blocks, each with 2 gauge-equivarian convolutional layers. To illustrate the computations within the convolutional layer, we use the example mesh from Figure and show only the computations for node b for clarity.
B.2 EMAN
For completeness, the full equations for EMAN are given in Equations (10)-(12) (see also Appendix C.5 in [10]..
16

---

EMA EMANItes M.A.NttesBlock
EMANNossDoook
EMANA Arcation
Regular Nonlin
Gasuge Convl over
Rognlar Nonim
bid lic
w............................... 8p, 8p2
EMANAllention
+ c)
GaugeConvLayer
f
p (0-44))
Krnigh (8.)
é)
GaugeConv
SelF- Self-Attenti Att-Attention
f.
Kilf
fa
(ga-s))
Korigh (0a)
Othe fc
fo
p(pc-sb)
Rongh (8a)
P (9c-bb)
niigh (f.)
Me
Oldd
f8
p(8cmb))
K Keiga (0z)
fa
p ((ga-1)
K neight (Oa)
mid
Figure 7: EMAN network architecture for the Wave PDE dataset. There are three message blocks, each with 2 attention and sauge-qquivariant convolutional layers. To illustrate the computations within the edge and node networks, we use the example mesh from Figure la and show only the computations for node b for clarity.
Sp Opp K Keself
value +
Orpo
Kneeihh (0g (0g)p(gn-pp)Gp
(10)
gE,Nr
(xceat fr))
quaurry
0pp = softmax
(11)
Van
(1LEAS,
Kay (0pg) p(gp-plta
(Keuary fp)
0g = softmax
(12)
VCau
where Kley Ki Kaane:& {self, neigh} are the usual key and value matrices for the self-interaction and neighbor interaction terms, and Kquery the query matrix. The llyeN, denotes a concatenation of the vectors for all Np-
Figure 7 shows the architecture for EMAN. There is an EMANAttention layer to compute the attention weights Otpy. which are passed to the aggregation function. Similarly to GemCNN, the implemented architecture does not use self-interaction kernels.
B.3 Hermes
We provide the proof of Proposition 1 (stated here again for convenience). Part of the proof uses the fact that anisotropic convolutions over the tangent space are equivariant under the appropriate kernel constraint shown in de Haan al. [9].
Proposition 2. Hermes is equivariant to local gauge transformations 9d for any p V of mesh.
Proof. Using Equations (3)-(7), we first consider the simpler case where the edge and node networks have one gauge-equivariant layer each, i.e. N. = N,, = Let G SO(2) be the gauge group. Let p E V be some vertex and E N₂ be some vertex adjacent to The inputs to the edge
17

---

network are fpr fq. Evg (the source node, target node, and edge features, respectively), which are all representations of G. With respect to the gauge transformation g" E G. they transform as
fp+ + pmp- P + pa(l=9")((a--p)
Cpq + Pma(-g")( pp
where L is first parallel transported to the gauge at p before applying the group action (Epg is assumed to already be in this gauge). As lig is the direct product of fp: p(ug-tp) Egy (Equation 3), also transforms as Rep + Pin Pinl-g))ap For the edge network to be equivariant with respect g", we need
to show that K1 Kneeihh (0 (Pia( Mppp))
Poun (-9²) (or (Knhighh (0a))pa))
(0
We compute
the left-hand side
a (Oniigh 9d (pt(lg9")Pppp = a (poml (-9") ( nengh (1g) (PpPpll"")aa((~>10)))))))))))))a pta(= )1pp))
Poun g-g" (K (1p)
desired, where have used the kernel constraint K meihh (9, g") Poun Baul-g9 (0g)))) g0) neigh (0,) Pin (gg-p) and the fact that is equivariant with respect to the Pout representation of G and thus commutes with Pou(-g7
For the node network, K self does not depend on 0g and so the kernel constraint is simpler: K Poal (oa(-g") (a (Kaathp)) and we
Poull Pou(-g9 g²)) K settPin(gg)) It is easy to see that a ( (K SaltPPal-g""le omit the proof as it is nearly identical to that of the edge network.
The remaining part is the aggregation (sum) (Equation 5). As p(-ge) is a linear group action (representation) of G. p(-g")mm, EGEN, ((-9")mpp
We have so far proved that Hermes is equivariant for No N, 1. This suffices to prove for all Ne 2 N, 2 1. The edge network de a composition of the gauge-equivariant convolutions and activations interleaved with each other. The regular nonlinearity is gauge-equivarian when the number of intermediate samples used in the discrete Fourier transform goes to co and is approximately gauge-equivariant with finite N. As the composition of equivariant functions is also equivariant, de is equivariant to any gd for any N₂ 2 1, and similarly, o, is equivariant to any gº for N₂, 2 1. Thus Hernes is equivariant any local gauge transformation gd G. in the limit N + co.
C Experiment Domains
C.1 Heat/Wave Equation
The heat and wave partial differential equations are given by du
8²u 212
alu, (Heat)
e² LLu,
(Wave)
(13)
where u is the temperature or the displacement for the heat and wave equations respectively, a is thermal diffusivity, e is constant, and ī. the symmetric cotangent Laplacian [70]. Note that i. is the discrete version of the Laplace- Belttr- operator over a Riemannian manifold. For scalar function L(u) is given as,
(i(w)); = 24,
Σ jEN(()
(cot + cot Bes)(u5 - uz),
(14)
where (i) denotes the adjacent vertices i, az and B,g are the angles opposite edge (i.j), and A, is the vertex area of where we use the barycentric cell area.
We use example meshes from the Vista software [30] and use 6 for training and 2 for testing generalization (test mesh) to unseen meshes. We further generate test sets to evaluate generalization on future timesteps (test time) and random initialization (test init) conditions. We simulate samples for T 200 timesteps, using 3 samples from T = O.... 150 for training and the remaining T= 151, 200 as test time samples. The unseen samples from 0, .., 200 are used for test init samples. Some meshes contained too many nodes to be computationally feasible so we simplify these meshes to be less than 5,000 nodes using quadric decimation [71]. We use a = c 1. See Tables 4, for visualizations of the meshes and how the features evolve over time.
18

---

The heat and wave equations were simulated using the finite difference method using the forward Euler scheme, As each mesh has a different Laplacian, we tune the time discretization dt for each mesh. To initialize, we randomly generate Gaussian bumps over the mesh (20% of all nodes for heat, 2.5% for wave). While such initialization is not necessarily realistic, we chose to incorporate more superposition and interference to generate more diverse dynamics. The task is to predict the temperature (heat) or displacement (wave) at each vertex at the next time step, given a short history of values. Both the inputs and outputs are trivial representations.
C.2 Cahn-Hilliard Equation
The Cahn-Hilliane equation [31] often represented by the following two coupled second-order equations:
dc Øt
of dc 1 + x = 0,
ML(μ) = 0,
(15)
where e is the fluid concentration, M is the diffusion coefficient, J is the chemical potential, L. is the Laplacian, f the double-well free energy function, and is a positive constant.
We use 4 meshes from the Vista software [30] and use 3 for training and for testing (test mesh). As in the heat and wave datasets, we use N = 5 samples with T 200 timesteps and split the training and test datasets similarly. Some meshes contained too many nodes to be computationally feasible we simplify such meshes using the same procedure in the heat and wave datasets. See Tables 6 for visualizations.
The Cahn-Hilliane equation was simulated using the DOLFINX software [72] with time discretization di = 5 x 10-6, M 1, randomize À uniformly from [0.01, 0.02] for each data sample. The initial concentrations are randomly sampled from A (0.6, 0.05) for each vertex. The solver formulates the coupled second-order PDEs in variational form, and solves it using the Newton-Krylov method [73].
100c2(1 - 2),, and we use the A-method with 0 = 0.5. We
The task is predict the concentration at each vertex at the next time step, given a short history of values. Both the inputs and outputs are trivial representations.
C.3 Shape Correspondence
The FAUST dataset [74] consists of 80 train and 20 test samples, where 2 unseen human subjects are used for testing. Following previous work [9], the inputs are æyz coordinates with representation Po and the network outputs trivial features, followed by linear post-processing layers. The task is shape correspondence between meshes, where for every mesh, each vertex is registered and denotes the same position on the body.
C.4 Object interactions
In this domain, we consider 50 objects on a mesh with 250 vertices, see Figure 8. Each mesh is generated parametrically using the random hills generation procedure provided in VTK [71]. For training, we generate 20 meshes using different random seeds but the same random hills parameters, randomly initialize the object positions and orientations, and then simulate for a 100 timesteps. At each timestep, we select the action with weighted probabilities: 10% turn left, 80% move forward, and 10% turn right. For testing, we generate 100 meshes with randomized hill parameters and simulate forward 20 timesteps. We represent the current state of the system using the vertex coordinates, one-hot embedding of object occupancy, and object orientations. The orientations are 2-dimensional vectors that represent the heading of the object on the tangent place at the current vertex, pointing towards the position of the neighboring node projected onto the tangent plane.
The task is to correctly predict the object occupancy and orientations of the next timestep, given the current state and action f(t, at) $141. The inputs to the graph network are Pa features, except for the orientations, which are Pl. The graph network outputs both the logits of the occupancy and the orientation as Po and #I features, respectively. As the occupancy is discrete, we add linear post-processing layer after the graph network for only the occupancy outputs and compute the negative log likelihood given the ground truth occupancy. For orientations, we use root mean square error (RMSE) from the orientations given by the graph network.
19

---

Table 4: Heat equation dataset of all meshes used with max 200. We simulate N = 5 samples for each mesh. Of the training meshes, we use 7 150, 200 as the test time dataset and unseen samples as the test init dataset. The armadillo and meshes were used as the test mesh dataset. Object
Split
No. vertices
t=0
t=50
t=200
Armadillo
Test 1,731
Bunny
Train
502
Lucy
Train
2,501
Sphere
Train
842
Spider
Train
4,670
Um
Train
2,454
Woman
Train
2,998
20

---

Table 5: Wave equation dataset of all meshes used with max 200. We simulate N = 5 samples for each mesh. the training meshes, we use T 150, .... 200 as the test time dataset and samples the test init dataset. The armadillo and urm meshes were used the test mesh dataset.
Object
Split
No. vertices
t=0
t=50
t=200
Armadillo
Test 1,731
Bunny
Train
502
Lucy
Train
2,501
Sphere
Train
842
Spider
Train
4,670
Um
Train
2,454
Woman
Train
2,998
21

---

Table 6: Cahn- Hilliard equation dataset of all meshes used with T Taax 200. We simulate N
samples for each mesh. Of the training meshes, we use T = 150, 200 as the test time dataset and
2 samples as the test init dataset. The armadillo and urn meshes were used as the test mesh dataset.
Object
Split
No. vertices
T=0
T=50
T=200
Bunny
Train
502
Ellipsoid
Train
1,942
Sphere
Train
842
Supertoroid
Test
2,940
Figure 8: Objects interactions mesh. Purple denotes an unoccupied vertex and other colors represent different objects. Images are different random initializations of the underlying mesh and object position and orientations.
D Training Details and Baselines
For classification tasks such as FAUST or for predicting object occupancy in the Objects dataset, post-processing linear layers are used. For FAUST, we find that training for longer (400 epochs vs 100 from [9]) benefits all models. We use cross entropy loss for FAUST, a combination of binary cross entropy and RMSE for Objects, and RMSE for all the PDE datasets. For the PDE datasets, use truncated backpropagation through time [75] and roll out the network predictions for 3 timesteps using history of timesteps.
use the regular nonlinearity [9] maintain gauge equivariance, with ReLU activations and 101 samples, for all models. We use band limit of 4 for the gauge-cquivariant kernels. Throughout,
22

---

Table Details of architecture and hyperparameters
Dataset
FAUST
Objects
Heat
Wave
Cahn-Hilliare
No. blocks
5
1
No. layers in &e per block
2
4
2
No. layers in On per block
0
3
Hidden representation
10p
@z 12ps
12ps
@ 12p:
@ 12p
No. post processing layers Epochs
0
400
100
100
100
100
Batch size
1
Optimizer
Adam
Learning rate
7 x 10
5 x 10
1 x 10
5 x 10
5 x 10 10-3
Table 8: Comparison table of relevant graph neural networks and their features.
Graph Flavors
Features
Models
Convolutional
Attentional
Message Passing
Eqqiviquia
Bouug--quivariant
Mesh Aware
GemCNN
EMAN
(w/ RelTan features)
Hermes
GCN
MPNN
MeshGraphNet
EGNN
SpiralNet++
Table 9: Number of learnable parameters
FAUST
Objects
Heat
Wave
Hill-arlliard
Flag
GemCNN
1,829,658
161,715
40,337
40,337
29,775
EMAN
1,839,476
165,013
40,093
40,093
30,882
Hermes
1,837,776
162,976
40,544
40,695
23,122
51,555
GCN
41,401
41,401
29,401
MPNN
41,449
41,449
29,673
McshGraphNet
44,993
44,993
35,457
45,313
EGNN
50.433
50,433
28,609
SpiralNet++
63,841
63,841
27,153
L2 regularization of the weights with coefficient of 10-5 used. A cosine learning rate scheduler was used for the Objects and Cahn-Hilliard datasets.
Table 7 contains the architecture details and hyperparameters used for each domain. Table 8 lists each method's flavor of message passing and its features. Table shows the number of learnable parameters used for each method and domain.
E
Results
Table 10 shows the results on FAUST, Objects, and FlagSimple datasets. On FAUST, GemCNN outperforms both EMAN and Hermes. We obtained slightly different results for EMAN than reported in the paper [10]. On Objects, Hermes outperforms GemCNN and EMAN, albeit slightly. On FlagSimple, Hermes considerably outperforms MeshGraphNNtt.
Table 10: Test results for each method using 5 runs. Gray numbers are 95% confidence intervals.
GemCNN
EMAN
Hermes
MeshGraphNet
FAUST
Accuracy (%)
99.1 10.1
97.5.10.4
97.5.10.2
Accuracy (%)
98.810.1
96.9
99.7.10.
OBJECTS
RMSE (x (x)
6.40+0.1
6.43+0.0
6.26
FLAGSIMPLE
RMSE (x10-3)
5.87=10.
9.01
23

---

E.1 Long-horizon prediction rollouts
Table shows additional qualitative samples generated autoregressively using only the initial conditions for 7 + 10, T - 30, 7 + 100 timesteps. Hermes predicts large spatial patterns accurately and outperforms GemCNN and EMAN on all datasets.
Table 11: Qualitative predictions rolled out to 10, 30, 100 timesteps using only initial conditions.
Timestep
GemCNN
EMAN
Hermes
Ground Truth
T+10
HEAT
T+30
T+100
T+10
WAVE
T+30
T+100
T+10
CALN-RILLIARD
T+30
T+100
24

---

E.2 Mesh Fineness
Figure shows visualizations the different mesh resolutions used and Figure 10 shows the results on the wave dataset. Hermes still outperforms GemCNN and EMAN each resolution. Unlike the heat dataset (Figure 4), there is an unexpected overall decreasing RMSE trend with a coarser mesh.
(a) 348
(b) 867
(c) 1,731
(d) 3,461
(e) 8,650
Figure 9: Visualization different mesh resolutions. The captions denote the number vertices.
x10
1.00
Test time RMSE
Test RMSE
0.75
0.50
GemCNNN
0.25
Hermes
0.00
8650
1731 No. vertices
348
8650
1731
867
No. vertices
Figure 10: Performance for varying mesh fineness for wave on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. The number of vertices decreases with increasing reduction ratio (see Figure 9).
E.3 Surface Roughness
Figure 11 shows visualizations of the different mesh roughness and Figure 12 shows the results on the wave dataset. Hermes outperforms GemCNN and EMAN roughness scales.
(a) 0.1
(b) 0.5
(c) 1.0
(d) 1.5
(c) 3.0
Figure 11: Visualization of different mesh roughnesses. The captions denote the roughness scale.
1.5
Test time RMSE
Test init RMSE
1.0-
GemCNNN
0.5-
Hermes
0.1
0.5
1.0
1.5
3.0
0.1
0.5
1.0
1.5
3.0
Scale
Scale
Figure 12: Performance for varying surface roughness for wave on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. Increasing scale increases the surface roughness of the mesh (see Figure 11).
25

---

E.4 Ablation on Residual Connection
Table 12 shows the ablation experiment with the residual connection. The results are mixed: having residual helps on Heat but not on Wave or should thus be considered task-dependent hyperparameter.
The decision to have a residual connection
Table 12: Ablation study on residual connection
With residual
Without residual
Test time (x10-3)
1.18
1.24
HEAT
Test init (x10-
1.16
1.22=0.5
Test mesh (x10
1.01
1.05#004
Test time (x10-3)
5.43
5.17
WAVE
Test init (x10-3)
3.72.113
2.38 10.3
Test mesh (x10-3)
3.79
2.5
Test time (x10 (x10-3)
4.23
3.08
CAHN-HILLIARD
Test init (x10
5.21#1.2
3.53#1.
Test mesh ( (x10-3)
5.34
5.73=1.2
26


Reviews:
[{"review_id": "jOuL7d4o5U", "reviewer": "Reviewer_GgMe", "summary": "This work proposes a novel graph neural network architecture for gauge-equivariant learning on meshes. Compared to prior work using convolutional or attentional methods, this paper shows a nonlinear message passing method for their gauge-equivariant graph neural network. A comparison with two baselines, GemCNN and EMAN, shows that the novel method, Hermes, outperforms baselines in a diverse set of example applications. ", "questions": "1. For Table 1 the reported values of RMSE feel less interpretable than perhaps relative errors, since it might not be clear if all the datasets are normalized for the PDEs. Would relative errors make sense in this context?\n\n2. What is the intuitive reason that all perform so poorly for the wave equation in rollout testing?\n\n3. As far as understood, both GemCNN and EMAN in this paper have the gauge-equivariant adjustments, correct? Would it make sense to have a baseline that is not gauge-equivariant?\n\n4. In Line 212 the authors mention how EMAN has far more parameters and constraining it would limit expressivity. Were the test examples hence using the constrained EMAN? I.e., did all models have the same number of parameters? It would also be interesting to see if the computation time during inference is different between the models, and how much they vary.", "limitations": "Very well detailed limitations are mentioned, and potential future work to tackle them. ", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Great introduction to the problem setting, background well-explained step-by-step, making it very easy for the reader to follow. Example applications are clear, and always have arguments for why they are relevant problems in practice. Ablation studies also provide a full overview of the proposed method. A well-varied set of problems was used to show the performance increase of the proposed method.", "weaknesses": "1. In Line 143 the authors noted the addition of a residual connection to the HermesBlocks, for more expressivity. Was this perhaps tested in an ablation study? Did the model perform noticably better?\n\n2. In Figure 3 the claim is that Hermes is accurate for all datasets, even though in the Wave example Hermes is diverging faster than EMAN. \n\n3. Line 312 mentions there is a runtime increase and parameter increase involved with Hermes, this would be good to add to the main paper to understand the drawbacks of the method, as well as scalability. Although somewhat high-dimensional meshes (170k vertices) have already been tried out in this paper, it is unclear how much larger the authors are considering when mentioning the limitation in scalability in Line 314.\n\n4. The paper concludes with an insightful remark, summarizing the findings from the paper. Ideally future work can be described as well, seeing what direction is most promising from the authors' opinion.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive and insightful feedback.\n> In Line 143 the authors noted the addition of a residual connection to the HermesBlocks ... tested in an ablation study?\n\nWe performed an ablation study and results are shown in Table 5 of the uploaded page. The impact on model performance was mixed: having a residual connection improves performance on Heat, but decreases performance slightly on Wave and Cahn-Hilliard. We will include these results in the final version and make note that the residual connection should be considered as a task-dependent design dimension. \n> In Figure 3 the claim is that Hermes is accurate for all datasets, even though in the Wave example Hermes is diverging faster than EMAN.\n\nThank you for pointing this out. We will adjust the wording to accurately reflect the results in the final version.\n> Line 312 mentions there is a runtime increase and parameter increase involved with Hermes ... mentioning the limitation in scalability in Line 314.\n\nWe show the mean forward computation time during inference on the test time dataset in Table 4 of the uploaded page. Interestingly, Hermes is actually slightly faster than GemCNN, likely because Hermes performs fewer aggregations than GemCNN when using a similar number of parameters. Adding more message-passing iterations would increase the runtime over GemCNN.\n\nRegarding the number of vertices, we reduced the original meshes (~170k vertices) to <5000 vertices to keep training tractable. It is well known that GNNs do not scale to larger graphs [A, B] due to memory constraints. We are not aware of any message passing graph networks that can scale up to 170k vertices on a single GPU and scaling to this many vertices likely requires distributed computing and/or graph subsampling. Applying such techniques on Hermes to scale to larger graphs would be a good future research direction.\n> Ideally future work can be described as well, seeing what direction is most promising from the authors' opinion.\n\nThank you for your comment. One future direction is to consider different dynamics such as non-stationary or chaotic dynamics and other PDEs important in real world applications. Another direction that is to analyze the design space of gauge-equivariant networks. While GNNs have been extensively studied, far less work exists for mesh methods. GNNs are often highly task-specific and there are many design dimensions (e.g. residual connections, message passing iterations, etc.) to consider [C]. It would be particularly helpful for practitioners to have guidelines on when to use gauge equivariance and/or message passing over simpler approaches. This work aims to be a first step in this direction by demonstrating Hermes as a good fit for predicting nonlinear dynamics on meshes.\n> For Table 1 the reported values of RMSE feel less interpretable than perhaps relative errors\n\nWe chose RMSE as it would emphasize large differences from the ground truth and is a standard way to measure error [D,F]. While relative error is also a standard method and may be more interpretable as a unitless quantity, it also depends on where the zero point of the units is (e.g. measurements in Celsius vs. Kelvin would give different relative errors even though the absolute error is equivalent) and therefore relative error may not always be preferred over RMSE. We observe roughly 2-4% relative absolute errors for Hermes on the Heat and Cahn-Hilliard datasets.\n\n>What is the intuitive reason that all perform so poorly for the wave equation in rollout testing?\n\nOne possible reason that all models diverge on long-term predictions is that the wave amplitude oscillates around 0 between [-1, 1] multiple times over the course of one training trajectory. On the other hand, the temperature in the heat PDE diffuses toward the mean and the concentration in the Cahn-Hilliard equation approach towards the boundaries. It is possible that it may be more difficult to predict the periodic nature of the wave and the exact times of the wave peaks.\n>Would it make sense to have a baseline that is not gauge-equivariant?\n\nYes, both GemCNN and EMAN are gauge-equivariant methods. We considered this comparison most relevant as our primary consideration was to elucidate when nonlinear message passing is more beneficial than linear messages with respect to meshes. We have also added several new baselines: two standard GNN variants (GCN and MPNN), a non-equivariant mesh method (MeshGraphNet [D]), an E(3)-equivariant baseline (EGNN [E]), and a non-equivariant, mesh-aware method (SpiralNet++ [F]). See Table 1 for the comparison of features of each method and Table 2 for the results of the uploaded page. The results show that Hermes outperforms EGNN and SpiralNet++ on all datasets and also MeshGraphNet on test mesh.\n> ... did all models have the same number of parameters? ... computation time during inference.\n\nYes, for our tests, we use a similar number of parameters for each method including EMAN (see Table 7 in the Appendix and Table 5 in the additional tables page). We also measured the mean computation time during inference on the  test time dataset and surprisingly find that Hermes is slightly better than GemCNN and roughly 1.5~2x than EMAN. All three gauge-equivariant methods are much slower than regular graph networks (e.g. GCN, EGNN, SpiralNet++). We will clarify the number of parameters and include the computation time in the final version.\n\nReferences:\n- [A] Wu, Z., et al. (2020). A comprehensive survey on graph neural networks. \n- [B] Duan, K., et al. (2022). A comprehensive study on large-scale graph training: Benchmarking and rethinking.\n- [C] You, J., et al. (2020). Design space for graph neural networks.\n- [D] Pfaff, T., et al. (2020). Learning Mesh-Based Simulation with Graph Networks.\n- [E] Satorras, V. G., et al. (2021). E (n) equivariant graph neural networks.\n- [F] Gong, S., et al. (2019). Spiralnet++: A fast and highly efficient mesh convolution operator."}, {"review_id": "0PSGMKTj29", "reviewer": "Reviewer_ASko", "summary": "This paper aims at solving complex partial differential equations on surfaces. Given the fact that most existing work neither incorporate surface geometry nor consider local gauge symmetries of the manifolds, this paper proposes a novel gauge equivariant network, known as Hermes, that can achieve higher performance than existing convolutional or attentional networks in certain cases. In addition, authors investigate in which cases their method has advantages over other methods. ", "questions": "- See the weakness part. I would like to raise the score if the authors can well address my concerns.", "limitations": "The limitations of this paper are well discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Propose a new gauge equivariant network, Hermes, for learning signal on meshes. \n- Hermes outperforms both convolutional and attentional architectures on complex and nonlinear dynamics such as surface PDEs.\n- Authors investigate in which situations nonlinear message passing should be preferred over convolutional or attentional counterparts.\n", "weaknesses": "- It is not clear what is the relationship between Hermes and previous methods that use graph networks to perform mesh-based simulation, such as [1]. Could authors elaborate on the differences with the method in [1]? And I would like to see the performance comparison between Hermes and [1] in some scenarios, for example, FlagDynamic and CylinderFlow in [1]. \n- In addition, Geo-FNO [2] can also perform PDE learning on irregular geometries, what is the advantages of Hermes over Geo-FNO in terms of accuracy in solving PDE on irregular domains. \n\n\n[1] Pfaff, Tobias, et al. \"Learning mesh-based simulation with graph networks.\" arXiv preprint arXiv:2010.03409 (2020).\n\n[2] Li, Zongyi, et al. \"Fourier neural operator with learned deformations for pdes on general geometries.\" arXiv preprint arXiv:2207.05209 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful references and hope to have addressed all of your questions.\n\n>It is not clear what is the relationship between Hermes and previous methods that use graph networks to perform mesh-based simulation, such as [1]. Could authors elaborate on the differences with the method in [1]? And I would like to see the performance comparison between Hermes and [1] in some scenarios, for example, FlagDynamic and CylinderFlow in [1].\n\nWe did not initially compare to GNN approaches such as [1] since our primary focus is on determining when nonlinear message passing would be beneficial over simpler linear message passing schemes for gauge-equivariant networks and how decoupling the network depth from the receptive field of the neurons improves expressivity. The main difference between Hermes and MeshGraphNet is that Hermes learns directly on the two-dimensional mesh (mesh space) and does not depend on the embedding to 3D space (world space). It performs convolutions in a way that preserves local gauge symmetries and thus incorporates the geometry of the mesh intrinsically. On the other hand, MeshGraphNet uses both node positions in mesh space and world space as input node features and additionally creates edges between nodes that are close in world space (see Figure 3 of [1]). It also normalizes the input node and edge features and encodes them before performing several iterations of message passing.\n\nWe agree that having more baselines to compare is useful. We have included MeshGraphNet [1] as a baseline for the PDE datasets. We scaled the hidden dimension and the number of message passing iterations so that the number of parameters is roughly equal to Hermes and keep the rest of the hyperparameters the same as in [1]. We do not use node/edge normalization to keep the RMSE values comparable. We also add two non-equivariant, non mesh-aware baselines (GCN and MPNN), a E(3)-equivariant non mesh-aware baseline (EGNN), and a non-equivariant, mesh-aware method (SpiralNet++). See Table 1 in the uploaded figures/tables page for the comparison of features of each method. Due to time constraints, we note that we were only able to conduct a coarse hyperparameter search for MeshGraphNet and mostly used the hyperparameters reported in the paper [4].\n\nTable 2 in the uploaded tables page shows that MeshGraphNet outperforms Hermes on Heat, does worse on Wave, and performs similarly or slightly worse on Cahn-Hilliard. Interestingly, we point out that MeshGraphNet does noticeably worse than Hermes on the test mesh dataset, suggesting that Hermes can more accurately learn the dynamics function and not just the specific to the seen trajectories. This property is important as one may use different discretizations and scales of the mesh when the underlying dynamics are the same.\nSince our method assumes triangular static meshes, we could not perform experiments on FlagDynamic and CylinderFlow in [1] and instead include FlagSimple. Table 3 shows that MeshGraphNet performs slightly better than Hermes. \n\nWe will include these additional results in the final version.\n\n> In addition, Geo-FNO [2] can also perform PDE learning on irregular geometries, what is the advantages of Hermes over Geo-FNO in terms of accuracy in solving PDE on irregular domains.\n\nThank you for the reference. The paper [2] indeed tackles similar problems. Geo-FNO assumes that there exists a diffeomorphic deformation between the embedding/input space and the computational mesh and then learns the deformation and then learns the Fourier neural operator in the computation space. A key difference between our method and Geo-FNO is that Hermes does not depend on the embedding space of the mesh (e.g. embedding a rough 2D mesh in 3D) and works directly on the intrinsic mesh surface. Geo-FNO mostly evaluates on flat 2D disks (with or without holes) where vertex normals would all be parallel and thus local symmetry would not be beneficial in these scenarios. We hypothesize that due to this difference, Hermes would outperform Geo-FNO on more irregular curvatures and rougher surfaces.  Furthermore, we note that the mesh sizes considered are < 2k vertices (we consider meshes with up to 4670 vertices). It would be interesting to test Hermes on different topologies such as the torus in future work. We will include this discussion in the final version."}, {"review_id": "9w92DeuME7", "reviewer": "Reviewer_yPBJ", "summary": "The authors propose a gauge equivariant method for simulating PDEs on the surface of meshes. Different from the convolutional and attentional prior works, the authors use non-linear message passing with gauge equivariant layers. They compare to the prior works in the FAUST shape classification and the simulation of three PDEs. They find that the convolutional method is best for the shape classification, while on the PDEs, their method achieves the best results.", "questions": "- In [3], it is noted in a footnote that the kernel could be made dependent on the radius, but that the authors found it not beneficial. The method proposed here seems to also have a kernel insensitive to distance. Have the authors verified whether this is still the best choice?\n- Could the authors clarify in their manuscript that the edge feature $e_{pq}$ is a feature situated on the fiber at $p$ ?\n- Could the authors add a discussion of the computational cost of the various method in their paper?\n- The regular non-linearity used in [3] is only approximately equivariant. Could the authors clarify that in their paper?\n\nReferences:\n- [1] Suk, Julian, Pim de Haan, Phillip Lippe, Christoph Brune, and Jelmer M. Wolterink. 2022. “Mesh Neural Networks for SE(3)-Equivariant Hemodynamics Estimation on the Artery Wall.”\n- [2] Basu, Sourya, Jose Gallego-Posada, Francesco Viganò, James Rowbottom, and Taco Cohen. 2022. “Equivariant Mesh Attention Networks.” \n- [3] De Haan, P., M. Weiler, T. Cohen, and M. Welling. 2020. “Gauge Equivariant Mesh CNNs: Anisotropic Convolutions on Geometric Graphs.”", "limitations": "The authors fairly reflect the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It's great to see the code included\n- I agree with the authors that it's good to have non-linear message passing as an additional gauge-equivariant method.\n- The paper is clearly written.", "weaknesses": "- The authors should include a reference to [1], which simulates fluid dynamics with gauge equivariant methods.\n- It appears like the PDE experiments merely using the scalar features without information on geometry. This seems insufficient. Both [1] and [2] have suggestions for gauge equivariant geometric input features.\n- The paper would benefit from an evaluation on more problems, as well as a comparison to pointcloud based methods (as done in [1]).\n\n\n\n----------\nThe weaknesses have been sufficiently addressed by the rebuttal. I increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and thoughtful review. Please see our response below.\n>The authors should include a reference to [1], which simulates fluid dynamics with gauge equivariant methods.\n\nThank you for pointing out this paper. It is indeed relevant and we will include it in the final version.\n>It appears like the PDE experiments merely using the scalar features without information on geometry. This seems insufficient. Both [1] and [2] have suggestions for gauge equivariant geometric input features.\n\nWe tried using the relative tangent features proposed in [2] and observe slightly worse performance. We note that although the input/output features are scalars in the PDE datasets, the intermediate latent features are not scalars and consider irreducible representations up to order 2. In both [1] and [2], equivariant input features are constructed using either the vertex normals and/or the relative distance vector or their projections onto the tangent plane at the source node. These features are already incorporated into our model: vertex normals are used to achieve gauge equivariance and positions are used as inputs. Thus the proposed equivariant geometric input features do not necessarily convey additional information and an expressive model should be able to learn such equivariant features from data. This is rather a question of feature engineering and we agree that incorporating these features may make it easier for the model to learn the task. However, we choose the simpler option and let the model learn such features on its own, without withholding any additional information. \n>The paper would benefit from an evaluation on more problems, as well as a comparison to pointcloud based methods (as done in [1]).\n\nWe agree that more evaluation would be beneficial and add the FlagSimple dataset from [A] where the task is to predict the positions of a flag blowing in the wind. We do not perform normalization or noisy training for simplicity. On this dataset, Hermes outperforms MeshGraphNet on the test dataset for this task (see Table 4 of uploaded page).\n \nAdditionally, we experiment with more baselines. Specifically we consider two non-equivariant, non mesh-aware baselines (GCN and MPNN), a SOTA non-equivariant, mesh-aware method (MeshGraphNet [A]), an E(3)-equivariant non mesh-aware baseline (EGNN [B]), and a non-equivariant, mesh-aware method (SpiralNet++ [C]). EGNN can be considered a point cloud based method as it can infer edges. See Table 1 in the uploaded tables page for the comparison of features of each method and Table 2 for the results. Hermes outperforms baselines in most settings and does slightly worse than MeshGraphNets on Heat.  We will include these results in the final version.\n>In [3], it is noted in a footnote that the kernel could be made dependent on the radius, but that the authors found it not beneficial. The method proposed here seems to also have a kernel insensitive to distance. Have the authors verified whether this is still the best choice?\n\nYes, the kernel used here is also independent of the radius. As the datasets contain generally homogeneous graphs with similar edge distances, we surmise that having a radius-dependent kernel would not change much and do not include it for computational efficiency. As our method easily accommodates edge features, we can make the messages radius dependent by adding the relative position vector and its distance as an edge feature. We tried this, but the performance was similar. For the FlagSimple dataset, we do use the edge features as described in [A].\n>Could the authors clarify in their manuscript that the edge feature e_pq is a feature situated on the fiber at p?\n\nYes, we will clarify this point in the manuscript. We do this because edge features are often vectors relative to p and/or the edge distance which is not necessarily tied to the geometry of the edge. One could conceivably use the fiber at q or use the midpoint between p and q and parallel transport the edge features accordingly, but we choose the simpler option. We note that we did not see much performance improvement by adding edge features to the PDE datasets.\n>Could the authors add a discussion of the computational cost of the various method in their paper?\n\nWe include the forward computation time during inference on the test time dataset in Table 4 of the additional page. Surprisingly, we find that Hermes is slightly faster computation-wise than GemCNN as we generally use a small number of message passing layers and use a similar number parameters. As expected, all 3 gauge-equivariant methods are significantly more computationally expensive than standard graph neural networks.\n>The regular non-linearity used in [3] is only approximately equivariant. Could the authors clarify that in their paper?\n\nWe will clarify in Section 3 and in Proposition 1 that the regular nonlinearity is only equivariant in the limit when the number of samples goes to infinity. In experiments, we used an increased number of intermediate samples (101 vs. 7) in each regular nonlinearity compared to [3], leading to an equivariance error to random gauge transformations of approximately 1e-5 for the entire model.\n\nReferences:\n- [A] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., & Battaglia, P. (2020, October). Learning Mesh-Based Simulation with Graph Networks. In International Conference on Learning Representations.\n- [B] Satorras, V. G., Hoogeboom, E., & Welling, M. (2021, July). E (n) equivariant graph neural networks. In International conference on machine learning (pp. 9323-9332). PMLR.\n- [C] Gong, S., Chen, L., Bronstein, M., & Zafeiriou, S. (2019). Spiralnet++: A fast and highly efficient mesh convolution operator. In Proceedings of the IEEE/CVF international conference on computer vision workshops (pp. 0-0)."}, {"review_id": "Pkgzl96TpI", "reviewer": "Reviewer_CzKN", "summary": "The authors describe a message-passing mesh-based gauge equivariant architecture. The architecture is described as the natural follow-up from previous equivariant architecture. In short, an edge network aggregates information between source, target nodes, and edge features. Then, these \"messages\" are aggregated with a gauge equivariant convolution to the target node, accounting also for self-interactions.\nSuch architecture is tested on 4 applications: shape correspondences, object interactions, PDE on meshes, modeling heat eq., and Cahn-Hilliard equation on a mesh. The authors compare it with other message-passing architecture proving the soundness and effectiveness of the technique.", "questions": "Overall I am quite happy with the paper, there are a few concerns I wish the authors would address:\n- has this method been tested on incredibly large meshes (>1M vertices)?\n- I would be interested to know what are the performance be wrt a method like MeshCNN in applications like mesh classification? both in terms of time and accuracy. (if the authors know this it would be great)\n- based on the visual results, it looks like the long-term roll out is far from GT, do the authors have any idea on how to fix this?\n- would alternating different message passing layers (conv->hermes->...) help in terms of speed up without excessively harming the performances?", "limitations": "I do not see any negative societal impact.\n\nI would recommend the authors add in the next submission (if any) other tasks to the paper such as mesh segmentation and classification and compare with know baselines. This should show the strength of the method. It would also be interesting to assess the robustness of the method with corrupted topologies, this could be also an interesting point for possible follow-up.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The described architecture is the natural follow-up from previous works (as the authors describe it). Its description and illustration are well crafted, and the text is clear and well-structured. The experiments are properly designed to show the benefits of this approach with respect to previous methods, hence showing numerical and qualitative improvements. Furthermore, the authors address important concerns such as mesh finesses and roughness.", "weaknesses": "The main weakness of the method is the requirement of a reasonable topology, which cannot be overcome. This may limit severely the application of these types of methods. \nOther than the above, as stated by the authors the method is relatively slow (or slower) compared to baselines (eg GemCNN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and positive feedback.\n> The main weakness of the method is the requirement of a reasonable topology ... the method is relatively slow (or slower) compared to baselines (eg GemCNN).\n\nWe respectfully argue that meshes can approximate any Riemannian manifold, which covers many possible surfaces and objects encountered in the real world. Furthermore, most objects can be approximated as manifold meshes where there are no singularities or unreasonable boundary conditions. Many previous mesh-based methods assume manifold meshes [A, B, C], including all 3 gauge-equivariant methods (GemCNN, EMAN, Hermes). We feel that the assumption of a manifold mesh is not a severe limitation.To handle unreasonable topologies or non-manifold meshes, one could perform a classing meshing technique [D] to make it a manifold. It may also be possible to modify Hermes to handle such meshes by not using gauge equivariance on the problematic vertices and edges.\n\nRegarding the computation time, we recorded the forward computation time of each model during inference on the test time dataset (Table 3 of the tables page). We find that Hermes actually has lower computation time in the forward pass than GemCNN, due to the fact that it performs a smaller number of aggregations than GemCNN and with a similar number of parameters. We will adjust the discussion to reflect this.\n> has this method been tested on incredibly large meshes (>1M vertices)?\n\nWe restricted our experiments to meshes with up to 4670 vertices. As meshes inherently contain more data than other 3D representations such as point clouds or voxels, they require more memory to process. Though we simplify this and process meshes as graphs, even graph neural networks cannot scale to large graphs due to memory constraints [E, F]. We are not aware of any message passing graph networks that can scale up to 1M vertices on a single GPU and scaling to such large meshes would likely require significant distributed training and/or multi-scale methods with graph subsampling. It would be a good avenue to explore to apply such techniques to Hermes.\n> ... the performance be wrt a method like MeshCNN in applications like mesh classification? both in terms of time and accuracy.\n\nAs a nonlinear message passing and gauge-equivariant method, Hermes is more suited to predicting complicated interactions and irregular, rough meshes, which is supported by the performance increase on surface PDE datasets. In mesh classification, it seems that it is more important to recognize local shapes and surface curvatures and we hypothesize that interactions between neighboring vertices would likely be close to linear. Methods such as MeshCNN would likely perform better in these scenarios. Regarding computation time, all three flavors of gauge-equivariant methods require more computation time than regular graph neural networks. As MeshCNN effectively performs a convolution over the mesh faces, it would likely be more computationally efficient than either GemCNN or Hermes, which perform convolutions over edges.\n> ... it looks like the long-term roll out is far from GT\n\nWe would argue that the long-term rollouts for Hermes do not look that far from GT as the local patches seem to match GT in size and sign, though the magnitudes may differ slightly. One possible way to improve performance is to use normalization for the node and edge input features or to predict differentials (e.g. x_t+1 - x_t) as targets as done in [3]. Another way could be to use multi-step predictions in the training loss. We note that these techniques are general and not specific to our method, and performance may depend on the task.\n>would alternating different message passing layers (conv->hermes->...) help in terms of speed up without excessively harming the performances?\n\nIt’s an interesting idea! It would combine the parameter/computational efficiency of the convolution but also retain the ability to compute nonlinear messages. One could also use the gauge-equivariant convolution from GemCNN to preserve gauge-equivariance for the entire model.\n>... other tasks to the paper such as mesh segmentation and classification and compare with know baselines ... assess the robustness of the method with corrupted topologies.\n\nAs mentioned above, mesh segmentation and classification tasks are likely not suited for our method. Node correspondence results on FAUST seem to support this. We do agree that adding more baselines and datasets would be useful and so we include several more baselines (GCN, MPNN, MeshGraphNet, EGNN, SpiralNet++)\n\nWe agree that experimenting with different corrupted topologies would strengthen the paper. One could also consider non-manifold meshes, with open geometries, internal faces, or non-manifold edges or vertices where we previously mentioned some possible techniques to process such meshes.\n\nReferences\n- [A] Hanocka, R., et al. (2019). Meshcnn: a network with an edge. ACM Transactions on Graphics.\n- [B] Pfaff, T., et al. (2020). Learning Mesh-Based Simulation with Graph Networks. In International Conference on Learning Representations.\n- [C] De Haan, P., et al. 2020. “Gauge Equivariant Mesh CNNs: Anisotropic Convolutions on Geometric Graphs.”\n- [D] Shimada, K., & Gossard, D. C. (1995). Bubble mesh: automated triangular meshing of non-manifold geometry by sphere packing. In Proceedings of the third ACM symposium on Solid modeling and applications.\n- [E] Wu, Z., et al. (2020). A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems.\n- [F] Duan, K., et al. (2022). A comprehensive study on large-scale graph training: Benchmarking and rethinking. Advances in Neural Information Processing Systems."}, {"review_id": "X6csqZWObG", "reviewer": "Reviewer_P6sJ", "summary": "This paper studies the problem of gauge equivariant convolutional and attentional architectures on meshes and proposes to introduce non-linear activations to enhance the model. The experiments on three models shows the performance of the proposed method. ", "questions": "See weaknesses. ", "limitations": "See weaknesses. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. The studied problem is important.\n\nS2. The presentation is good. \n\nS3. Equivariance is an important property in message passing neural networks.", "weaknesses": "W1. The most important point is the novelty. Actually, Equivariant Mesh Attention Networks have combined Gauge Equivariance with MPNN. Your work adds non-linear activation [16] into the original models. The technical contribution is week.\n\nW2. Given that your claim that \"the combination of nonlinear message passing and gauge equivariance has not been proposed\", you should introduce \"nonlinear\" into the title and give more emphasis on the significance of it. \n\nW3. Besides the performance, I didn't see too many deep insights about the benefit of nonlinear terms. For example, how to choose the non-linear activation? How the \"non-linear\" activation function influence the solution of non-linear PDE function. Since the difference of our method is minor, you should introduce deep insights to strength your contribution. \n\nW4. How about add non-linear activation in different places? I hope to get some deep results.\n\nW5. The performance of your methods seems to not good on FAUST used in EMAN. Why you skip TOSCA? Can your method compared with [R1,R2,R3] on some standard benchmarks reported in [R1,R2,R3].  \n\n\n[R1] Learning Mesh-Based Simulation with Graph Networks, ICLR 21\n\n[R2] EAGLE- Large-scale Learning of Turbulent Fluid Dynamics with Mesh Transformers, ICLR 23\n\n[R3] Predicting Physics in Mesh-reduced Space with Temporal Attention, ICLR 22.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the detailed feedback and hope to have addressed all of your concerns.\n\n> The most important point is the novelty.\n\nWe respectfully disagree that our work only differs from [16] by adding a non-linear activation. Our work explores non-linear message passing with gauge-equivariance on meshes, an important design decision to consider with respect to the three flavors of message passing. Previous works have highlighted the importance of nonlinear message passing for modeling complex interactions [A, B] and also for equivariant graph networks [C]. It is thus natural to explore how nonlinear message passing would benefit gauge-equivariant methods. There is also a clear conceptual difference to Hermes than simply adding a nonlinearity to GemCNN. Nonlinear message passing effectively decouples the hop distance in the graph from the number of nonlinear layers and gives practitioners the ability to choose the receptive field freely from the network depth.  This makes the input graph and computational different which may be useful for certain tasks. Throughout the literature, various methods have been proposed to perform this decoupling such as graph sparsification [D], diffusion [E], or dynamic rewiring [F]. In our work, we demonstrate that this decoupling may be crucial in modeling complex interactions and surface dynamics on meshes.\n\n> Given that your claim that \"the combination of nonlinear message passing and gauge equivariance has not been proposed\", you should introduce \"nonlinear\" into the title \n\nWe agree and we will include the word “nonlinear” in the title of the final version.  We adopted our terminology from [G] which labels the three flavors of GNN as convolutional, attentional, or message-passing.  Sometimes the last is also referred to as “general message passing,” and we tried to be consistent with the literature.\n\n> Besides the performance, I didn't see too many deep insights about the benefit of nonlinear terms. \n\nWe emphasize that the contribution of our method is not the addition of specific nonlinearity, but rather the proposal of nonlinear functions for the message and update functions to compute nonlinear interaction messages in gauge-equivariant networks. This stems from an important design decision to compute nonlinear interactions between vertices, which we demonstrate to be important in predicting complex surface dynamics. Furthermore, this also decouples one axis of architecture design (network depth) from the receptive field of the network, which is important in improving model performance without additional learnable parameters. The specific nonlinearity used is a hyperparameter in our method, dependent on the task.\n\n>How about add non-linear activation in different places?\n\nWe point out that adding non-linear activation in different places does not change the nonlinearity of the message and update functions. The “degree” of nonlinearity depends on the network depth and so nonlinear activations should be interspersed with the convolution layers in the architecture. Thus the axis to consider is the number of layers in the message and update functions. In all datasets, we tuned the Hermes architecture so that the message and update functions consist of a different number of layers and activations. \n\n>Why you skip TOSCA? Can your method compared with [R1,R2,R3] on some standard benchmarks reported in [R1,R2,R3].\n\nWe were unable to obtain the TOSCA dataset as the public download URL is down. We clarify that we are not chasing state-of-the-art results but propose a new architecture that combines nonlinear message passing with gauge equivariance for meshes. We demonstrate the importance of computing nonlinear messages in predicting complex surface dynamics. However, we agree that having more baselines would be beneficial. We have included MeshGraphNet [A] as a baseline and also add two non-equivariant, non mesh-aware baselines (GCN and MPNN), an E(3)-equivariant non mesh-aware baseline (EGNN), and a non-equivariant, mesh-aware method (SpiralNet++). All methods use a similar number of parameters. See Table 1 in the uploaded figures/tables page for the comparison of features of each method. Results show that Hermes outperforms other baselines on all tasks, with the exception of MeshGraphNet on the Heat dataset. Interestingly, Hermes significantly outperforms MeshGraphNet on the test mesh dataset for Wave and Cahn-Hilliard, suggesting that Hermes can more accurately learn the dynamics function without being mesh-specific. \n\nWe also add the FlagSimple dataset from [A] and the results show that Hermes outperforms MeshGraphNet. We will include these additional results in the final version.\n\nReferences:\n- [A] Battaglia, P., Pascanu, R., Lai, M., & Jimenez Rezende, D. (2016). Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems.\n- [B] Kipf, T., Fetaya, E., Wang, K. C., Welling, M., & Zemel, R. (2018). Neural relational inference for interacting systems. In International conference on machine learning.\n- [C] Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. J., & Welling, M. (2021). Geometric and Physical Quantities improve E (3) Equivariant Message Passing. In International Conference on Learning Representations.\n- [D] Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. Advances in neural information processing systems.\n- [E] Gasteiger, J., Weißenberger, S., & Günnemann, S. (2019). Diffusion improves graph learning. Advances in neural information processing systems.\n- [F] Gutteridge, B., Dong, X., Bronstein, M. M., & Di Giovanni, F. (2023). DRew: Dynamically Rewired Message Passing with Delay. In International Conference on Machine Learning.\n- [G] Bronstein, M. M., Bruna, J., Cohen, T., & Veličković, P. (2021). Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.\n"}]

Author rebuttal:
## Summary of Response

We thank the reviewers for the detailed feedback and constructive suggestions and hope that we have addressed all concerns. The reviewers all appreciate the importance of the problem and the clear benefit of combining nonlinear message passing with a gauge-equivariant method for predicting complex dynamics. The main concern raised by reviewers seems to be more comparisons with standard baselines and on more datasets, and also an evaluation of computation time amongst the gauge equivariant methods.

We have added several new baselines with different features and also a new dataset for a stronger evaluation. We display the main results here and have also added all results in the additional figures/tables page. 

Summary of additions:

1. Added several new baselines: non-equivariant GNN variants (GCN and MPNN), a SOTA message passing, mesh-aware method  (MeshGraphNet), an equivariant, non mesh-aware message passing network (EGNN), a non-equivariant, mesh-aware baseline (SpiralNet++). Results are shown in Table 2 of the uploaded figures/tables page.
2. Included a detailed comparison of features of each method in Table 1 of the uploaded figures/tables page.
3. Added a new FlagSimple dataset to compare Hermes and MeshGraphNet (Table 2 of uploaded page).
4. Recorded the average computation time during inference for each method in Table 3 and discussed the difference in computation between GemCNN, Hermes, and non gauge-equivariant methods.
5. Performed an ablation study on residual connections in Hermes.

Additional results with new baselines and new dataset:

|  |   | Hermes | GCN | MPNN | MeshGraphNet | EGNN | SpiralNet++
|---|---|---|---|---|---|---|---|
|Heat  | Test time ($\times 10^{-3}$) | $1.18 \scriptstyle \pm 0.3$ | $152 \scriptstyle \pm 1.2$ | $2.66 \scriptstyle \pm 0.8$|$\textbf{0.93} \scriptstyle \pm 0.2$ | $3.09 \scriptstyle \pm 1.2$ | $2.82 \scriptstyle \pm 0.2$
|   | Test init ($\times 10^{-3}$) | $1.16 \scriptstyle \pm 0.3$ | $152 \scriptstyle \pm 0.9$ | $2.63 \scriptstyle \pm 0.8$|$\textbf{0.93} \scriptstyle \pm 0.2$ | $3.07 \scriptstyle \pm 1.2$ | $6.44 \scriptstyle \pm 0.1$
|   | Test mesh ($\times 10^{-3}$) |$\textbf{1.01} \scriptstyle \pm 0.3$ | $127 \scriptstyle \pm 2.2$ | $2.36 \scriptstyle \pm 0.7$|$2.41 \scriptstyle \pm 1.1$ | $8.96 \scriptstyle \pm 5.0$ | $22.0 \scriptstyle \pm 0.2$
|Wave  | Test time ($\times 10^{-3}$) | $\textbf{5.43} \scriptstyle \pm 0.8$ | $162 \scriptstyle \pm 5.0$ | $9.07 \scriptstyle \pm 1.2$|$6.26 \scriptstyle \pm 0.9$ | $45.9 \scriptstyle \pm 6.1$ | $8.88 \scriptstyle \pm 1.2$
|   | Test init ($\times 10^{-3}$) | $\textbf{3.72} \scriptstyle \pm 1.3$ | $158 \scriptstyle \pm 5.9$ | $5.24 \scriptstyle \pm 1.1$|$4.24 \scriptstyle \pm 0.6$ | $12.1 \scriptstyle \pm 3.5$ | $8.47 \scriptstyle \pm 0.6$
|   | Test mesh ($\times 10^{-3}$) | $\textbf{3.79} \scriptstyle \pm 1.3$ | $164 \scriptstyle \pm 5.1$ | $6.29 \scriptstyle \pm 1.3$ | $7.01 \scriptstyle \pm 1.9$ | $54.5 \scriptstyle \pm 18$ | $10.8 \scriptstyle \pm 0.8$
|Cahn-Hilliard  | Test time ($\times 10^{-3}$) | $\textbf{4.23} \scriptstyle \pm 0.9$ | $250 \scriptstyle \pm 7.6$ | $7.25 \scriptstyle \pm 3.1$|$\textbf{4.49} \scriptstyle \pm 0.6$ | $8.36 \scriptstyle \pm 1.4$ | $11.6 \scriptstyle \pm 3.3$
|   | Test init ($\times 10^{-3}$) | $5.21 \scriptstyle \pm 1.2$ | $383 \scriptstyle \pm 6.0$ | $7.52 \scriptstyle \pm 3.0$|$\textbf{4.64} \scriptstyle \pm 0.6$ | $10.6 \scriptstyle \pm 1.1$ | $12.9 \scriptstyle \pm 2.8$
|   | Test mesh ($\times 10^{-3}$) | $\textbf{5.34} \scriptstyle \pm 0.9$ | $391 \scriptstyle \pm 8.6$ | $7.63 \scriptstyle \pm 3.0$|$18.7 \scriptstyle \pm 7.2$ | $9.38 \scriptstyle \pm 1.7$ | $13.4 \scriptstyle \pm 2.5$
|Flag  | Test ($\times 10^{-3}$) | $\textbf{5.87} \scriptstyle \pm 0.0$| - | - |$\textbf{9.01} \scriptstyle \pm 0.1$ | - | - |

Average forward computation time during inference of each considered method on the test time dataset.

|  | GemCNN | EMAN  | Hermes | GCN | MPNN | MeshGraphNet | EGNN | SpiralNet++
|---|---|---|---|---|---|---|---|---|
|Heat (s) | 0.0124 | 0.0195 | 0.0108 | 0.0015 | 0.0012 | 0.0022 | 0.0014 | 0.0009 |
|Wave (s) | 0.0125 | 0.0177 | 0.0105 | 0.0014 | 0.0013 | 0.0022 | 0.0013 | 0.0009 |
|Cahn-Hilliard (s) | 0.0069 | 0.0092 | 0.0062 | 0.0019 | 0.0015 | 0.0021 | 0.0016 | 0.0013 |


Meta-review: All reviewers but one are strongly in favor of this work, and it appears to have both mathematical and practical value. The AC agrees with the positive assessments. In the final revision, please incorporate promised changes from the rebuttal, and importantly please include the new experiments introduced in the rebuttal.
Decision: Accept (poster)