SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Compared to other non-gauge equivariant baselines, Hermes outperforms all methods, except for certain cases with MeshGraphNet. Hermes is worse than MeshGraphNet on Heat, outperforms on Wave, and performs similarly on Cahn-Hilliard. It performs substantially better on all test mesh datasets, which may indicate that Hermes can generalize to the true dynamics function, rather than the specific dynamics seen in the training trajectories. In other words, Hermes is better adapted to use the underlying geometry while MeshGraphNet overfits to specific geometries.
Table 1: RMSE on PDE domains, using 5 runs. All values are expressed in  $\times 10^{-3}$  and gray denotes  $95\%$  confidence intervals. Hermes generally outperforms baselines, except for some cases with MeshGraphNet (MGN). Hermes outperforms MeshGraphNet on all test mesh datasets.
Hermes	GemCNN	EMAN	GCN	MPNN	MGN	EGNN	SpiralNet++
HEAT	Test time	1.18±0.3	3.88±0.8	2.93±0.9	152±1.2	2.66±0.8	0.93±0.2	3.09±1.2	2.82±0.2
Test init	1.16±0.3	3.85±0.8	2.90±0.9	152±0.9	2.63±0.8	0.93±0.2	3.07±1.2	6.44±0.1
Test mesh	1.01±0.3	3.50±0.6	2.47±0.7	127±2.2	2.36±0.7	2.41±1.1	8.96±5.0	22.0±0.3
WAVE	Test time	5.43±0.8	12.2±1.5	19.0±3.0	162±5.0	9.07±1.2	6.26±0.9	45.9±6.1	8.88±1.2
Test init	3.72±1.3	7.28±0.7	15.3±3.6	158±5.9	5.24±1.1	4.24±0.6	12.1±3.5	8.47±0.6
Test mesh	3.79±1.3	8.23±0.6	15.8±3.7	164±5.1	6.29±1.3	7.01±1.9	54.5±18	10.8±0.8
CAHN-HILLIARD	Test time	3.48±0.9	13.8±12	8.41±4.0	250±7.6	7.25±3.1	4.49±0.6	8.36±1.4	11.6±3.3
Test init	4.23±1.5	14.9±12	8.75±4.0	383±6.0	7.52±3.0	4.64±0.6	10.6±1.1	12.9±2.8
Test mesh	5.41±0.8	14.1±12	43.8±27	391±8.6	7.63±3.0	18.7±7.2	9.38±1.7	13.4±2.5
Figure 3: Errors from long-horizon prediction rollouts on unseen meshes, given only initial conditions. Error bars denote standard error over 5 runs averaged over unseen meshes (2 meshes for heat/wave, 1 for Cahn-Hilliard). GemCNN has exploding errors with increasing  $t$  on Heat and Wave. Hermes generally outperforms GemCNN and EMAN, with the exception of EMAN on Wave.
On the FlagSimple dataset (Table 10), Hermes outperforms MeshGraphNet considerably, suggesting that Hermes is not limited to static meshes and can handle temporally changing meshes well. Note that MeshGraphNet was modified to have a similar number of parameters as Hermes and so the results are different than reported in Pfaff et al. [20].
5.1 Long-horizon prediction rollouts
Using a representative random run, we generate predictions from each model autoregressively given only the initial conditions. We generate predictions on the unseen meshes (test mesh) and roll out the entire trajectory of the PDE ( $T_{\mathrm{max}} = 200$ ). Figure 3 shows how the errors change with the increasing rollout horizon. For GemCNN, the errors accumulate quickly and the predictions diverge for Heat and Wave. EMAN performs similarly to Hermes on Heat but underperforms on Cahn-Hilliard. All methods eventually diverge on Wave; this may be due to the fact that the wave amplitude oscillates multiple times over the longer horizon and so it may be more difficult to predict the periodic nature.
Table 2 shows prediction samples at  $T + 50$  generated autoregressively given only initial conditions. GemCNN fails on the Heat dataset and does not produce the correct spatial patterns for Wave and Cahn-Hilliard. EMAN performs well on Wave, but fails on Cahn-Hilliard. Hermes gives fairly realistic predictions on all datasets. See Appendix E.1 for more samples with varying  $T$ .
5.2 Mesh Fineness
Here we investigate how mesh fineness impacts performance with respect to the three flavors of message passing. Our intuition is that as meshes become finer, the features at each node become more similar. Thus the dynamics between nodes would become more linear and convolutional approaches may perform comparably with nonlinear message passing.
For this experiment, we solve the heat and wave equations on a single mesh ("armadillo"). To generate different mesh resolutions, we simplify the original mesh (\# vertices  $= 172$ , 974) to  $\{348, 867, 1731, 3461, 8650\}$  vertices using quadric decimation [39] (1731 vertices were used for the main results in Table 1). See Figure 9 in Appendix E.2 for data visualizations. We generate 15 trajectories with  $T_{\mathrm{max}} = 100$  and use 5 of the 15 trajectories as test init and use  $T = 81, \ldots, 100$  of the remaining 10 trajectories for the test time. As we use a single mesh, we do not test generalization to unseen meshes. We use 3 random seeds for each method.
Table 2: Qualitative prediction samples rolled out to the full path using only initial conditions. GemCNN completely fails on Heat, while EMAN fails on Wave. Hermes predicts the spatial patterns accurately and outperforms GemCNN and EMAN on all datasets.
Dataset	GemCNN	EMAN	Hermes	Ground Truth
Heat (T+50)
Wave (T+50)
Cahn-Hilliard (T+50)
Figure 4: Performance for varying mesh fineness for heat on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. The errors for GemCNN and EMAN increase as the mesh becomes coarser, while Hermes performs similarly throughout.
Figure 4 shows the results for the heat dataset. RMSE on both the test time and test init datasets increase for GemCNN and EMAN as the mesh becomes coarser (decreasing number of vertices). This coincides with the intuition that coarser meshes have more nonlinear dynamics between nodes and thus contributing to the increase in errors. On the other hand, Hermes still quantitatively outperform GemCNN and EMAN across all mesh resolutions. It is also robust to varying mesh fineness and has similar error values throughout.
On the wave dataset, we find that Hermes still outperforms GemCNN and EMAN at each resolution though the gap is not as large, see Figure 10 (Appendix E.2). There is a surprising decreasing trend in errors for all methods as the mesh becomes finer. This indicates that mesh fineness may not be the only factor at play: it is possible that the type of dynamics (PDE) used and the specific architecture (e.g. graph receptive field) can affect the results. Another possible explanation is that as we use a single simplified mesh, there may be some artifacts that affect the wave PDE simulations.
5.3 Surface Roughness
We investigate whether surface roughness affects the performance gap between GemCNN and Hermes. We use the same armadillo mesh with 1,731 vertices from before and extract the vertex normals. The vertex normals are then randomly scaled using a Gaussian distribution  $\mathcal{N}(0,s^2)$ , where  $s\in \{0.1,0.5,1,1.5,3\}$ . The vertex coordinates are then modified by adding these scaled normals to the original coordinates, resulting in different surface roughnesses. See Figure 11 in Appendix E.3 for visualizations. We use the same settings as in the fineness experiments in Section 5.2.
Similarly to the results on mesh fineness, we find that Hermes retains the performance advantage over GemCNN and EMAN across all roughness scales. Its RMSE errors are also nearly constant
Figure 5: Performance for varying surface roughness for heat on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. Increasing scale increases the surface roughness of the mesh. Hermes outperforms GemCNN and EMAN across different roughness scales and is much more robust.
Table 3: Mean forward computation time in seconds (ms) during inference on test time PDE datasets.
GemCNN	EMAN	Hermes	GCN	MPNN	MeshGraphNet	EGNN	SpiralNet++
HEAT (ms)	12.4	19.5	10.8	1.5	1.2	2.2	1.4	0.9
WAVE (ms)	12.5	17.7	10.5	1.4	1.3	2.2	1.3	0.9
CAHN-HILLIARD (ms)	6.9	9.2	6.2	1.9	1.5	2.1	1.6	1.3
throughout indicating robustness to surface roughness. Surprisingly, GemCNN seems to perform increasingly better as the surface becomes rougher while EMAN seems to perform roughly equal. On the wave dataset, Hermes still outperforms other baselines at most roughness scales, see Figure 12 in Appendix E.3. There does not seem to be any correlation in errors with surface roughness.
5.4 Computation Time
As the edge and node networks  $\phi_{n},\phi_{e}$  consist of multi-layer perceptrons, one might think Hermes requires more computation time than its simpler convolutional or attentional counterparts. Table 3 shows that this is not the case; Hermes has a lower computation time than both GemCNN and EMAN. As we control for a similar number of parameters in each architecture, there are fewer aggregations in Hermes than in GemCNN or EMAN resulting in a lower average computation time in the forward pass. For EMAN in particular, due to its attention mechanism, we find that it uses many more parameters than either GemCNN or Hermes. Thus constraining EMAN to have a similar number of parameters as GemCNN restricts its expressivity. On the other hand, Hermes is very flexible as it can use a different number of layers for the edge and node networks. Even though Hermes uses a slightly lower hidden dimension than GemCNN in all domains, the results show that this does not hamper model expressivity and it still achieves higher performance. As expected, all three gauge equivariant methods are significantly more computationally expensive than standard graph neural networks.
5.5 Residual connection ablation
As Hermes models the self-influence of nodes, we also test whether a residual connection is necessary. Table 12 in Appendix E.4 shows the ablation of the residual connection. Having a residual connection improves performance on Heat, but decreases performance slightly on Wave and Cahn-Hilliard. Thus the residual connection seems to be task-dependent and should be considered a hyperparameter. In our experiments, a residual connection was used for each message passing block.
Learning over meshes Several different mesh operators have been studied within computer graphics, in the context of shape classification [40-42], dense shape correspondence [43, 36], mesh segmentation [44-46]. Due to the success of graph neural networks (GNNs) [26, 27, 38], several approaches use GNNs to process meshes [21, 22, 47, 36] and incorporate geometric information via geodesic convolutions, anisotropic kernels, dual graphs, or spiral operators. Pfaff et al. [20] introduce a state of the art method for learning simulations with meshes by representing a graph in mesh space and in world space. In this work, we introduce a novel architecture that combines nonlinear message passing with explicit equivariance to local gauge transformations.
Gauge Symmetry Most works on non-Euclidean manifolds have used convolutions on discretized patches of the manifold, approximating them to be Euclidean. Masci et al. [21] define the patch operator using local geodesic coordinate systems. Monti et al. [23] use a mixture of parametric Gaussian kernels. All of these works use (linear) convolutions, unlike our work. Boscaini et al. [22] uses anisotropic diffusion kernels, but uses the principal curvature direction as the preferred gauge which may be ill-defined for some shapes. On the other hand, several works have considered equivariance to gauge symmetry. Cohen et al. [6] proposes gauge equivariant convolutions on the icosahedron. Most similar to our work is that of [9, 48, 10], which explicitly incorporate gauge equivariant kernels for meshes. de Haan et al. [9] use convolutions while He et al. [48] and Basu et al. [10] use attention to discrete and continuous gauge transformations, respectively. In this work, we complete the picture and propose general nonlinear message passing with gauge equivariance for meshes. For more in-depth theory and discussion of local gauge equivariance on manifolds, see [49].
Solving PDEs on surfaces With the rise of physics-informed neural networks [50-52] and their increased sample efficiency and performance over classical methods, several recent works have focused on solving complex partial differential equations with deep learning, e.g. fluid dynamics [53-55], thermodynamics [56], structural mechanics [57, 58], and material science [59, 60]. Some approaches learn the finite-dimensional [51, 53, 54] or infinite-dimensional [61, 17, 62] solution operators. However, there has been less work on applying deep learning to solving PDEs on surfaces embedded in 3D space using meshes, with some exceptions [63-65]. Fang and Zhan [63] solve the Laplace-Beltrami equation over a 3D surface with neural networks while Tang et al. [64] propose an extrinsic approach. However, both methods are mesh-free and only use points and their normals, discarding any connectivity information. Li et al. [65] extend Fourier neural operator [66] and learns a diffeomorphic deformation between the input space and the computational mesh. While Geo-FNO depends on the embedding space of the mesh (e.g. embedding a rough 2D mesh in 3D), our method works directly on the intrinsic mesh surface. Perhaps most relevant to this paper is [67], which extends GemCNN [9] in several ways to predict hemodynamics on artery walls. In this work, we propose a nonlinear message passing architecture for meshes that retains the underlying geometry and demonstrate their effectiveness in predicting a variety of surface dynamics.
We introduce a novel architecture, Hermes, that performs gauge equivariant, nonlinear message passing for meshes. Hermes complements convolutional GemCNN and attentional EMAN and completes development of the 3 flavors of message passing in the gauge equivariant setting. In the context of meshes, similar to GNNs, there seems to be a tradeoff between simple linear operations such as convolutions versus nonlinear message passing. Convolutions are computationally efficient and can perform well on simpler tasks such as shape correspondence, see Table 10 in Appendix E. However, when the interactions are more complicated such as in surface PDEs, nonlinear message passing surpass linear schemes. By decoupling the degree of nonlinearity from the depth of the network and receptive field, Hermes outperforms GemCNN and EMAN significantly on the PDE datasets and produces realistic predictions given only initial conditions.
A limitation is that Hermes may use more parameters depending on the architectures of the edge and node networks. In particular, both EMAN and Hermes cannot scale well to meshes with a large number of vertices naively, and one may need to consider more sophisticated approaches such as multi-scale or graph expander approaches. Furthermore, the architecture search space for Hermes is larger than that of GemCNN or EMAN, as one needs to consider different combinations of numbers of layers in the edge and node networks, along with the number of message passing blocks.
For future work, one direction is to consider different dynamics such as non-stationary or chaotic dynamics, and other PDEs important in real world applications such as Navier-Stokes for climate or blood flow. Another direction is to analyze the design space of gauge equivariant networks. While GNNs have been extensively studied, far less work exists for mesh methods. GNNs are often highly task-specific and there are many design dimensions (e.g. residual connections, message passing iterations, etc.) to consider [68]. It would be particularly helpful for practitioners to have guidelines on when to use gauge equivariance or message passing over simpler approaches. This work aims to be a first step in this direction by demonstrating Hermes as a good fit for predicting nonlinear dynamics on meshes.
We sincerely thank Vedanshi Shah for the initial data generation code and Pim de Haan for clarifying parts of GemCNN. This work is supported in part by NSF 2107256 and 2134178. This work was completed in part using the Discovery cluster, supported by Northeastern University's Research Computing team.
[1] Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. *Polygon mesh processing*. CRC press, 2010.
[2] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192, 2020.
[3] Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15272-15281, 2021.
[4] Franco Dassi, Bree Ettinger, Simona Perotto, and Laura M Sangalli. A mesh simplification strategy for a spatial regression analysis over the cortical surface of the brain. Applied Numerical Mathematics, 90:111-131, 2015.
[5] Mayur Mudigonda, Sookyung Kim, Ankur Mahesh, Samira Kahou, Karthik Kashinath, Dean Williams, Vincen Michalski, Travis O'Brien, and Mr Prabhat. Segmenting and tracking extreme climate events using neural networks. In Deep Learning for Physical Sciences (DLPS) Workshop, held with NIPS Conference, 2017.
[6] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pages 1321-1330. PMLR, 2019.
[7] Kenneth G Powell, Philip L Roe, and James Quirk. Adaptive-mesh algorithms for computational fluid dynamics. In Algorithmic trends in computational fluid dynamics, pages 303-337. Springer, 1993.
[8] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Velicković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[9] Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=Jnspzp-oIZE.
[10] Sourya Basu, Jose Gallego-Posada, Francesco Viganò, James Rowbottom, and Taco Cohen. Equivariant mesh attention networks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=3IqqJh2Ycy.
[11] Marcelo Bertalmio, Li-Tien Cheng, Stanley Osher, and Guillermo Sapiro. Variational problems and partial differential equations on implicit surfaces. Journal of Computational Physics, 174 (2):759-780, 2001.
[12] Thomas D Economon, Francisco Palacios, Sean R Copeland, Trent W Lukaczyk, and Juan J Alonso. Su2: An open-source suite for multiphysics simulation and design. Aiaa Journal, 54 (3):828-846, 2016.
[13] Steffen Marburg and Bodo Nolte. Computational acoustics of noise propagation in fluids: finite and boundary element methods, volume 578. Springer, 2008.
[14] Amy Novick-Cohen. Chapter 4 the cahn-hilliard equation. volume 4 of Handbook of Differential Equations: Evolutionary Equations, pages 201-228. North-Holland, 2008. doi: https://doi.org/10.1016/S1874-5717(08)00004-2. URL https://www.sciencedirect.com/science/article/pii/S1874571708000042.
[68] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009-17021, 2020.
[69] O. Tange. Gnu parallel - the command-line power tool. ;login: The USENIX Magazine, 36(1): 42-47, Feb 2011.
[70] Martin Reuter, Silvia Biasotti, Daniela Giorgi, Giuseppe Patanè, and Michela Spagnuolo. Discrete laplace-beltrami operators for shape analysis and segmentation. Computers & Graphics, 33(3):381-390, 2009.
[71] Will Schroeder, Kenneth M Martin, and William E Lorensen. The visualization toolkit an object-oriented approach to 3D graphics. Prentice-Hall, Inc., 1998.
[72] A. Logg and G. N. Wells. DOLFIN: automated finite element computing. ACM Transactions on Mathematical Software, 37, 2010. doi: 10.1145/1731022.1731030.
[73] Carl T Kelley. Solving nonlinear equations with Newton's method. SIAM, 2003.
[74] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Piscataway, NJ, USA, June 2014. IEEE.
[75] Ronald J Williams and David Zipser. Gradient-based learning algorithms for recurrent. Backpropagation: Theory, architectures, and applications, 433:17, 1995.
A.1 Local Coordinate Frame
As discussed in the main text, the reference neighbor  $d$  defines the basis  $\{e_1^b,e_2^b\}$  for the tangent space  $T_{b}M$  (see Figure 1a). Denote by  $\log_b(x_d) = \mathrm{Proj}_{T_bM}(x_d) - x_b$  the vector from  $x_{b}$  (the coordinates of  $b$ ) to the orthogonal projection of  $x_{d}\in \mathbb{R}^{3}$  (the coordinates of  $d$ ) onto the tangent plane  $T_{b}M$ . Let  $e_1^b = \log_b(x_d) / \| \log_b(x_d)\|$  and  $e_2^b = n_b\times e_1^b$ . Then the orientation of every neighbor  $v\in \mathcal{N}_b$  can be represented with polar angles, where
$$
\theta_ {v} = \operatorname {a t a n 2} \left(e _ {2} ^ {b} ^ {\top} \log_ {b} (v), e _ {1} ^ {b} ^ {\top} \log_ {b} (v)\right) \tag {8}
$$
is the angle between  $v$  and the reference neighbor after projection to the tangent plane.
A.2 Parallel Transport
Let  $f_{p}$  and  $f_{q}$  be feature vectors at nodes  $p, q$ , respectively, where  $q$  is adjacent to  $p$ . As features at different nodes live in different tangent spaces and have different bases, in order for the anisotropic kernel to be applied to  $f_{q}$  for each  $q \in \mathcal{N}_{p}$ , each  $f_{q}$  must be parallel transported to  $T_{p}M$  and be in the same gauge.
Let  $f_{p}$  and  $f_{q}$  be feature vectors at nodes  $p, q$ , respectively, where  $q$  is adjacent to  $p$ . As mentioned in the main text, the parallel transporter  $g_{q\rightarrow p}$  first aligns the tangent plane  $T_{p}M$  to the tangent plane  $T_{q}M$  by a 3D rotation and then transforms the gauge at  $q$  to the gauge at  $p$  by a planar rotation.
Let  $n_p, n_q$  be the normal at  $p$  and  $q$ , and  $R_{qp} \in SO(3)$  be the 3D rotation required to align  $T_qM$  and  $T_pM$ . Then  $g_{q \to p} \in [0,2\pi)$  is given by
$$
g _ {q \rightarrow p} = \operatorname {a t a n 2} \left(\left(R _ {q p} e _ {2} ^ {q}\right) ^ {\top} e _ {1} ^ {p}, \left(R _ {q p} e _ {1} ^ {q}\right) ^ {\top} e _ {1} ^ {p}\right), \tag {9}
$$
where  $(e_1^q, e_2^q)$  and  $(e_1^p, e_2^p)$  are the frames at node  $q$  and  $p$ , respectively.
B Gauge-Equivariant Message Passing Methods
In this section, we provide the architectures of GemCNN and EMAN for the wave dataset to compare with Hermes (see Figure 2), and prove that Hermes is equivariant to local gauge transformations.
B.1 GemCNN
The architecture for GemCNN is given in Figure 6. Note that while the equations in GemCNN (see Equations 1) allow for self-interactions, the implemented architecture does not actually use self-interaction kernels, possibly due to residual connections.
Figure 6: GemCNN network architecture for the Wave PDE dataset. There are three message blocks, each with 2 gauge-equivariant convolutional layers. To illustrate the computations within the convolutional layer, we use the example mesh from Figure 1a and show only the computations for node  $b$  for clarity.
B.2 EMAN
For completeness, the full equations for EMAN are given in Equations (10)-(12) (see also Appendix C.5 in [10]).
Figure 7: EMAN network architecture for the Wave PDE dataset. There are three message blocks, each with 2 attention and gauge-equivariant convolutional layers. To illustrate the computations within the edge and node networks, we use the example mesh from Figure 1a and show only the computations for node  $b$  for clarity.
$$
f _ {p} ^ {\prime} = \alpha_ {p p} K _ {\text {v a l u e}} ^ {\text {s e l f}} f _ {p} + \sum_ {q \in \mathcal {N} _ {p}} \alpha_ {p q} K _ {\text {v a l u e}} ^ {\text {n e i g h}} \left(\theta_ {q}\right) \rho \left(g _ {q \rightarrow p}\right) f _ {q}, \tag {10}
$$
$$
\alpha_ {p p} = \operatorname {s o f t m a x} \left(\frac {\left(K _ {\text {k e y}} ^ {\text {s e l f}} f _ {p}\right) ^ {\top} \left(K _ {\text {q u e r y}} f _ {p}\right)}{\sqrt {C _ {\text {a t t}}}}\right), \tag {11}
$$
$$
\alpha_ {p q} = \operatorname {s o f t m a x} \left(\frac {\left(\left\| _ {q \in \mathcal {N} _ {p}} K _ {\text {k e y}} ^ {\text {n e i g h}} \left(\theta_ {p q}\right) \rho \left(g _ {q \rightarrow p}\right) f _ {q}\right) ^ {\top} \left(K _ {\text {q u e r y}} f _ {p}\right)}{\sqrt {C _ {\text {a t t}}}}\right), \tag {12}
$$