SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $K_{\mathrm{key}}^i, K_{\mathrm{value}}^i, i \in \{\mathrm{self}, \mathrm{neigh}\}$  are the usual key and value matrices for the self-interaction and neighbor interaction terms, and  $K_{\mathrm{query}}$  is the query matrix. The  $\|_{q \in \mathcal{N}_p}$  denotes a concatenation of the vectors for all  $q \in \mathcal{N}_p$ .
Figure 7 shows the architecture for EMAN. There is an EMANAttention layer to compute the attention weights  $\alpha_{pq}$ , which are passed to the aggregation function. Similarly to GemCNN, the implemented architecture does not use self-interaction kernels.
B.3 Hermes
We provide the proof of Proposition 1 (stated here again for convenience). Part of the proof uses the fact that anisotropic convolutions over the tangent space are equivariant under the appropriate kernel constraint as shown in de Haan et al. [9].
Proposition 2. Hermes is equivariant to local gauge transformations  $g^p$  for any  $p \in \mathcal{V}$  of a mesh.
Proof. Using Equations (3)-(7), we first consider the simpler case where the edge and node networks have one gauge-equivariant layer each, i.e.  $N_{e} = N_{n} = 1$ . Let  $G = SO(2)$  be the gauge group. Let  $p \in \mathcal{V}$  be some vertex and  $q \in \mathcal{N}_p$  be some vertex adjacent to  $p$ . The inputs to the edge
network are  $f_{p}, f_{q}, e_{pq}$  (the source node, target node, and edge features, respectively), which are all representations of  $G$ . With respect to the gauge transformation  $g^{p} \in G$ , they transform as
$$
f _ {p} \mapsto \rho_ {\text {i n}} (- g ^ {p}) f _ {p}, \quad \rho (g _ {q \rightarrow p}) f _ {q} \mapsto \rho_ {\text {i n}} (- g ^ {p}) \rho (g _ {q \rightarrow p}) f _ {q}, \quad e _ {p q} \mapsto \rho_ {\text {i n}} (- g ^ {p}) e _ {p q},
$$
where  $f_{q}$  is first parallel transported to the gauge at  $p$  before applying the group action ( $e_{pq}$  is assumed to already be in this gauge). As  $h_{pq}$  is the direct product of  $f_{p}, \rho(g_{q \to p})f_{q}, e_{pq}$  (Equation 3), it also transforms as  $h_{pq} \mapsto \rho_{\mathrm{in}}(-g^{p})h_{pq}$ . For the edge network to be equivariant with respect to  $g^{p}$ , we need to show that  $\sigma\left(K_{\mathrm{neigh}}^{1}(\theta_{q} - g^{p})(\rho_{\mathrm{in}}(-g^{p})h_{pq})\right) = \rho_{\mathrm{out}}(-g^{p})\left(\sigma\left(K_{\mathrm{neigh}}^{1}(\theta_{q})h_{pq}\right)\right)$ . We compute the left-hand side
$$
\begin{array}{l} \sigma \left(K _ {\text {n e i g h}} ^ {1} \left(\theta_ {q} - g ^ {p}\right) \left(\rho_ {\text {i n}} (- g ^ {p}) h _ {p q}\right)\right) = \sigma \left(\rho_ {\text {o u t}} (- g ^ {p}) K _ {\text {n e i g h}} ^ {1} \left(\theta_ {q}\right) \rho_ {\text {i n}} (g ^ {p}) \rho_ {\text {i n}} (- g ^ {p}) h _ {p q}\right) \\ = \rho_ {\text {o u t}} (- g ^ {p}) \sigma \left(K _ {\text {n e i g h}} ^ {-} (\theta_ {q}) h _ {p q}\right) \\ \end{array}
$$
as desired, where we have used the kernel constraint  $K_{\mathrm{neigh}}^{1}(\theta_{q} - g^{p}) = \rho_{\mathrm{out}}(-g^{p})K_{\mathrm{neigh}}^{1}(\theta_{q})\rho_{\mathrm{in}}(g_{q\to p})$  and the fact that  $\sigma$  is equivariant with respect to the  $\rho_{\mathrm{out}}$  representation of  $G$  and thus commutes with  $\rho_{\mathrm{out}}(-g^p)$ .
For the node network,  $K_{\mathrm{self}}^{1}$  does not depend on  $\theta_q$  and so the kernel constraint is simpler:  $K_{\mathrm{self}}^{1} = \rho_{\mathrm{out}}(-g^{p})K_{\mathrm{self}}^{1}\rho_{\mathrm{in}}(g^{p})$ . It is easy to see that  $\sigma \left(K_{\mathrm{self}}^{1}\rho_{\mathrm{in}}(-g^{p})h_{p}\right) = \rho_{\mathrm{out}}(-g^{p})\left(\sigma \left(K_{\mathrm{self}}^{1}h_{p}\right)\right)$  and we omit the proof as it is nearly identical to that of the edge network.
The remaining part is the aggregation (sum) (Equation 5). As  $\rho(-g^p)$  is a linear group action (representation) of  $G$ ,  $\rho(-g^p)m_p = \sum_{q \in \mathcal{N}_p} \rho(-g^p)m_{pq}$ .
We have so far proved that Hermes is equivariant for  $N_{e} = N_{n} = 1$ . This suffices to prove for all  $N_{e} \geq 1, N_{n} \geq 1$ . The edge network  $\phi_{e}$  is a composition of the gauge-equivariant convolutions and activations interleaved with each other. The regular nonlinearity is gauge-equivariant when the number of intermediate samples used in the discrete Fourier transform goes to  $\infty$  and is approximately gauge-equivariant with finite  $N$ . As the composition of equivariant functions is also equivariant,  $\phi_{e}$  is equivariant to any  $g^{p}$  for any  $N_{e} \geq 1$ , and similarly,  $\phi_{n}$  is equivariant to any  $g^{p}$  for  $N_{n} \geq 1$ . Thus Hermes is equivariant to any local gauge transformation  $g^{p} \in G$ , in the limit  $N \to \infty$ .
C Experiment Domains
C.1 Heat/Wave Equation
The heat and wave partial differential equations are given by
$$
\frac {\partial u}{\partial t} = \alpha \tilde {L} u, \quad (\text {H e a t}) \quad \frac {\partial^ {2} u}{\partial t ^ {2}} = c ^ {2} \tilde {L} u, \quad (\text {W a v e}) \tag {13}
$$
where  $u$  is the temperature or the displacement for the heat and wave equations respectively,  $\alpha$  is thermal diffusivity,  $c$  is a constant, and  $\tilde{L}$  is the symmetric cotangent Laplacian [70]. Note that  $\tilde{L}$  is the discrete version of the Laplace-Beltrami operator over a Riemannian manifold. For a scalar function  $u$ ,  $\tilde{L}(u)$  is given as,
$$
\left(\tilde {L} (u)\right) _ {i} = \frac {1}{2 A _ {i}} \sum_ {j \in \mathcal {N} (i)} \left(\cot \alpha_ {i j} + \cot \beta_ {i j}\right) \left(u _ {j} - u _ {i}\right), \tag {14}
$$
where  $\mathcal{N}(i)$  denotes the adjacent vertices of  $i$ ,  $\alpha_{ij}$  and  $\beta_{ij}$  are the angles opposite edge  $(i,j)$ , and  $A_i$  is the vertex area of  $i$ , where we use the barycentric cell area.
We use 8 example meshes from the PyVista software [30] and use 6 for training and 2 for testing generalization (test mesh) to unseen meshes. We further generate test sets to evaluate generalization on future timesteps (test time) and random initialization (test init) conditions. We simulate 5 samples for  $T_{\mathrm{max}} = 200$  timesteps, using 3 samples from  $T = 0, \dots, 150$  for training and the remaining  $T = 151, \dots, 200$  as test time samples. The unseen 2 samples from  $T = 0, \dots, 200$  are used for test init samples. Some meshes contained too many nodes to be computationally feasible so we simplify these meshes to be less than 5,000 nodes using quadric decimation [71]. We use  $\alpha = c = 1$ . See Tables 4, 5 for visualizations of the meshes and how the features evolve over time.
The heat and wave equations were simulated using the finite difference method using the forward Euler scheme. As each mesh has a different Laplacian, we tune the time discretization  $dt$  for each mesh. To initialize, we randomly generate Gaussian bumps over the mesh (20% of all nodes for heat, 2.5% for wave). While such initialization is not necessarily realistic, we chose to incorporate more superposition and interference to generate more diverse dynamics. The task is to predict the temperature (heat) or displacement (wave) at each vertex at the next time step, given a short history of values. Both the inputs and outputs are trivial representations.
C.2 Cahn-Hilliard Equation
The Cahn-Hilliard equation [31] is often represented by the following two coupled second-order equations:
$$
\frac {\partial c}{\partial t} - M L (\mu) = 0, \quad \mu - \frac {d f}{d c} + \lambda L (c) = 0, \tag {15}
$$
where  $c$  is the fluid concentration,  $M$  is the diffusion coefficient,  $\mu$  is the chemical potential,  $L$  is the Laplacian,  $f$  is the double-well free energy function, and  $\lambda$  is a positive constant.
We use 4 meshes from the PyVista software [30] and use 3 for training and 1 for testing (test mesh). As in the heat and wave datasets, we use  $N = 5$  samples with  $T_{\mathrm{max}} = 200$  timesteps and split the training and test datasets similarly. Some meshes contained too many nodes to be computationally feasible so we simplify such meshes using the same procedure as in the heat and wave datasets. See Tables 6 for visualizations.
The Cahn-Hilliard equation was simulated using the DOLFINx software [72] with time discretization  $dt = 5 \times 10^{-6}$ ,  $M = 1$ ,  $f = 100c^2 (1 - c^2)$ , and we use the  $\theta$ -method with  $\theta = 0.5$ . We randomize  $\lambda$  uniformly from [0.01, 0.02] for each data sample. The initial concentrations are randomly sampled from  $\mathcal{N}(0.6, 0.05)$  for each vertex. The solver formulates the coupled second-order PDEs in variational form, and solves it using the Newton-Krylov method [73].
The task is to predict the concentration at each vertex at the next time step, given a short history of values. Both the inputs and outputs are trivial representations.
C.3 Shape Correspondence
The FAUST dataset [74] consists of 80 train and 20 test samples, where 2 unseen human subjects are used for testing. Following previous work [9], the inputs are  $xyz$  coordinates with representation  $\rho_0$  and the network outputs trivial features, followed by linear post-processing layers. The task is shape correspondence between meshes, where for every mesh, each vertex is registered and denotes the same position on the body.
C.4 Object interactions
In this domain, we consider 50 objects on a mesh with 250 vertices, see Figure 8. Each mesh is generated parametrically using the random hills generation procedure provided in VTK [71]. For training, we generate 20 meshes using different random seeds but the same random hills parameters, randomly initialize the object positions and orientations, and then simulate for a 100 timesteps. At each timestep, we select the action with weighted probabilities:  $10\%$  turn left,  $80\%$  move forward, and  $10\%$  turn right. For testing, we generate 100 meshes with randomized hill parameters and simulate forward 20 timesteps. We represent the current state of the system using the vertex coordinates, a one-hot embedding of object occupancy, and object orientations. The orientations are 2-dimensional vectors that represent the heading of the object on the tangent place at the current vertex, pointing towards the position of the neighboring node projected onto the tangent plane.
The task is to correctly predict the object occupancy and orientations of the next timestep, given the current state and action  $f(s_{t},a_{t}) = \hat{s}_{t + 1}$ . The inputs to the graph network are  $\rho_0$  features, except for the orientations, which are  $\rho_{1}$ . The graph network outputs both the logits of the occupancy and the orientation as  $\rho_0$  and  $\rho_{1}$  features, respectively. As the occupancy is discrete, we add a linear post-processing layer after the graph network for only the occupancy outputs and compute the negative log likelihood given the ground truth occupancy. For orientations, we use root mean square error (RMSE) from the orientations given by the graph network.
Table 4: Heat equation dataset of all meshes used with  $T_{\mathrm{max}} = 200$ . We simulate  $N = 5$  samples for each mesh. Of the training meshes, we use  $T = 150, \dots, 200$  as the test time dataset and 2 unseen samples as the test init dataset. The armadillo and urn meshes were used as the test mesh dataset.
Object	Split	No. vertices	t=0	t=50	t=200
Armadillo	Test	1,731
Bunny	Train	502
Lucy	Train	2,501
Sphere	Train	842
Spider	Train	4,670
Urn	Train	2,454
Woman	Train	2,998
Table 5: Wave equation dataset of all meshes used with  $T_{\mathrm{max}} = 200$ . We simulate  $N = 5$  samples for each mesh. Of the training meshes, we use  $T = 150, \dots, 200$  as the test time dataset and 2 samples as the test init dataset. The armadillo and urn meshes were used as the test mesh dataset.
Object	Split	No. vertices	t=0	t=50	t=200
Armadillo	Test	1,731
Bunny	Train	502
Lucy	Train	2,501
Sphere	Train	842
Spider	Train	4,670
Urn	Train	2,454
Woman	Train	2,998
Table 6: Cahn-Hilliard equation dataset of all meshes used with  $T_{\mathrm{max}} = 200$ . We simulate  $N = 5$  samples for each mesh. Of the training meshes, we use  $T = 150, \dots, 200$  as the test time dataset and 2 samples as the test init dataset. The armadillo and urn meshes were used as the test mesh dataset.
Object	Split	No. vertices	T=0	T=50	T=200
Bunny	Train	502
Ellipsoid	Train	1,942
Sphere	Train	842
Supertoroid	Test	2,940
Figure 8: Objects interactions on a mesh. Purple denotes an unoccupied vertex and other colors represent different objects. Images are different random initializations of the underlying mesh and object position and orientations.
D Training Details and Baselines
For classification tasks such as FAUST or for predicting object occupancy in the Objects dataset, post-processing linear layers are used. For FAUST, we find that training for longer (400 epochs vs 100 from [9]) benefits all models. We use cross entropy loss for FAUST, a combination of binary cross entropy and RMSE for Objects, and RMSE for all the PDE datasets. For the PDE datasets, we use truncated backpropagation through time [75] and roll out the network predictions for 3 timesteps using a history of 5 timesteps.
We use the regular nonlinearity [9] to maintain gauge equivariance, with ReLU activations and 101 samples, for all models. We use a band limit of 4 for the gauge-equivariant kernels. Throughout,
Table 7: Details of architecture and hyperparameters
Dataset	FAUST	Objects	Heat	Wave	Cahn-Hilliard
No. blocks	5	1	1	2	1
No. layers in φe per block	2	3	4	2	4
No. layers in φn per block	0	0	3	1	1
Hidden representation	⊕2i=010ρi	⊕2i=012ρi	⊕2i=012ρi	⊕2i=012ρi	⊕2i=012ρi
No. post processing layers	2	1	0	0	0
Epochs	400	100	100	100	100
Batch size	1	2	1	1	1
Optimizer	Adam
Learning rate	7 × 10-3	5 × 10-4	1 × 10-4	5 × 10-4	5 × 10-3
Table 8: Comparison table of relevant graph neural networks and their features.
Graph Flavors	Features
Models	Convolutional	Attentional	Message Passing	E(3)-Equivariant	Gauge-Equivariant	Mesh Aware
GemCNN	✓	✓	✓	✓
EMAN	✓	(w/ RelTan features)	✓	✓
Hermes	✓	✓	✓
GCN	✓
MPNN	✓
MeshGraphNet	✓	✓
EGNN	✓	✓
SpiralNet++	✓	✓
Table 9: Number of learnable parameters
FAUST	Objects	Heat	Wave	Cahn-Hilliard	Flag
GemCNN	1,829,658	161,715	40,337	40,337	29,775	-
EMAN	1,839,476	165,013	40,093	40,093	30,882	-
Hermes	1,837,776	162,976	40,544	40,695	23,122	51,555
GCN	-	-	41,401	41,401	29,401	-
MPNN	-	-	41,449	41,449	29,673	-
MeshGraphNet	-	-	44,993	44,993	35,457	45,313
EGNN	-	-	50,433	50,433	28,609	-
SpiralNet++	-	-	63,841	63,841	27,153	-
L2 regularization of the weights with a coefficient of  $1 \times 10^{-5}$  was used. A cosine learning rate scheduler was used for the Objects and Cahn-Hilliard datasets.
Table 7 contains the architecture details and hyperparameters used for each domain. Table 8 lists each method's flavor of message passing and its features. Table 9 shows the number of learnable parameters used for each method and domain.
E Results
Table 10 shows the results on FAUST, Objects, and FlagSimple datasets. On FAUST, GemCNN outperforms both EMAN and Hermes. We obtained slightly different results for EMAN than reported in the paper [10]. On Objects, Hermes outperforms GemCNN and EMAN, albeit slightly. On FlagSimple, Hermes considerably outperforms MeshGraphNet.
Table 10: Test results for each method using 5 runs. Gray numbers are  ${95}\%$  confidence intervals.
GemCNN	EMAN	Hermes	MeshGraphNet
FAUST	Accuracy (%)	99.1±0.1	97.5±0.4	97.5±0.2	-
OBJECTS	Accuracy (%)	98.8±0.1	96.9±0.5	99.7±0.0	-
RMSE (×10-2)	6.40±0.1	6.43±0.0	6.26±0.0	-
FLAGSIMPLE	RMSE (×10-3)	-	-	5.87±0.0	9.01±0.1
E.1 Long-horizon prediction rollouts
Table 11 shows additional qualitative samples generated autoregressively using only the initial conditions for  $T + 10, T + 30, T + 100$  timesteps. Hermes predicts large spatial patterns accurately and outperforms GemCNN and EMAN on all datasets.
Table 11: Qualitative predictions rolled out to 10, 30, 100 timesteps using only initial conditions.
Timestep	GemCNN	EMAN	Hermes	Ground Truth
HEAT	T+10
T+30
T+100
WAVE	T+10
T+30
T+100
CAHN-HILLIARD	T+10
T+30
T+100
E.2 Mesh Fineness
Figure 9 shows visualizations of the different mesh resolutions used and Figure 10 shows the results on the wave dataset. Hermes still outperforms GemCNN and EMAN at each resolution. Unlike the heat dataset (Figure 4), there is an unexpected overall decreasing RMSE trend with a coarser mesh.
(a) 348
(b) 867
(c) 1,731
(d) 3,461
(e) 8,650
Figure 9: Visualization of different mesh resolutions. The captions denote the number of vertices.
Figure 10: Performance for varying mesh fineness for wave on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. The number of vertices decreases with an increasing reduction ratio (see Figure 9).
E.3 Surface Roughness
Figure 11 shows visualizations of the different mesh roughness and Figure 12 shows the results on the wave dataset. Hermes outperforms GemCNN and EMAN at most roughness scales.
(a) 0.1
(b) 0.5
(c) 1.0
(d) 1.5
(e) 3.0
Figure 11: Visualization of different mesh roughnesses. The captions denote the roughness scale.
Figure 12: Performance for varying surface roughness for wave on the test time (left) and test init (right) datasets. Error bars denote standard error over 3 runs. Increasing scale increases the surface roughness of the mesh (see Figure 11).
E.4 Ablation on Residual Connection
Table 12 shows the ablation experiment with the residual connection. The results are mixed: having a residual helps on Heat but not on Wave or Cahn-Hilliard. The decision to have a residual connection should thus be considered a task-dependent hyperparameter.
Table 12: Ablation study on residual connection
With residual	Without residual
HEAT	Test time (×10-3)	1.18±0.3	1.24±0.5
Test init (×10-3)	1.16±0.3	1.22±0.5
Test mesh (×10-3)	1.01±0.3	1.05±0.4
WAVE	Test time (×10-3)	5.43±0.8	5.17±0.6
Test init (×10-3)	3.72±1.3	2.38±0.3
Test mesh (×10-3)	3.79±1.3	2.52±0.4
CAHN-HILLIARD	Test time (×10-3)	4.23±0.9	3.08±0.8
Test init (×10-3)	5.21±1.2	3.53±1.3
Test mesh (×10-3)	5.34±0.7	5.73±1.2