Learning Physical Simulation with Message Passing Transformer

## Abstract

Abstract Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph’s Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.

## 1 Introduction

In recent advancements in physical systems simulation, an increasing number of neural network-based methods are challenging traditional numerical solvers. These methods, notable for being several orders of magnitude faster and maintaining low error rates, have garnered substantial attention from researchers
[]
[]
[]
[]
. Graph Neural Networks (GNNs) have garnered widespread interest and research attention due to their unique features, including node-wise independent updates and aggregation operations that closely resemble iterations in traditional simulations. Additionally, particle-based and mesh-based simulations can be easily converted into graphs through neighbor interactions [] and topology structures []. This growing body of research can be collectively termed as GNN for Simulation (GNN4Sim)
[]
[]
[]
[].

Unlike the fields of Natural Language Processing (NLP) and Computer Vision (CV), physical systems are more complex and unstable, where every piece of information is crucial. This implies that issues like over-smoothing and over-squashing might be more pertinent in physical simulations. A common solution to mitigate these issues is mapping the original information to a high-dimensional latent space for processing. This approach has become a widely used architecture in most GNN4Sim applications
[, , , , ]
, known as the encoder-processor-decoder (EPD), which, due to its focus on nodes and edges rather than graph structure, has a broad range of applications.

Since the Attention mechanism achieved significant success in the Natural Language Processing (NLP) domain
[]
, many Transformer-based network architectures have been developed in computer vision
[, ]
, and Social and Information Networks
[]
. To enhance the capabilities of EPD and to leverage the fixed number of message passing iterations, we replace the commonly used Multi-Layer Perceptron (MLP) with Hadamard-Product Attention. This variant of the Dot-Product Attention mechanism is tailored for inputs with a finite maximum sequence length, enabling more nuanced processing of the relative importance of features within the message passing framework of the processor.

Incorporating theoretical methodologies from graph signal processing, we have innovatively applied the Graph Fourier Transform (GFT)
[, ]
to the domain of physical simulations through our introduction of Graph Fourier Loss (GFL). This novel loss function optimizes model performance by leveraging the unique spectral properties of graphs. Rather than applying a Fourier Transform directly to the model, we employ preprocessing Laplacian eigenvector matrix within the loss function, thereby keeping the model’s inference time nearly unchanged.

## 2 Related Work

##### Neural Approaches in Physical Systems Simulation

In recent years, Several neural-based methods have achieve great research interest in physical systems simulation due to their high computational speed and low error rates. Among these, Convolutional Neural Networks (CNNs)
[, , , , , ]
have been utilized to infer the dynamics of physical systems, demonstrating the capability of neural networks in accurately and efficiently simulating complex phenomena.
Physics-Informed Neural Networks (PINNs)
[, , , , ]
, which leverage implicit representations and physical condition constraints, can be trained without traditional datasets, illustrating a groundbreaking approach to model training that is particularly valuable in scenarios where empirical data is scarce or difficult to obtain.
Neural operators
[, , , ]
offer a novel methodology for predicting the physical state at any given time step directly from initial conditions. This represents a significant shift from traditional simulation methods, enabling more efficient and flexible simulations across various scales and conditions.
Graph Neural Networks (GNNs)
[, , , ]
facilitate information exchange between nodes through message passing. This mechanism effectively captures the interactions within physical systems, allowing for the detailed simulation of complex dynamics.

##### Advancements in GNN Architectures for Simulation

Recent studies in GNN4Sim have introduced various improvements. Multiscale methods
[, , , ]
, benefiting from simplified latent graph structures, have significantly accelerated training and inference while maintaining quality. At its core, this involves enhancing the efficiency of message passing, which entails modifying the topological structure of the graph. This change aims to concurrently increase processing speed and enhance accuracy in the simulations. Similarly, FIGNet
[]
by adding face-face edges, changes the graph structure to improve collision accuracy. Han et al.
[]
through uniform sampling, simplifies the graph structure, applying scaled dot-product attention to the entire graph but requires multiple prior temporal steps information. TIE
[]
streamlines interaction modeling in Message Passing Neural Networks, utilizing a modified attention mechanism to efficiently process particle dynamics without explicit edge representations. LAMP
[]
uses reinforcement learning to adapt to the varying relative importance of the trade-off between error and computation at inference time. C-GNS
[]
focus on model the constraints of the physical system.

##### Bridging Graph Theory and Signal Processing

Graph Signal Processing (GSP) extends traditional signal processing techniques to signals defined on graphs
[]
[]
, a paradigm shift that has unlocked new avenues in analyzing complex data structures. Bruna et al.
[]
introducing Spectral Networks for graph data learning, establishing foundational techniques for GNNs.
Sandryhaila and Moura
[]
and Hammond et al.
[]
introduced the concept of applying wavelet transforms on graphs, offering a powerful tool for signal analysis and processing on irregular domains. Further, the development of Graph Convolutional Networks (GCNs)
[, , ]
, simplified the application of convolutional neural networks to graph data, enabling efficient learning of graph-structured data. These foundational studies emphasize the significance of spectral methods in understanding and leveraging the inherent structure of data represented as graphs.

## 3 Problem Formulation and Preliminaries

This section introduces the problem formulation and the necessary preliminary concepts. It begins with the representation of physical systems using graph structures and the optimization goal for a learnable simulator in Section 3.1. It then delves into the Graph Fourier Transform (GFT), which facilitates the analysis of graph signals in the spectral domain in Section 3.2.

### 3.1 Problem Formulation

We consider graph $G^{t}=(V^{t},E^{t})$ to represent a physical system with $t$ taking discrete values $t=0,1,\ldots$, where $V$ denotes the set of nodes with node attributes $v_{i}$ for each $v_{i}\in V$, and $E$ denotes the set of edges with edge attributes $e_{ij}$ for each $e_{ij}\in E$. The graph is characterized by a fixed number of nodes $N$, and the mesh topology of the corresponding graph $G$ does not vary over time. This implies a constant structure where the connectivity pattern among nodes is preserved across all time steps.
We also define a total of $M$ Message Passing iterations, with $k=0,1,\ldots,M$. During the $k$-th Message Passing iteration, the attributes of nodes and edges are denoted by $v_{k,i}$ and $e_{k,ij}$, respectively, for each $v_{i}\in V$ and $e_{ij}\in E$.

The learnable simulator $f_{\theta}$, parameterized by $\theta$, can be optimized towards training objective. The goal of the learnable simulator is to predict the next state of the system, $G^{t+1}$, based on the previous prediction of graph $G^{t}$ at time step $t$, denoted by $G^{t+1}=f_{\theta}(G^{t})$, or $G^{0}\rightarrow G^{1}\rightarrow\cdots\rightarrow G^{t}$.

### 3.2 Graph Fourier Transform

The Graph Fourier Transform (GFT) transforms signals on a graph from the spatial vertex domain to the spectral frequency domain. For a signal defined on the vertices of the graph, GFT leverages the eigenvectors of the graph’s Laplacian matrix, projecting the signal onto the orthogonal basis formed by these eigenvectors. This projection allows us to analyze and process the signal in a domain where convolution and filtering can be performed algebraically.

#### 3.2.1 Mathematical Definitions

The adjacency matrix of $G$ is denoted by $A$, where $A_{ij}$ represents the weight of the edge between vertices $i$ and $j$. The degree matrix $D$ is a diagonal matrix where $D_{ii}=\sum_{j}A_{ij}$. The Laplacian matrix of the graph is defined as $L=D-A$.

The eigenvalues and eigenvectors of $L$ are denoted by $\lambda_{i}$ and $u_{i}$, respectively, where $i=1,2,\ldots,N$, and $N$ is the number of vertices in the graph.

Given a signal $x\in\mathbb{R}^{N}$ defined on the vertices of the graph, the GFT of $x$ is given by

$$ $\hat{x}=U^{T}x$ $$

where $U=[u_{1},u_{2},\ldots,u_{N}]$ is the matrix of eigenvectors of $L$, and $U^{T}$ is its transpose. The signal $x$ can be reconstructed from its GFT $\hat{x}$ using the inverse GFT, given by

$$ $x=U\hat{x}$ $$

## 4 Method

Figure: Figure 1: Model Architecture of the Message Passing Transformer, visualizing the information processing procedure for the first of four Message Passing ($k=0,M=4$) times. The encoder module transposes inputs into a latent space and the decoder predicts future states by extrapolating these encoded representations. The processor unit conducts numerous iterations, each treated as a regression problem, to refine node and edge attributes.
Refer to caption: extracted/5655542/picture/GNN4Sim.png

In Section 4.1, we present the overall architecture of the model, followed by a detailed description of the Hadamard-Product Attention and Graph Fourier Loss in Sections 4.2 and 4.3, respectively. The Hadamard-Product Attention is introduced to address aggregation bias and enable more fine-grained feature processing. The Graph Fourier Loss is introduced to balance the high-energy and low-energy components in the spectral domain, thereby enhancing the model’s capacity to learn complex physical phenomena.

### 4.1 Message Passing Transformer

The Message Passing Transformer architecture incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and utilizes Graph Fourier Loss for model optimization. Figure. 1 visualizes the computational process of the model.

##### Encoder

The node and edge attributes are transformed into a latent space by $f_{1}$ and $f_{2}$, respectively.

$$ $v_{0,i}\leftarrow f_{1}(v_{i}),\quad e_{0,ij}\leftarrow f_{2}(e_{ij})$ $$

##### Processor

The edge features are updated by $f_{3}$, incorporating features from adjacent nodes. Node features are then updated by $f_{4}$, which aggregates information across multiple tokens using Hadamard-Product Attention. Each token represents node and aggregated edge features from a particular Message Passing iteration:

$$ $e_{k+1,ij}\leftarrow f_{3}(e_{k,ij},v_{k,i},v_{k,j}),\quad v_{k+1,i}\leftarrow f _{4}\left(v_{k,i},\bigoplus_{m=0}^{k}\left(v_{m,i},\sum_{j}e_{m,ij}\right)\right)$ $$

where $\bigoplus$ denotes the sequential concatenation of tokens $\left(v_{m,i},\sum_{j}e_{m,ij}\right)$ for each $m$ from 0 to $k$, forming the input sequence for $f_{4}$.

##### Decoder

After $M$ times Message Passing, the latent node features are mapped back to the original attribute space by $f_{5}$, culminating in the update of the graph state to the next time step.

$$ $v_{i}^{\prime}\leftarrow f_{5}(v_{M,i}),\quad G^{t+1}=\text{UPDATE}\left(G^{t} ,v_{i}^{\prime}\right)$ $$

Here, $f_{1}$, $f_{2}$, $f_{3}$, and $f_{5}$ are all shallow MLPs. During training, finally, we compute the model loss using our Graph Fourier Loss and update the weights accordingly.

### 4.2 Hadamard-Product Attention

The Message Passing Transformer conceptualizes node updates as an autoregressive problem, utilizing past message-passing attributes for keys (K) and values (V), with the current state as the query (Q).

##### Scaled Hadamard-Product Attention

To address effectively sidestepping the aggregation bias introduced by the summation operations typical of matrix multiplication, we introduce the Scaled Hadamard-Product Attention (HPA) mechanism strategically utilizes element-wise multiplication followed by a linear transformation to finalize the attention computation.

Our Attention mechanism is tailored for inputs with a finite maximum sequence length, enabling more nuanced processing of the relative importance of features within the message passing framework of the processor. Assume query matrix ${Q}\in\mathbb{R}^{b\times d_{k}}$, key matrix ${K}\in\mathbb{R}^{b\times s\times d_{k}}$, and value matrix ${V}\in\mathbb{R}^{b\times s\times d_{k}}$, where $b$ represents the batch size, $s$ denotes the sequence length, and $d_{k}$ signifies the feature dimension. Particularly, $Q$ represents the feature vector of an individual node. $K$ and $V$ constitute interlaced sequences where node attributes and aggregated edge attributes alternate. By performing broadcasting and the Hadamard product between the query matrix ${Q}$ and the key matrix ${K}$, we obtain the score matrix ${S}$.

The Hadamard product is executed to compute the score matrix $S$. Thus, for each $i,j,k$, we have:

$$ ${S}_{i,j,k}=\frac{{Q}_{i,j,k}\circ{K}_{i,j,k}}{\sqrt{d_{k}}}$ $$

where $Q$ is expanded from a shape of $b\times d_{k}$ to $b\times s\times d_{k}$, such that for all $i\in\{1,\dots,b\}$, $j\in\{1,\dots,s\}$, and $k\in\{1,\dots,d_{k}\}$, the expanded ${Q}_{i,j,k}={Q}_{i,k}$. ${S}\in\mathbb{R}^{b\times s\times d_{k}}$ represents the result of element-wise multiplication between each query and key. Notice that if calculating the sum of the feature dimensions in ${S}$, the score matrix is identical to that of the Dot-Product Attention. This approach circumvents the summation aggregation inherent in matrix multiplication, thereby providing a more fine-grained representation.

Our attention mechanism that diverges from the traditional softmax application along the sequence dimension in $S$, opting instead for execution along the feature dimension.
This attention methodological shift implies that, contrary to the Scaled Dot-Product Attention, the computation of attention weights places emphasis on the allocation of weights across each feature dimension rather than assigning a weight to each position in the sequence relative to others.
Consequently, the contribution of each dimension to the Value vector’s computation is determined by its relative importance across the dimension, not by its position within the sequence.

$$ ${attn}_{i,j,k}=\frac{\exp({S}_{i,j,k})}{\sum_{k^{\prime}=1}^{d_{k}}\exp({S}_{i ,j,k^{\prime}})}$ $$

Subsequently, a Hadamard product between the attention weights $attn$ and the value matrix $V$ is performed, resulting in a weighted value matrix $w\in\mathbb{R}^{b\times s\times d_{k}}$.

$$ ${w}_{i,j,k}={attn}_{i,j,k}\circ{V}_{i,j,k}$ $$

To finalize the transformation and align the output dimensions with the input, this weighted value matrix undergoes a linear transformation to reshape $w$ from $b\times s\times d_{k}$ back to a dimension of $b\times d_{k}$.

##### Multi-Head Hadamard-Product Attention

Incorporating the principles of Scaled Hadamard-Product Attention, we introduce the Multi-Head Hadamard-Product Attention (MHHA) as an adaptation of the conventional Multi-Head Attention mechanism. MHHA modifies the traditional approach by applying the Hadamard-Product Attention mechanism within each head, thereby facilitating a more nuanced processing of information across multiple representation subspaces.

$$ ${Q}_{i}={Q}W_{i}^{Q},\;{K}_{i}={K}W_{i}^{K},\;{V}_{i}={V}W_{i}^{V}$ $$

where $W_{i}^{Q}\in\mathbb{R}^{d_{model}\times d_{k}}$, $W_{i}^{K}\in\mathbb{R}^{d_{model}\times d_{k}}$, and $W_{i}^{V}\in\mathbb{R}^{d_{model}\times d_{k}}$ are the weight matrices for the $i$-th head. Scaled Hadamard-Product Attention is then applied to each head’s transformed inputs, capturing detailed interactions within that subspace and finally concatenated and passed through a linear transformation $W^{O}\in\mathbb{R}^{(d_{k}\times s)\times d_{k}}$:

$$ ${head}_{i}=\text{HPA}({Q}_{i},{K}_{i},{V}_{i})$ $$

$$ $\text{MHHA}(Q,K,V)=\text{Concat}({head}_{1},...,{head}_{h})W^{O}$ $$

### 4.3 Graph Fourier Loss

Graph Fourier Loss (GFL) leverages the spectral properties of graphs to enhance model infering efficacy. By incorporating the graph’s Laplacian eigenvectors in preprocessing and employing a unique loss function during training, GFL aims to mitigate computational burdens while improving model performance.

##### Preprocessing

When the model does not alter the graph’s topological structure, the inherent topological properties of the graph, such as the Laplacian matrix and its eigenvalues and eigenvectors, remain unaltered throughout the training process. To avoid the substantial increase in computation time caused by calculating eigendecompositions during training, we preprocess the training set before commencing model training. For each trajectory, the graph’s Laplacian matrix is calculated and subsequently decomposed into eigenvectors $U$ , which are stored and utilized during training.

##### Compute Graph Fourier Loss during training

To circumvent the significant computational overhead of calculating eigenvectors during inference, we propose the Graph Fourier Loss (GFL) as the loss function. This strategy ensures the inference speed of the model remains unaffected.

Initially, we perform GFT on both the model’s output $y^{\text{train}}\in\mathbb{R}^{N\times d_{k}}$ and the target output $y\in\mathbb{R}^{N\times d_{k}}$, transforming the signals from the time domain to the frequency domain:

$$ $\hat{y}=U^{T}y,\quad\hat{y}^{\text{train}}=U^{T}y^{\text{train}}$ $$

Subsequently, we calculate the energy of each dimension of the transformed signals and sum them up to obtain the total energy for each signal across all nodes and dimensions:

$$ $E=\sum_{k=1}^{d_{k}}|\hat{y}_{:,k}|^{2},\quad E_{\text{train}}=\sum_{k=1}^{d_{ k}}|\hat{y}^{\text{train}}_{:,k}|^{2}$ $$

$E$ and $E_{\text{train}}$ represent the total energy of the target and model output signals in the frequency domain, respectively.

The energy $E$ are then sorted, and using the hyperparameter segment rate $s_{r}$, it is divided into high $E_{\text{high}}$ and low $E_{\text{low}}$ energy components. An adjustment factor $\alpha$ is computed based on the mean energy of these partitions:

$$ $\alpha=\sqrt{\frac{\text{mean}(E_{\text{low}})}{\text{mean}(E_{\text{high}})+ \epsilon}}\cdot\lambda$ $$

The constant $\epsilon$ is employed to prevent division by zero, while the regularization parameter $\lambda$ controls the strength of the adjustment. When $\alpha>1$, high-energy regions are amplified, resulting in the model emphasizing high-energy components. Conversely, when $\alpha<1$, this emphasis is reduced. Both manual setting of the regularization parameter $\lambda$ and incorporation it as a learnable parameter have been tested, found in Experiment 5.3

Finally, we adjust the signals and compute the mean squared error (MSE) directly in the frequency domain.
It is not necessary to perform the inverse GFT on the errors for the MSE calculation, as the norm is preserved under the Fourier transform. Consequently, the gradients computed in the frequency domain are equivalent to those computed in the time domain.:

$$ $\hat{y}^{\text{train}^{\prime}}=\text{adjust}(\hat{y}^{\text{train}},\alpha), \quad\hat{y}^{\prime}=\text{adjust}(\hat{y},\alpha)$ $$

$$ $\text{MSE}_{\text{freq}}=\frac{1}{N}\|\hat{y}^{\prime}-\hat{y}^{\text{train}^{ \prime}}\|_{2}^{2}$ $$

where the $\text{adjust}(\cdot)$ function operates on signals transformed to the frequency domain and multiplies their high-energy components with an adjustment factor $\alpha$, which aims to balance the energy distribution across the spectrum. Further explanation can be found in Appendix A.

## 5 Experiments

Figure: Figure 2: Comparison of RMSE of velocity norm between the Lagrangian system FlagSimple and the Eulerian system CylinderFlow using our MPT model and the MGN model [].
Refer to caption: extracted/5655542/picture/compar.png

In Section 5.1, we describe the datasets used and the implementation. We then describe baseline models for comparison and present and discuss the evaluation results and ablation studies conducted to assess the contributions of specific model components in Sections 5.2 and 5.3. We also visualize the RMSE of a Lagrangian system and an Eulerian system respectively, as shown in Figure 5.

### 5.1 Task setups

##### Datasets Description

We evaluated our method in the representation of both Lagrangian and Eulerian dynamical systems. The Lagrangian systems involve the datasets FlagSimple and DeformingPlate, while the Eulerian systems include CylinderFlow and Airfoil, with all datasets sourced from MeshGraphNet [].

- •
FlagSimple models a flag blowing in the wind, utilizing a static Lagrangian mesh with a static topology structure and ignores collisions.
- •
DeformingPlate Utilizes a quasi-static simulator to model the deformation of a hyper-elastic plate by a kinematic actuator. The dataset is structured with a Lagrangian tetrahedral mesh.
- •
CylinderFlow simulates the flow of an incompressible fluid around a fixed cylinder in a 2D Eulerian mesh.
- •
Airfoil focuses on the aerodynamics around an airfoil wing section, employing a 2D Eulerian mesh to monitor the evolution of momentum and density.

##### Implementation

Our framework is built using PyTorch [] and PyG (PyTorch Geometric) []. The entire model is trained and inferred on a single Nvidia RTX 4090. Detailed information, including network hyperparameters, input and output formats, and noise injection methods, can be found in appendix C.

**Table 1: RMSE of our method, MeshGraphNet (MGN), Bi-Stride Multi-Scale GNN (BSMS-GNN) and Transformer with Implicit Edges (TIE) for different rollout steps. Our method achieves state-of-the-art in all datasets**
| Measurements | Dataset | Ours (MPT) | MGN<br>[] | BSMS<br>[] | TIE<br>[] |
| --- | --- | --- | --- | --- | --- |
| RMSE-1 [1e-2] | Cylinder | 2.01E-01 | 5.83E-01 | 5.26E-01 | 4.97E-01 |
|  | Airfoil | 2.47E+02 | 3.15E+02 | 3.05E+02 | 3.32E+02 |
|  | Plate | 1.03E-02 | 2.67E-02 | 2.81E-02 | 3.54E-02 |
|  | Flag | 1.20E-02 | 6.53E-02 | 6.98E-2 | 5.84E-2 |
| RMSE-50 [1e-2] | Cylinder | 6.33E-01 | 1.42 | 3.68 | 7.15 |
|  | Airfoil | 4.14E+02 | 5.46E+02 | 1.37E+03 | 6.19E+03 |
|  | Plate | 9.26E-02 | 1.73E-01 | 2.95E-01 | 3.58E-01 |
|  | Flag | 1.91 | 2.28 | 2.67 | 2.33 |
| RMSE-all [1e-2] | Cylinder | 3.75 | 4.32 | 1.52E+1 | 2.89E+01 |
|  | Airfoil | 1.68E+03 | 2.15E+03 | 1.03E+04 | 1.57E+05 |
|  | Plate | 1.13 | 1.60 | 4.72 | 9.63 |
|  | Flag | 2.08 | 2.57 | 3.58 | 1.24E+01 |

### 5.2 Comparison with Baselines

##### Baselines

In our evaluation, we compared against several state-of-the-art models. The Bi-Stride Multi-Scale Graph Neural Network (BSMS) [] introduces multiscale methods to enhance the efficiency of message passing, significantly accelerating training and inference while maintaining simulation quality. MeshGraphNet (MGN) [] leverages a mesh-based approach for graph representation, enabling detailed and accurate physical simulations. Finally, the Transformer with Implicit Edges (TIE) model [] streamlines interaction modeling in Message Passing Neural Networks by utilizing a modified attention mechanism to efficiently process particle dynamics without explicit edge representations, improving computational efficiency. For the baseline models, we adopted the original parameter settings as specified in their respective papers.

##### Evaluation

Table 1 demonstrates the superiority of our model across all datasets. The CylinderFlow dataset at RMSE-1 reveals a pronounced improvement with our model, which shows a reduction in error by 59.6% compared to TIE, the nearest competitor. For longer simulations, the comparative model shifts. At RMSE-50 and RMSE-all, MeshGraphNet (MGN) performs better than other baselines. Our model continues to exhibit superior performance, showing a reduction in error by 55.4% at RMSE-50 and by 13.2% at RMSE-all when compared to MGN. These improvements illustrate the robustness of our model, particularly in complex dynamic simulations where accurate initial predictions are of the utmost importance for the generation of reliable longer-term forecasts.

In the context of the Airfoil dataset, our model remains state-of-the-art. At the RMSE-50 condition, the model’s error rate is reduced by 24.1% compared to MGN. This illustrates the model’s capacity to maintain accuracy over prolonged sequences, which is an essential feature for simulations requiring stability over extended temporal spans. At RMSE-all, the improvement reaches its zenith with an error reduction of 21.8%.

For the DeformingPlate and FlagSimple datasets, our model displays similar trends. In the Plate dataset, the RMSE-1 shows an improvement of 61.4% over MGN, with continued dominance in longer simulations, indicated by a 46.4% error reduction at RMSE-50. The Flag dataset demonstrates a more modest but consistent improvement across all metrics. The most notable reduction is 81.6% at RMSE-1.

It is notable that TIE has higher errors due to the fact that it does not utilise edge information from the dataset. Instead, it computes attention over all points within a set window distance, which introduces approximation errors.

### 5.3 Ablation Studies

To rigorously evaluate the influence of specific model components and configurations on overall performance, systematic ablation studies were undertaken. These included: (1) Evaluating the contributions of individual components by comparing the effects of Hadamard-Product Attention (HPA) and Graph Fourier Loss (GFL), (2) comparing Dot-Product Attention with Hadamard-Product Attention, (3) assessing the efficacy of learnable lambda parameters $\lambda$ versus manual setting of $\lambda$, and (4) investigating how varying segmentation rates $s_{r}$ affect model performance.

**Table 2: Ablation study conducted on the CylinderFlow dataset to evaluate the contributions of individual components within our architecture. We test the effects of Hadamard-Product Attention (HPA), Graph Fourier Loss (GFL), and their combination. Results indicate that integrating both HPA and GFL leads to reductions in error.**
| Measurements | Without HPA and GFL | HPA only | GFL only | HPA + GFL (ours) |
| --- | --- | --- | --- | --- |
| RMSE-1 [1e-2] | 5.83E-1 | 2.64E-1 | 2.27E-1 | 2.01E-1 |
| RMSE-50 [1e-2] | 1.42 | 9.10E-1 | 6.96E-1 | 6.33E-1 |
| RMSE-all [1e-2] | 4.32 | 3.94 | 3.89 | 3.75 |

Figure: Figure 3: (a) Comparison of Dot-Product Attention and Hadamard-Product Attention in CylinderFlow. HPA demonstrates a lower average RMSE across all rollout steps compared to the Dot-Product Attention. (b) Comparison of learnable and manual $\lambda$ settings in FlagSimple. Learnable $\lambda$ achieves lower error compared to manual settings. (c) The impact of varying segmentation rates $s_{r}$ in FlagSimple. Different segmentation rates do not significantly impact the final results.
Refer to caption: extracted/5655542/picture/DPA_and_HPA_comparison.png

##### Effectiveness of Graph Fourier Loss and Hadamard-Product Attention

In the ablation study focusing on the CylinderFlow dataset, wwe test the effects of Hadamard-Product Attention (HPA), Graph Fourier Loss (GFL), and their combination on multiple dimensions of performance, including predictive accuracy as measured by RMSE.

In terms of predictive accuracy, the model without HPA and GFL performed the worst, demonstrating significantly higher error rates across all RMSE measures. The integration of both HPA and GFL demonstrated the highest improvement in reducing error rates across all RMSE measures when compared to standalone implementations of GFL and HPA.

##### Effectiveness of Hadamard-Product Attention

Figure 3 compares the performance of Dot-Product Attention [] and Hadamard-Product Attention (HPA) on the CylinderFlow dataset. HPA consistently outperforms Dot-Product Attention across all rollout steps, demonstrating a lower average RMSE. This indicates that HPA’s finer-grained feature dimension weighting is more effective in capturing the dynamics of the system, leading to more accurate predictions.

##### Effectiveness of Learnable λ 𝜆 \lambda italic_λ

Figure 3 compares the performance of models with learnable $\lambda$ settings against manual $\lambda$ settings in the FlagSimple dataset. The results show that the learnable $\lambda$ achieves a lower final average RMSE compared to manual settings. This highlights the advantage of allowing the model to adaptively adjust $\lambda$ during training, leading to better overall performance.

##### Segmentation Rate Selection

In Figure 3, we investigate the impact of varying segmentation rates $s_{r}$ on the FlagSimple dataset. The results indicate that different segmentation rates do not significantly impact the final results. This robustness to segmentation rate selection demonstrates that our model can maintain high performance regardless of the specific value of $s_{r}$, simplifying the hyperparameter tuning process.

## 6 Conclusion and Limitation

The Message Passing Transformer (MPT) has achieved notable advancements in the accuracy of physical system simulations, leveraging Hadamard-Product Attention and Graph Fourier Loss to capture complex dynamics effectively. While MPT offers substantial improvements in simulation fidelity, it does face challenges with increased memory consumption, slower computational speeds, and limitations due to the fixed sequence length of the Hadamard-Product Attention. These issues, though significant, represent areas for future optimization and refinement. Continuing to develop MPT could lead to broader applications in dynamic system modeling and enhance its utility in scientific and engineering fields, driving forward the capabilities of learnable simulation technologies.

## 7 Appendix

## Appendix A Analysis of Learnable λ 𝜆 \lambda italic_λ in Graph Fourier Loss

In our experiments, we observed a notable phenomenon: when using a learnable $\lambda$ within the Graph Fourier Loss (GFL) framework, $\lambda$ did not tend to zero. However, when applying a similar adjustment directly to the mean squared error (MSE) in the frequency domain, the value of $\lambda$ quickly diminished to zero. This appendix section provides a comprehensive analysis of this observation and elucidates the underlying reasons.

The key difference between the GFL approach and direct MSE adjustment lies in how $\lambda$ interacts with the frequency domain energy components. In GFL, $\lambda$ is indirectly involved through the calculation of an adjustment factor $\alpha$, which is applied separately to the model’s output $y^{\text{train}}$ and the target output $y$. This can be expressed as:

$$ $\text{MSE}_{\text{freq}}=\frac{1}{N}\left\|\text{adjust}(U^{T}y^{\text{train}} ,\alpha)-\text{adjust}(U^{T}y,\alpha)\right\|_{2}^{2}$ $$

In this formulation, $\lambda$ helps balance the energy distribution across different frequency components, ensuring that it remains non-zero to maintain the desired balance between high and low-frequency components. The frequency domain energy for each signal is computed, and high and low-frequency components are adjusted based on $\alpha$. Since $\alpha$ is based on the mean energies and $\lambda$ affects the balance of these means, the gradient of $\lambda$ directly impacts the adjusted signals. This intricate adjustment mechanism precludes the possibility of a simple decline in the value of $\lambda$, as it plays a pivotal role in maintaining equilibrium between the high and low-frequency energies.

Conversely, when $\lambda$ is directly applied to the MSE in the frequency domain using the combined error $y^{\text{train}}-y$, the lack of separate intermediate adjustments for $y^{\text{train}}$ and $y$ leads to a different effect. This can be expressed as:

$$ $\text{MSE}_{\text{freq}}=\frac{1}{N}\left\|\text{adjust}(U^{T}(y^{\text{train} }-y),\alpha)\right\|_{2}^{2}$ $$

Here, $\lambda$ scales the error term without the intermediate balancing mechanism, causing $\lambda$ to quickly diminish as it directly attempts to minimize the combined error in the frequency domain. When directly adjusting the error and computing the Fourier transform, $\lambda$ primarily affects the adjustment of the error. As the model’s prediction $y^{\text{train}}$ and the target $y$ become closer, the error $y^{\text{train}}-y$ becomes smaller. In this scenario, the optimizer tends to reduce $\lambda$ to zero to minimize the loss function.

## Appendix B Dataset Details

Figure: (a) Flag Simple
Refer to caption: extracted/5655542/picture/flagsimple/flag_simple_rollout_visualization_0_397_minclim0_maxclim0.25.png

**Table 3: Dataset Specifications**
| Dataset | System | Mesh Type | Dimensions | # Steps | time step $\Delta t$ |
| --- | --- | --- | --- | --- | --- |
| FlagSimple | Lagrangian | triangle | 3D | 400 | 0.02 |
| DeformingPlate | Lagrangian | tetrahedral | 3D | 400 | — |
| CylinderFlow | Eulerian | triangle | 2D | 600 | 0.01 |
| Airfoil | Eulerian | triangle | 2D | 600 | 0.008 |

##### Dataset Specifications

Our models are trained and evaluated across four distinct datasets: FlagSimple, DeformingPlate, CylinderFlow, and Airfoil. Each dataset consists of 1000 training trajectories, 100 validation trajectories, and 100 test trajectories, with each trajectory comprising between 250 to 600 time steps. The FlagSimple dataset models a flag fluttering in the wind using a static Lagrangian mesh with a fixed topology, ignoring collision effects. The DeformingPlate dataset simulates the deformation of a hyper-elastic plate driven by a kinematic actuator with a quasi-static simulator, structured on a Lagrangian tetrahedral mesh. The CylinderFlow dataset involves the simulation of incompressible The Airfoil dataset focuses on the aerodynamic properties around an airfoil section, utilizing a 2D Eulerian mesh to track changes in momentum and density over time. Table 3 details for each dataset include: System, indicating whether the simulation is Lagrangian for solid mechanics or Eulerian for fluid dynamics; Mesh Type, specifying the geometric configuration such as triangular or tetrahedral; Dimensions, indicating whether the simulation is in 2D or 3D; and # Steps, the total number of simulation steps in each trajectory reflecting the depth of time-dependent analysis. The time step $\Delta t$ column specifies the simulation time increment between each step.

## Appendix C Model Details

##### Model Hyperparameters

The training of our model employs a batch size of 1. The Message Passing (MP) time is all set at $15$ steps. For optimization, the Adam optimizer is used with an initial learning rate of $10^{-4}$, which decays exponentially to $10^{-6}$ over the course of 2 million training steps from a total of 5 million steps. The model employs four functions, namely, $f_{1}$, $f_{2}$, $f_{3}$, and $f_{5}$, which are all configured as ReLU-activated two-hidden-layer MLPs. Each of these MLPs has both a hidden and an output layer with a size of 128. The Hadamard-Product Attention mechanism implemented uses four heads and includes a dropout rate of $0.1$ to prevent overfitting. The segmentation rate, denoted by $s_{r}$, is set at $0.5$. Rather than employing a manual setting of the parameter $\lambda$, we have chosen to utilize learnable lambda parameters, denoted by $\lambda$.

**Table 4: Model Input and Output Specifications**
| Dataset | edge inputs $e^{M}_{ij}$ | edge inputs $e^{W}_{ij}$ | node inputs $v_{i}$ | output |
| --- | --- | --- | --- | --- |
| FlagSimple | $x_{m,ij}$, $|x_{m,ij}|$,<br>$x_{w,ij}$, $|x_{w,ij}|$ | $x_{w,ij}$, $|x_{w,ij}|$ | $n_{i},\dot{x_{i}}$ | $\ddot{x}_{i}$ |
| DeformingPlate | $x_{m,ij}$, $|x_{m,ij}|$,<br>$x_{w,ij}$, $|x_{w,ij}|$ | $x_{w,ij}$, $|x_{w,ij}|$ | $n_{i}$ | $\dot{x_{i}},\sigma_{i}$ |
| CylinderFlow | $x_{w,ij}$, $|x_{w,ij}|$ | – | $n_{i},w_{i}$ | $\dot{w_{i}}$ |
| Airfoil | $x_{w,ij}$, $|x_{w,ij}|$ | – | $n_{i},w_{i},\rho_{i}$ | $\dot{w_{i}},\dot{\sigma_{i}}$ |

##### Model Input and Output

In table 4, several specific terms and symbols define the structure of input and output data for each dataset involved in the simulations. The edge inputs $e^{M}_{ij}$ and $e^{W}_{ij}$ represent interactions associated with edges between nodes $i$ and $j$, where $x_{m,ij}$ denotes the world edge position and $x_{w,ij}$ indicates the mesh edge position. The node inputs $v_{i}$ include $n_{i}$, representing node types, and $x_{i}$, indicating node positions. Other node-specific properties include momentum ($w_{i}$) and density ($\rho_{i}$), while outputs encompass acceleration ($\ddot{x}_{i}$), velocity ($\dot{x}_{i}$), and von Mises stress ($\sigma_{i}$).

**Table 5: Noise Scale and World Edge Radius Specifications**
| Dataset | Noise Scale | World Edge Radius $r_{w}$ |
| --- | --- | --- |
| FlagSimple | pos: 1e-3 | — |
| DeformingPlate | pos: 3e-3 | 0.03 |
| CylinderFlow | momentum: 2e-2 | — |
| Airfoil | momentum: 1e1, density: 1e-2 | — |

##### Noise Injection and World Edge Radius Settings

To enhance the robustness of our model against noisy inputs and to simulate real-world data conditions more accurately, we implemented a strategy for noise injection into the training process. These noise scales are consistent with the settings used in MeshGraphNet [], as detailed in Table 5. Additionally, the world edge radius $r_{w}$ column specifies the radius used for defining mesh interactions in the DeformingPlate dataset.

## Appendix D Rollout Visualizations

Figure: Figure 5: Rollout results of MPT: the FlagSimple visualizes only the world position, while other colors represent the velocity norm.
Refer to caption: extracted/5655542/picture/rollout.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1) What does MHHA refer to in the figure 1, I believe it is the Scaled Hadamard Product Attention (HPA)?

Q2) Can this model be used effectively with the pretrain and then finetune paradigm (especially on completely different equations/regimes) ? How generalizable is the model across equations, regimes, meshes etc?

Q3) How would this GNN-Transformer modelling of the system behave if we wanted to incorporate physics-informed loss components (Especially as the loss is GFL and not standard MSE)?
soundness: 3
strengths: S1) Novel Graph Fourier Loss which helps learn complex physical phenomena effectively across the energy spectrum of the system. This helps avoid using the Graph Fourier Transform in both training and inference

S2) Modified Attention mechanism focusing on obtaining importance of features as scores by using softmax along the dimension and not along the sequence. This is one of the major contributors to the results as the graph structure helps with message passing/interaction and hence softmax can be applied along the dimension instead of sequence.

S3) Detailed experiments and theoretical/mathematical justification alongwith ablations of all the design choices considered. The combination of HPA and GFL seems to achieve superior performance by learning in both spatial and spectral domains.
summary: This work proposes a new hybrid Graph Neural Network based autoregressive Transformer architecture. The authors use an Encoder-Processor-Decoder (EPD) architecture to model physical simulations.
The major contributions of the work include the scaled Hadamard Product Attention (along feature dimensions) and a novel loss function which utilises Graph Fourier Transform.
The authors have shown the effectiveness of the proposed model on both eulerian and lagrangian dynamical systems by achieving superior performance over previous GNN and Transformer based methods.
weaknesses: W1) Comparison between the current method and previous methods in terms of the wall clock time and memory footprint have not been included (The authors have mentioned this as a limitation and also in section F of the appendix). A thorough quantitative analysis of time and memory for each component would be useful for the current and future research as well.

W2) Although the GFL seems very effective, how one arrives at the formulation is not very clear. How does one arrive at the expression of $\alpha$ and and the concept of segment rate as well  in section 4.3 is not straightforward. This makes it unclear of how to build further on this work.

**Presentation/Typos/Corrections**

Section 1 line 47-48
*we replace the commonly used Multi-Layer Perceptron (MLP) with Hadamard-Product Attention.* 
From what I understand in Figure 1, we are still having MLP (FFN) layers right. Should this not be *we replace the commonly used Scaled Dot Product Attention with Hadamard-Product Attention.* instead? 

In Figure 1, I believe it should be *xM* times instead of *xMP* times as MP is not a quantity and refers to Message Passing

Section 2 line 63 *methods have achieved* or *methods achieve* instead of *methods have achieve*

Section 3 line 116-117 For the clarity of notation, can we have a superscript $t$ to denote time along with the currently defined iteration $k$ as a subscript. I feel this will greatly enhance readability (Also make the changes elsewhere)

Section 4.2.1 line 257-259 Is the expression for the gradient computation correct? I believe either a parenthesis is missing or the $\sigma$ should be moved after the first term.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Seeing that the entire model is trained, as well as inferred on a single RTX 4090, do you see a way to compare the performance against the other evaluated models across training compute, as well as the cost of inference? As is, it feels like this computational efficiency is not captured properly by the evaluation design. Specific comparisons I think would help to emphasize this point would be 

* Rollout performance on a fixed training budget on aforementioned RTX 4090. If you were to e.g. take only the training budget of the introduced model, and retrain the compared to models with only this budget, how would rollout performance compare?
* At inference time, one could for example evaluate the time to first sample as a comparative metric on the chosen hardware, my intuition woulds be that the optimizations done by you with the precomputation & caching should aid your proposed approach considerably.
soundness: 3
strengths: Where the paper shines is the theoretical background of its approach, the entire model architecture, as well as formulation intricacies are described in detail, and allow for an in-depth look at the architecture, the notable differences to preceding work like the Hadamard-Product attention, and the graph Fourier loss. As far as the reviewer can tell, the architecture should be fully reproducible from this exhibition.
summary: The authors introduce the message passing transformer, a graph neural network transformer hybrid introducing two novel components, the Hadamard product attention, and a novel graph Fourier loss, to improve the algorithm's training speed the Laplacian eigenvectors are precomputed. Rollouts of the spatiotemporal dynamics are tested on the datasets of the Meshgraphnet paper, with long-term rollouts being found to be more stable across Eulerian, as well as Lagrangian dynamics.
weaknesses: Where this paper falls short in its current form is the evaluation, and its embedding into present literature.

Maybe slightly too focussed on PINN literature, it misses two landmark works of the past year:

* The Universal Physics Transformers of Alkin et al., which also have a GNN-core and hence fall squarely into the category of a GNN-Transformer hybrid like the presented architecture
* Poseidon: Efficient Foundation Models for PDEs by Herde et al., which is also built to spatio-temporally evolve physical systems These two works, as well as the datasets they were introduced with are presently not represented in the paper.

It would improve the related work greatly to contrast these approaches to highlight the architectural differences, and if on purpose to make sure that it is clear to the reader why the proposed architecture is not compared to the previous two. Both models are freely available, and could be fine-tuned to the datasets used in this paper. Leading into the evaluation, the reviewer would furthermore question whether the models evaluated against are representative of the current state-of-the-art.

There have been a number of new datasets since MGN, not only the two previously mentioned works, but furthermore other datasets for PDEs/Physical systems which are more challenging than MGN's datasets. I would urge the authors to consider extending their evaluation to these more recent datasets. Concretely, the following datasets could provide a good starting point:
* _PDEBENCH: An Extensive Benchmark for Scientific Machine Learning_ by Takomoto et al.
* _LagrangeBench: A Lagrangian Fluid Mechanics Benchmarking Suite_ by Toshev et al.
* _PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs_ by Hao et al. includes benchmarks simulated with spectral methods, which should lend itself to the style of the message-passing Transformer.
* _Toward multi-spatiotemporal-scale generalized PDE modeling_ by Gupta et al.

More directly with the two previously mentioned models, I would urge the reviewers to consider extending their evaluations to more related models to allow for a better comparison against contemporary work. Specifically for the problem of long rollouts in the context of machine learning models for particle dynamics, _Neural SPH: Improved Neural Modeling of Lagrangian Fluid Dynamics_ was introduced at ICML'24 by Toshev et al., alongside its datasets, and would lend itself to comparison. It is unclear to the reviewer why the proposed architecture was not compared to this state-of-the-art architecture for longterm rollouts of particle dynamics. Either comparing to this architecture, or

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - What does "weight of the edge" mean in Line 131?
- How is the weighted value matrix $w$ reshaped back to a dimension of d in detail? Is the process implemented with a learned MLP?
- During each step of message passing, are the networks $f_1, f_2, f_3, f_4, f_5$ shared or independent, respectively?
soundness: 3
strengths: - The proposed Hadamard-Product Attention offers a fine-grained approach to attention by assigning weights to each feature dimension. The experiments show that it brings an improvement over traditional Dot-Product Attention. 
- The application of Graph Fourier Loss to balance spectral components is novel, leveraging graph signal processing to enhance model accuracy over extended rollouts in physical simulations.
summary: The paper introduces the Message Passing Transformer for physical simulations. MPT employs an Encoder-Processor-Decoder framework and integrates two innovative components: Hadamard-Product Attention and Graph Fourier Loss. Hadamard-Product Attention assigns attention weights at the feature level rather than by sequence position. Graph Fourier Loss is introduced as a spectral loss function that balances high- and low-energy graph components. By precomputing the graph’s Laplacian eigenvectors, GFL maintains computational efficiency, as this precomputation does not impact inference time.
weaknesses: - The computational requirements of the proposed method are significantly greater than all the baselines. Moreover, the computational cost for precomputing Laplacian eigenvectors is not discussed.
- From my understanding, the precomputation of Laplacian eigenvectors is not feasible for dynamic graphs that undergo frequent topological changes, such as some of the datasets regarding dynamic flags proposed in the MGN paper.
- The ablation study in Table 2 is only conducted on the CylinderFlow dataset.
- In Figure 3(a), is there something wrong with the label on the graph? It's quite confusing that the figure aims to show the effectiveness of Hadamard-Product Attention while it compares the results of different segment rate.
- In Figure 8, it would be better if the authors could also present the ground truth results for better comparsion.
- Typo: Line 481, "wwe"

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In line 113, it is said the graph topology does not vary over sequence. However, for lagrangian systems like deforming plate and flag simple that construct lagrangian edges based on world positions under certain radius, how do you compute eigenvectors of the graph laplacian matrix. 
2. In more complex scenarios that require mesh refinement, such as the sphere and flag dynamics datasets, the graph topology will also change over time. How can Graph Fourier Loss be applied in these situations?
3. In Sec 4.3, why is the loss calculated on the results after the adjust function rather than directly on the results from the Graph Fourier Transform?
soundness: 3
strengths: The use of Hadamard-Product Attention (HPA) and Graph Fourier Loss (GFL) enhance the accuracy and robustness of physical simulations. The results show that MPT has superior performance on four datasets.
summary: This paper introduces the Message Passing Transformer for physical simulation. It employs Hadamard-Product Attention for fine-grained feature processing and Graph Fourier Loss for optimization. The model achieves  accuracy gains in long-term simulations of both Lagrangian and Eulerian dynamical systems compared to existing methods.
weaknesses: 1. The paper suggests using Graph Fourier Transform for preprocessing by calculating the eigenvectors of the graph Laplacian matrix, assuming a fixed graph topology. However, for Lagrangian systems like deforming plates or flags, the graph topology changes over time based on world positions and relationships. This inconsistency raises questions about the validity of the methods and experiments presented, as the assumption of a constant structure contradicts the dynamic nature of Lagrangian systems. 
2. The introduction of HPA significantly increases computational overhead in training and inference. The computation overhead largely depends on the sequence length, there is a lack of comprehensive computation analysis for Sec 4.2.
3. The proposed method should compare more attention-based baselines, such as [1], [2].
4. There are missing annotations in Figure 3a, making it unclear which one is HPA.

[1] Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers. 
[2] Learning flexible body collision dynamics with hierarchical contact mesh transformer

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: -  Small perturbations to the Laplacian could result in completely different eigenspaces [1]. Do you think the introduction then of a Graph Fourier loss (GFL) would lead to difficulties for remeshing or to generalise when the underlying connectivity/mesh differs? 

- There is quite substantial differences between the results of MGN in your work and that quoted in the original paper [2] on the same datasets. Why is this the case? 

- When you do ablations and state your results "without" GFL it is not clear what you mean. Do you mean replacing this loss with an MSE on the velocities?

- You state in the introduction that a common solution to over-smoothing and over-squashing in similar works is to map to a latent space before preprocessing. However, if you map node features to a latent representation and then use a standard GNN which only incorporates local updates then this would not solve over-squashing. How does your method solve this issue? 

- The words "new" and "universal" to describe your architecture are not clear to me. Is your model universal? Additionally, the use of message-passing and a transformer itself is not new (see second point of weaknesses). What is novel about the architecture which you use outside of the loss function?

- Given the above point and the many other methods which use attention and message-passing, the term 'message passing transformer' to describe the method is not clear and confuses other terms and methods used in the Graph Machine Learning literature. Why would you say the term 'message passing transformer' describes your architecture better than what is done with these other approaches? 

- You mention in the paper that HPA and GFL "synergistically" optimize the model’s performance. Why is it the case the the methods in combination are more effective than the sum of their parts? 

- Some recent work [6] has also shown improvements over these baselines on these datasets. Can you say that your model is 'State-of-the-art' without also having newer methods such as this as a baseline? 

[1] Huang et al. ON THE STABILITY OF EXPRESSIVE POSITIONAL ENCODINGS FOR GRAPHS. ICLR 2024.

[2] Pfaff et al. LEARNING MESH-BASED SIMULATION WITH GRAPH NETWORKS. ICLR 2021. 

[3] Rampášek et al. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. 

[4] Veličković et al. Graph Attention Networks. ICLR 2018.

[5] He et al. A GENERALIZATION OF VIT/MLP-MIXER TO GRAPHS. ICML 2023. 

[6] Luo et al. CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation. NeurIPS 2023.
soundness: 3
strengths: - The Graph Fourier Loss is a novel loss term for learning dynamic systems (to the best of my knowledge). It is well motivated and discussed in detail throughout the paper and in the appendix. Additionally, it is shown to improve performance on a range of datasets over some baselines. 

- The ablation studies clearly show the performance improvements of incorporating the attention mechanism and GFL, highlighting the benefits of the outlined contributions.
summary: The paper applies Hadamard-Product Attention and a Graph Fourier Loss within a ‘Message Passing Transformer’ framework to address feature importance in both the spatial and spectral domains. The authors show that this leads to accuracy improvements for modelling dynamical systems.
weaknesses: - The main weakness in the paper is the impractical computation runtime of the approach which is substantially slower than the baselines (Table 6). Given that methods such as MGN [2] also focus on achieving substantial computational speed ups, it is not clear from the paper what the practical benefit is in improving the accuracy over such a method whilst being much slower and having larger memory requirements. Additionally, the paper hides some of the training time in a pre-processing step calculating the eigenvalues and eigenvectors. In order to have a fair comparison, the paper should quote this pre-processing time as its a required step for training. 

- Using the Hadamard-Product Attention has already been shown to be beneficial in graph representation learning over dot-product attention (Appendix F of [5]). Additionally, the use of message-passing and a transformer itself is not new. For example, GraphGPS [3] uses both local message-passing and global transformer updates. Graph Attention Networks [4] uses attention to weight local updates. Graph MLP-MIXER [5] uses Hadamard-Product Attention between local patch encodings. The paper needs to be more clear what are its contributions in relation to other Graph Neural Networks (such as those which I have listed and others). It seems the only difference and contribution is the Graph Fourier Loss which can lead to a computational burden in the pre-processing step and suffers from instabilties (see Question 1).

Meta-review: This paper introduces a novel hybrid graph neural network transformer (Message Passing Transformer, MPT). At the technical level, the main contributions consist in using a scaled Hadamard product attention and a new loss function employing the graph Fourier transform. Numerical results on Eulerian and Lagrangian dynamics show promising improvements over the state of the art. 

The reviewers have appreciated the novelty in the approach and the improvements demonstrated in experiments. However, several issues related to the benefits of the method in comparison with related work and to its computational costs have been raised. In response, the authors have provided a rather thorough rebuttal, presenting a number of additional experiments. This was appreciated by the reviewers that have updated their scores, making the paper borderline. 

As the paper has changed significantly (even at the level of the abstract and of the description of the main contributions), my opinion -- which is shared with various reviewers -- is that it still needs another round of thorough reviewing to ensure a proper evaluation. It is in fact not entirely clear that all the main objections of the reviewers have been positively resolved. For this reason, I suggest a rejection at this stage but would like to encourage the authors to resubmit an improved version to a future venue.

Decision: Reject