Learning Physical Simulation with Message Passing Transformer

Authors: Zeyi Xu, Yifei Li
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Physical Simulation, Graph Neural Network, Learned Simulation, Message Passing

Abstract:
Machine learning methods for mesh-based physical simulation have achieved significant success in recent years. We propose the Historical Message-Passing Integration Transformer (HMIT), an architecture based on Graph Neural Networks that incorporates a message passing framework and applies Graph Fourier Loss (GFL) for model optimization. To mitigate over-squashing, capture fine-grained details, and scale linearly with node count, we introduce Historical Message-Passing Attention (HMPA), which integrates multi-step historical message-passing information for each node with feature-wise softmax and employs a decoder-only architecture. Additionally, to modulate loss at specific frequencies and handle varying energy levels, we introduce GFL, which uses a frequency-domain energy adjustment schedule. To improve computational efficiency, we precompute the graph's Laplacian eigenvectors before training. Our architecture achieves significant accuracy improvements in shart- and long-term rollouts for both Lagrangian and Eulerian dynamical systems compared to current methods.

Decision: reject
Reviewer scores: 6, 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730117299909

thread_type:
official_review

<thread_object>
creator: reviewer_znqh

<content>
summary:
The paper applies Hadamard-Product Attention and a Graph Fourier Loss within a ‘Message Passing Transformer’ framework to address feature importance in both the spatial and spectral domains. The authors show that this leads to accuracy improvements for modelling dynamical systems.

strengths:
- The Graph Fourier Loss is a novel loss term for learning dynamic systems (to the best of my knowledge). It is well motivated and discussed in detail throughout the paper and in the appendix. Additionally, it is shown to improve performance on a range of datasets over some baselines. 

- The ablation studies clearly show the performance improvements of incorporating the attention mechanism and GFL, highlighting the benefits of the outlined contributions.

weaknesses:
- The main weakness in the paper is the impractical computation runtime of the approach which is substantially slower than the baselines (Table 6). Given that methods such as MGN [2] also focus on achieving substantial computational speed ups, it is not clear from the paper what the practical benefit is in improving the accuracy over such a method whilst being much slower and having larger memory requirements. Additionally, the paper hides some of the training time in a pre-processing step calculating the eigenvalues and eigenvectors. In order to have a fair comparison, the paper should quote this pre-processing time as its a required step for training. 

- Using the Hadamard-Product Attention has already been shown to be beneficial in graph representation learning over dot-product attention (Appendix F of [5]). Additionally, the use of message-passing and a transformer itself is not new. For example, GraphGPS [3] uses both local message-passing and global transformer updates. Graph Attention Networks [4] uses attention to weight local updates. Graph MLP-MIXER [5] uses Hadamard-Product Attention between local patch encodings. The paper needs to be more clear what are its contributions in relation to other Graph Neural Networks (such as those which I have listed and others). It seems the only difference and contribution is the Graph Fourier Loss which can lead to a computational burden in the pre-processing step and suffers from instabilties (see Question 1).

questions:
-  Small perturbations to the Laplacian could result in completely different eigenspaces [1]. Do you think the introduction then of a Graph Fourier loss (GFL) would lead to difficulties for remeshing or to generalise when the underlying connectivity/mesh differs? 

- There is quite substantial differences between the results of MGN in your work and that quoted in the original paper [2] on the same datasets. Why is this the case? 

- When you do ablations and state your results "without" GFL it is not clear what you mean. Do you mean replacing this loss with an MSE on the velocities?

- You state in the introduction that a common solution to over-smoothing and over-squashing in similar works is to map to a latent space before preprocessing. However, if you map node features to a latent representation and then use a standard GNN which only incorporates local updates then this would not solve over-squashing. How does your method solve this issue? 

- The words "new" and "universal" to describe your architecture are not clear to me. Is your model universal? Additionally, the use of message-passing and a transformer itself is not new (see second point of weaknesses). What is novel about the architecture which you use outside of the loss function?

- Given the above point and the many other methods which use attention and message-passing, the term 'message passing transformer' to describe the method is not clear and confuses other terms and methods used in the Graph Machine Learning literature. Why would you say the term 'message passing transformer' describes your architecture better than what is done with these other approaches? 

- You mention in the paper that HPA and GFL "synergistically" optimize the model’s performance. Why is it the case the the methods in combination are more effective than the sum of their parts? 

- Some recent work [6] has also shown improvements over these baselines on these datasets. Can you say that your model is 'State-of-the-art' without also having newer methods such as this as a baseline? 

[1] Huang et al. ON THE STABILITY OF EXPRESSIVE POSITIONAL ENCODINGS FOR GRAPHS. ICLR 2024.

[2] Pfaff et al. LEARNING MESH-BASED SIMULATION WITH GRAPH NETWORKS. ICLR 2021. 

[3] Rampášek et al. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022. 

[4] Veličković et al. Graph Attention Networks. ICLR 2018.

[5] He et al. A GENERALIZATION OF VIT/MLP-MIXER TO GRAPHS. ICML 2023. 

[6] Luo et al. CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation. NeurIPS 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730282101100

thread_type:
official_review

<thread_object>
creator: reviewer_ybdw

<content>
summary:
This paper introduces the Message Passing Transformer for physical simulation. It employs Hadamard-Product Attention for fine-grained feature processing and Graph Fourier Loss for optimization. The model achieves  accuracy gains in long-term simulations of both Lagrangian and Eulerian dynamical systems compared to existing methods.

strengths:
The use of Hadamard-Product Attention (HPA) and Graph Fourier Loss (GFL) enhance the accuracy and robustness of physical simulations. The results show that MPT has superior performance on four datasets.

weaknesses:
1. The paper suggests using Graph Fourier Transform for preprocessing by calculating the eigenvectors of the graph Laplacian matrix, assuming a fixed graph topology. However, for Lagrangian systems like deforming plates or flags, the graph topology changes over time based on world positions and relationships. This inconsistency raises questions about the validity of the methods and experiments presented, as the assumption of a constant structure contradicts the dynamic nature of Lagrangian systems. 
2. The introduction of HPA significantly increases computational overhead in training and inference. The computation overhead largely depends on the sequence length, there is a lack of comprehensive computation analysis for Sec 4.2.
3. The proposed method should compare more attention-based baselines, such as [1], [2].
4. There are missing annotations in Figure 3a, making it unclear which one is HPA.

[1] Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers. 
[2] Learning flexible body collision dynamics with hierarchical contact mesh transformer

questions:
1. In line 113, it is said the graph topology does not vary over sequence. However, for lagrangian systems like deforming plate and flag simple that construct lagrangian edges based on world positions under certain radius, how do you compute eigenvectors of the graph laplacian matrix. 
2. In more complex scenarios that require mesh refinement, such as the sphere and flag dynamics datasets, the graph topology will also change over time. How can Graph Fourier Loss be applied in these situations?
3. In Sec 4.3, why is the loss calculated on the results after the adjust function rather than directly on the results from the Graph Fourier Transform?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730358122816

thread_type:
official_review

<thread_object>
creator: reviewer_wc4g

<content>
summary:
The paper introduces the Message Passing Transformer for physical simulations. MPT employs an Encoder-Processor-Decoder framework and integrates two innovative components: Hadamard-Product Attention and Graph Fourier Loss. Hadamard-Product Attention assigns attention weights at the feature level rather than by sequence position. Graph Fourier Loss is introduced as a spectral loss function that balances high- and low-energy graph components. By precomputing the graph’s Laplacian eigenvectors, GFL maintains computational efficiency, as this precomputation does not impact inference time.

strengths:
- The proposed Hadamard-Product Attention offers a fine-grained approach to attention by assigning weights to each feature dimension. The experiments show that it brings an improvement over traditional Dot-Product Attention. 
- The application of Graph Fourier Loss to balance spectral components is novel, leveraging graph signal processing to enhance model accuracy over extended rollouts in physical simulations.

weaknesses:
- The computational requirements of the proposed method are significantly greater than all the baselines. Moreover, the computational cost for precomputing Laplacian eigenvectors is not discussed.
- From my understanding, the precomputation of Laplacian eigenvectors is not feasible for dynamic graphs that undergo frequent topological changes, such as some of the datasets regarding dynamic flags proposed in the MGN paper.
- The ablation study in Table 2 is only conducted on the CylinderFlow dataset.
- In Figure 3(a), is there something wrong with the label on the graph? It's quite confusing that the figure aims to show the effectiveness of Hadamard-Product Attention while it compares the results of different segment rate.
- In Figure 8, it would be better if the authors could also present the ground truth results for better comparsion.
- Typo: Line 481, "wwe"

questions:
- What does "weight of the edge" mean in Line 131?
- How is the weighted value matrix $w$ reshaped back to a dimension of d in detail? Is the process implemented with a learned MLP?
- During each step of message passing, are the networks $f_1, f_2, f_3, f_4, f_5$ shared or independent, respectively?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730826332916

thread_type:
official_review

<thread_object>
creator: reviewer_l6sr

<content>
summary:
The authors introduce the message passing transformer, a graph neural network transformer hybrid introducing two novel components, the Hadamard product attention, and a novel graph Fourier loss, to improve the algorithm's training speed the Laplacian eigenvectors are precomputed. Rollouts of the spatiotemporal dynamics are tested on the datasets of the Meshgraphnet paper, with long-term rollouts being found to be more stable across Eulerian, as well as Lagrangian dynamics.

strengths:
Where the paper shines is the theoretical background of its approach, the entire model architecture, as well as formulation intricacies are described in detail, and allow for an in-depth look at the architecture, the notable differences to preceding work like the Hadamard-Product attention, and the graph Fourier loss. As far as the reviewer can tell, the architecture should be fully reproducible from this exhibition.

weaknesses:
Where this paper falls short in its current form is the evaluation, and its embedding into present literature.

Maybe slightly too focussed on PINN literature, it misses two landmark works of the past year:

* The Universal Physics Transformers of Alkin et al., which also have a GNN-core and hence fall squarely into the category of a GNN-Transformer hybrid like the presented architecture
* Poseidon: Efficient Foundation Models for PDEs by Herde et al., which is also built to spatio-temporally evolve physical systems These two works, as well as the datasets they were introduced with are presently not represented in the paper.

It would improve the related work greatly to contrast these approaches to highlight the architectural differences, and if on purpose to make sure that it is clear to the reader why the proposed architecture is not compared to the previous two. Both models are freely available, and could be fine-tuned to the datasets used in this paper. Leading into the evaluation, the reviewer would furthermore question whether the models evaluated against are representative of the current state-of-the-art.

There have been a number of new datasets since MGN, not only the two previously mentioned works, but furthermore other datasets for PDEs/Physical systems which are more challenging than MGN's datasets. I would urge the authors to consider extending their evaluation to these more recent datasets. Concretely, the following datasets could provide a good starting point:
* _PDEBENCH: An Extensive Benchmark for Scientific Machine Learning_ by Takomoto et al.
* _LagrangeBench: A Lagrangian Fluid Mechanics Benchmarking Suite_ by Toshev et al.
* _PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs_ by Hao et al. includes benchmarks simulated with spectral methods, which should lend itself to the style of the message-passing Transformer.
* _Toward multi-spatiotemporal-scale generalized PDE modeling_ by Gupta et al.

More directly with the two previously mentioned models, I would urge the reviewers to consider extending their evaluations to more related models to allow for a better comparison against contemporary work. Specifically for the problem of long rollouts in the context of machine learning models for particle dynamics, _Neural SPH: Improved Neural Modeling of Lagrangian Fluid Dynamics_ was introduced at ICML'24 by Toshev et al., alongside its datasets, and would lend itself to comparison. It is unclear to the reviewer why the proposed architecture was not compared to this state-of-the-art architecture for longterm rollouts of particle dynamics. Either comparing to this architecture, or

questions:
Seeing that the entire model is trained, as well as inferred on a single RTX 4090, do you see a way to compare the performance against the other evaluated models across training compute, as well as the cost of inference? As is, it feels like this computational efficiency is not captured properly by the evaluation design. Specific comparisons I think would help to emphasize this point would be 

* Rollout performance on a fixed training budget on aforementioned RTX 4090. If you were to e.g. take only the training budget of the introduced model, and retrain the compared to models with only this budget, how would rollout performance compare?
* At inference time, one could for example evaluate the time to first sample as a comparative metric on the chosen hardware, my intuition woulds be that the optimizations done by you with the precomputation & caching should aid your proposed approach considerably.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730882497196

thread_type:
official_review

<thread_object>
creator: reviewer_g8yr

<content>
summary:
This work proposes a new hybrid Graph Neural Network based autoregressive Transformer architecture. The authors use an Encoder-Processor-Decoder (EPD) architecture to model physical simulations.
The major contributions of the work include the scaled Hadamard Product Attention (along feature dimensions) and a novel loss function which utilises Graph Fourier Transform.
The authors have shown the effectiveness of the proposed model on both eulerian and lagrangian dynamical systems by achieving superior performance over previous GNN and Transformer based methods.

strengths:
S1) Novel Graph Fourier Loss which helps learn complex physical phenomena effectively across the energy spectrum of the system. This helps avoid using the Graph Fourier Transform in both training and inference

S2) Modified Attention mechanism focusing on obtaining importance of features as scores by using softmax along the dimension and not along the sequence. This is one of the major contributors to the results as the graph structure helps with message passing/interaction and hence softmax can be applied along the dimension instead of sequence.

S3) Detailed experiments and theoretical/mathematical justification alongwith ablations of all the design choices considered. The combination of HPA and GFL seems to achieve superior performance by learning in both spatial and spectral domains.

weaknesses:
W1) Comparison between the current method and previous methods in terms of the wall clock time and memory footprint have not been included (The authors have mentioned this as a limitation and also in section F of the appendix). A thorough quantitative analysis of time and memory for each component would be useful for the current and future research as well.

W2) Although the GFL seems very effective, how one arrives at the formulation is not very clear. How does one arrive at the expression of $\alpha$ and and the concept of segment rate as well  in section 4.3 is not straightforward. This makes it unclear of how to build further on this work.

**Presentation/Typos/Corrections**

Section 1 line 47-48
*we replace the commonly used Multi-Layer Perceptron (MLP) with Hadamard-Product Attention.* 
From what I understand in Figure 1, we are still having MLP (FFN) layers right. Should this not be *we replace the commonly used Scaled Dot Product Attention with Hadamard-Product Attention.* instead? 

In Figure 1, I believe it should be *xM* times instead of *xMP* times as MP is not a quantity and refers to Message Passing

Section 2 line 63 *methods have achieved* or *methods achieve* instead of *methods have achieve*

Section 3 line 116-117 For the clarity of notation, can we have a superscript $t$ to denote time along with the currently defined iteration $k$ as a subscript. I feel this will greatly enhance readability (Also make the changes elsewhere)

Section 4.2.1 line 257-259 Is the expression for the gradient computation correct? I believe either a parenthesis is missing or the $\sigma$ should be moved after the first term.

questions:
Q1) What does MHHA refer to in the figure 1, I believe it is the Scaled Hadamard Product Attention (HPA)?

Q2) Can this model be used effectively with the pretrain and then finetune paradigm (especially on completely different equations/regimes) ? How generalizable is the model across equations, regimes, meshes etc?

Q3) How would this GNN-Transformer modelling of the system behave if we wanted to incorporate physics-informed loss components (Especially as the loss is GFL and not standard MSE)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732190149903

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response Part 1.

comment:
## General response:
We thank all reviewers and the AC for their time and effort in reviewing our work and for providing insightful comments to strengthen it. We have addressed typographical and grammatical errors throughout the paper, as noted by [Reviewer G8Yr, Reviewer wc4g]. We have also updated figure labels and representations to improve clarity and accuracy, as suggested by [Reviewer G8Yr, Reviewer wc4g, Reviewer YBDW]. Additionally, we have added visualizations of ground truth results to enhance experimental clarit [Reviewer wc4g]. **If further experimental evidence, including additional open-source baselines or datasets, is required, we will provide these as soon as possible.**

1. **Contributions**
    1. **[Motivation]** 
        * To mitigate over-squashing, capture fine-grained details, and scale linearly with node count, we purpose HPA. 
        * To modulate loss at specific frequencies and handle varying energy levels, we introduce GFL.
    3. **[Method]** 
        * Our Hadamard-Product Attention (HPA) integrates multi-step historical message-passing information with feature-wise softmax and a decoder-only architecture. 
        * Our Graph Fourier Loss (GFL) uses a frequency-domain energy adjustment schedule, similar to a piecewise function.
    4. **[Experiments]** Extensive experiments and ablations show that HPA and GFL boost performance across datasets, outperforming traditional attention mechanisms [Reviewer wc4g, Reviewer YBDW].
    5. **[Presentation]** Our manuscript includes detailed architectural and theoretical explanations, ensuring reproducibility and clarifying innovations[Reviewer L6SR].


2. **New Results**
    1. **[Method: Complexity Analysis]**. We have modified the paper and replaced the section '**Gradient Computation in HPA**' with '**Complexity Analysis**' [Reviewer G8Yr, Reviewer YBDW].
    2. **[Experiment: Precomputation Cost]**. We presented the precomputation time for the graph Laplacian matrix [Reviewer G8Yr, Reviewer L6SR, Reviewer wc4g, Reviewer YBDW, Reviewer ZnqH].
    3. **[Experiment: Ablation in FlagSimple]**. We tested the individual and combined performance of model components on the FlagSimple dataset [Reviewer wc4g].
    4. **[Experiment: Additional Baselines Comparison]**. We additionally ran the Mesh Transformer[1], HCMT[3], Graph MLP-MIXER[4] to demonstrate the superiority of our method [Reviewer L6SR, Reviewer YBDW, Reviewer ZnqH].
    5. **[Experiment: Additional Dataset Test]**. We selected the Dam Problem from CFDBench [2] as an additional dataset for performance comparison [Reviewer L6SR].
    6. **[Experiment: Runtime Consistency Performance]**. We tested runtime performance for models with varying hidden dimensions (128 and 256) on the DeformingPlate dataset [Reviewer L6SR].
    7. **[Experiment: GFL Robustness Performance]**. We evaluated energy changes after randomly removing different proportions of edges on the Airfoil dataset to demonstrate the robustness of GFL to graph topology variations [Reviewer ZnqH].

3. **Complexity Analysis[Reviewer G8Yr, Reviewer YBDW]**
    We calculate the FLOPs for the HPA mechanism, with $N$ as the number of nodes, $s$ as the sequence length, and $d$ as the feature dimension. Constructing the key and value matrices $K$ and $V$ requires linear transformations across each sequence step, amounting to $2Nsd^2$ FLOPs. The calculation of attention weights $a$, including the element-wise product and softmax operation, requires $2Nsd$ FLOPs. The element-wise multiplication of $a$ and $V$, followed by a linear transformation to reshape the result back to dimension $d$, adds $Nsd + Nsd^2$ FLOPs. Summing these components, the total FLOPs for HPA is $3Nsd^2 + 3Nsd$. This complexity scales **linearly** with the number of nodes $N$, with a fixed sequence length $s$. Compared to a global self-attention mechanism, our decoder-only architecture avoids the $O(N^2)$ complexity, making it especially efficient and advantageous for large-scale simulations.


4. **[Experiment: Precomputation Cost]**[Reviewer G8Yr, Reviewer L6SR, Reviewer wc4g，Reviewer YBDW, Reviewer ZnqH]: 

    | Dataset   | Eigen Decomposition Time per Sample| Total Processing Time  |
    |-----------|------------------------------------|------------------------|
    | Cylinder  | 0.026 s                            | 27.21 s                |
    | Airfoil   | 0.204 s                            | 218.34 s               |
    | Plate     | 0.016 s                            | 19.19 s                |
    | Flag      | 0.020 s                            | 21.82 s                |

    While we provide a precomputation cost table for clarity, this data processing only occurs once and can be considered part of the dataset generation process.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732190187017

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response Part 2.

comment:
5. **[Experiment: ablation in FlagSimple]**[Reviewer wc4g]: 

    | Measurements    | Without HPA and GFL | HPA only | GFL only | HPA + GFL (Ours) |
    |-----------------|---------------------|----------|----------|------------------|
    | RMSE-1 [1e-2]   | 6.47E-2             | 1.51E-2  | 2.29E-2  | **1.12E-2**      |
    | RMSE-50 [1e-2]  | 2.29                | 1.97     | 2.03     | **1.87**         |
    | RMSE-all [1e-2] | 2.45                | 2.16     | 2.21     | **2.05**         |

    The ablation study focusing on the FlagSimple dataset.

6. **[Experiment: Additional Baslines Comparison]**[Reviewer L6SR, Reviewer YBDW, Reviewer ZnqH]: 

    | Measurements    | Dataset   | Ours (MPT)         |Mesh Transformer[1]| HCMT[3]          |Graph MLP-MIXER[4]|
    |-----------------|-----------|--------------------|-------------------|------------------|------------------|
    | RMSE-1 [1E-2]   | Cylinder  | **2.03 E-01**      | 3.05 E-01         | -                | 4.12 E-01        |
    |                 | Airfoil   | **2.61 E+02**      | 2.96 E+02         | -                | 3.05 E+02        |
    |                 | Plate     | **1.00 E-02**      | 2.38 E-02         | 1.15 E-02        | 3.28 E-02        |
    |                 | Flag      | **1.12 E-02**      | 4.73 E-02         | -                | 6.89 E-02        |
    | RMSE-50 [1E-2]  | Cylinder  | **6.32 E-01**      | 1.07              | -                | 4.61             |
    |                 | Airfoil   | **4.08 E+02**      | 5.21 E+02         | -                | 6.32 E+02        |
    |                 | Plate     | **9.25 E-02**      | 1.30 E-01         | 9.68 E-01        | 6.19 E-01        |
    |                 | Flag      | **1.87**           | 2.04              | -                | 3.90             |
    | RMSE-all [1E-2] | Cylinder  | **3.78**           | 4.26              | -                | 2.05 E+01        |
    |                 | Airfoil   | **1.64 E+03**      | 2.00 E+03         | -                | 3.97 E+03        |
    |                 | Plate     | **1.09**           | 1.28              | 1.15             | 8.22             |
    |                 | Flag      | **2.05**           | 2.32              | -                | 7.25             |

    We additionally evaluated ours MPT against the Mesh Transformer[1], HCMT[3], and Graph MLP-MIXER[4] across four datasets.

7. **[Experiment: Additional Dataset Test]**[Reviewer L6SR]: 

    | Measurements    |Ours (MPT)    | MGN       | BSMS        | TIE       |
    |-----------------|--------------|-----------|-------------|-----------|
    | RMSE-1          | **1.08 E-01**| 2.25 E-01 | 1.87 E-01   | 1.62 E-01 | 
    | RMSE-50         | **2.57 E-01**| 6.03 E-01 | 5.34 E-01   | 5.71 E-01 |
    | RMSE-all (100)  | **4.63 E-01**| 9.25 E-01 | 8.36 E-01   | 8.49 E-01 |

    We additionally conducted an experiment on the Dam Problem from the 2023 CFDBench [2] benchmark.

8. **[Experiment: Runtime Consistency Performance]**[Reviewer L6SR]: 

    | Measurements    | Ours (MPT)    | MGN (128)  | MGN (256)  | BSMS (128)  | BSMS (256)  | TIE (128)  | TIE (256)  |
    |-----------------|---------------|------------|------------|-------------|-------------|------------|------------|
    | RMSE-1          | **1.00 E-02** | 2.69 E-02  | 2.14 E-02  | 2.83 E-02   | 2.65 E-02   | 3.56 E-02  | 3.21 E-02  |
    | RMSE-50         | **9.25 E-02** | 1.73 E-01  | 1.62 E-01  | 2.81 E-01   | 2.43 E-01   | 3.61 E-01  | 3.37 E-01  |
    | RMSE-all (100)  | **1.09**      | 1.61       | 1.52       | 4.52        | 4.34        | 9.62       | 9.05       |


    We trained and tested MGN, BSMS, and TIE with a hidden layer dimension of 256 on the DeformingPlate dataset, resulting in a training budget slightly higher than our method. 

9. **[Experiment: GFL Robustness Performance]**[Reviewer ZnqH]:

    | Edge Removal Percentage  | High Energy Mean          | Low Energy Mean           | Energy Ratio (High/Low)        |
    |--------------------------|---------------------------|---------------------------|--------------------------------|
    | 1%                       | 3.388615 ± 0.014727       | 3.391265 ± 0.014988       | 1.000784 ± 0.000418            |
    | 5%                       | 3.387550 ± 0.014215       | 3.389730 ± 0.014485       | 1.000643 ± 0.000392            |
    | 10%                      | 3.387155 ± 0.013993       | 3.389425 ± 0.014275       | 1.000670 ± 0.000385            |


    Energy changes after randomly removing different proportions of edges on the Airfoil dataset. 

[1] Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers

[2] CFDBench: A Comprehensive Benchmark for Machine Learning Methods in Fluid Dynamics

[3] Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer

[4] A GENERALIZATION OF VIT/MLP-MIXER TO GRAPHS
</content>
</thread_object>

</thread>
<thread>
creation date:
1732716733862

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The revised paper has been uploaded

comment:
We appreciate the time and effort of all reviewers and the AC in reviewing our work, as well as their insightful comments, which have significantly strengthened it.

# Updates and Changes
## Introduction
- Added citations of **The Universal Physics Transformers**, **Poseidon: Efficient Foundation Models for PDEs**, **GraphGPS**, **GAT**, and **Graph MLP-Mixer**.
- More clearly defined the key challenges our Historical Message-Passing Attention (HMPA) addresses:
  - **Mitigating over-squashing**
  - **Capturing fine-grained details**
  - **Scaling linearly with node count**

## Problem Formulation and Preliminaries
- Revised Section 3.1 (*Problem Formulation*) for improved clarity. Corresponding symbols were updated, and the assumption that graph topology remains constant over time was removed.
- Clarified ambiguous descriptions in Section 3.2.1 (*Mathematical Definitions*), particularly regarding the adjacency matrix.

## Method
- Removed Sections 4.2.1 (*Gradient Computation in HPA*) and 4.2.3 (*Dynamic Feature Importance*).
- Added Section 4.2.1 (*Complexity Analysis*).
- Refined and improved expressions for better clarity.

## Experiment
- **[Experiment: Precomputation Cost]** was added as part of Section 5.3 (*Precomputation Cost*), along with Table 2 and corresponding analysis.
- **[Experiment: Ablation in FlagSimple]** was added to Section 5.4 (*Ablation Studies*), included as Table 3.
- **[Experiment: Additional Baselines Comparison]** was incorporated into Section 5.2 (*Comparison with Baselines*), included as Table 1 with corresponding revisions. While we evaluated HCMT [1] during the rebuttal phase, HCMT was tested only on the *DeformingPlate* dataset because its **CMT** and **HMT** modules were specifically designed for collision cases. Therefore, HCMT is not included in the final paper.
- **[Experiment: Additional Dataset Test]** was added to *Appendix G (Performance of HMIT on Dam Flow)* as Table 9 with corresponding analysis.
- **[Experiment: Runtime Consistency Performance]** was not included in the paper. During our response to Reviewer ZnqH, we identified further optimization opportunities, making Table 1 more comparable in terms of parameter count, runtime, and memory consumption. Consequently, comparisons with baselines using hidden layers of 256 dimensions were no longer necessary.
- **[Experiment: GFL Robustness Performance]** was added to *Appendix B (Experimental Analysis of Graph Fourier Loss)* as Table 4, along with corresponding analysis.
- The new **[Experiment: Memory Consumption and Computational Speed]** was added to *Appendix F (Memory Consumption and Computational Speed)* as Table 8 with corresponding updates.
-  We have added a clarification regarding the fact that world edges do not participate in the Laplacian eigendecomposition in the paragraph 'Noise Injection and World Edge Radius Settings' in Appendix E (Model Details).

## Visualizations
- **Figure 1**: Corrected the label error and clarified the abbreviation.
- **Figure 3(a)**: Corrected the label error.
- **Appendix H**: Updated Rollout Visualizations, now including comparisons between Ground Truth and ours HMIT.


[1] Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel hybrid graph neural network transformer (Message Passing Transformer, MPT). At the technical level, the main contributions consist in using a scaled Hadamard product attention and a new loss function employing the graph Fourier transform. Numerical results on Eulerian and Lagrangian dynamics show promising improvements over the state of the art. 

The reviewers have appreciated the novelty in the approach and the improvements demonstrated in experiments. However, several issues related to the benefits of the method in comparison with related work and to its computational costs have been raised. In response, the authors have provided a rather thorough rebuttal, presenting a number of additional experiments. This was appreciated by the reviewers that have updated their scores, making the paper borderline. 

As the paper has changed significantly (even at the level of the abstract and of the description of the main contributions), my opinion -- which is shared with various reviewers -- is that it still needs another round of thorough reviewing to ensure a proper evaluation. It is in fact not entirely clear that all the main objections of the reviewers have been positively resolved. For this reason, I suggest a rejection at this stage but would like to encourage the authors to resubmit an improved version to a future venue.