Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a graph and attention based architecture to model the dynamics of complex physical systems. Their experiments indicate a significant (in cases, upto two order of magnitude) outperformance of a state-of-the-art graph network baseline.
main_review: This paper tackles the problem of simulating physics using graph neural networks (GNNs). GNNs have recently shown to be extremely efficient and fast alternatives to physical simulators. The current state-of-the-art of physical simulation with neural networks, is MeshGraphNets, which uses a GNN archictecture to learn one-time-step updates. These updates are integrated over time to produce arbitrary length simulations.

However, the feedforward nature of MeshGraphNets brings two issues to the fore. 
* Long-term stability: Since MeshGraphNets do not incorporate any feedback mechanism, they operate in an open-loop system, often resulting in eventual divergence of predicted velocities/accelerations.
* Physical implausibility: The training recipes of MeshGraphNets requires adding noise to the state vectors (e.g., perturbations to positions velocities etc.) without which the approach overfits to a set of reference trajectories. Adding noise to these physical quantities makes the learning setup physically implausible (i.e., the function we learn does not map physically accurate initial states to physically accurate subsequent states.

This paper addresses both these issues by incorporating a self-attention mechanism between the graph encoder and decoder layers.

Another key novel aspect of the paper is in the reduction step, where an input graph is reduced to a much smaller, but representative ‘summary graph’. Learning dynamics over the summary graph, in principle, is sufficient to represent dynamics over the full mesh.

The experiments and results presented in this paper provide compelling evidence to both claims, and as such I would recommend this paper for acceptance. However, there are a few minor discussion points / analyses I believe will further strengthen the paper.

Is there any insight on how succinct the learned representation might be? For instance, in the case of rigid bodies, the input meshes may be subsumed into a single node (the center-of-mass) and yet represent the full system dynamics.

Does the sampling strategy of the summary nodes have a significant impact on the resulting learned physical model? For objects that are inherently non-uniform density, the graph network might benefit from sampling in proportion to the object density as opposed to uniformly. Any thoughts/insights on this?
summary_of_the_review: This paper is very well-written, tackles an important problem, and overcomes two shortcomings in existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a GNN-based model with temporal attention to auto-regressively predict flow velocity of 3 test cases in fluid dynamics. Prediction results and some diagnostic analysis on the components of the network have been done. 
main_review: 1. The paper's abstract starts with "Auto-regressive sequence models for physics prediction are often restricted to
low-dimensional systems, as memory cost increases with both spatial extents and sequence length."-- This is not true. A plethora of work in computational physics, fluid dynamics, and climate dynamics have looked at autoregressive prediction for very high-dimensional chaotic systems, and quite successfully with deep learning. The authors should do a more thorough literature review looking at these works. 

2. The cases presented are not that high-dimensional. In fact, the cylinder flow can be decomposed to its first 3 POD modes which are enough to describe the dynamics of the system. Case 2 is moderately complicated, Case 3 is a more suitable test case. So, motivating this work from a perspective of high-dimensional physics is not quite fair. It is undoubtedly a good starting point, but there have been many works on much more complicated systems.  
 
3. The authors should dig some literature to look at methods used in what can be considered state-of-the-art in this field of data-driven physics. The baselines should include some of them.

4. The best thing about the paper is the effort to diagnose what is going on in different components. I think that is a big step forward in this field. I feel this section of the paper to be very insightful and interesting. 
summary_of_the_review: I recommend acceptance because of the diagnostic section of the paper. While there are a few concerns, I think with a bit of revision and better baselines, the paper presents a nice set of results especially with the diagnostics section
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new algorithm combining graph-neural-network (GNN) and auto-regressive sequence models for physics prediction problems. The authors first use GNNs to compress the physical graphs, then use transformers to predict the next steps of the compressed representations, and finally use GNNs to recover the graph representations from the predicted representations. Through empirical studies, the authors show that this method outperform the previous SOTA model (MeshGraphNet) significantly, especially in the long-rollout prediction scenarios. The authors further analyze the models to understand the success and find that the ability of the transformer model to replay the earlier sequences seems to be the critical for the better performance especially in the scenarios with oscillations.
main_review: Strengths:
1.	The proposed model which uses GNNs as auto-encoders for the graphs and then uses transformers as dynamic models seems to be innovative.
2.	The empirical results on the three scenarios tested also support the better performance of the newly proposed model compared to MeshGraphNet, especially in the first and the third scenarios.

Weaknesses:
1.	The main critic (or confusion) I have for this model is its ability to capture complicated dynamics beyond oscillations. The transformer architecture makes this model good at handling oscillations, as earlier states can be easily copied to the current states through the attention mechanism. But what about scenarios beyond oscillations? In fact, just as the paper itself shows in the second scenario (Sonic flow), where the oscillations seem to be less happening, the difference between this algorithm seems to be less compared the MeshGraphNet (or even worse compared to it). The initial higher error also indicates that the transformer model actually fails to capture the actual dynamics. I think the authors need to show more scenarios not mainly about oscillations and whether this method still outperform the other one.
2.	More results on how the performance of the GNNs for compressing the graph representations into hidden states are needed. The authors mention that the initial higher errors might be due to the loss of the states from the auto-encoders, can the auto-encoders be modified to make this loss lower? What are some key hyperparameters on this structure that would influence the performance?
3.	Can authors also provide some failure examples of this model to help better understand its strength and weakness?
4.	This might be because I am not familiar about this specific field, but what is the main algorithm difference between this model and MeshGraphNet? Was the GNN autoencoder model also used there? Or is that newly proposed in this model?

summary_of_the_review: This work proposes a new model for prediction physics dynamics. This new model uses graph neural networks to compress the physic graphs into lower-dimensional representations and then uses transformer models to predict the dynamics. Through empirical studies on three physical scenarios, the authors show the advantage of this method compared to the previous SOTA. However, more results need to be presented to show the power of this method in complicated physical scenarios beyond oscillations, where the transformer models indeed gain advantage by their nature.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use auto-regressive sequence models with attention mechanisms to predict the evolution of physical simulations. Mesh-based discretizations are targeted, and hence the employed networks take the form of GNNs. The paper focuses on encode-process-decode structures, and a transformer is proposed as main method to predict future states in the latent space. This approach is evaluated with three CFD settings: an unsteady wake flow, a sonic backwards facing step, and a medical flow scenario. The results are additionally analyzed in terms of latent space and attention weight content.

main_review: The use of a transformer architecture with attention in the context of physical simulations is interesting, and not widely studied (as far as I know). As such I was curious to see how well it works. The training methodology leaves a few open questions: is the network actually trained with sequences of different length than used for the evaluations later on? The autoregressive training described in 3.3 sounds like the network is trained on "full sequences". However, what would be important to demonstrate is that the models are trained for sequences of length n, and then evaluated for much longer sequences, e.g. 2n, 3n or higher for input parameters not seen at training time. If the training was successful, these should still give reasonable predictions.

The paper then compares a the proposed transformer with GRUs and LSTMs and comes to the conclusion that the transformer performs best across all scenarios. Unfortunately, my main worries stem from these simulation scenarios. 

Interestingly, both LSTM and GRU are on-par for the medical example. This example seems quite artificial: it's basically the cylinder wake flow case, but repeated with a vessel like geometry. That seems to make it even simpler than the original wake flow case. It would have been more interesting to model something like a simple cardiac cycle.

While the summary of the different simulation cases makes sense in the main paper, the details provided in the appendix are insufficient. Neither governing equations, nor variables are properly explained. As thus, I find it very difficult to estimate whether the simulations are meaningful, which parameters were varied for the training data sets, or which cases were used for the test evaluations.

My guess is that the compressible "sonic" case is the most interesting one, but it's content remains unclear, as does the "T_,0" parameter that seems to be varied for the models. The two evolutions shown are very similar, though, and this case seems to consist of only 40 evaluation steps. That is probably too short to really leverage the attention architectures.

Despite this, the analysis of the latent space content and attention weights provide some interesting insights, and it's good to see that the dominant CFD frequency is present in the attention data.

The appendix gives additional details of the network architectures used. Unfortunately, model sizes (in terms of trainable parameters) are missing. 

summary_of_the_review: Overall, I think the paper targets a very interesting direction, and I can encourage the authors to continue their work in this direction. However, in the current form I find it difficult to argue for accepting the paper straight away. The training and simulation setups leave too many open questions to properly evaluate whether the attention approach actually provides benefits or not.

Post rebuttal: the authors have answered my questions, and added two interesting simulation results in the appendix. So I've adjusted my score to a more positive one, and (seeing the other two positive reviews) I'd be fine with acceptance.
review_title: null

Meta-review: This paper proposes a model to predict the spatiotemporal dynamics of physical simulations on irregular meshes. The observations are modeled as a sequence of graph representations, each graph corresponding to a snapshot of the observation sequence at time t. This model uses two components, a graph encoder-decoder to compress the observations and an autoregressive transformer to model the dynamics. The two components are trained sequentially. At inference time, given an initial hidden state infered from the data and some additional conditional information, a sequence of states is predicted in an auto-regressive manner in the hidden space, each state of the sequence is then decoded to produce a prediction in the original observation space. The originality of the model lies in the encoder-decoder graph and in the use of a transformer for the prediction. Tests are performed on three fluid dynamics simulation data sets.

All reviewers pointed out some original contributions in the proposed method, in particular the use of transformers in the learned hidden space. In the rebuttal, the authors provided substantial additional results and further details and explanations. Their responses led two reviewers to increase their scores. All reviewers ultimately agree that the paper presents interesting results and conclusions.
Decision: Accept (Poster)