Predicting Physics in Mesh-reduced Space with Temporal Attention
Authors: XU HAN
Han Gao
Tobias Pfaff
Jian-Xun Wang
Liping Liu
Venue: ICLR 2022 Poster (2022)
Keywords: fluid dynamics
graph neural network
attention neural network

Abstract: Auto-regressive sequence models for physics prediction are often restricted to low-dimensional systems, as memory cost increases with both spatial extents and sequence length. On the other hand, graph-based next-step prediction models have recently been very successful in modeling complex high-dimensional physical systems on irregular meshes, but suffer from error accumulation and drift, due to their short temporal attention span. In this paper, we present a method that marries the strengths of both approaches. We use a GNN to locally summarize features and create coarsened, compact mesh representation of the system state, onto which we apply a transformer-style temporal attention module. We use a second GNN to decode these predictions back to a full-sized graph and perform fine-scale updates. Our method outperforms a competitive GNN baseline on three complex fluid dynamics prediction tasks, from sonic shocks to vascular flow. We demonstrate stable rollouts without the need for training noise and show perfectly phase-stable predictions even for very long sequences. More broadly, we believe our approach paves the way to bringing the benefits of attention-based sequence models to solving high-dimensional complex physics tasks.

Published as a conference paper at ICLR 2022
PREDICTING PHYSICS IN MESH-REDUCED SPACE WITH TEMPORAL ATTENTION
Xu Han
Han Gao"
Tobias Pffaf"
Tufts University
University of Notre Dame hgaol @nd.edu
DeepMind tob.pfaff@ gmail.com
Xu.Han @ tufts.edu
Jian-Xun Wang
Li-Ping Liu Tufts University Liping. Liu@ tufts.edu
University of Notre Dame swang33@nd.cod
ABSTRACT
Graph-based next-step prediction models have recently been very successful in modeling complex high-dimensional physical systems on irregular meshes. How- ever, due to their short temporal attention span, these models suffer from error accumulation and drift. In this paper, we propose a new method that captures long-tern dependencies through transformer-styy temporal attention model. We introduce an encoder-decoder structure to summarize features and create a com- pact mesh representation of the system state, allow the temporal model op- erate on a low-dimensional mesh representations a memory efficient manner. Our method outperforms competitive GNN baseline on several complex fluid dynamics prediction tasks, from sonic shocks to vascular flow. We demonstrate stable rollouts without the need for training noise and show perfectly stable predictions even for very long sequences. More broadly, we believe our approach paves the way to bringing the benefits of attention- based sequence models solv- ing high-dimensional complex physics tasks.
INTRODUCTION
There has been an increasing interest in many scientific disciplines, from computational fluid dy- namics [3, 40] over graphics [43, 41] to quantum mechanics [21, 1], to accelerate numerical simu- lation using learned models. In particular, methods based Graph Neural Networks (GNN) have shown to be powerful and flexible. These methods can directly work with unstructured simulation meshes, simulate systems with complex domain boundaries, and adaptively allocate computation to the spatial regions where it is needed [8, 39, 35, 52].
Most models for complex physies prediction tasks, in particular those on unstructured meshes, are next-step prediction models; that is, they predict the next state utt+1, 1, x) of physical system from the current state utt, x). As next-step models suffer from error accumulation, mitigation strategies such as training noise [39] or augmented training data using a
[42] have to be used keep rollouts stable. These remedies are not without drawbacks- training noise can be hard to tune, and ultimately place bound on the achievable model accuracy. And worse, next-step models also tend to show drift, which is not as easily mitigated. Failure examples include failure to conserve volume energy, shift in phase, or loss of shape information (see c.g., the failure case example in [39]).
On the other hand, auto-regressive sequence models such Recurrent neural networks (RNNs), or more recently transformers, have been hugely successful in predicting sequences in NLP and image applications [36, 31]. They capture stochastic dynamics and work with partial observations [50]. Furthermore, their long attention span allows them to better preserve phase and conserved quantities [18]. However, as memory cost for full-sequence transformer models scales with both
"Equal contribution.

---

Published as a conference paper at ICLR 2022
sequence length and spatial extents, it is hard to directly extend such models to predict physical systems defined on large unstructured meshes.
This paper combines powerful GNNs and transformer to model high dinensninnnii physical systems on meshes. The key idea is creating a locally coarsened yet more expressive graph. to limit memory consumption of the sequence model, and allow effective training. We first a GNN to aggregate local information solution fields of a dynamic system into node representations by performing several rounds of message passing, and then coarsen the output graph to a small set of pivotal nodes. The pivotal nodes' representations form a latent that encodes the system state in a low-dimensional space. We apply a transformer model on this latent, with attention over the whole sequence, and predict the latent for the next step. We then use a second GNN to recover the full-sized graph by up-sampling and message passing. This procedure of solving on coarse scale, upsampling, and performing updates on a fine-scale is related to V-cycle in multigrid methods [4].
We show that this approach can outperform the state-of-the-arr MeshGraphNets(33]] baseline on accuracy over a set of challenging fluid dynamies tasks, We obtain stable rollouts without the need to inject training noise, and unlike the baseline, do not observe strong error accumulation or drift in the phase of vortex shedding.
2 RELATED WORK
Developing and running simulations of complex, high-dimensional systems can be very time- intensive. Particular for computational fluid dynamics, there is considerable interest using neural networks for accelerating simulations of aerodynamics [3] or turbulent flows [22, 46). Fast predic- tions and differentiable learned models can be useful for tasks from airfoil design [40] over weather prediction [37] to visualizations for graphics [43, 41, 7]. While many of these methods use convo- lutional networks on 2D grids, recently GNN-based learned simulators, in particular, have shown to be very flexible approach, which can model wide range of systems, from artieulated dynamics [38] to dynamics of particle systems [24, 391. In particular, GNNs naturally enable simulating on meshes with irregular or adaptive resolution [35, 8].
These methods are either steady-state or next-step prediction models. There are, however, strong benefits of training on the whole predicted sequence: next-step models tend to drift and accumu- late eπrоr, while sequence models can use their long temporal range to detect drift and propagate gradients through time to prevent error accumulation. Models based on RNNs have been success- fully applied to 1D time series and small n-body systems [6, 54] or small 2D) systems [50]. More recently, transformer models [44], which have been hugely successful for NLP tasks [36], are be- ing applied to low-dimensional physics prediction tasks [18]. However, applying sequence mod- els to predict high-dimensional systems remains a challenge due to their high memory overhead. Dimensionality reduction techniques, such CNN autoencoders [34, 33, 27, 23, 30, 17, 12, 28), POD [45, 49, 5, 32, 19, 9, 48, 11], or Koopman operators [25, 10, 15] can be used to construct a low- dimensional latent space. The regressvee sequence model then operates
these linear (POD modes) or nonlinear (CNNs) latents. However, these methods cannot directly handle unstructured data with moving varying-size meshes, and many of them do not consider parameter variations. For example, POD cannot operate on state vectors with different lengths (e.g., variable mesh sizes data in Fig.2 2 bottom). On the other hand, CNN auto-encoders can only be applied rectangular domains and uniform grids; and rasterization of complex simulation domains to uniform grids are known to be inefficient and are linked to many drawbacks as discussed [13].
In contrast, our method reduces the input space locally by aggregating information on a coarsened graph. This plays the strength of GNNs to learn local universal rules and is very closely related to lelti-level methods [51, 16] and graph coarsening [14, 53, 2].
METHODOLOGY
3.1 PROBLEM DEFINITION AND OVERVIEW
We are interested in predicting spatiotemporal dynamics of complex physical systems (c.g., fluid dy- namics), usually governed by set of nonlinear, coupled, parameterized partial differential equations
2

---

Published as a conference paper at ICLR 2022
Y
Deooder
Interpolation
Initialize pivotal nodes
Layer
Euly-Coneected
Representation
Select Nodes
Encoder
Layer Normalization
40
S/-1
The diagram of the proposed model, SMMR-Tranffomer-GMMG. We first represent the domain as a graph and then select pivotal nodes (red//reem/yellow)) to encode information over the entire graph. The encoder GMR runs Message passing along graph edges that the pivotal nodes collect information from nearby nodes. The latent vector 2, summarizes information at the pivotal nodes, and represents the whole domain the current step. The transformer will predict 2++1 based on all previous state latent vectors, Finally, we decode 1++1 through message passing to obtain the next-step prediction Y 1+1
Figure 1:
(PDEs) as shown in its general form,
@u(ar, Flu((,t);;)) a (0, Tel. 01
where u(æ,0, 1) E Rd represents the state variables (e.g., velocity, pressure, or temperature) and F is partial differential operator parameterized by μ.. Given initial and boundary conditions, unique spatiotemporal solutions of the system can be obtained. One popular method of solving these PDE systems is the finite volume method (FVM) [29], which discretizes the simulation domain Ω into an unstructured mesh consisting of N cells (C; :i i= 1,..., N). At time i, the discrete state field Y. {ui, center. Traditionally, solving this discretized system involves sophisticated numerical integration over time and space. This process is often computationally expensive, making it infeasible for real- time predictions or applications requiring multiple model queries, e.g., optimization and control. Therefore, our goal to learn a simulator, which, given the initial state Y, and system parameters μr, can rapidly produce rollout trajectory of states Yr..Y..
1, ....., ......NN
can thus be defined by the state value u² each cell
As mentioned ahove, solving these systems with high spatial resolution by traditional numerical solvers can be quite expensive. In particular, propagating local updates over the fine grid, such as pressure updates incompressible flow, can require many solver iterations. Commonly used technique to improve the efficiency include multigrid methods [47], which perform local updates on both the fine grid, as well as one or multiple coarsened grids with fewer nodes, to accelerate the propagation of information. One building block of multigrid methods is the V-cycle, which consists of nown-sampling the fine to a coarser grid, performing a solver update, up-sampling back on the fine grid, and performing an update the fine grid.
Noting that GNNs excel at local updates while the attention mechanism over temporal sequences allows long-term dependency (see remark A.2), we devise algorithm inspired by the multigrid V-cycle. For each time step, we use a GNN to locally summarize neighborhood information on our fine simulation mesh into pivotal nodes, which form a coarse mesh (section 3.2). We use temporal attention over the entire sequence in this lower dimee--inniionall space (section 3.3), upsample back onto the fine simulation, and perform local updates using mesh recovery GNN (section 3.2). This com- bination allows us to efficiently make use of long-term temporal attention for stable, high-fidelity dynamics predictions.
3.2 GRAPH REPRESENTATION AND MESH REDUCTION
Graph Representation. We construct a graph G = (V,E) to represent snapshot of dynamic system at time step t. Here each node E V corresponds the mesh cell C, so the graph size is |V] N. The set of edges are derived from neighboring relations of cells: if two cells C, and C
3

---

Published as a conference paper at ICLR 2022
neighbors, then two directional edges (1,j) and (j. i) are both in &. We fix this graph for all time steps. At each step i, each node V uses the local state vector Ui as its attribute. Therefore, (G, (Yo...... Yr) forms temporal graph sequence. The goal of the learning model is to predict
(Y,,..., Yr) (Y,....,yy) given G and yo.
Mesh Reduction, After representing the system a graph, we GNN summarize and extract a low-dimensional representation 2, from Y, for each step t. In this part of discussion, we omit the subscript for notational simplicity. We refer the encoder as Graph Mesh Reducer (GMR) since its role is coarsening the mesh graph. GMR first selects a small set S C V of pivotal graph nodes and locally encodes the information of the entire graph into representations at these nodes. By operating on rich, summarized node representations, the dynamics of the entire system is -ell-ooproximated even on this coarser graph.
There are a few considerations for selection of pivotal nodes, for example, the spatial spread and the centrality of nodes in the selection. We generally use uniform sampling to select S from V. This effectively preserves the density of graph nodes over the simulation domain, i.e. pivotal nodes are more concentrated important regions of the simulation domain. More details and visualizations on the node selection process can be found in section A.6.
GMR is implemented as Encode Proile-Proesss-eee Decode (EPD) GraphNet [39]. GMR first extracts node features and edge features from the system state using the node and edge Multi-Layer Perceptrons (MLPs).
of mlpp Y [i].
e m (p(i) - p(j)).
Here [ is the i-th row of Y, i.e. the state vector at each node, and p(i) is the spatial position of cell C..
Then GMR uses L GraphNet processer blocks [38] to further refine node representations through message passing. In this process node can receive and aggregate information from all neighboring nodes within graph distance L. Each processor updates the node and edge representations as
f-1 e', = mlp (el of 10--1,0 o0 of = =mlp; of
Σ JEN:
f 1, L,...,, L.
(3)
13
Here vf, ei, are the outputs of Equation 2, and N denotes all neighbors of node The two functions (-p,(-) and mip" respectively concatenate their arguments as vectors and then apply MLPs. The calculation in equation 2 and equation 3 computes new set of node representations V (vf : V) for the graph G.
Finally GMR applies an MIPP to representations of the pivotal nodes in S only to "summarize" the entire graph onto a coarse graph:
h. = mlp, iES
(4)
We concatenate these vectors into a single latent = = concat i E S) as reduced vector represen- tation of the entire graph. We collectively denote these three computation steps GMR(G, Y). The latents can be computed independently for each time step t and will be used as the represen- tation for the attention-basee simulator.
Mesh Recovery To recover the system state from the coarse vector representation 2, we define a Graph Mesh Up-Sampling (GMUS) network. The key step of GMUS is to restore information on the full graph from representations of pivotal nodes. This procedure is the inverse of operation of GMR. We first set the representations at pivotal nodes by splitting z, that is, r; = h i S. We then compute representations of non- non-pivotal nodes by spatial interpolation [2]: for non-pivotal node j. we choose a set N₁ of nearest pivotal nodes in terms of spatial distance and compute the its representations F by
Wigh:
= Σ VEN;
Wij
(5)
ΣIEN; Wij"
d(j, 0)2
Here d(j, i) is the spatial distance between cells C, and C, Then every node E V has a represen- tation r;, and all nodes" representations are collectively denoted as R (r: E V).
4

---

Published as a conference paper at ICLR 2022
Similar to GMR, GMUS applies EPD GraphNet to the initial node representation R restore Y on the full graph g. We denote the chain of operations far Y GMUS(G, Details about information flows in GMR and GMUS can be found in section A.1.
all time steps and sequences. For each time step We train GMR and GMUS as auto-encoder and minimize the reconstruction loss
we compute Y, GMUS(G, GMR(G, Y:))
T
graaph
Σ Y 12
-
(6)
With these two modules we can encode system states (Yi.... Yr) latents (Z1,..., ZT) as a low- dimensional representation of system dynamics. In the next section, we train transformer model predict the sequences of latents.
3.3 AFTENTION-BASED SIMULATOR
As a dynamics model, we learn a simulator which can predict the sequence (z 2T) autore- gressively based on an initial state 2g GMR(G, Yo) and the system parameters μ.. The system parameters include conditions such as different Reynolds numbers, initial temperatures, or/and pa- rameters of the spatial domain (see Table for details). The model conditioned on μ., to be able to predict flows with arbitrary system parameters at test time, We encode into special parameter token z_1 of the same length as the state representation vectors 2t
2-1 = mp),()))
(7)
Then, the transformer model [44], trans()) (-) predicts the subsequent latent vectors in an autoregres- sive manner.
2 trans(==11 20); 2. trans(=-1, 1, 2o, 21, ........ 1
Specifically, we single layer of multi-head attention in transformer model, which found sufficiently powerful for the environments we studied. We first describe the computation of a single attention head: At step the prediction ž;-1 from the previous step issues query to compute attention (((-)) over latents (2-1, 20, ..,..., 21-1).
2T_ W W2 Vd
a(--1) softmax
[z-1, 20, 21..,, 2
Here Wi and W2 are learnable parameters, and denotes the length of the vector 21-1. This equation corresponds the popular inner-product attention model [44].
Next, the attention head computes the prediction vector
W3 11 20, ....... 2)--1)a a:-1
(10)
with the learned parameter W3.
Our multi-head attention model uses K parallel attention heads with separate parameters to compute K vectors (g.......9) 9. predict the residual of &, over 2; ž,- mlp,, (concat(g) .......
described above. These vectors are concatenated and fed into an MLP to
(11)
We train the autoregressive model entire sequences by minimizing the prediction loss
T
Lott
Σ 1 1=1
(12)
Compared to next-step models such as MeshGraphNet [35], this model can propagate gradients through the entire sequence, and can use information from previous steps to produce stable predictions with minimal drift and error accumulation.
The proposed model also contains next-step models as special cases: if we select all nodes as pivotal nodes and set the temporal attention model such that it predicts the next step with only physical parameters and the current step. then the proposed model becomes a next-step model. The proposed model shows advantage when reduces the dimension of representation by using pivotal nodes and expands the input range when predicting the next step. A.2 shows how this method reduces the amount of computation and memory usage.
5

---

Published as a conference paper at ICLR 2022
3.4 MODEL TRAINING AND TESTING
We train the mesh reduction and recovery modules GMR, GMUS by minimizing equation 6 and the temporal attention model by minimizing equation 12. We obtain best results by training those components separately, as this provides stable targets for the autoregressive predictive module. A second reason is memory and computation efficiency; we can train the temporal attention method on full sequences in reduced space, without needing to jointly train the more memory-intensivy GMR/GMUS modules.
The transformer model is trained by incrementally including loss terms computed from different time steps. For every time step t, we train the transformer by minimizing the objective Ei || Z Ee 11. Until the objective reaches a threshold, we add the next loss term || 2 || to the objective and continue to train the model We found that this incremental approach leads to much more stable training than directly minimizing the overall objective.
As the transformer model can directly attend all previous steps via equation 10, don't suffer as much from vanishing gradients, compared to e.g. LSTM sequence models. By predicting in mesh- reduced space, the memory footprint is limited, and the model can be trained without approximate gradient calculations such as teacher forcing or limiting the attention history. Details for training procedure can be found in section A.3.
Once we have trained our models, we can make predictions for new problem. We first represent system parameters as token x-1 by equation 7 and encode the initial state into a token zo GMR(G, Yo), then we predict the sequence 21...., Er by equation 8, and finally we decode the system state Y by
= GMUS(E)),, t= I....,
(13)
4 RESULTS
Datasets. We tested the proposed method to three datasets of fluid dynamic systems: (1) flow over a cylinder with varying Reynolds number; (2) high-speed flow over a moving wedge with different initial temperatures; (3) flow in different vascular geometries. These datasets correspond to applications with fixed, moving, and varying unstructured mesh, respectively. A detailed description of the datasets can be found in section A.4.
Methods. The proposed methods is compared to the state-of-the-arr MeshGraphNet method [35], which next-step model and has been shown to outperform a list of previous methods. Two versions of MeshGraphNet are considered here, with and without Noise Injection (NI). We also study variants of our model with LSTM and GRU (GMR-LSTM and GMR-GRU) instead temporal attention. These variants also operate in mesh-reduced space, and use the same encoder (GMR) and decoder (GMUS) as our main method. Model details can be found in section A.5. And section A.6 shows pivotal nodes for each dataset.
Stable and visually accurate model predictions. We tested the proposed method on three fluid systems. Figure 2 compares our model's predictions for velocity against the ground truth (CFD) on six test set trajectories. Our model shows stable predictions even without noise injection, and model rollouts are visually very closely the CFD reference. On the other hand, without mitiga- tion strategies, next-step models often struggle to predict such long sequences stably and accurately. For cylinder flow, our model can successfully learn the varying dynamics under different Reynolds numbers, and accurately predict their corresponding transition behaviors from laminar to vortex shedding. In supersonic flow over a moving wedge, our model can accurately capture the shock bouncing back and forth on moving mesh. In particular, the small discontinuities at the leeward side of the edge are discernible in our model predictions. Lastly, in vascular flow with a variable-size circular thrombus, and thus different mesh sizes, the model is able to predict the flow accurately. In both cylinder flow and vascular flow, both frequency and phase of vortex shedding is accurately cap- tured. The excellent agreement between model predictions and reference simulation demonstrates the capability of our model when predicting long sequences with varying-sizz size meshes. Results for pressure can be found in section A.9.

---

Published as a conference paper at ICLR 2022
Re 307 (Cylinder Flow)
Re 993 (Cylinder Flow)
8=85
t= 215
298
t=0
[=85
t= 215
t=298
OOurs Truth
T(0) 201 (Sonic Flow)
T(0)
299 (Sonic Flow)
1=0
1=15
t= 30
1=40
|=0
1=15
t= 30
1=40
Ours Truth
R 0.31
(Vascular flow) t= 160
R 0.49 (Vascular flow)
4=4
[=80
250
1=40
[=80
t= 160
t=00
Truth
Ours
Figure 2: Contours of the velocity field, predicted by model versus the ground truth (CFD). Our model accurately predicts long rollout sequences under varying system parameters.
Table 1: The average relative rollout eπror of three systems, with unit of x10-3 3, We compare MeshGraphNet with or without noise injection (NT), to three variants of the our model (LSTM, GRU, or Transformer), which all shared the same GMR/GMUS mesh reduction models. Our model significantly outperform MeshGraphNet on datasets with long rollouts.
Dataset-rollout step
Cylinder flow-400
Sonie flow-40
Vascular flow-250
Variable
u
p
T
NI
25
778
136
1.71
3.67
0.4
0.027
57
133
55
MeshGraphNet
without NI
98
2036
673
4.12
6.13
0.24
0.020
3117
1771
601
GRU
114
1491
1340
1.34
4.59
0.59
0.37
8.2
11.2
23.6
Ours
LSTM
124
1537
1574
1.57
5.8
0.69
0.45
8.4
11.1
23.3
Transformer
4.9
89
38
0.95
2.8
0.43
0.39
7.3
10
22
Error behavior under long rollouts To quantitatively assess the performances our models and baselines, we compute the relative mean square елror (RMSE), defined as RMMS(GGrediccioon ftruth) Elprreciation -a 2.ruthy2 over the full rollout trajectory. Table I com-
E(ppradiction
pares average prediction errors on all three domains. Our attention-based model outperforms base- lines and model variants in most scenarios. For sonic flow, which has 40 time steps, our model performs better than MeshGraphNet when predicting velocity values (u and v). The quantities and I' are relatively simple predict, and all models show very low relative error rates 10-*), though our model has slightly worse performance. The examples with long rollout sequences (predicting cylinder flow and vascular flow) highlight the superior performance of our attention iitention-based method compared with baseline next-step models, and the RMSE of our method is significantly lower than baselines.
Figure shows how error accumulates in different models. Our model has higher error for the first few iterations, which can explained by the information bottleneck of mesh reduction. However, the error only increases very slowly over time. MeshGraphNet error accumulation, rendering less accurate for long aag--ppn predictions. Noise injection partially addresses the issue, but error accumulation is still considerably higher compared to our model. We can attribute this better long-span error behavior of our sequence model, to being able to pass gradients all previous steps, which is possible in next-step models. We also note that the
the other hand suffers from strong

---

Published as a conference paper at ICLR 2022
10a
10
RMSE
10-1
10-3
10"3
10
10
10-3
10-3
10
Time Step
Time Step
Time Step
Figure 3: Averaged error over all state variables on cylinder flow (left), sonic flow (middle) and vas- cular flow (right), for the models MeshGrrahhNetsMMGNN --.)), MGN-NI(--) Ours-GRU (-). Ours-LSTM (--.-), Ours-Transfore
(-). Our model, particularly the transformer, show much less error accumulation compared to the next-step model.
Re = 405, / = 250 (Cylinder flow)
R= 0.31, / 228 (Vascular flow)
Next-Step
Ours
Truth
Next-Step
Ours
Truth
Figure 4: Predictions of the next-step MeshGraphNet model and our model, compared to ground truth. The next-step model fails to keep the shedding frequency and show drifts on cylinder flow. On vascular flow, we notice the left inflow diminishing over the time, while our model remains close to the reference simulation.
transformer model performs much better compared to the GRU and LSTM model variants. We believe its ability to directly attend to long-term history helps to stabilize gradients in training, and also to identify recurrent features the flow, as discussed later this section.
In Figure 4 we illustrate the difference in the models' abilities predict long rollouts. For cylinder flow, our model is able to retain phase information. MeshGraphNet on the other hand drifts to an opposite vortex shedding phase at t 250, though its prediction still looks plausible. For vascular flow, our model gives an accurate prediction at step = 228, while the prediction MeshGraphNet is significantly different from the ground-trute truth at this step: the left inflow diminishes, making the prediction less physically plausible.
Diagnosis of state latents. The dinensinnio latent vectors, computed by the encoder-decoder (GMR and GMUS), can be viewed as lossy compression of high-dimensional physical data on ir- regular unstructured meshes. In experiment, observe the compression ratio is 4% while the loss of the information measured by RMSE is below 1.5%. More details are in section A.7.
We also visualize state latents z-s from the cylinder flow dataset and show that the model can distin- guish rollouts from different Reynolds numbers. We first use Principal Component Analysis (PCA) reduce the dimension of z-s two and plot them in Figure 5 (left). Rollouts with two different Reynolds numbers have two clearly distinctive trajectories. They eventually enters their respective periodic phase. We also apply PCA directly to the system states (Y) and get 2-d latent vectors di- rectly from the original data. We plot these vectors in Figure 5 (right). Latent vectors from PCA start in the center and form circle. The two trajectories from two different Reynolds numbers are hard to distinguish, which makes it challenging for the learned model to capture parameter variations. Al- though CNN- based embedding methods [15, 51, 30] are also nonlinear, they cannot handle irregular geometries with unstructured meshes due to the limitation of classic convolution operations. Using pivotal nodes combined with GNN learning, the proposed model is both flexible dealing data with irregular meshes and effective in capturing state transitions in the system.
Diagnosis of attention weights. The attention vector a, in equation plays an important role in predicting the state of the dynamic system. For example, if the system enters perfect oscillating
8

---

Published as a conference paper at ICLR 2022
Paousater Token Allention
0.7
0.0
Time Step
Figure 5: 2-D principle subspace of the latent vectors from GMR (left) and PCA (middle) for flow past cylinder system: Re 307 (>), Re 993 +) start from and respectively.
Figure 6: Attention values of the parameter tokens versus time, Re 307 (-), and Re 993 (---).
Freq al Max Mag
0.16
Magnitude
0.14
0.5
0.5
0.12
0
0.2
0.4
300
650
1,000
Frequency
Frequency
Reynolds Number
Figure 7: Attention (-+) CFD data
(left) Re versus frequency for attention and CFD data. Fourier transform for both attention and CFD data at (middle) Re = 307 and (right) Re = 993.
stage, the model could directly look up and re-use the encoding from the last cycle as the prediction, make predictions with little deterioration or drift.
We can observe the way our model makes predictions via the attention magnitude over time. Figure 6 shows attention weights for the initial parameter token 2-1, which encodes the system parameters μe, over one cylinder flow trajectory. In the initial transition phase, we see high attention to the parameter token, as the initial velocity fields different Reynold numbers are hard to distinguish. As soon vortex shedding begins around 120, and the flow enters oscillatory phase, and attention shift away from the parameter token, and locks onto these cycles. This can be seen in Figure 7 (left): perform a Fourier analysis of the attention weights, and notice that the peak of the attention frequency (blue) coincides with the vortex shedding frequency in the data (purple). this
That our dynamics model learned to attend previous cycles in vortex shedding, and information to make accurate predictions that keep both frequency and phase stable. Figure 7 (right) shows that this observation holds for the whole range Reynolds numbers, and their corresponding vortex shedding frequencies.
For sonic flow, the system state during the first few steps much easier discern even without knowing the system parameters and hence, the attention to z-1 quickly decays. This shows that the most physically useful signals, whether
the transformer can adjust the attention distribution that is parameters or system state. Analysis on this, as well as the full attention weights maps can be found in section A.
5 CONCLUSION
In this paper introduced graph-based mesh reduction method, together with temporal at- tention module, to efficiently perform autoregressive prediction of complex physical dynamics. By attending whole simulation trajectories, method more easily identify conserved properties or fundamental frequencies the system, allowing prediction with higher accuracy and less drift compared to next-step methods.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
Han Gao and Jianxun Wang are supported by the National Science Foundation under award numbers SMMI-1934300 and OAC-2047127. Li-Ping Liu are supported by NSF 1908617. Xu Han was also supported by NSF 1934553.
REFERENCES
[I] MS Albergo, G Kanwar, and PE Shanahan. Flow-based generative models for markov chain monte carlo in lattice field theory. Physical Review D, 100(3):034515, 2019.
[2] Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Lozano-Perez, and Leslie Kaelbling. Graph element networks: adaptive, structured compu- tation and memory. In International Conference on Machine Learning, pp. 212-222. PMLR, 2019.
[3] Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):525-544 2019.
[4] Dietrich Braess and Wolfgang Hackbusch. A new convergence proof for the multigrid method including the v-cycle. SIAM journal on numerical analysis, 20(5):967-975, 1983.
[5] Ashesh Chattopadhyay, Pedram Hassanzadeh, and Devika Subramanian. Data-driven predic- tions of multiscale lorenz 96 chaotic system using machine-learning methods: reservoir com- puting, artificial neural network, and long short-term memory network. Nonlinear Processes Geophysics, 27(3):373-333 2020.
[6] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint arXiv: 806607366, 2018.
[7] Xiaohui Chen, Xu Han, Jiajing Hu, Francisco JR Ruiz, and Liping Liu. Order matters: Prob- abilistic modeling of node sequence for graph generation. arXiv preprint arXiv:2106.06 2021.
[8] Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems, pp. 7178-7189, 2018.
[9] Hamidreza Eivazi, Hadi Veisi, Mohammad Hossein Naderi, and Vahid Esfahanian. Deep neu- ral networks for nonlinear model order reduction of unsteady flows. Physics of Fluids, 32(10): 105104, 2020.
[10] Hamidreza Eivazi, Luca Guastoni, Philipp Schlatter, Hossein Azizpour, and Ricardo Vinuesa. Recurrent neural networks and koopman-based frameworks for temporal predictions in a low- order model of turbulence. International Journal of Heat and Fluid Flow, 90:108816, 2021.
[11] Stefania Fresca and Andrea Manzoni. Pod-dl-rom: enhancing deep learning- based reduced order models for nonlinear parametrized pdes by proper orthogonal decomposition. Computer Methods in Applied Mechanics and Engineering, 388: 14181, 2022.
[12] Kai Fukami, Koji Fukagata, and Kunihiko Taira. Assessment of supervised machine learn- ing methods for fluid flows. Theoretical and Computational Fluid Dynamics, 34(4):497-519, 2020.
[13] Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: ahysics-informed geometry-adaplen convolutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal Computational Physicx, 428:1 10079, 2021.
[14] Hongyang Gao and Shuiwang Ji. Graph u-nets. international conference on machine leam- ing, pp. 2083-2092. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2022
[15] Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. arXiv preprint arXiv:2010.03957, 2020.
[16] Dirk Hartmann, Christian Lessig, Nils Margenberg, and Thomas Richter. A neural network multigrid solver for the navier-stokes equations. arXiv preprint 1.xxv::008..152222 2020.
[17] Kazuto Hasegawa, Kai Fukami, Takaaki Murata, and Koji Fukagata. Machine- learning -basd reduced-order modeling for unsteady flows around bluff bodies of various shapes. Theoretical and Computational Fluid Dynamics, 34(4):367-383, 2020.
[18] Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Tch, and Hyunjik Kim. Lietransformer: equivariant self-attention for lie groups. In International Conference on Machine Learning, pp. 4533-4543. PMLR, 2021.
[19] Pierre Jacquier, Azzedine Abdedou, Vincent Delmas, and Azzeddine Soulaïmani. Non- intrusive reduced-order modeling using uncertainty-aware deep neural networks and proper orthogonal decomposition: Application to flood modeling. Journal of Computational Physics, 429:40 2021.
[20] Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods in numerical dynamics, volume 1000, pp. 1-20. IUC Dubrovnik Croatia, 2007.
[21] Gurtej Kanwar, Michael S. Albergo, Denis Boyda, Kyle Cranmer, Daniel C. Hackett, Sébastien Racanière, Danilo Jimenez Rezende, and Phiala E. Shanahan. Equivariant flow-based sampling for lattice gauge theory. Phys. Rev. Lett., 125:121604 Sep 2020. doi: 10.1103/PhysRev... 125.121601.
[22] Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamies. Proceedings of the Na- lional Academy of Sciences, 118(21), 2021.
[23] Angran Li, Ruijia Chen, Amir Barati Farimani, and Yongjie Jessica Zhang. Reaction diffusion system prediction based on convolutional neural network. Scientific reports, 10(1):1-9, 2020.
[24] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In Interna- tional Conference on Learning Representations, 2019.
[25] Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):1-10, 2018
[26] Luis F Gutiérrez Marcantoni, José P Tamagno, and Sergio A Elaskar. High speed flow simula- tion using openfoam. Mecánica Computucional, (1(16))2939-990 29599, 2012.
[27] Romit Maulik, Bethany Lusch, and Prasanna Balaprakash. Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders. Physics of Fluids, 33(3):033106 2021.
[28] Masaki Morimoto, Kai Fukami, Kai Zhang, Aditya G Nair, and Koji Fukagata. Convo- lutional neural networks for fluid flow analysis: toward effective metamodeling and low- dimensionalization. arXiv preprint arXiv:2101.22535, 2021.
[29] Fadl Moukalled, L Mangani, Marwan Darwish, et al. The finite volume method in computa- tional fluid dynamics, volume 113. Springer, 2016.
[30] Takaaki Murata, Kai Fukami, and Koji Fukagata. Nonlinear mode decomposition with convo- lutional neural networks for fluid dynamics. Journal of Fluid Mechanics, 882, 2020.
[31] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp- 4055-4064 PMLR, 2018
11

---

Published as a conference paper at ICLR 2022
[32] Suraj Pawar, SM Rahman, H Vaddireddy, Omer San, Adil Rasheed, and Prakash Vedula. A deep learning enabler for nonintrusive reduced order modeling of fluid flows. Physics of Fluids, 31(8):085101, 2019.
[33] Jiang-Zhou Peng, Siheng Chen, Nadine Aubry, Zhi-Hua Chen, and Wei- Tao Wu. Timi-variant prediction of flow over an airfoil using deep neural network. Physics of Fluids, 32(12):123602, 2020.
[34] liang-Zhor Peng, Siheng Chen, Nadine Aubry, Zhihua Chen, and Wei-Tao Wu. Unsteady reduced-order model of flow over cylinders based on convolutional and deconvolutional neural network structure. Physics of Fluids, 22669) 23609, 2020.
[35] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Repre- sentations, 2021.
[36] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
[37] Stephan Rasp and Nils Thuerey. Data-driven medium-range weather prediction with a resnet pretrained on climate simulations: A new model for weatherbench. Journal of Advances in Modeling Earth Systems, 13()::20200S002200, 2021.
[38] Alvaro Sanzhez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Ried- miller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference Machine Learning, pp. 4470-4479. PMLR, 2018
[39] Alvaro Sanchez-Gonzaldd Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Pe- ter Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning. pp. 8459- 8468. PMLR, 2020.
[40] Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):55-36, 2020.
[41] Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. In Computer Graphics Forum, volume 37, pp. 171-182. Wiley Online Library, 2018
[42] Kiwon Um, Robert Brand, Philipp Holl, Nils Thuerey, et al. Solver-in-the-loop: Learning from differentiable physics to interact with iterative pde-solvers. arXiv preprint arXiv:2007.0000 10006 2020.
[43] Benjamin Ummenhofer, Lukas Prantl, Nils Thürey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference Learning Represen- tations, 2020.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural informa- tion processing systems, pp. 5998-6008, 2017.
[45] Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumoutsakos. Data-driven forecasting of high-dimensional chaotic systems with long short- term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and 70844, 2018.
Engineering Sciences,
[46] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics-informed deep learning for turbulent flow prediction, 2020.
[47] Pieter Wesseling. Introduction to multigrid methods. Technical report, INSTITUTE FOR COMPUTER APPLICATIONS IN SCIENCE AND ENGINEERING HAMPTON VA, 1995.
12

---

Published as a conference paper at ICLR 2022
[48] Pin Wu, Junwu Sun, Xuting Chang, Wenjie Zhang, Rossella Arcucci, Yike Guo, and Christo- pher Pain. Data-driven reduced order model with temporal convolutional neural network. Computer Methods in Applied Mechanics and Engineering, 360: 12766, 2020.
[49] Xuping Xie, Guannan Zhang, and Clayton G Webster. Non-intrusive inference reduced order model for fluids using deep multistep neural network. Mathematics, 7(8):557, 2019.
[50] SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
[51] Jiayang Xu and Karthik Duraisamy. Multi-level convolutional autoencoder networks for para- metric prediction of spatio-temporal dynamics. Computer Methods in Applied Mechanics and Engineering, 112:113379, 2020.
[52] Jiayang Xu, Aniruddhe Pradhan, and Karthik Duraisamy. Conditionally parameterized, discretization-awarr neural networks for mesh-based modeling of physical systems. arXiv preprint vrxiv:2109.09510 2021.
[53] Rex Ying. Jiaxuan You, Christopher Morris, Xiang Ren, William Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling, arXiv preprint 100.11800.08000 2018.
[54] Ruiyang Zhang, Yang Liu, and Hao Sun. Physics-informed multi-lstm networks for metamod- eling of nonlinear structures. Computer Methods in Applied Mechanics and Engineering, 369: 113226, 2020.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 INFORMATION FLOW IN GMR AND GMUS
Graph Mesh Reducer
Graph Mesh Up-Sampling
Figure 8: Schematic of information flow in the Graph Mesh Reducer (GMR) and Graph Mesh Up-Sampling (GMUS). Red and blue nodes pivotal nodes. The node size increase represents information aggregation from surroundings.
Figure 8 shows how the information flows in Graph Mesh Reducer (GMR) and Graph Mesh Up- Sampling (GMUS). This is analogous to a V-cycle in multigrid methods: Consider a graph with seven nodes (DV) layers. The increased size of pivotal nodes after a message- pessage-passing layer represents the information aggregation from surrounding nodes.
to be reduced and recovered by GMR and GMSU with message-passing
A.2 COMPUTATIONAL COMPLEXITY
The computation of the proposed model is from the three components: GMR, GMUS, and the transformer. The graph contains N nodes and also has ((N) edges because of special structure. Then GMR and GMUS L-layer graph neural networks have computation time O Ld"), with d' being the maximum number of hidden units.
The transformer takes time O (d')²) make prediction: prediction needs to consider O(T) previous steps, and the computation over a time step is ((d''?²). Note that is the length of the latent vector at each step.
In our actual calculation, the transformer takes much less time than GMR or GMUS since the latent dimension d' is small.
However, it would be very expensive the model did not use GMR and directly applied a trans- former to the original system states. In this case, d' would be Nd, and the total complexity would be (2N22,) which would not be feasible for typical computation servers.
14

---

Published as a conference paper at ICLR 2022
Cylinder Fow
[b) Sonio Flow
(o) Vascuar Flow
Figure 9: Overview of datasets of cylinder flow, (b) sonic flow, and (c) vascular flow.
A.3 MODEL TRAINING
The GMR/GMUS and temporal attention model are trained separately. We first train GMR/GMUS by minimizing the reconstruction loss Leraph defined by Eqn. Then, the parameters of the trained encoder-decoder are frozen, and only the parameters of the temporal attention model will be trained by minimizing the prediction loss Las (Eqn. 12) with early stopping. The parameter token encoder mlp, is trained jointly with temporal attention. The training algorithm is detailed in Algorithm 1.
Algorithm 1 Training Process
Input: Domain graph G. Node features over time [Yo...., YT). GMRs, GMUS,,.ttention, RMSE threshold R², MLPp, condition parameter Learned parameters 0, and vb
Output:
repeat
for Y, Yo...., Yr) do
Y. = (GMR&
end for
Compute Vo.a + Vo (0, o. Yo.... Yrl, Po,... Fr1)
Update o, 0 using the gradients V.. V
until convergence of the parameters (0, Φ)
(20, 21......21 2r GMR YY..... Yrl), 2-1 (L)
for t E ,1, do
T')
repeat
E....., 2.] = Attention ([--1, 20l)
Compute V..v < Va. Late (w, 1/. [1,
22), (1..... 22))
Update w. v using the gradients V.. V
until RMSE([E)..... 2t], [.I...., 2r1) < Re'
end for
A.4 DATASET DETAILS
Three flow datasets, cylinder flow, sonic flow and vascular flow, are used in our numerical ex- periments, and the schematics of flow configurations are shown in Figure 9. Cylinder and vaseu- lar flow are governed by incompressible Navier-Stokes (NS) equations, whereas compressible NS equations govern sonic flow. The ground truth datasets are generated by solving the incompress- ible/compressible NS equations based on the finite volume method (FVM), We use the open-source FVM library OpenFOA.M [20] to conduct all CFD simulations. All flow domains are discretized by triangle (tri) mesh cells and the mesh resolutions
adaptive the richness of flow features. The case setup details can be found Table 2. Both cylinder flow and sonic flow datasets consist of 51 training and 50 test trajectories, while vascular flow consists of 11 training and 10 test trajectories.
15

---

Published as a conference paper at ICLR 2022
Table 2: Simulation details of datasets
Dataset
PDEs
Cell type
Meshing
# nodes
steps
oi (sec)
Cylinder flow
Incompr. NS
Fixed 1699
400
0.5
Sonic flow
Compr. NS
tri
Moving
1900
40
5 x 10
Vascular flow
Incompr. NS
tri
Small-varying 7561 (avg.)
250
0.5
Table 3: Input-output information of networks: u, p, T, T,P: P. velocity, pressure, temperature, density and cell volume, respectively. x denote the cell coordinates. To, Re, r denote the initial temperature (sonic flow), Reynolds number (cylinder flow) and radius of the thrombus (vascular flow).
denote X-axis velocity, Y-axis
GMR node input
GMR
GMUS node output
Parameter token model input
Nodal embed dim
edge
selected node
Dataset
input
2; Zj
Cylinder flow
U4 U4, Ps. m4, Re
Ug Pa
Re
4
256
U4 U/ Pc, T; Pi
2; zj.
Uj Dc,
Sonic flow
m4, T,0
D 2.y1
Tsp
T,
256
2; J.
Vascular flow
Uy U4 Ps. 77,
C; x
Ug 04.
400
Here we summarize all input and outputs of the GMR/GMUS models, as well as the attention simu- lator for each dataset. GMR encodes the nigh-dimensional system state onto the node presentations on pivotal nodes, while GMR decodes latents back to the high-dimensional physical state (Figure 8). The parameter token encoder takes the physical parameter μ as input, and outputs a parameter token. The attention model takes the parameter token, together with the latent vectors of states at all previous time steps z, to predict the latents of the next time step. All input and output variables for each dataset are detailed in Table 3.
The governing equations for all fluid dynamic cases can be summarized as follows,
0p
+ V (pv) = 0,
0t
K(v) Ōt
+ V. (pvv)= = V (NVv) Vp + V (m(Vv)?)
V v), 3
(14)
0
Dcp Dp
2
Øt (pcp T) + (Pc, = V (kVT) + pT
3 u + p
Dt
Dt
where is the velocity vector, / is the viscosity, l is the thermal conductivity, Cp is the specific heat of the fluid and the definitions of the stream scalar V, the potential scalar & are referred to the chapter in [29]. Only the sonic flow (compressible flow) involves solving the third equation.
For cylinder flow, the simulation mesh topology remains the same for all trajectories, and the Reynolds number varies. In sonic flow, topology also remains fixed; however, the mesh node coor- dinates change over the course of the simulation to simulate the moving ramp. Here, we vary the initial temperature. Finally, for vascular flow, vary the radius of the thrombus, which means that each trajectory will have a different mesh topology, and the simulation model should be able to work with variable-sized inputs in this case.
A.5 MODEL DETAILS
Node and edge representations in our GraphNets are vectors of width 128. The node and edge functions (mlp,. mlp,, mip,)) are MLPs with two hidden layers of size 128, and ReLU activation. The input dimension of mlp, is based on the number of input features in each node (see details in table 3), and the input dimension of mlp, is three due the one-layer neighboring edge. We use = GraphNet blocks for both GMR and GMUS. The node and edge functions mlp; and mlp;' used in the GraphNet blocks are L-yer U-attvvaee ML.Ps with hidden size of 128. The output
16

---

Published as a conference paper at ICLR 2022
(a)
(b)
Figure 10: Flow past cylinder flow system: (a) all cells in the FV mesh; (b) pivotal nodes
(b)
(c)
(d)
(e)
Figure 11: Sonic flow system: (a-c) all cells the mesh for different time steps; (d-f) pivotal
nodes for 3 different time steps.
size 128 for the all MLPs, except for the final layer, which of size 4 for Cylinder, Sonic and for Vascular.
We use a single layer and four attention heads in our transformer model. The embedding sizes of for each dataset (cylinder flow, sonic flow, and vascular flow) 1024, 1024, and 800, respectively. This is calculated by #pivotal nodes X Nouts in which Nout is the output size of the last GraphNet block. The mip, used encode the system parameter μ MLP with hidden layers of size 100, and with output dimensions that match z, i.e. 1024, 1024, and 800 for Cylinder Flow, Sonic Flow, and Vascular Flow, respectively.
Table 4: Number of parameters for each model
Dataset
MeshGraphNet
GRU
LSTM
Transformer
GMR-GMU
Cylinder flow
2.2M
9.4M
11.5M
14.1M
1.2M
Sonic flow
2.2M
9.4M
11.5M
14.1M
1.2M
Vascular flow
2.2M
5.8M
7.0M
8.5M
1.2M
A.6 PIVOTAL POINTS SELECTION
In general, pivotal nodes are selected by uniformly sampling from the entire mesh as preserves the mesh density distribution, which is designed based on the flow physics observed in training data. For our datasets, we select 256 pivotal nodes out of 1 699 cells for cylinder flow, 256 pivotal nodes out of 1900 cells for sonic flow, and 400 pivotal nodes out of 7561 cells for vascular flow. The pivotal nodes in vascular flow are manually reduced the aneurysm region, where flow features not rich. The pivotal nodes distributions for the three flow systems shown in Figures 10, 11, and 12, respectively.
17

---

Published as a conference paper at ICLR 2022
XXx
XX
(a)
(b)
X
X
x xx
(c)
(d)
x
xx
XX
Figure 12: Vascular flow system: all cells in the FV mesh (left), pivotal nodes (right), r = 0.3 (a-b),
= 0.4 (c-d), 0.5 (e-f).
18

---

Published as a conference paper at ICLR 2022
Table 5: Report of data compressing: Qol data represent original data of the variables of interest; Embedding represents the saved embedding vectors from the GMR (encoder); Decoder is the saved GMUS model which is used to decode the embeddings; Compression Ratio is the ratio of Qol data size the sum of the embedding and decoder sizes. The RMSE is the relative mean square error of the reconstructed data from embedding.
Dataset
Qol Data
Embedding
Decoder
Compression Ratio
RMSE (x 10-3)
Cylinder flow
3570 MB
82.1MB
12.8MB
37
14.3
Sonic flow
651MB
8.5MB
18 MB
25
1.11
Vascular flow
195IMB
8.5MB
49.3 MB
33
10
A.7 DATA COMPRESSION
The Graph Mesh Reducer (GMR) and Graph Mesh Up-Sampling (GMUS) can be treated as an encoder and decoder for compressing high-dimensional physical data on irregular unstructured meshes. For any given mesh data (g. Y). we can encode it into a latent vector 2 = GMR (g. Y) through GMR. Therefore, a high-dimensional spatiotemporal trajectory (G, Yo. Yr) can be compressed into a low-dimensional representation (G, 20...... Zr), which significantly reduces memory for storage. Similar other compressed sensing techniques, the original data can be re- stored by GMUS from latent vectors. Table 5 shows that the decoder can significantly reduce the size of the original data about 3% with a slight loss of accuracy after being restored by the de- coder. Note that only the GMR/GMUS are included in the data compressing process, the transformer dynamics model not involved.
A.8 INFERENCE TIME
Our ground truth solver uses the finite volume method to solve the governing PDEs. Specifically, the "pressure implicit with splitting of operators" (PISO) algorithm and "semi-implicit method for pressure linked equations" (SIMPL.F) [29] used simulate the unsteady cylinder and vascular flows, and a PISO-based compressible solver [26] is used to simulate the high-speed sonic flow. All these traditional numerical methods involve large number of iterations and time integration steps. In contrast, the trained neural network model is able to make predictions without the need of sub- timestepping. Hence, these predictions can be are significantly faster than running the ground-truth simulator. The online evaluation of our learned model at inference time only involves two steps,
Step 1: 21 = trans(z-1, =0), 2, = trans(z 2- 2n, 21, 21-1)
(15)
Step 2: Y, GMUS(2)),
t= 1..,..,, T
which are completely decoupled. We compare the evaluation cost of the learned model with the FV-based numerical models Table and observe significant speedups for all three datasets.
Table 6: Wall-clock time of finite volume (FV) model and Invention-GMUL to simulate trajectory for three flow cases.
Dataset
GMUS
Transformer
GT Model
Speed-up
Cylinder flow (for 400 time steps)
2sec
0.4758sec
16s8.59sec
682
Sonic flow (for 40 time steps) Vascular flow (for 250 time steps)
0.2sec
0.047sec
25.026sec
100
2. 2.6sccc
0.31sec
2451sec
800
A.9 OTHER VARIABLE CONTOUR
Figure 13 shows the pressure contours.
19

---

Published as a conference paper at ICLR 2022
Re 307 (Cylinder Flow)
Re 993 (Cylinder Flow)
85
t= 213
[= 298
[=83
t= 215
t= E228
OOurs Truth
T(0) 201 (Sonic Flow)
T(0) 299 (Sonic Flow)
1=0
1=15
t= 30
[=40
|=0
1=15
t= 30
[=40
Ours Truth
R = 0.31 (Vascular flow)
R 0.49 (Vascular flow)
[=4
[=80
[= 160
250
1=40
[=80
t= 160
250
Truth
Ours
Figure 13: The pressure contours of rollouts predicted by model versus the ground truth (CFD).
With different condition settings, the model accurately predict long sequences of rollouts.
A.10 FURTHER ANALYSIS OF LATENT ENCODINGS AND ATTENTION
Figure analyzes the latent encoding and attention for the sonic flows in similar vein to section 4. Different initial temperatures (as the system parameter) lead to fast diverging of state dynamics, and thus the parameter tokens overlap for different parameters. However, the attentions to z-1 also rapidly decay, demonstrating that the transformer can adjust the attention distribution to capture most physically useful signals.
Parameter Token Áttention
0.5
0.5
0.5
0
-0.5
0.5
0.5
0
0.5
0
20
40
Co
Co
Time Step
(a)
(b)
(c)
Figure 14: 2-D principle subspace of the embedded vectors from GMR (a) and PCA (b) for the high- speed sonic flow system: T(0) 201 trajectory (-)), (0)) 299 trajectory (-+) start from their parameter tokens - and -, respectively. (c) Attention values the parameter tokens versus
time, (()) 201 (-), and T(0) = 299 (---).
Figure 15 shows the overview of attention value distribution in time-time time diagram for all three fluid systems. The attentions of the state to that at different time step is plotted as log-scale contour. Note that the of attention values in the same equals to due to the softmax layer (Eqn. 9).
20

---

Published as a conference paper at ICLR 2022
0
0
-5
100
10-
-10
50
Tine
200
-10
Time
20
-15
100
-20
150
-5
200
-00
400
100
200
300
400
10
20 Time
30
40
250
0
100
200
Time
Time
(a)
(b)
(c)
Figure 13: Examples of attention value (log value contour) for (a) flow past cylinder, (b) sonic flow and (c) vascular flow.
Table 7: The average relative rollout error for the cylinder flow, with unit of x x10-3 We compare MeshGraphNet with noise injection (NI) to our model with transformer. We train the model on the training trajectories with 400 steps, and test it the unseen trajectories with 800 steps.
rollout step
800
Variable
"
p
MeshGraphNet-NN 43
1274 258
Ours- Transformer
158 48
A.11 LONGER ROLLOUTS
We perform additional experiment to test how the model performs when rolling out of the training range. We train a model for the cylinder flow setup (n=400 steps), and evaluate it for lengths of 2n.
To limit the memory footprint for very long rollouts,
sliding window of for the attention, ie., we always attend to the last n time steps. This way allow us to retain the same complexity for the training. Fig. 16 illustrates this procedure: always the parameter embedding token #_1 as the first entry of the attention sequence, followed by moving window of the n last timesteps.
Table 7 compares the performance of the proposed model with MeshGraphNet a longer rollout length (2n = 2 x 400 steps). We find that the proposed model still outperforms the baseline for all u, and p trajectories.
Figure 16: Sliding window for attention mechanism on latent representations. The parameter em- bedding token ž_1 is always included in the first entry.
A.12 PREDICTION ON CRNVECTION-DIFFUSIIN FLOW
Many of the systems study in the main paper have oscillatory behaviors, which hard to capture by next-step models stably. To further demonstrate the capability of the model on predicting non-oscillatory dynamics, add additional example: convection-diffusionn flow, governed by
dc
v
- (vc) a
2
c).
(16)
Pe
where cis the concentration of the transport scalar, v = [1, 1] is the convection velocity, and Pe the Peclet number, which controls the ratio of the advective transport rate its diffusive transport rate.
21

---

Published as a conference paper at ICLR 2022
Convection Flow
Diffusion Flow
1=0
2=25
t= 30
[=00
t=0
F=
t= 10
Truth
Ours
Figure 17: The concentration contours of rollouts predicted by our model versus the ground truth (CFD).
As shown in Figure 17, the model can accurately predict the non-oscillatory sequences of rollouts Quantitatively, our model (RMSE 1.03 x 10-5) still outperforms MeshGraphNet (RMSE 2.79 x 3)
A.13 CARDIAC FLOW PREDICTION
We test the proposed model on cardiac flows (pulsatile flows) with varying viscosity in multiple cardiac cycles, which is more realistic for cardiovascular systems. An idealized spatial-temporal inflow condition is defined as
v(z, t) 0.6(x (E Imma)
2nt (sin(
)) + 0.15
(17)
(Jmmx
Tmin )2
1.8
to simulate the pulsating inflow from the heart. The model can accurately predict 10 cardiac cycles (Figure 18). The comparison with MeshGraphNet is listed in Table 8, showing significantly better performance for our model.
Table 8: The average relative rollout error for cardiac flow, with unit of x
rollout step
300
Variable
t P
MeshGraphNet--NI 194 71 944
Transoranformmr
4.7 2.3 106
Cardiac Flow
1=90
2(X)
t=270
Truth
Ours
Figure 18: The velocity contours of rollouts predicted by our model versus the ground truth (CFD) for a cardiac flow.
22


Reviews:
[{"review_id": "yM427EN4sEV", "reviewer": "Reviewer_7ipR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper's abstract starts with \"Auto-regressive sequence models for physics prediction are often restricted to\nlow-dimensional systems, as memory cost increases with both spatial extents and sequence length.\"-- This is not true. A plethora of work in computational physics, fluid dynamics, and climate dynamics have looked at autoregressive prediction for very high-dimensional chaotic systems, and quite successfully with deep learning. The authors should do a more thorough literature review looking at these works. \n\n2. The cases presented are not that high-dimensional. In fact, the cylinder flow can be decomposed to its first 3 POD modes which are enough to describe the dynamics of the system. Case 2 is moderately complicated, Case 3 is a more suitable test case. So, motivating this work from a perspective of high-dimensional physics is not quite fair. It is undoubtedly a good starting point, but there have been many works on much more complicated systems.  \n \n3. The authors should dig some literature to look at methods used in what can be considered state-of-the-art in this field of data-driven physics. The baselines should include some of them.\n\n4. The best thing about the paper is the effort to diagnose what is going on in different components. I think that is a big step forward in this field. I feel this section of the paper to be very insightful and interesting. ", "summary_of_the_paper": "The paper presents a GNN-based model with temporal attention to auto-regressively predict flow velocity of 3 test cases in fluid dynamics. Prediction results and some diagnostic analysis on the components of the network have been done. ", "summary_of_the_review": "I recommend acceptance because of the diagnostic section of the paper. While there are a few concerns, I think with a bit of revision and better baselines, the paper presents a nice set of results especially with the diagnostics section", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sP0AWlrCxTV", "reviewer": "Reviewer_X3ET", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The use of a transformer architecture with attention in the context of physical simulations is interesting, and not widely studied (as far as I know). As such I was curious to see how well it works. The training methodology leaves a few open questions: is the network actually trained with sequences of different length than used for the evaluations later on? The autoregressive training described in 3.3 sounds like the network is trained on \"full sequences\". However, what would be important to demonstrate is that the models are trained for sequences of length n, and then evaluated for much longer sequences, e.g. 2n, 3n or higher for input parameters not seen at training time. If the training was successful, these should still give reasonable predictions.\n\nThe paper then compares a the proposed transformer with GRUs and LSTMs and comes to the conclusion that the transformer performs best across all scenarios. Unfortunately, my main worries stem from these simulation scenarios. \n\nInterestingly, both LSTM and GRU are on-par for the medical example. This example seems quite artificial: it's basically the cylinder wake flow case, but repeated with a vessel like geometry. That seems to make it even simpler than the original wake flow case. It would have been more interesting to model something like a simple cardiac cycle.\n\nWhile the summary of the different simulation cases makes sense in the main paper, the details provided in the appendix are insufficient. Neither governing equations, nor variables are properly explained. As thus, I find it very difficult to estimate whether the simulations are meaningful, which parameters were varied for the training data sets, or which cases were used for the test evaluations.\n\nMy guess is that the compressible \"sonic\" case is the most interesting one, but it's content remains unclear, as does the \"T_,0\" parameter that seems to be varied for the models. The two evolutions shown are very similar, though, and this case seems to consist of only 40 evaluation steps. That is probably too short to really leverage the attention architectures.\n\nDespite this, the analysis of the latent space content and attention weights provide some interesting insights, and it's good to see that the dominant CFD frequency is present in the attention data.\n\nThe appendix gives additional details of the network architectures used. Unfortunately, model sizes (in terms of trainable parameters) are missing. \n", "summary_of_the_paper": "This paper proposes to use auto-regressive sequence models with attention mechanisms to predict the evolution of physical simulations. Mesh-based discretizations are targeted, and hence the employed networks take the form of GNNs. The paper focuses on encode-process-decode structures, and a transformer is proposed as main method to predict future states in the latent space. This approach is evaluated with three CFD settings: an unsteady wake flow, a sonic backwards facing step, and a medical flow scenario. The results are additionally analyzed in terms of latent space and attention weight content.\n", "summary_of_the_review": "Overall, I think the paper targets a very interesting direction, and I can encourage the authors to continue their work in this direction. However, in the current form I find it difficult to argue for accepting the paper straight away. The training and simulation setups leave too many open questions to properly evaluate whether the attention approach actually provides benefits or not.\n\nPost rebuttal: the authors have answered my questions, and added two interesting simulation results in the appendix. So I've adjusted my score to a more positive one, and (seeing the other two positive reviews) I'd be fine with acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eal_XSyhCKa", "reviewer": "Reviewer_PRtV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles the problem of simulating physics using graph neural networks (GNNs). GNNs have recently shown to be extremely efficient and fast alternatives to physical simulators. The current state-of-the-art of physical simulation with neural networks, is MeshGraphNets, which uses a GNN archictecture to learn one-time-step updates. These updates are integrated over time to produce arbitrary length simulations.\n\nHowever, the feedforward nature of MeshGraphNets brings two issues to the fore. \n* Long-term stability: Since MeshGraphNets do not incorporate any feedback mechanism, they operate in an open-loop system, often resulting in eventual divergence of predicted velocities/accelerations.\n* Physical implausibility: The training recipes of MeshGraphNets requires adding noise to the state vectors (e.g., perturbations to positions velocities etc.) without which the approach overfits to a set of reference trajectories. Adding noise to these physical quantities makes the learning setup physically implausible (i.e., the function we learn does not map physically accurate initial states to physically accurate subsequent states.\n\nThis paper addresses both these issues by incorporating a self-attention mechanism between the graph encoder and decoder layers.\n\nAnother key novel aspect of the paper is in the reduction step, where an input graph is reduced to a much smaller, but representative ‘summary graph’. Learning dynamics over the summary graph, in principle, is sufficient to represent dynamics over the full mesh.\n\nThe experiments and results presented in this paper provide compelling evidence to both claims, and as such I would recommend this paper for acceptance. However, there are a few minor discussion points / analyses I believe will further strengthen the paper.\n\nIs there any insight on how succinct the learned representation might be? For instance, in the case of rigid bodies, the input meshes may be subsumed into a single node (the center-of-mass) and yet represent the full system dynamics.\n\nDoes the sampling strategy of the summary nodes have a significant impact on the resulting learned physical model? For objects that are inherently non-uniform density, the graph network might benefit from sampling in proportion to the object density as opposed to uniformly. Any thoughts/insights on this?", "summary_of_the_paper": "This paper presents a graph and attention based architecture to model the dynamics of complex physical systems. Their experiments indicate a significant (in cases, upto two order of magnitude) outperformance of a state-of-the-art graph network baseline.", "summary_of_the_review": "This paper is very well-written, tackles an important problem, and overcomes two shortcomings in existing approaches. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8pWxq6GWFHp", "reviewer": "Reviewer_UQq7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe proposed model which uses GNNs as auto-encoders for the graphs and then uses transformers as dynamic models seems to be innovative.\n2.\tThe empirical results on the three scenarios tested also support the better performance of the newly proposed model compared to MeshGraphNet, especially in the first and the third scenarios.\n\nWeaknesses:\n1.\tThe main critic (or confusion) I have for this model is its ability to capture complicated dynamics beyond oscillations. The transformer architecture makes this model good at handling oscillations, as earlier states can be easily copied to the current states through the attention mechanism. But what about scenarios beyond oscillations? In fact, just as the paper itself shows in the second scenario (Sonic flow), where the oscillations seem to be less happening, the difference between this algorithm seems to be less compared the MeshGraphNet (or even worse compared to it). The initial higher error also indicates that the transformer model actually fails to capture the actual dynamics. I think the authors need to show more scenarios not mainly about oscillations and whether this method still outperform the other one.\n2.\tMore results on how the performance of the GNNs for compressing the graph representations into hidden states are needed. The authors mention that the initial higher errors might be due to the loss of the states from the auto-encoders, can the auto-encoders be modified to make this loss lower? What are some key hyperparameters on this structure that would influence the performance?\n3.\tCan authors also provide some failure examples of this model to help better understand its strength and weakness?\n4.\tThis might be because I am not familiar about this specific field, but what is the main algorithm difference between this model and MeshGraphNet? Was the GNN autoencoder model also used there? Or is that newly proposed in this model?\n", "summary_of_the_paper": "This work proposes a new algorithm combining graph-neural-network (GNN) and auto-regressive sequence models for physics prediction problems. The authors first use GNNs to compress the physical graphs, then use transformers to predict the next steps of the compressed representations, and finally use GNNs to recover the graph representations from the predicted representations. Through empirical studies, the authors show that this method outperform the previous SOTA model (MeshGraphNet) significantly, especially in the long-rollout prediction scenarios. The authors further analyze the models to understand the success and find that the ability of the transformer model to replay the earlier sequences seems to be the critical for the better performance especially in the scenarios with oscillations.", "summary_of_the_review": "This work proposes a new model for prediction physics dynamics. This new model uses graph neural networks to compress the physic graphs into lower-dimensional representations and then uses transformer models to predict the dynamics. Through empirical studies on three physical scenarios, the authors show the advantage of this method compared to the previous SOTA. However, more results need to be presented to show the power of this method in complicated physical scenarios beyond oscillations, where the transformer models indeed gain advantage by their nature. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model to predict the spatiotemporal dynamics of physical simulations on irregular meshes. The observations are modeled as a sequence of graph representations, each graph corresponding to a snapshot of the observation sequence at time t. This model uses two components, a graph encoder-decoder to compress the observations and an autoregressive transformer to model the dynamics. The two components are trained sequentially. At inference time, given an initial hidden state infered from the data and some additional conditional information, a sequence of states is predicted in an auto-regressive manner in the hidden space, each state of the sequence is then decoded to produce a prediction in the original observation space. The originality of the model lies in the encoder-decoder graph and in the use of a transformer for the prediction. Tests are performed on three fluid dynamics simulation data sets.

All reviewers pointed out some original contributions in the proposed method, in particular the use of transformers in the learned hidden space. In the rebuttal, the authors provided substantial additional results and further details and explanations. Their responses led two reviewers to increase their scores. All reviewers ultimately agree that the paper presents interesting results and conclusions.
Decision: Accept (Poster)