Unifying Predictions of Deterministic and Stochastic Physics in Mesh-reduced Space with Sequential Flow Generative Model
Authors: Luning Sun
Xu Han
Han Gao
Jian-Xun Wang
Liping Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: AI4Science
Fluid Dynamics
Generative Models
Graph Neural Network
TL;DR: Propose a unified framework to predict the behavior of both deterministic and stochastic systems by generative model

Abstract: Accurate prediction of dynamical systems in unstructured meshes has recently shown successes in scientific simulations. Many dynamical systems have a nonnegligible level of stochasticity introduced by various factors (e.g. chaoticity), so there is a need for a unified framework that captures both deterministic and stochastic components in the rollouts of these systems. Inspired by regeneration learning, we propose a new model that combines generative and sequential networks to model dynamical systems. Specifically, we use an autoencoder to learn compact representations of full-space physical variables in a low-dimensional space. We then integrate a transformer with a conditional normalizing flow model to model the temporal sequence of latent representations. We evaluate the new model in both deterministic and stochastic systems. The model outperforms several competitive baseline models and makes more accurate predictions of deterministic systems. Its own prediction error is also reflected in its uncertainty estimations. When predicting stochastic systems, the proposed model generates high-quality rollout samples. The mean and variance of these samples well match the statistics of samples computed from expensive numerical simulations.

Unifying Predictions of Deterministic and Stochastic Physics in Mesh-reduced Space with Sequential Flow Generative Model
Luning Sun' Lawrence Livermore National Lab Livermore, CA 94550 sun42011nl.gom
Xu Han Tufts University Medford, MA 02155 xu www.hunnttu...... edu
Han Gao
Jian-Xun Wang
Li-Ping Liu
Harvard University Cambridge, MA 02138 hgao1@seas hwr.aardd.... edu
University of Notre Dame
Tufts University Medford, MA 02155 liping wwwttufts....... edu
Notre Dame, IN 46556 jwang338nd edu
Abstract
Accurate prediction of dynamical systems in unstructured meshes has recently shown successes in scientific simulations. Many dynamical systems have nonneg- ligible level of stochasticity introduced by various factors (e.g. chaoticity), so there is need for unified framework that captures both deterministic and stochastic components in the rollouts of these systems. Inspired by regeneration learning, we propose a model that combines generative and sequential networks model dynamical systems. Specifically, we use an autoencoder to learn compact representations of full-space physical variables in low-dimensional space. We then integrate transformer with conditional normalizing flow model to model the temporal sequence of latent representations. We evaluate the new model in both deterministic and stochastic systems. The model outperforms several competitive bascline models and makes more accurate predictions of deterministic systems. Its own prediction error is also reflected in its uncertainty estimations. When predicting stochastic systems, the proposed model generates high-quality rollout samples. The mean and variance of these samples well match the statistics of samples computed from expensive numerical simulations.
Introduction
Accurate prediction of long-term dynamics of physical systems is of great interest in many science and engineering fields. The classical simulation of a system heavily relies on spatial/temporal discretization of the space and the numerical solution from the governing equation [ However, due to the multi-scale, stochastic nature of the complex physics and the complexity of the geometry, the simulation of large-scale real-time applications is extremely expensive in computation (2 In recent years, deep learning models have been applied predict rollouts of large and complex physical systems thanks to their flexibility and scalability 3]. Moreover, they can handle uncertainties and non- coneeiiy in physical problems 44 more effectively than traditional methods.
mnittedimmeniioall algebraic system derived
Previous work has studied two types of complex dynamical systems: deterministic ones and stochastic ones. A deterministic system is often considered under perfectly controlled experiment with exactly
'quual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This type of system can be modeled by autoregressive predicting models. stochastic system such as quantum mechanics [1] and statistical physics [5] has stochastic rollouts. When there are stochastic forcing terms or IC/BC terms in the governing equations, researchers have developed various models for predicting the stochastic state variables, such as turbulence velocity and stock prices (06) Given that there is not a clear boundary between the two types of systems, it is highly desirable that a unified model can model either deterministic or stochastic system automatically. However, such models so far are limited to classical numerical models such as OpenFOAM (2I. There an urgent need to develop a unified deep-leaming model that model both types of systems.
known PDE terms and initial conditions (IC) or boundary conditions (BC) 311331
In this work, we propose unified framework based on deep generative models predict deterministic and stochastic systems. The new model is based on graph-structure state representations [T], which can handle irregular spatial areas commonly seen in dynamical systems. It uses an autoencoder to encode state representation to a low-dimensional latent vector. Accurate encoding and decoding critical for recovering the state of the dynamical system. This work makes several innovations to enhance the autoencoder's ability to preserve the information in system states. We provide new approach to encoding the graph-structured representation with a vector with a fixed length. We also get some inspiration from regeneration learning [22] and train our autoencoder with self-supervised learning.
To describe stochasticity in the system, we use a sequential probabilistic model for the latent representations. We integrate a transformer and a normalizing flow construct a step-wise predictive neural network in the latent space. When there is stochasticity, the model will learn the conditional distribution of the next latent state; when there no stochasticity, can place probabilities correct deterministic predictions and still can minimize the predictive error. By including the ability to simulate both deterministic and stochastic systems in a uniform framework, reduces the effort of developing separate models for different problems.
We evaluate our proposed framework in an extensive empirical study. The results indicate the proposed model outperforms the SOTA baselines on deterministic datasets regarding accuracy. More importantly. for the first time, we introduce several alternative evaluation metrics other than normalized RMSE for stocahstic fluid dynamics, which help improve comparisons between different methods in this domain. The proposed framework can produce high-quality samples for stochastic systems in the mesh space.
2 Background
2.1 Problem definition
Let's consider general partial differential equation (PDE) defined on the d-dimensional space and one-dimensional time domain,
du Øt
j 2., in Ω [0, Tema]
(1)
where Ω C Rd is the spatial domain, Tend is the endpoint of time, : öΩ x [0, T] is the random
parameter over time for stochastic systems (e.g., boundary conditions). and u : 12 x [0. T] is the primary solution variable (e.g., velocity and pressure of fluid flow). Here μ D is the global physical system parameter and is time-invariant (e.g., Re number) for a given rollout, j Ω x [0, T) D - Ω x [0. 7] is an aggregation of the spatial terms of conservation laws (e.g., source and flux). To numerically solve the conservation law in Equation T let C₁ be a mesh of Ω that is, C, = (C2 E Ω: =, 1,...... .V} is a collection of non-overlapping cells that cover Q. We further use W4 to denote the evaluation of u at the cell center of C at time step t.
Using the mesh described above, we apply finite volume discretization to yield the parametrized, nonlinear dynamical system. Here, the dynamical system can be computationally intensive because a fine-level mesh will lead to discretization with a large degree of freedom. For stability issues, the numerical time step also needs to be very small. Therefore, numerical simulations with traditional methods require extremely expensive computation and storage.
In this paper, we are interested in two different scenarios: deterministic dynamics with invariant parameter over time (c.g., fixed boundary condition ), and stochastic dynamics with random

---

parameter & over time (e.g., perturbations in boundary conditions of turbulent flow). Both cases will also include the global physical parameters μ.. The scope our paper is build a unified data-driven surrogate model for generating/predictin parametric PDE solutions.
2.2 Normalizing flow models
normalizing flow model constructs flexible probabilistic distribution by applying learnable bijective mapping to simple random variable (e.g. Gaussian distributed), Suppose the mapping is f(z) with æ E Rd being the simple imput variable, then we have the probability p(z) of z as follows:
p(z) p(x) det (D(((a))
(2)
a(w)
Here [(f(mm)) is the Jacobian of f at a:. The function f is usually a neural network with a layered 8(x)
structure, with each layer being a bijective mapping. Then the determinant @(f(=)) is the product
O(w)
of determinants of these layers' Jacobian matrices. With a special design of layer structures, their
determinants can be efficiently computed. For example, RealNVP model [23] is constructed by
stacking several coupling layers, each of which has a lower triangular Jacobian matrix. With h E Rd as the input, coupling layer ft runs the following calculation:
(h) concat (h(1 dd), [[(d' + 1) d @ exp(s(h(1 : d'))) + =(h[1:
(3)
Here concat concatenates its arguments one vector, and d' is usually about one-half of d. In this calculation, the first half of the vector directly copied to the output. The second half goes through an entry-wise linear transformation: the operation O is the Hadamard product, the neural network provides scaling coefficients, and the neural network provides biases. With such coupling layers, we have transformation that defines flexible distribution p(z).
p(z) = p(x) (de((
Here h" = x, and ht fe(he-1). At the same time, p(z) also has an efficient sampling procedure given by
Methodology
Our new predicting model has three components: encoder that compresses graph representation of the state into fixed-length vector, a sequential model that predicts next-step representations the latent space, and a decoder that decodes spatial states from graph representations. The encoder is an improved version of the GMR -MMS encoder [IT). The sequential model is a conditional flow model. The encoder and decoder are trained via self-supervised learning in regeneration learning.
3.1 Graph representation learning
Following GMR-GMUS we use graph G (V,E) encode a snapshot of a deterministic or stochastic dynamical system at time step l. Here each node i E y corresponds to the mesh cell C., and each edge (i,j) E E represents neighboring relationship between two cells. At each step I, the solution ug at cell becomes the feature vector of the node i E V at time L. Because the mesh is pre-defined and fixed, the graph representation (G, (wi,e, E V)) preserves the full information of the mesh Cj, at time f. Here use Y, (u., i E V) to denote snapshot at time t.
The key task for the encoder is encode the graph into a low-dimensional vector 2;. For this task, we use an improved version of the encoder of EMR-GMUS. The neural network over the graph representation to learn node representations. Then takes node vectors of a selection of "pivotal" nodes and concatenates them to get 2,. In our new encoder, we select set locations in the spatial space instead of graph nodes to aggregate spatial information: each selected location aggregates vectors of nearby nodes to get its representation. It decouples the graph representation and the aggregation operation so that select location can encode nodes within an arbitrary distance. To improve the training stability, we also improve the graph neural network's
encoder runs graph
3

---

Pbc
PbGMUs
Batch Norm
Coupling Layer
Batch Norm
Coupling
Layer
4r+1
1+
Cy+-
Layer Normalization
p(a++1
||, 20x 21s
Multl-Head Attention
PbGMA
Layer
Layer Normalization
Multi-Heas Attention
Multi-Head Attention
Y
20
21
4/-1
Figure The diagram of the proposed model, which first compresses the whole graph into the latent representation z, by PbGMR using selected positions. During the generation process, a transformer encodes physical parameters and previous latent representations into a condition vector Cr+11 from which normalizing flow model describes the conditional probability of Z Finally, the decoder PbGMUS decodes Z:41 through to obtain the next-step prediction Y1+1-
architecture by adding residual connections between its message-passing layers. We call this new encoder Position-based Graph Mesh Reducer (PbGMR), These two modifications clearly improve the encoder's ability preserve the state information, which will be empirically shown the experiment section.
The architecture of PbGMR is specified as follows. For notational convenience, omit the time from the subscript. For each graph node E V, we first extract node and edge features as follows.
= mlp,(u.);
e (pos(()si)) - pos(j))-
(5)
Here pos(1) is the spatial location of the cell center of C.. After that, we apply message-passing layers.
egy = eij layernorm(mnlp
(e
of-1
(6)
of = of-1 layernorm
mlp;
é-1 D
Σ
is
e=
JEN.
Here N₁ denotes all neighbors of node i, mlp; and mlp; are two separate multi-layer perceptrons, and layernorm is the noymalaormaiization operation. When perceptron accepts multiple arguments, its arguments are first concatenated into single vector. Each layer here is similar to a GraphNet block 24 but with slight differences. The prominent are residual connections and layer normalization in the update of node and edge representations: they help to stabilize the training procedure [25 27
After have learned node representations, represent the entire state. We randomly select a small set of centers the spatial area. For each center S, select & nearest mesh cells knn(c) based spatial distance and then compute the position representation h. by interpolation
need to aggregate them into single vector to
28)
Wey
h
Σ jEknn(c)
Wej
d(c,j)² .
(
jjcknn(e) Wej
Here d(c,j) is the spatial distance between cell Cj and the center Given that the calculation is scaled by the sum of Wey-s, the unit of the spatial distance does not change the calculation here. The number i of neighbors is hyper-parameter and we fix it to 10. Finally, concatenate all representations of centers into a single vector x = concat(h, /h, : E S) as the latent for the entire graph. Note that the centers in S are fixed for problem.
As a comparison, the encoder in GMR-GMUS uses graph nodes as centers only considers connected neighbors of center. Then the number of neighbors in the interpolation operation is

---

limited by the graph structure. As result, each center vector from can only represent information short range. Our new encoder overcomes this issue by decoupling the neighbors in interpolation and the neighbors in the graph representation and then gives the interpolation operation more freedom to represent a state of the system.
3.2 Decoding and self-supervised training
We devise decoder PbMUS recover node features on the graph from latent representation 2: Y = PbGMUS(z). Here we also consider the computation at a single time step and omit time indices. We first split and get vectors at interpolation centers: (he c E S) z and then compute the initial node representation r by spatial interpolation from centers:
Wiche
Wie
c(i,c)2 , VEVE
j jEkun'( (4) Zecknn'())
Wse
Here knn'(i) are k centers that are nearest to the cell center Then we apply message-passing layers to compute Y = gnn(G, (rf i V)), with gnn representing the network layers. These layers have the same architecture as PbGMR but use different learnable parameters.
Self-supervised training. Without considering the sequential property of the data, first train the encoder and decoder on single steps with self-supervised training. This training method shares the same spirit of regeneration learning and improves encoder and decoder's s abilities capture spatial patterns in the data. In particular, we use the reconstruction error as the minimization objective to train the encoder-decoder pair.
min
Σ || PbGMUS(P6GMR(X)))
(10)
3.3 Attention-based temporal conditioned generative model
Now we consider the sequence of latent representations from the encoder and devise sequential generative model to model the sequence of latent representations. The lowddimensional latent space reduces the modeling difficulty, and the vector form of latent representations z, avoids the graph structure and allows more model choices. We first decompose the sequence as follows.
T
P(zur [1, zo) = p(z1 ,r, 20) II
pl 2 Ir, 20, 21:-1)--
(11)
t=2
The key is to devise model for the conditional p(=z) 20, 11.-1)) We take two steps construct this conditional: we first represent the condition (J4, 20, 21:t-1) with single vector e and then adapt RealNVP 23 normalizing flow model, to describe the conditional p( p(zelc.).
Calculate the conditional vector with transformer. Because physical parameters μ control entire system, and the initial condition 2g contains information about the initial condition, we consider them special and use them in the prediction of z; for each I. We structure the problem as a "ttranslation" problem and use the transformer [29] to run the calculation. In our case, the input "sentence" is (H, 20), the first few "tokens" in the target sentence are (21,.... #t-1), and the "next token" to be predicted is c.
C = transformer((4s 20), (zi,..., z(-1)
(12)
While the computation is exactly the same one step in translation task, the rationale is very different. First, the translation task directly gets the z, from the transformer, but our model needs to send to conditional flow model to get the prediction an informative input sequence, but our model uses less informative input and depends on the transformer get reasonable output sequence.
Second, the translation task has
5

---

Predict 2; with a conditional flow model. Once we have vector representation of the condition, we can construct a conditional flow model from RealMVP. Specifically, we append the condition vector to the input of the scaling function and the bias function in equation 3 in each layer.
fe hh[1 d"), h/d
d @ exp(ssh)) d'], c.)) t(h(1 c()]
(13)
From these layers, we have the flow model p(z, ct). By chaining up the transformer and the conditional flow model, we have our sequential model for conditional probability plz,|,, 20, =1:r-1),
During training, train the sequential model on (2u..... zr) that computed from our P&GMR encoder. During inference, we can efficiently sample x, from the sequential model. The whole training and inference processes can be found in Appendix A.2 and Appendix A.3
Related Work
Deep learning for physics. For a deterministic system, one-step learning simply takes the current state the input and outputs the next-step prediction 3 32). To further improve the accuracy of long-term forecasts, dimension reduction is applied together with sequence nets, aiming to solve long dynamical systems on regular domain 33 Such works often adopt CNNs encoders that can not be applied to irregular mesh space. To further work on mesh data directly with dimension
reduction, the GNN encoder/decoder was initially introduced by 11] Various GNN architectures were also proposed to facilitate the learning of physics. [13]
T3
36 39
For physical stochastic
systems, CNN is mainly used for probabilistic prediction [8 solving stochastic system in the graph space.
40
However, it lacks researches
Regeneration learning and generative modeling. Regeneration learning is a new learning paradigm for data generation, which first generates latent representation for the data. Then the generation process happens on the latent space. There are many recently popular generative models for images 43), videos 44], speeches[45] and music (46] are built this paradigm. We notice that it should be further investigated for graph generation tasks.
5
Experiments
5.1 Deterministic dynamics
We benchmark our proposed model with three datasets from three flows: flows over the cylinder, high-speed flows over the moving edge, and vascular flows ([]] Note that for test cases, feed the model physical parameters and the snapshot at the beginning time to generate the whole trajectory. Detailed description
be found in Appendix A.5
We compare our new method against five SOTA models for fluid dynamics, including two variants of MeshGraphNet 3] and three variants of the GMR-GMUS(II, We use relative mean square error
(RMSE) as the evaluation metric: RMSE = (iprediction ..outbi?? More details can be found in the
Eiapredesiony
Appendix A.2 For cylinder and vascular flows, we calculate the RMSE with respect to velocity variables u, U, and the pressure variable p. For high-speed flows over the moving edge, we also calculated the RMSE for the temperature T.
The results on prediction errors. Table
shows the comparison of different models. Our model outperforms the baseline models in all the scenarios. For the cylinder flow case, our model has improvements of 22%. 17%, and 47% for u, v, and p. For the sonic
Table 1:
The average relative reconstruction enror of three systems, with the unit of x 10 33
Dataset
CMR-GMUS
BNGMR-GMUS
Cylinder flow
14.3
1.9
flow case, our model has improvements of 61%, 70% for the u and variables, and improvements of 82% and 50% respectively
Sonic flow
1.11
0.24
Vascular flow
10
2.8
for the p and T variables. For the vascular flow case, model achieves improvements of 52%, 45%, and 95% for u, v, and p. The ablation

---

Re 307 (Cylinder Flow)
Re
993 (Cylinder Flow)
[=75
t=
150
1=295
1=75
t=
119
t=33]
Sample
Std
T(0)
201 (Sonic Flow)
T(0)
299 (Sonic Flow)
1=20
t= 30
1=40
[=20
t= 30
(=40
Sample
Std
Figure 2: For each the first is one of our predicted samples for velocity. The second row the predicted standard deviation for velocity. Our podeel-predittd sample accurately reflects the physical systems. Mcanwhile, the predicted standard deviation also has physical spatial-tempon pattern.
study in the Appendix section (Table indicates that both the new encoder and the conditional normalizing flow are essential to improve prediction accuracy.
The error in the time axis is shown in Figure 3 The performance of model is shown with the solid black line. The model has small initial errors in all three tasks. Then its error accumulates over time, but the accumulation is much slower than that of competing models, so model consistently has the lowest error in the final time step.
To better understand our model, have also done extensive ablation study reported in the appendix. Here we only show a comparison of our model with GMR-GMUS in the encoding- decoding tasks. The results in Table. [ show that the new model has clearly better encoding and decoding capabilities. In one encoding and decoding case, the reconstruction error of our model is less than fifth of that of the EMR-GMUS model in the appendix. The ablation study also shows that both residual connections and encoding centers enhance the mucoding-decoding accuracy (please Table & in the Appendix section).
Contour analysis. As probabilistic model, we plotted the contour of the predicted sample and the standard deviation (std) for all three datasets. v visualizes the velocity predictions from two datasets. In each subplot, the top row is one prediction (e.g. one sample from our model), and the bottom row visualizes the standard deviation of predictions. To clearly compare different predictions, use the same color bar within the same dataset. With visual inspection, the sample predictions reasonable and stable for all three datasets.
The standard deviations also show reasonable spatial-tempont patterns consistent with our expectations. For cylinder flow with Re 307, the standard deviation larger around the vortex-shedding region, indicating the model is less certain about the fast-changing dynamics. Moreover, the standard deviation tends to grow with time, reflecting the error accumulation behavior for long-time rollouts. Moreover, the standard deviation is smaller when Re is higher, indicating that the model more certain about the prediction. The same trends can also be seen in other datasets. Therefore, our proposed model can accurately predict the dynamics for deterministic systems while providing
uncertainty estimate, potentially improving the interpretability of deep learning systems. More contours can be found in Appendix A.6
5.2 Stochastic dynamics
Dataset. We apply the proposed method solve a stochastic dynamical system governed by
unsteady-state incompressible Ntvier-Stokes equations. We also compare our method to the unsteady Reynolds-averaged Navier-Stokes equations (URANS), the most applied method in the industry due to the balance of accuracy and computational efficiency. Let Ω be the channel with backward- facing

---

Table 2: The average relative rollout error of three systems, with the unit x 10-3, compare with five different baselines. Two variances of MeshGraphNet with without noise injection (NI) and three variants of GMR-GMUS (LSTM, GRU, or Transformer).
Dataset-rollout step Variable
Cylinder flow-400
Sonie flow-40
Vascular flow-250
M
v
p
v
T
"
v
p
NI
25
778
136
1.71
3.67
0.4
0.027
57
133
55
MeshGraphNet
without
2036
673
4.12
6.13
0.24
0.020
3117
1771
601
GRU
114 1491
1340
1.34
4.59
0.59
0.37
8.2
11.2
23.6
GMR-GMUS
LSTM
124
1537
1574
1.57
5.8
0.69
0.45
8.4
11.1
23.3
decoding-onle
4.9
89
38
0.95
2.8
0.43
0.39
7.3
10
22
PMGMR-GMUS aacoding-decoding
3.8
74
20
0.37
0.85
0.079
0.01
3.49
5.47
1.05
10a
10-2
109
RMSE
10
19-
10-2
10-3
10
10
10-3
Time Step
Time Step
Time Step
Figure 3: Averaged error over all state variables cylinder flow (left), sonic flow (middle) and
vascular flow (right), for the models MeshGrraphNessMMGNN -. MGN-NI p. GMR-GMUS
GRU GMR-GMUS-LSTMM -...), TARR--GMUS-Transfffre
Ours model
step shown in Figure The goal of the problem is to generate the temporal unsteady velocity field close to the ones simulated by large eddy simulation (LES) with the inflow velocity along the inlet boundary subject to uncertainty. The inflow at every time step is uncertain and sampled from a 60-dimensional stochastic space subject to a uniform distribution as iniltt U u 10 u', u' ((0 1)}.
inlet iulltt
Analysis of fluid motion. To visually study the generated results, we plot several time steps of the stream-wise velocity (u) from collection of model samples in Figure The proposed method clearly generates diverse turbulent flow samples. Specifically, the multiscale structure of the vortex can be seen in the contour plots. To quantitatively examine the model performance on turbulent statistics, we plot and compare the mean and variance of the velocity profiles from the URANS, the proposed model, and LES results in Figure The mean velocity profile predicted by the URANS model (gray lines) has significant discrepancies compared to the LES result (cyan lines), particularly in the right half of the domain where the backward- facing step's geometry is less important and small-scale turbulent features dominant. As for the velocity variance, the URANS model underestimates the fluctuation the whole domain. On the contrary, both the LES velocity mean and variance can be captured well by our generated samples (purple lines).
Comprehensive evaluations. To evaluate the temporal energy transport, we plot the power spectral density using Welch's method [47] (left. The high wave numbers reflect the rapid changes of fluid motion (c.g., instantancous fluctuations), and the low wave numbers reflect the slow changes, such as the re-circulation introduced by the backward-facce -fcing step. By leveraging the sequence models,
Figure 4: Velocity contour at time step 40, 80, 120, 160 (left to right) from different samples.

---

(²(u)
-5
0
4
5
7
8
9
10
20
Figure 5: Evaluation lines: Inlet
Outlet
Wall zz-evaluation lines
Quanti-
ties from LES URANS and our model
the multiscale problem in the time domain can be automatically learned. However, URANS directly infinite experiments, and it not capture stochastic behavior.
calculate ensemble average Therefore, the energy (E(k)) of high- high-wave-nune wave-number (k) signals is underpredicted by URANS while the proposed model has consistent energy spectrum pattern with the LES simulations. Moreover, the turbulent kinetic energy (TKE) is computed to evaluate the quality of generated samples. Physically, the TKE is characterized by measured root-mean-squee -quure velocity fluctuations. By applying the decoder, the point-wise instantaneous flow can be recovered from the latent vector to the physical solution reasonably well. The TKE metric can be defined scalar:
Sa Sa [(u u)² + (v d)3ld Joz Jio.T.ma [ les u (ulea les yl
TKE-
(14)
indicates the preservation of the TKE benchmarked by the LES simulations. Our model can generate flow field preserving 99% of the energy, whereas URANS only do less than 20%. As for the temporal mean field, although URANS aims to solve directly, it is inevitable to introduce the bias from the LES result due the ergodicity assumption of RANS. Instead, the probabilistic problem to learn the distribution of the spatiotemporal field. We calculate the error of the temporal mean field and find that although URANS leverages the conservation laws, it still obtains much larger етгог than our generated samples. Finally, we further propose to measure the quality of predicted rollouts using evaluation metries of video generation computer vision. In particular, we consider two metrics: the continuous ranked probability score (CRPS) [48] 49 and Frechet Video Distance (FVD) (50]. We use CRPS to assess the respective accuracy of our probabilistic forecasting and LES models. Since our model considers the distribution of the forecasts as whole, it outperforms URANS, which only focuses on the mean of the distribution. Our model is also better evaluated in the FVD) metric. This indicates the learned distribution by our model is close the samples from LES.
directly formulate
We have also experimented with two deep learning methods, MeshGraphNet and GMR-GMUS, which are designed for deterministic systems. Their predictions both far from LES simulations, not to mention that they can only make deterministic predictions. More results can be found in Appendix A.13
Conclusion
We propose new learning model to predict/generate deterministic and stochastic fluid dynamics over unstructured mesh uniform way. With an integration of novel graph auto-encoder, a transformer, and a normalizing flow model, the new model decomposes temporal and spatial correlations in a dynamical system. outperforms competitive baseline models for deterministic systems while providing reasonable spatial-temporal pattern of forward uncertainty estimations. The samples

---

Table Energy cascade spectrum (2.58, 0l21)( and criteria of generation quality(righr).
URANS
LES
Ours
Quantity of interest
URANS
Ours
100
CRPS (Card)
3.24
1.28
CRPS (Cor)
2.0
1.08
E(k)
10 10-7
FVD (darl)
228112
1262
FVD (dw.4)
137860
397
10
mean flow error (Eurl)
0.31
0.0176
100
10
10a
mean flow error (eor4)
0.94
0.176
4:
turbulent energy (TKE,1)
0.192
0.99
from the model trained on stochastic systems capture the rich physical patterns of expensive LES simulations. The current model still has one limitation: can not accurately model stochastic variables close to boundary areas, which is shown in Appendix A.12 In future work, we will design new learning architectures overcome this issue.
Acknowledgement
We thank all reviewers for their insightful feedback. Liu was supported by NSF CAREER 2239869, Wang's group would like to acknowledge the funds from Office of Naval Research under award numbers N00014-23-1-207 and National Science Foundation under award numbers OAC-2047127.
References
[1] Fadl Moukalled, Luca Mangani, Marwan Darwish, F Moukalled, L Mangani, and M Darwish. The finite volume method. Springer, 2016.
[2] Stephen B Pope and Stephen B Pope. furbulentflows. Cambridge university press, 2000.
[3] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh- based simulation with graph networks. International Conference on Learning Representations, 2021.
[4] Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physics, page 11 110922 2023.
[5] Yarin Gal, Petros Koumoutsakos, Francois Lanusse, Gilles Louppe, and Costas Papadimitriou. Bayesian uncertainty quantification for machine-learned models in physies. Nature Reviews Physics, 4(9):573-577, 2022.
[6] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. simple baseline for bayesian uncertainty in deep learning. Advances neural information processing systems, 32, 2019.
[7] Jayaraman Thiagarajan, Rushil Anirudh, Vivek Sivaraman Narayanaswamy, and Timo Bre- mer. Single model uncertainty estimation via stochastic data centering. Advances in Neural Information Processing Systems, 35:8662-8674, 2022.
[8] James Dolezal, Andrew Srisuwananukomn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Brittany Cody, Aaron S Mansfield, Sagar Rakshit, Radhika Bansal, Melanie C Bois, et al. pnncetainty-informe deep learning models enable high-confidence predictions for digital histopathology. Nature communications, 13(1))6572, 2022.
[9] Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-ifformed deeponets. Journal of Computational Physics, 475:1 11855, 2023.
[10] Zhong Yi Wan, Leonardo Zepeda-Núñes Anudhyan Boral, and Fei Sha. Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems. arXiv preprint arXiv:2301.10391, 2023.
10

---

[11] Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.09113. 2022.
[12] Rishikesh Ranade, Chris Hill, Lalit Ghule, and Jay Pathak. A composable machine-learning approach for steady-state simulations on high-resolution grids. arXiv preprint arXi:22210008837, 2022.
[13] Jiayang Xu, Aniruddhe Pradhan, and Karthikeyan Duraisamy. Conditionally parameterized, discretization-awaa neural networks for mesh-based modeling of physical systems. Advances in Neural Information P Processing Systems, 34: 14:1634-1645, 2021.
[14] Dmitrii Kochkov, Tobias Pfaff, Alvaro Ganchez-Gonzalee Peter Battaglia, and Bryan K Clark. Learning ground states of quantum hamiltonians with graph networks, arXiv preprint arX:v:2110.06390, 2021.
[15] Alvaro Sanchez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference machine learning, pages 8459-8468. PMLR, 2020.
[16] Nicholas Geneva and Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows. arXiv preprint 200x:22006.04731, 2020.
[17] Michael McCabe and Jed Brown. Learning to assimilate in chaotic dynamical systems. Advances in Neural Information Processing Systems, 34:12237-12250, 2021.
[18] Mustafa Z Yousif, Linqi Yu, and HeeChang Lim. Physics-guided guiided deep learning for generating turbulent inflow conditions. Journal of Fluid Mechunics, 936:A21, 2022.
[19] Deniz A Bezgin and Nikolaus A Adams. Normalizing flows as novel pdf turbulence model. arXiv preprint arXiv:2101.03590 2021.
[20] Mengyuan Yang, Xiaolin Zheng, Qianqiao Liang, Bing Han, and Mengying Zhu. smart trader for portfolio management based on normalizing flows. IJCAL, 2022.
[21] Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods in numerical dynamics, volume 1000, pages 1-20, 2007.
[22] Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, and Yoshua Bengio. Regeneration learning: A learning paradigm for data generation. arXiv preprint arXiv:3301.08846 2023.
[23] Laurent Dinh, Jascha Sohl- Dickkstiin and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605.v6600..800 2016
[24] Alvaro miller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physies engines for inference and control. In International Conference on Machine Learning, pages 4479. PMLR, 2018
Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Ried-
[25] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams, The shattered gradients problem: If resnets are the answer, then what is the question? International Conference on Machine Learning, pages 342-350. PMLR, 2017.
[26] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems,
2021.
[27] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[28] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
11

---

[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[30] Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In Conference on Robot Learning, pages 1 1177-1167. PMLR, 2023.
[31] Meire Fortunato, Tobias Pfaff, Peter Wirnsberger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets. arXiv preprint arXiv:2210.00612,, 2022.
[32] Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint 22Xiv:2202.00728. 2022.
[33] Han Gao, Jian-Xun Wang, and Matthew J Zahr. Non-intrusive model reduction of large- scale, nonlinear dynamical systems using deep learning. Physica D: Nonlinear Phenomena, 412:132614, 2020.
[34] Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Neural Networks, 146:272-289, 2022.
[35] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems, 35:2240 2253, 2022.
[36] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[37] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01 2018.
[38] Marco Maurizi, Chao Gao, and Filippo Berto. Predicting stress, strain and deformation fields in materials and structures with graph neural networks. Scientific Reports, 12(1):21834, 2022.
[39] Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. Advances in Neural Information Processing Systems, 35:26256-26268, 2022.
[40] Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[42] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[44] Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi, and David J Fleet. Vidco diffusion models. arXiv preprint arXiv:2204.0344 03445, 2022.
[45] Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779 4773,, IEEE, 2018
12

---

[46] Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, and Yan Liu. Telemelody: Lyric-to-melody generation with template-based two-stage method. arXiv preprint
2021.
[47] Peter Welch. The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacousticc, 15(2):00-73, 0000000-33310 1967.
[48] James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science, 22(10): 1087-1096, 1976.
[49] Christopher AT Ferro, David S Richardson, and Andreas P Weigel. On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorological Applications: journal of forecasting, practical applications, training techniques and modelling, 15(1):19-24, 2008.
[50] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[52] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[53] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-17) 1780, 1997.
[54] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
13

---

A
Appendix
A.1 Proof
The Jacobian of the transformation in Equation
can be written as:
(x)
Idd
0 diag(exp [ ( (oncal .e)])
(a) (x)
(15)
(x)
fwhere diag(expp))) (exp(s "fonlas("="conna((((( ((:) cc]]] is the diagonal matrix whose diagonal elements corre- spond to the vector exp s ( ci-ccoon lid" .e) Since the concatenation operation still keeps the matrix triangular, the determinant can be computed as:
det(0)((z)00- = exp(sun(ss(onnca((**11) c)))
(16)
A.2 Training
As regeneration learning method, there are two stages during the training. The first stage is to train a graph auto-encoder, GUGMRRGMUS, in this case, through self-supervised learning. This training process happens all time steps and sequences in the training set. And for given sequence .Y....,, Yr], we compute Y, PbGMUS(C, PbGMR(G, Y)) each step and minimize the reconstruction loss:
C ggraph
Σ Y Y112
(17)
In the second stage, the parameters in P6GMR-GMUS will be fixed. And the attention- based sequence model, well as the conditional flow model, will be trained. We maximize the log-likelihood in the latent space:
Cp
log (z)le))
(18)
We don't set a time window so that the whole trajectory will be trained. The entire training process can be found in Algorithm I Particularly in the second stage, such a teacher forcing-like like strategy allows a faster training process compared with the previous work [1] which adopts a rollout strategy during the training.
A.3 Inference
For inference we first encode the first two snapshots into latent space [zo, 21 The condition vector is computed from the braetd sequence model. By sampling noise vector æ from an isotropic Gaussian, and putting it going backward through the flow, we can obtain a new sample from (x)(())) as latent representation at each step. We describe the procedure of training in Algorithm 2
A.4 Latent analysis
Pigure the latent z, and conditional vectors e, for the stochastic BFS cases. For visualization purposes, we plot the first two modes of the PCA result. Moreover, we plot two different realizations of the stochastic process with clearly distinctive trajectories, as shown in the e plot (right). Since the cases are inherently from the same distribution, the sampled latent trajectories z, plot (left) have lot of overlaps after the few initial development steps. The plots of different cases visually conform to the physical intuition and indicate that model can learn the relation between different cases well.
14

---

Algorithm 1 Training Process
Input: Domain graph G, Node features time YY.... Yr). PbGMRR. PbGMUS Attention
encoder MHA Attention decoder Masked-MHAJ, conditional flow model (-))),, physical condition parameter μ
Output: Learned parameters 0, 6, w and ve repeat
for Yi YY...., Yr) do = -PbGMUS (P 9))
end for
Compute Vo. +- Ve,e (0, Φ. Yo.
Yrl, YY....., Yr))
Update Φ, 0 using the gradients
until convergence of the parameters (8, 6)
/Self-supervised learning)
[20, 21..... 2T] = GMRR( (Y...... Yrl)
repeat
út, 20 = M (11 zn)
C2, cq] Masked- ((21, 21,1), (0),20)
Compute W.,A, &- V..d. (p X, vh C2 (c.....,ct Cel, 12 (2,,..., 21))
Update w, to using the gradients V.. VA. V
until convergence of the parameters (w, X, vf)
Algorithm 2 Inference Process
Domain graph G, The firt two snapshot of node features Y,, Yi). PbGMR,, PыGMUS g. Attention encoder MHA,, Attention decoder Mas, conditional flow model p=(.)) physical condition parameter μr, sample length T
Input:
Output: A trajectory sample (Y,..... Yr]
[z0, 21 PbGMR& ([o, Y)).9) ja, ž0 = MHA zo) for / [2..... T do
I= (ūt, [0]] Sample latent representation from conditional flow 2, (z(()))))
C: Masked-MMHA
end for
Recover physical parameters on the original space (Y,.... Yrl P 2rl)
20
0
5
-10
-20
25
-20
5
0
Co
Co
(a)
(b)
Figure 6: 2-D principle subspace of the (a) latent vectors and (b) conditional latent vectors for the stochastic BFS cases: trajectory trajectory 2 start from
and respectively.
15

---

A.5 Dataset
Four fluid simulation datasets, cylinder flow, sonic flow, vascular flow and stochastic backward- faeing step (BFS) are used in the experiments. The labeled data are generated by solving the Navier Stokes equation by OpenFOAM [2I]. a commonly used finite volume method-basee open-source solver for fluid simulation. Cylinder and sonic flow consist of 50 cases in the training dataset and 50 cases in the test dataset. The vascular flow cases have 10 training and 10 test cases. The stochastic BFS case has 5 training cases. Unlike the first three deterministic cases, the stochastic BFS case is stochastic case. The simulation details can be found Table Moreover, the details input and output of the PbGMR-GMUS are listed in Table 5
The governing equation for the first three cases is the same as the one listed in For the last stochastic BFS case, the governing equation is listed in Equation (19). One significant difference in this case is that introduces stochasticity by perturbing the inlet u velocity at every time step.
Table 4: Simulation details of datasets
Dataset
Meshing
nodes
# steps
type
Cylinder flow
Fixed
1699
400
deterministic
Sonic flow
Moving 1900
40
deterministic
Vascular flow
Small-varying
7561 (avg.)
250
deterministic
Stochastic backward-facing step
Fixed
22500
240
stochastic
Table Input-output information of PbGMR-GMUS: u: X-axis velocity, v: Y-axis velocity, p: pressure, T: temperature, density, m: cell volume. To, Re, ugi denote the initial temperature, Reynolds number, radius of the thrombus, and boundary velocities, respectively. BFS denotes backward-face farinng-fingg steps.
P6GMR node input
PЫGMUS node output
Nodal embed dim
Dataset
# centers
Cylinder flow
Ug Us. Pt, m4, Re
Ug 0/s Ps
256
Ui, Vi, Ps. Ti, Pi
Us U2, Pi
Sonic flow
m4, T.o
Tú, Pi
256
Vascular flow
U4. U/ Pa, "k,""
U4 02, Pi
4
400
Stochastic BFS
ma, UBB
U/, Ub
4
256
Ou
dv
Ox +
0,
Uu
Ou
Ou
0p
8 u
8²u
+ u
+
U
+ vl
(19)
tt
Ox
Dy
dx
0y
Iu
dv
du
1 8p
8ºv
O2 0y2
vl
Ht
oy
p By
Ox?
subject to us uinlet = 10 + w', ~ u(0, 1) at every time step.
A.6 Additional visualization results
Apart from the contour shown in Figure 2 additional contour plots for the cylinder, sonic flow and vascular flow are shown this section. Figure shows the sampled velocity and std of the velocity for vascular flow cases. Figure 8 presents the sampled pressure for cylinder, sonic and vascular flow. Additionally, Figure shows the sampled temperature variable in sonic flow.
16

---

R
0.31 (Vascular flow)
R 0.49 (Vascular flow)
[=90
80
1=230
90
150
200
Sample
Std
Figure 7: Velocity contour for the vascular flow case, the first row is one of predicted samples.
The second row is the predicted standard deviation.
Re 307 (Cylinder Flow)
Re
993 (Cylinder Flow)
[=75
30
(=295
1=75
199
Sample
Std
T(0 201 (Sonic Flow)
T(0)
299 (Sonic Flow)
[=20
t=
30
[=20
t= 30
(=40
Std Sample
R 0.31 [= 180
(Vascular flow)
1 = 0.49 (Vascular flow)
[=00
1=230
1=90
[= 150)
(=200
Sample
Std
Figure 8: Pressure contour for benchmark cases
T(0) 201 (Sonic Flow)
T(0) 299 (Sonic Flow)
[=20
t=
30
[=20
t= 30
[=40
Std Sample
Figure Temperature contour for sonic flow case.
17

---

A.7 Multi-head attention
Transformer [29 has been proven successful in the NLP field. The design of the multi-hews attention (MHA) layer is based on the attention mechanism with Query-Key Value (QKV). Given the packed matrix representations of queries Q, keys K, and values V, the scaled dot-product attention used by Transformer is given by:
QKT VDx
A K, V) = softmax
V
(20)
where D3 represents the dimensions queries and keys.
The multi-head attention applies heads of attention, allowing a model to attend to different types of information.
MHA K, CONCAT (head1..... hee... W
ATTENTION (QW? KW, vW.) , I,..., H. (21)
where
head;
A.8 Additional details for experimental setups
We descrihed the details the experiments of PbGMR-GGM -MMUS and attention-based conditional flow model. We provide the hyperparameters used in the experiments in Table
Cylinder
Sonie
Vaseular
Stochastic BFS
PGGMR-GMUS Optimization
Learning rate
1 10
1x 10-5
1 10-4
1 10-4
Optimizer
Adam [51]
Batch size Number of epochs
1500
20000
700
600
MHA/Flow model Optimization
Learning rate
10-4 -4
x 10-5
1 x 10-5
1 x 10-4
Optimizer
Adam
Batch size
5 90000
50
10 240000
5 220000
Number of epochs Weight decay
240000
10-5
N/A
PbGMR-GMUS Architecture
Layers of message-passing
3 128
Hidden dimension
Output dimension
256
256
400
256
Activation function
relu
MНА Architecture
Layers of Encoding MHA
2
Layers of Decoding Masked-MHA
1
Hidden dimension
1024
1024
1600
1024
Activation function
gelu
[52]
Number of heads
4
8
4
4
Flow model Architecture
Conditioning length
1024
Hidden size
1024
Number of coulping layers
2
Table 6: Hyperparameters
18

---

1e-2
1.0
u error
V error
0.8
p error
RMSE
0.6
0.4
0.2
0.0
50
100
150
200
250
300
350
400
# pivotal positions
Figure 10: Reconstructor error RMSE with different numbers of centers for cylinder flow, with the training epoch = 300.
A.9 Performance impact of substituting Transformer with LSTM
In this research, we conduct ablation study in which we replace the proposed attention- based sequence model in our framework with Long Short-Term Memory (LSTM) [53] architecture. The aim is to evaluate the impact of the underlying sequence modeling system. The result is listed Table 7 Following this change, we observe that there a notable decrease in the performance metrics, suggesting less optimal fit for the task at hand compared to Transformer-based model. This outcome underscores the usefulness attention-based sequence models, like the Transformer, which appears capture dependencies in the data more effectively than the LSTM. The attention mechanism inherent to Transformers allows the model to focus on different parts of the input sequence dynamically, which may explain the observed performance superiority.
the performance of our
Table 7: The average relative rollout error of two systems, with the unit of x 10-3
Dataset-rollout step
Sonie flow-40
Vascular flow-250
Variable
u
V p
T u p
P5GMR-GMUS LSTM 10.3 29.7
1.7 11.1 43.0 46.6 16.6
A.10 Performance impact of number centers
We conduct an ablation study to investigate the effect of the number of centers on the RMSE of our framework's 's predictions for cylinder flow recovery. The result is in Figure I0 As we progressively increased the number of centers, the recovery RMSE exhibited decreasing trend, demonstrating an enhancement prediction accuracy. Interestingly, upon reaching 256 centers, the rate RMSE decrease started to decelerate significantly. This suggests that the optimal number of centers for cylinder flow is around 256. Beyond this point, further increments do not contribute as much to the improvement in mncoding-decoding cost. Hence, a count of 256 centers appears to be the sweet spot, offering a good trade-off between recovery accuracy and computational efficiency.
accuracy, while simultancously increasing the computational
19

---

Figure 11: The first row shows meshes and the second row shows the distribution for the centers.
A.11
Centers
The centers distribution for Stochastic BFS can be found in Figure 11
A.12
Analysis for stochastic BFS
Backward- faccinggg-agggggg- step flow refers to fluid flow configuration characterized by sudden contraction followed by an expansion in a channel or pipe. This flow configuration is commonly encountered in various engineering applications, such in heat exchangers, combustion chambers, and aerodynamic systems. In wackward-facing step flow, the fluid initially enters the channel or pipe through an inlet, where encounters a step or abrupt contraction. As the fluid passes over the step, its velocity increases, and the flow separates, forming recirculation zone downstream. This recirculation zone is often referred to as the "backflow region." The backward- facing step flow has been extensively studied due to its complex flow characteristics and its relevance in understanding phenomena like separation, reattachment, and flow control.
The simulations and generated samples exhibit remarkable similarity in the contours of both stream- wise velocity and wall-normal velocity (Figure T2 L 14). This close resemblance demonstrates the high fidelity of the simulation results and the accuracy of the generated samples. The level of detail captured these contours highlights the effectiveness of the proposed generative framework.
Figure 12: Velocity contour (+) time step 40, 80, 120, 160 (left right) from different samples.
Figure 13: Velocity contour (u) time step 40, 80, 120, 160 (left right) from different LES simulations.
We also randomly choose nine spatial locations analyze the learning performance, and the coor- dinates of these points are plotted in Figure 15 Figure 16 and 17 presented the comparison of u, velocity between the LES simulation, URANS simulation and our model. Generally, the pointwise velocity temporal signals obtained from the generated sample provide comprehensive representation of the flow characteristics and look visually similar those obtained from LES simulations. However, the traditional URANS modeling approach cannot capture the intricate fluctuations in the flow field. Note that we also found the generated velocity time signals at points and have relatively large discrepancy from the LES simulation, and it is explainable. Point 1 is very close to the inlet boundary: therefore, the a velocity here is very random because of our stochastie velocity inlet condition. Point 2 locates in region where the BES flow is not fully developed yet and in this region the velocity
20

---

Figure 14: Velocity contour (v) at time step 40, 80, 120, 160 (left to right) from different LES simulations.
6
8
4
-1
Figure 15: Spatial location of the random chosen evaluation points. Point (-4.98, 0.016), Point 2: (-1.65. 0.55), Point 3: (1.68, -0.88), Point 4: (0.016, -0.32), Point 5: (3.35, 0.22), Point 6:
(1.68, 0.78), Point 7: (5.01, -0.65), Point (8.35, -0.11), Point 9: (6.68, 0.45)
10.8
10.8
10
10.6
10.6
10.4
10.4
10.2
10
10
10.2
100
200
100
200
0
100
200
14
15
12
10
10
8
0
100
200
0
100
200
0
100
200
20
10
15
10
10
10
100
200
0
100
200
100
200
t
Figure 16: Time signal from LES points (top to bottom, left to right).
URANS
flow model
of u evaluated
21

---

-1022
0.2
5
5
0.1
0
0
-0.1
0.2
100
200
100
200
200
2
10
0
0
2
100
200
100
200
100
200
10
5
10
0
0
0
-5
10
100
200
100
200
100
200
6
Figure 17: Time signal from LES
URANS
flow model
of u evaluated at
points (top to bottom, left to right).
magnitude is very small. These conditions make it hard capture accurate distribution of the velocity at these 2 points. Finally, we also compare the velocity distribution between our generated samples and the LES simulation results, shown in Figure 18 19 Except for points 1 and 2, the distribution of our generated samples aligns very well with the LES simulations.
22

---

Ours
Ours LES
Ours
180
122
12.
10.4
163
124
103
15.4
18.5
306
13.8
-30
-3
40
Ours
LES
LES
0.00
600
Ours
LES
-10
-35
03
0.00
308
133
175
0354
Ours LES
Ours
LES
LES
E15
L100
600
3224
6005
Figure 18: Comparison of the distribution velocity between the LES result and our model at evaluation points
Ours
Ours LES
Ours
613
000
3.3
8.0
077
560
634
Ours
Ours
332
614
LES
LES
012
074'
6.0
0..3
-1
0.0
0.1
Ours LES
Ours LES
Ours
0.2
LES
616
013
Figure 19: Comparison of the distribution velocity between the LES result and our model at evaluation points
23

---

A.13
Ablation study
Comprehensive ablation studies are included in this section demonstrate the effectiveness each parts. The velocity reconstruction error the Backward-facing step flow dataset with or without residual connection and centers shown in T The result indicates that each component can improve the graph reconstruction accuracy. We also test the influence of BDGMR-GMEL and the attention conditional flow model to the final performance of the deterministic task, as shown in T The result shows that both the encoder-decoder part and the flow model are necessary for the success of the whole framework.
Table 8: The reconstruction error of stochastic flow with the same training epoch number for Backward facing step flow dataset. The residual connection and centers can decrease the reconstruc- tion error significantly.
residual connection
centers
V
X
85
/
19.2
15.8
Table 9: Ablation study: the average relative rollout error with/without each proposed component for vascular flow. CNF: using conditional normalizing flows for sequence prediction. In variant 2, remove the CNF block and transformer only structure for predictions, similar to [11] Our
model with residual connection and centers have the best performance.
residual connection centers
CNF
p
GMR-GMUS
x
7.3
10
22
Variaint1
/
6.1
8.7
9.2
Variaint2
V
x
8
3
Ours
3.49
5.47
1.05
To efficiently incorporate the temporal information, we conceive a temporal conditional normalizing flows to account for the long-time temporal dependency. The original RealNVP is only probabilistic model, and is not straighforward to include temporal information. To achieve this, we propose an encoding deeddinn transformer structure to incorporate fixed physical parameters and all previous steps into condition vector c. In this ablation study, we remove transformer structures and replace condition vector e by the current step latent vector 2 when predicts next step =t. This variant can be treated as a probabilistic 1-step prediction method. The result in Table I0 shows that the proposed structure significantly improve the accuracy compared with such 1-step method, while still easy to calculate the probability of a coupling layer if we concatenate condition vector at every coupling layer, shown in Appendix A.I
Table 10: Ablation study: effect of attention-based temporal conditional model for Sonic dataset, NE: normalizing flows only condition on previous step. The proposed model outperforms step NE, indicating the design of attention based temporal model is necessary for accurate time series prediction.
p
T
-step NF
4.85
16.36
1.09
5.50
Ours
0.37
0.85
0.079
0.01
Our model can benefit from longer time dependencies, since we take all previous steps in the experiment setting. Also, we can use the moving window to reduce the cost further. We did one additional experiment of reducing the number of inputs during the inference time, as shown in Table 11 When reducing the window size from 400 to 150, the degradation in accuracy is not significant. So the model also applicable cases with thousands or even longer steps with the sliding window.
24

---

Table 11: Ablation study: effect of window length (WL) for Cylinder dataset, reducing the window length to 150 doesn't It increase eπror significantly.
WL
2
0
p
150
4.14
80.48
22.99
400
3.8
74
20
Though we only tested RealNVP in the experiment part, other normalizing flows are also feasible here. In Table T2] we test another normalizing flow model MAF (54), and find the result is comparable to ReaINVP. indicates that the proposed framework is flexible.
Table 12: Ablation Study:
the average rollout using different normalizing flow models. MAF: Masked Autoregressive Flow. The performance comparable with RealNVP.
Dataset-rollout step
Cylinder flow-400
Sonic flow-40
Vascular flow-250
Variable Ours
V
p
p T
p 1.05 0.02 3.82 5.84
RealNVP 3.8 74
20
0.37 0.85 0.079 0.01 3.49 5.47
Ours
MAF 3.8 72
19.13
0.33 0.69 0.055
1.16
As a probabilistic model, the proposed BOGMR-GMUS conditional flow model is trying to fit such stochastic process and get various samples with the same physical statisties. To demonstrate these, we test MGN and GMR-GMUS on the stochastic dataset. In Table 13, we can see that the performance of the proposed model is better than other learning-based models all metrics. In Figure 20, we visualize the mcan and variance of velocity from each model, and find the MGN can capture the real distribution all. The result of GMR-GMUS model, though looks reasonable in statistics, can only produce the same output given the same input condition, which doesn' reflect the stochasticity of the last dataset. Figure 21, we get two samples from EMR--MUS, and find the two samples are exactly the same.
Table 13: Criteria of generation quality for the stochastic flow for URANS and different learning- based models, the proposed model has the best performance.
CRPS (C..4)
CRPS (C...)
FVD (dg.J)
FVD (d,,.)
MFE (Cu-l)
MFE (e...)
TE (TKE,1)
URANS
3.24
2.0
228112
137860
0.31
0.94
0,192
MeshCiraphNer
5.06
2.47
1159950
179206
0.82
1.60
0.87
GMR-GMUS
4976
3740
0.167
Ours
1.28
1262
397
0.0176
Figure 20: Evaluation lines of mean (left) and variance (right) streamwise velocity w: r-evaluation URANS MeshGraphNet GMR-GMUS
lines (- --- Quantities from LES
and the proposed model
Figure 21: Velocity magnitude time steps 0, 50, 100 (left right) from different samples using GMR-GMUS + transformer for the stochastic dataset. 1st row for sample 1. 2nd row for sample The deterministic model can only produce the same output while the proposed model can generate different realizations.
25


Reviews:
[{"review_id": "1RxV1OjpN5", "reviewer": "Reviewer_fgSk", "summary": "The authors present a new approach for modeling fluid dynamics. The approach involves using GNNs to derive global latent space representations and construct a transformer-based conditional generative models for the dynamics. The resulting model is able to generate stochastic predictions from given initial conditions and system parameters at inference time. The proposed framework is compared against some competitive baselines on a few deterministic and stochastic benchmarks.", "questions": "Questions\n* How much of the improvements can be attributed to using *pivotal positions* and how much to using a different architecture for the latent dynamical model? It would be nice to have some ablation studies\n* What is the advantage of using the RealNVP vs other generative models (e.g. there is a number of normalizing flows [here](https://github.com/VincentStimper/normalizing-flows))?\n\nComments\n* Ln 168 - I believe it’s inaccurate to say the geometry of the graph is not included in the latent vector - it’s really embedded implicitly in the representation.\n* Ln 243 - “.. potentially improving the interpretability of deep learning systems” - UQ and interpretability are different concepts.\n\nTypos\n* Equation (1), $j$ is not defined in text. Based on what follows, it can be a stochastic operator as well?\n* Ln 95 - functions that *are* parameterized by neural networks\n* Equation (5) - consider not using $p$ for position to avoid confusion with probability densities?\n* Typeset errors on ln 219 and 289?\n* Figure 3 - axis labels are missing\n* Table 3 - what is referred to by “Flow” is your model? I don’t believe this is referenced elsewhere in the text.\n* In appendix, Ln 589 - the results are in Figure 10 not 11", "limitations": "The authors do present ablation studies showing that a certain number of *pivotal positions* are required. Other potential areas to discuss are scalability, applicability to more complex systems, sampling requirements and parameter extrapolations.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "*  This work seeks to perform probabilistic modeling of high-dimensional dynamical systems originating from parametric PDEs. This is a very important problem as traditional methods are typically quite expensive and lacking ways to quantify uncertainties. The method proposed here is a nice attempt that addresses both challenges.\n* The method presented is very flexible and easily adapts to various problems and configurations. Because of the complexity of the cases, it has the potential to serve as a fast surrogate for modeling turbulence useful for inner-loop applications.\n* The introduced regeneration learning framework is quite novel as it effectively integrates different blocks (graph representation learning, attention-based model, normalizing flows). Such formulation offers an efficient and attractive alternative to directing learning long-time spatiotemporal fields on the original mesh. Moreover, the framework can handle deterministic/stochastic systems in a unified way.\n* The presentation is very clear, logical and easy to understand in general. The plots are of high quality and informative.\n* Claims are backed by strong and convincing empirical evidence. The examples in the numerical experiments section demonstrate sufficient complexity and variety to help justify the value of the proposed method.", "weaknesses": "* A number of minor typos in texts and figures (see section below).\n* More discussions on the limitations would strengthen the paper further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer's supportive evaluation of our work. We are trying our best to address your concerns with the following answers.\n\n> **Q1:**: A number of minor typos in texts and figures (see section below).\n\n**Response:** Thanks for your careful reading. We already fixed them in the revised version.\n\n> **Q2:**:More discussions on the limitations would strengthen the paper further.\n\n**Response:** Thanks for this concern. We agree it is necessary to discuss the limitations of the proposed model. Please refer to global rebuttal **Q3** for details.\n\n> **Q3:** How much of the improvements can be attributed to using pivotal positions and how much to using a different architecture for the latent dynamical model? It would be nice to have some ablation studies\n\n**Response:** Thanks for asking this question. We already made more comprehensive ablation studies and the reviewer can refer to our global rebuttal and new supplementary experiment table 8 and table 9. Ablation studies show that pivotal positions are helpful to improve reconstruction accuracy. And the proposed conditional flow model performs better against a single transformer model with the same encoder-decoder structure. We believe these new ablation studies are helpful in identifying the use of each component in our new design.\n\n\n\n\n> **Q4:** What is the advantage of using the RealNVP vs other generative models (e.g. there is a number of normalizing flows here)?\n\n**Response:** Thanks for the reviewers' insightful question on different normalizing flows models and providing the links. The key design for the time series prediction model is to adopt a transformer-based encoding-decoding structure to capture temporal conditions as well as global physical parameters. The reason we adopt the flow model is that there are fewer parameters to compute and it is easier to get samples at each step compared with other probabilistic models such as Gaussian distribution. We think other normalizing flows are also feasible here. In supplementary experiment table 12, we test another normalizing flow model MAF, and find the result is comparable to RealNVP. It indicates that the proposed framework is flexible.\n\n\n> **Q5:** Ln 168 - I believe it’s inaccurate to say the geometry of the graph is not included in the latent vector - it’s really embedded implicitly in the representation.\nLn 243 - “.. potentially improving the interpretability of deep learning systems” - UQ and interpretability are different concepts.\n\n**Response:** Thanks for reading our script carefully and giving constructive feedback. We agree that the geometry is implicitly embedded into the representation. Moreover, we agree that UQ and interpretability are different concepts. The current model can capture the UQ for deterministic systems and generate physical realizations for stochastic systems. We already fixed these in the revisions. \n\nWe've made updates based on the feedback provided, and we believe that these changes substantially improve the manuscript. We're grateful for the valuable comments and thank you for your time and expertise. "}, {"review_id": "e2mNhRqm5E", "reviewer": "Reviewer_YdeK", "summary": "The authors present a unified framework for solving deterministic and stochastic physical dynamical systems on high-dimensional mesh space. Instead of updating values at each discretized mesh, the paper introduces an approach that evolves states in a low dimensional latent space through encoding states and physical parameters. This encoding is done by a message passing graph neural network and multi-head attention (MHA) model, which encodes physics parameters into a conditional vector. The encoded state is then evolved using a conditional normalizing flow, dependent on the encoded conditional vector. The effectiveness of this method is shown through experiments, in which it outperforms other mesh-based ML models in terms of accumulation error in deterministic problems, with applications in stochastic problems also being demonstrated.", "questions": "Please have a look at the weakness mentioned in the strength and weakness section and address these. Overall the idea seems interesting, the authors need to substantiate their claims in light of existing literature and possibly more experiments.\n", "limitations": "Yes, the limitations are discussed in the appendix. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method is simple and applicable to wide range of PDE-simulation problems with discretized domain. Wide variety of experiments are conducted, and both quantitative and qualitative evaluations are provided.", "weaknesses": "About the novelty, it is still unclear that what components of the proposed model enables solving both deterministic and stochastic  systems efficiently. Are there any limitations that prevent existing models from being applied to both systems?\n\nIt is concerning that the proposed method take all the previous (latent) vectors as input to predict the next state, as opposed to fixed small number of input vectors of MeshGraphNets reported in Section 5, that may result in unfair advantages over other methods. Also, although in Table 1 the reconstruction error of PbGMR-GMUS are compared against GMR-GMUS, it is still unclear how the proposed model would compare against an ablation model whose encoder is GMR-GMUS.\n\nThe proposed model is compared against baselines which evolve states in original space. There is a possibly missing reference [1], and the motivation of using latent evolution is similar to the presented one. How does this work compare against it? Is the proposed method also applicable to grid space?\n\n[1] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution (NeurIPS 2022)\n\n\n**Minor comments**\n\nIn Appendix A.10, Figure 11 is cited, but seemingly wrong. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer is positive about the application and performance to wide range of PDE-simulation problems of the proposed framework. This is an excellent support to our work. And reviewer encourages us to further experiment to highlight our novelty over previous models. Here are our responses to your questions.\n\n> **Q1:** It is still unclear that what components of the proposed model enable solving both deterministic and stochastic systems efficiently. Are there any limitations that prevent existing models from being applied to both systems?\n\n**Response:** This is a great and important question. Please refer to our global rebuttal **Q2**. The previous models' most challenging part is predicting stochastic systems in mesh space. Our new experiment results indicate previous SOTA models can not capture close distribution or get different samples with the same physical statistics. (Table 13, Figure 20, and Figure 21 in new supplementary experiments)\n\nAlso, existing CNN-based models, such as video generation models, can not handle irregular mesh space. To overcome the above limitations, we first propose PbGMR-GMUS to encode the graph space into latent space with highly accurate reconstruction. A novel attention-based temporal conditioned flow model is then developed as the probabilistic model. At each time step, only a low-dimensional vector needs to be predicted. And the model can capture spatiotemporal dependencies and stochastics efficiently. \n\n We did comprehensive ablation studies, as shown in Table 8 and Table 9 in the one-page rebuttal. The results demonstrate that the proposed model can increase the accuracy of deterministic training. Moreover, the proposed flows model enables learning stochastic processes and generates different realizations for the stochastic systems with low cost. Furthermore, we test MGN and GMR-GMUS on the stochastic dataset. In Fig 20 and Table 13, the performance of the proposed model is better than other learning-based models, such that the MGN can not capture the real distribution at all. The result of GMR-GMUS with a sequential model, though looks reasonable in statistics, can only reproduce the same output given the same input, which doesn't reflect the stochasticity.\n\n> **Q2:** It is concerning that the proposed method take all the previous (latent) vectors as input to predict the next state, as opposed to fixed small number of input vectors of MeshGraphNets reported in Section 5 ...\n\n**Response:** MeshGraphNet's in/output are defined on the high-dimensional mesh space instead of low-dimensional vector space. In contrast, the proposed model makes such predictions on latent space with vectors as inputs. Though our model takes more latent vectors as input, the computation is even less than MeshGraphNets during the roll-out stage if the steps are not very large. Since the model can benefit from longer time dependencies, we take all previous steps in the experiment setting. Also, we can use the moving window to reduce the cost further. We did one additional experiment of reducing the number of inputs during the inference time, as shown in supplementary experiments Table 11. The result of reducing the window size from 400 to 150, the degradation in accuracy is not significant. So the model is also applicable to cases with thousands or even longer steps with the sliding window.\n\n\n> **Q3:** Although in Table 1 the reconstruction error of PbGMR-GMUS are compared against GMR-GMUS, it is still unclear how the proposed model would compare against an ablation model whose encoder is GMR-GMUS.\n\n**Response:** Since PbGMR-GMUS improves GMR-GMUS, we prove that PbGMR-GMUS has a better reconstruction error than GMR-GMUS. The following time sequence model can benefit from it. In Table 1, our model performs better on all three datasets compared with [11], whose encoder is GMR-GMUS. To help evaluate the performance against an ablation model with a GMR-GMUS encoder, we listed ablation study results in Table 8 (in new supplementary experiments) and Table 9. In Table 8, residual connection and pivotal positions are necessary for a highly accurate reconstruction of graphs. In Table 9, even with the same attention-based conditional normalizing flow model, the model (Variant 1) with GMR-GMUS as encoder has a worse performance compared with the proposed model.\n\n> **Q4:** The proposed model is compared against baselines which evolve states in original space. There is a possibly missing reference [1]. How does this work compare against it? Is the proposed method also applicable to grid space?\n\n**Response:** Thanks for pointing out this related reference. We added it to the related work. The two works have a similar motivation to make predictions in the latent space for effective computation. However, this reference uses convolutional neural networks to encode the original space, which poses difficulties in applications to irregular mesh spaces. Using CNN to encode images is a very mature technique, but it is difficult to encode graphs. That is also the motivation of this work to propose a general method to encode graphs better. Also, [1] only concentrates on deterministic processes and did not demonstrate on stochastic physics. Still, this reference predicts the PDE process in the latent space, and it will be helpful to readers to know more related works in this area. It is not straightforward to compare the proposed method since we concentrate on systems defined on the irregular mesh. The proposed method is also applicable to the grid space. But the existing CNN already works very well on such tasks.\n\n> **Q5:** In Appendix A.10, Figure 11 is cited, but seemingly wrong.\n\n**Response:** Thanks for pointing it out; we already corrected it in the appendix.  It should be Figure 10. \n\nFollowing the reviewer's comment improves the quality of the submission significantly. We genuinely appreciate the reviewer's effort and help."}, {"review_id": "aJlulfCiNP", "reviewer": "Reviewer_9WF1", "summary": "This paper introduces a novel mesh-based machine-learning approach for modeling stochastic fluid dynamics. Similar to [11], the state transitions are modeled in a compact latent space (referred to as the mesh-reduced space) rather than the high-dimensional mesh space. Compared with [11], this paper makes several technical contributions below. \n1. It proposes to use virtual pivotal positions instead of selecting pivotal nodes from the mesh topology.\n2. It employs a deep generative model based on a normalizing flow approach, RealNVP, to model the stochastic dynamics. The model also incorporates multi-head attention over sequential latent states. \n\nThe experimental results demonstrate that the proposed model significantly outperforms [11] in the deterministic setup, while in the stochastic setup, the model is compared against an off-the-shelf CFD method named URANS, rather than learning-based models.", "questions": "1. In Line 118, the claim that the proposed model enables the enlargement of the detection distance compared to GMR-GMUS lacks detailed explanations. Further elaboration on this point would be beneficial.\n2. In Line 87, it is mentioned that the stochastic dynamic system includes a random parameter $\\mu$ that varies over time, such as perturbations in the boundary conditions of turbulent flow. However, in Line 161, $\\mu$ is referred to as the global physical system parameter and seems to be time-invariant. It would be helpful for the authors to clarify whether $\\mu$ is time-varying across the input and output sequences.\n3. Regarding $\\mu$, as well as the positions $p(i)$ of the mesh cells, it is not clear from the provided context whether they are given in the future prediction time horizon. \n4. I am not very familiar with RealNVP, but based on the information provided, should it be written as $P(z_{1:T} \\mid \\mu, z_0, x; \\theta)$ in Equation (10) instead of $P(z_{1:T} \\mid \\mu, z_0; \\theta)$?\n\n\n\n", "limitations": "No discussions on limitations or broader societal impacts are presented in the current text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: To the best of my knowledge, this paper introduces the first learning-based probabilistic model that simulates fluid systems upon the mesh topology. The key innovation lies in integrating a flow method for stochastic temporal modeling into the GMR-GMUS [11] framework, which presents a reasonable and creative combination of existing approaches.\n\nQuality and clarity: Overall, the paper is clear and easy to follow.\n\nSignificance: The proposed model is shown to be effective in deterministic setups, which consistently outperforms MeshGraphNet and GMR-GMUS [11] by substantial margins. ", "weaknesses": "Main concerns:\n\n1. The proposed model appears to be a straightforward extension of GMR-GMUS [11] combined with RealNVP [25]. \n2. Ablation studies for using virtual pivotal positions, residual connections, and the conditional normalizing flow are lacking, which makes it difficult to determine the specific contributions of these components to the performance improvement shown in Table 1. It is suggested that the authors address this issue and provide a more comprehensive analysis.\n3. Although the paper addresses the new problem of probabilistic modeling of fluids, the lack of comparison with existing learning-based models such as GMR-GMUS and MeshGraphNet in Section 5.2 raises doubts about the true challenges of the prediction task for stochastic fluid systems and the superiority of the proposed approach.\n\nMinor concerns:\n\n1. Accurate citations with corresponding publication sources should be provided. For example, GMR-GMUS [11] was published at ICLR 2022. his will enhance the credibility and traceability of the referenced papers.\n2. It would be valuable to include a brief discussion regarding the limitations of the research to provide readers with a broader perspective on the potential areas for improvement or expansion.\n3. In Line 62, the authors introduce GMR-GMUS without relevant sources, which made me confused until I came to Section 3.1.\n4. In Lines 113-121, it would be good if the authors could describe the learning method by referring to specific components or steps in Figure 1, so the readers can better grasp the concepts and processes discussed in the text. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we express our heartfelt gratitude for your warm acknowledgment of the novelty of the learning-based probabilistic model, meticulously simulating fluid systems across intricate mesh topologies. Here are our responses to your questions.\n\n> **Q1:** The proposed model appears to be a straightforward extension of GMR-GMUS [11] combined with RealNVP [25].\n\n**Response:**  Our endeavors transcend the boundaries of the deterministic system set by the prior SOTA works, significantly improved the SOTA reconstruction technique, and more importantly, without loss of the generalizability, unified the deterministic and stochastic system. \n\nBy replacing pivotal nodes with pivotal positions, we can enlarge the detection distance for each position embedding, thus aggregating information more efficiently. Also, a residual connection is added to the message-passing layer, which can make the training more stable. Such advantages make PbGMR-GMUS's reconstruction accuracy higher against GMR-GMUS. And low reconstruction error is significant for the success of latent generative models. To test the improvement brought by PbGMR-GMUS, we conducted comprehensive ablation studies, and the results are shown in Tables 8 and 9 (supplementary experiments). The result shows that all the newly added components can improve the performance. Detailed analysis is shown in the global rebuttal.\n\nFurthermore, our attention-based temporal conditioned generative model is not a simple extension of RealNVP. The original RealNVP is only a probabilistic model, and it is not straighforward to include temporal information. To achieve this, we propose an encoding-decoding transformer structure to incorporate fixed physical parameters and all previous steps into condition vector $c$. Then we prove that it is still easy to calculate the probability of a coupling layer if we concatenate condition vector $c$  at every coupling layer. Actually, the key design is attention-based temporal conditional model, not the selection of RealNVP. The proposed framework can be combined with other probabilistic model. To prove this, we replace RealNVP with MAF, another flow model, Table 12 indicates the results are comparable to RealNVP. So the whole framework is a flexible and useful paradigm. We also test within a single RealNVP, removing the temporal conditional model. The performance in Table 10 drops very obviously.\n\n\n> **Q2:** Ablation studies for using virtual pivotal positions, residual connections, and the conditional normalizing flow are lacking ...\n\n**Response:** We obey your command. We have added the ablation studies on virtual pivotal positions, residual connections, and conditional normalizing flows and the results are in Table 8-10 (supplementary experiment results). In general, all of the proposed components help improve the performance. Detailed analyses are in global rebuttal **Q1**.\n\n> **Q3:** The lack of comparison with existing learning-based models such as GMR-GMUS and MeshGraphNet for stochastic fluid systems ...\n\n**Response:** The reviewer's concern is absolutely reasonable. We agree that more learning-based baselines are needed for section 5.2 and train the GMR-GMUS[11] and MeshGraphNet[3] to address the concern. The results are attached in Table 13 and Figure 20. Our model has the best performance, and the deterministic model cannot capture the critical stochastic nature of the underlying physics. Detailed analyses are shown in the global rebuttal **Q2** due to the word limitation. \n\n> **Q4:** ... clarify whether $\\mu$ is time-varying across the input and output sequences.\n\n**Response:** Thanks for your help with clarification. The correct form for Equation 2 is $\\frac{\\partial{u}}{\\partial{t}} = j(\\mathbf{u}, \\mathbf{\\mu}, \\mathbf{\\iota})$ where the $\\iota$ is the random parameter over time for stochastic systems (e.g., boundary conditions). So $\\mu$ in lines 71, 86, and 87 should be replaced with $\\iota$.\nIn Line 161, the $\\mu$ is the global physical system parameter and is time-invariant (e.g., Re number). We revised the notations to make it more clear. We appreciate this excellent comment.\n\n> **Q5:** Regarding $\\mu$, as well as the positions $p(i)$ of the mesh cells, it is not clear from the provided context whether they are given in the future prediction time horizon.\n\n**Response:** Thank you for asking the clarification. During time series prediction in the latent space, $\\mu$ (time-invariant global physical parameter) is given through the transformer encoder in equation (12). While the positions of the mesh cells are not given during the prediction in time. We only need to guarantee that the node order is fixed when concatenating them into latent vector $z$.\n\n>  **Q6:** Should it be written as ... in Equation (10) instead of ...?\n\n**Response:** Thanks for raising this discussion. RealNVP is a probabilistic model. We define such conditional probability only on latent space. It should be noticed that here 'latent space' is not the same as mentioned in section 2.2, which describes internal operation of RealNVP. In Equation 10, we only care about how to define such temporal conditional probability, and there is no relation with specific probabilistic model. Though we select normalizing flow model in this work, it can be defined by other conditional distributions.\n\n> **Q7:** It would be valuable to include a brief discussion regarding the limitations of the research.\n\n**Response:** Thanks for this concern. We agree it is necessary to discuss the limitation of the proposed model. Please refer to global rebuttal **Q3** for details.\n\n> **Q8:** Accurate source of GMR-GMUS [11], and proper reference for GMR-GMUS in Line 62.\n\n**Response:** Thanks for pointing it out. We revised several references with accurate sources and added a reference in Line 62.\n\nWe believe your suggestions are helpful. Thanks a lot for your careful reading and efforts on our submission. We revised the minors in our manuscript."}, {"review_id": "lcRzMZ7YYi", "reviewer": "Reviewer_8v2N", "summary": "The paper proposes PbGMR-GMUS to encode the graphs of physics systems into low-dimensional features, which are further reconstructed into the desire graph representations. An attention-based model is integrated into flow-based generative models to predict the future dynamics in the latent space. The proposed method is able to handle both deterministic and stochastic fluid dynamics thanks to the probabilistic model and achieve superior performance over previous methods.", "questions": "* How to prove the statement about the first benefits at L163-164? Is there any further explanation?\n* Is the main difference to between simulating deterministic and stochastic fluid dynamics that the $\\mu$ in Equation 14 would change with time for stochastic dynamics?\n* Any video demos for the predictions of both deterministic and stochastic fluid dynamics? The temporal consistency can be better illustrated by videos.\n\n", "limitations": "The proposed PbGMR-GMUS is technically limited, which is a variant of GMR-GMUS with simple modifications. Video demos are missing to verify the temporal consistency of the predictions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* An effective model to learn the graph representations in low-dimensional space.\n* The simulation of dynamics is achieved by a probabilistic method using generative models, which rollout vivid predictions for long-term predictions.\n* The method is able to solve both deterministic and stochastic fluid dynamics.", "weaknesses": "* Miss citation at L62 for GMR-GMUS.\n* On top right of Figure 1, is it \"Coupling layer\" instead of \"Couling layer\"?\n* Unclear caption for Figure 5. Which one is the variance?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your important support. And we seriously address your concerns accordingly; they are extremely helpful to the submission.\n\n> **Q1:** Miss citation at L62 for GMR-GMUS.\n\n**Response:** Thanks for pointing it out. We already added reference in the revised version.\n\n> **Q2:** On top right of Figure 1, is it \"Coupling layer\" instead of \"Couling layer\"?\n\n**Response:** Yes, you are right. It should be \"coupling  layer\" and we already fixed it. \n\n> **Q3:** Unclear caption for Figure 5. Which one is the variance?\n\n**Response:** We agree that the caption should be more clear. The upper is the mean of the velocity, and the bottom is the variance of the velocity. We added more explanations in the revised version.\n\n\n> **Q4:** How to prove the statement about the first benefits at L163-164? Is there any further explanation? \n\n**Response:** As a stochastic process, probabilistic models on the original graph space $p(Y_t|Y_{t+1})$ face severe one-to-many mapping problems (ill-posed or ill-condition problem)[24]. That is, given the previous step state $Y_t$, it can correspond to many different next step $Y_{t+1}$, which increases the learning difficulty. Incorrect modeling on the\nill-posed problem could result in overfitting on the training set and poor generalization on the test set. Thus, converting $Y$ into more compact $z$ will make the mapping between $z_t$ and $z_{t+1}$ less ill-posed and ease the model learning. We will add more explanation in the revised version.\n\n> **Q5:** Is the main difference to between simulating deterministic and stochastic fluid dynamics that the $\\mu$ in Equation 14 would change with time for stochastic dynamics? \n\n**Response:** Thanks for asking the clarification. Here $\\mu$ is a time-invariant global physical parameter, such as $Re$. And the numerical simulation has another time-varying parameter such as the stochastic boundary condition for the stochastic fluid dynamics. As for our model, predictions of deterministic and stochastic physics are unified; the model can automatically capture the stochasticity conditioning on $\\mu$ during the training.\n\n>  **Q6:** Any video demos for the predictions of both deterministic and stochastic fluid dynamics? The temporal consistency can be better illustrated by videos. \n\n**Response:** Thanks for this good suggestion. We already uploaded some video demos. This link is anonymized and contains not author information. Based on the policy, we first sent it to the AC in the official comment. \n\nWe appreciate your careful reading and suggestions for this paper. We believe these revisions and new supplementary experiments help improve the manuscript a lot. And your support means a lot to us."}]

Author rebuttal:
Dear reviewers,

We would like to express our gratitude for your constructive feedback on our manuscript. The insights provided have been invaluable in refining our work. We have uploaded one-page supplementary experimental results in response to your comments.

Here we want to highlight several contributions mentioned in the reviews:

- The proposed method is simple and applicable to wide range of PDE-simulation problems with discretized domain. Wide variety of experiments are conducted, and both quantitative and qualitative evaluations are provided.

- This paper introduces the first learning-based probabilistic model that simulates fluid systems upon the mesh topology. 

- This work seeks to perform probabilistic modeling of high-dimensional dynamical systems originating from parametric PDEs. This is a very important problem as traditional methods are typically quite expensive and lacking ways to quantify uncertainties. The method proposed here is a nice attempt that addresses both challenges.

- The introduced regeneration learning framework is quite novel as it effectively integrates different blocks (graph representation learning, attention-based model, normalizing flows). Such formulation offers an efficient and attractive alternative to directing learning long-time spatiotemporal fields on the original mesh. Moreover, the framework can handle deterministic/stochastic systems in a unified way.

- The simulation of dynamics is achieved by a probabilistic method using generative models, which rollout vivid predictions for long-term predictions. The proposed method is able to handle both deterministic and stochastic fluid dynamics thanks to the probabilistic model and achieves superior performance over previous methods.

Also, based on constructive suggestions, we conducted several ablation studies to identify the use of each component and compare the results on stochastic physics with the learning-based model MeshGraphNet[3] and GMR-GMUS + transformer[11]. Here, we want to solve the issues the reviewers are concerned mostly.

> **Q1** There should be ablation studies to identify the use of each component. (Reviewer YdeK, 9WF1, and 8v2N)

**Response:**  We appreciate the reviewers' suggestions and believe such ablation studies are helpful. We first check the velocity reconstruction error on the Backward-facing step flow dataset with or without residual connection and pivotal positions. The result of Table 9 indicates each component can improve the graph reconstruction accuracy. We also test the influence of PbGMR-GMUS and the attention conditional flow model to the final performance of the deterministic task. Table 10 in supplemental experiments indicates both the encoder-decoder part and the flow model is necessary for the success of the whole framework.

> **Q2** The model should be compared against other learning-based models, such as MeshGraphNet and GMR-GMUS, on the stochastic dataset to prove the superiority of the proposed approach. What are the true challenges of the prediction task for stochastic fluid systems?(Reviewer 9WF1, YdeK)

**Response:** It is very hard for the previous models to predict stochastic mesh-based physical systems. Defining a probabilistic model on such sequential data is not easy. If we use a Gaussian distribution for the targeted system, which contains $N$ meshes, at each time step, the model should predict $N$ means and $N$x$N$ covariance. Also, the model should take temporal- and spatial- dependencies into consideration. Both make the computation too heavy to be applicable. Also, sampling from such distribution is computation-consuming. Most of the previous models, such as MeshGraphNet[3], and GMR-GMUS decoding-only[11], are not probabilistic models, so they can not be applied to any stochastic system. Given the same initial state and physical parameters, they always produce the same prediction. In contrast, as a probabilistic model, the proposed PbGMR-GMUS + conditional flow model is trying to fit such a stochastic process and get various samples with the same physical statistics. To demonstrate these, we test MGN and GMR-GMUS on the stochastic dataset. In Table 13, we can see that the performance of the proposed model is better than other learning-based models on several metrics. In Figure 20, we visualize the mean and variance of velocity from each model, and find the MGN can not capture the real distribution at all. The result of GMR-GMUS model, though looks reasonable in statistics, can only produce the same output given the same input condition, which doesn't reflect the stochasticity of the last dataset.  In Figure 21, we get two samples from GMR-GMUS[11], and find the two samples are exactly the same. 

> **Q3** More discussions on the limitations would strengthen the paper further. (Reviewer 9WF1, fgSk)

**Response:** We discussed the limitation of the proposed model in Appendix, section A.12. Reviewer YdeK
also mentioned that the limitations are discussed in the appendix. The model can not capture an accurate distribution for stochastic systems close to boundary areas and regions where flow is not fully developed. (Details and figures are also in this part) We agree that readers may ignore such discussions, and we will add some analysis to the main paper and give a reference to the appendix.

In conclusion, we have incorporated the insightful feedback provided by the reviewers, enhancing the overall quality of our paper. We sincerely value the time and effort invested in reviewing our work and believe that the revised manuscript and supplementary experiments address the concerns raised.





Meta-review: Reviewers agreed about paper's excellent presentation of integrating many different ideas to achieve compelling results on the hard problem of learning stochastic dynamics from LES data. I especially liked the inclusion of alternative evaluation metrics other than normalized RMSE in this domain and will go a long way in helping improve comparisons between different methods in this domain.
Decision: Accept (spotlight)