Title: Unifying Predictions of Deterministic and Stochastic Physics in Mesh-reduced Space with Sequential Flow Generative Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: AI4Science

Fluid Dynamics

Generative Models

Graph Neural Network
Abstract: Accurate prediction of dynamical systems in unstructured meshes has recently shown successes in scientific simulations. Many dynamical systems have a nonnegligible level of stochasticity introduced by various factors (e.g. chaoticity), so there is a need for a unified framework that captures both deterministic and stochastic components in the rollouts of these systems. Inspired by regeneration learning, we propose a new model that combines generative and sequential networks to model dynamical systems. Specifically, we use an autoencoder to learn compact representations of full-space physical variables in a low-dimensional space. We then integrate a transformer with a conditional normalizing flow model to model the temporal sequence of latent representations. We evaluate the new model in both deterministic and stochastic systems. The model outperforms several competitive baseline models and makes more accurate predictions of deterministic systems. Its own prediction error is also reflected in its uncertainty estimations. When predicting stochastic systems, the proposed model generates high-quality rollout samples. The mean and variance of these samples well match the statistics of samples computed from expensive numerical simulations.

Unifying Predictions of Deterministic and Stochastic Physics in Mesh-reduced Space with Sequential Flow Generative Model

Abstract

Accurate prediction of dynamical systems in unstructured meshes has recently shown successes in scientific simulations. Many dynamical systems have a nonnegligible level of stochasticity introduced by various factors (e.g. chaoticity), so there is a need for a unified framework that captures both deterministic and stochastic components in the rollouts of these systems. Inspired by regeneration learning, we propose a new model that combines generative and sequential networks to model dynamical systems. Specifically, we use an autoencoder to learn compact representations of full-space physical variables in a low-dimensional space. We then integrate a transformer with a conditional normalizing flow model to model the temporal sequence of latent representations. We evaluate the new model in both deterministic and stochastic systems. The model outperforms several competitive baseline models and makes more accurate predictions of deterministic systems. Its own prediction error is also reflected in its uncertainty estimations. When predicting stochastic systems, the proposed model generates high-quality rollout samples. The mean and variance of these samples well match the statistics of samples computed from expensive numerical simulations.

1 Introduction

Accurate prediction of long-term dynamics of physical systems is of great interest in many science and engineering fields. The classical simulation of a system heavily relies on a spatial/temporal discretization of the space and the numerical solution to a finite-dimensional algebraic system derived from the governing equation [1]. However, due to the multi-scale, stochastic nature of the complex physics and the complexity of the geometry, the simulation of large-scale real-time applications is extremely expensive in computation [2]. In recent years, deep learning models have been applied to predict rollouts of large and complex physical systems thanks to their flexibility and scalability [3]. Moreover, they can handle uncertainties and non-linearity in physical problems [4-8] more effectively than traditional methods.

Previous work has studied two types of complex dynamical systems: deterministic ones and stochastic ones. A deterministic system is often considered under a perfectly controlled experiment with exactly

known PDE terms and initial conditions (IC) or boundary conditions (BC) [9, 10, 3, 11-13]. This type of system can be modeled by autoregressive predicting models. A stochastic system such as quantum mechanics [14] and statistical physics [15] has a stochastic rollouts. When there are stochastic forcing terms or IC/BC terms in the governing equations, researchers have developed various models for predicting the stochastic state variables, such as turbulence velocity and stock prices [16-20]. Given that there is not a clear boundary between the two types of systems, it is highly desirable that a unified model can model either a deterministic or a stochastic system automatically. However, such models so far are limited to classical numerical models such as OpenFOAM [21]. There is an urgent need to develop a unified deep-learning model that can model both types of systems.

In this work, we propose a unified framework based on deep generative models to predict deterministic and stochastic systems. The new model is based on graph-structured state representations [11], which can handle irregular spatial areas commonly seen in dynamical systems. It uses an autoencoder to encode a state representation to a low-dimensional latent vector. Accurate encoding and decoding is critical for recovering the state of the dynamical system. This work makes several innovations to enhance the autoencoder's ability to preserve the information in system states. We provide a new approach to encoding the graph-structured representation with a vector with a fixed length. We also get some inspiration from regeneration learning [22] and train our autoencoder with self-supervised learning.

To describe stochasticity in the system, we use a sequential probabilistic model for the latent representations. We integrate a transformer and a normalizing flow to construct a step-wise predictive neural network in the latent space. When there is stochasticity, the model will learn the conditional distribution of the next latent state; when there is no stochasticity, it can place probabilities to correct deterministic predictions and still can minimize the predictive error. By including the ability to simulate both deterministic and stochastic systems in a uniform framework, it reduces the effort of developing separate models for different problems.

We evaluate our proposed framework in an extensive empirical study. The results indicate the proposed model outperforms the SOTA baselines on deterministic datasets regarding accuracy. More importantly, for the first time, we introduce several alternative evaluation metrics other than normalized RMSE for stocahstic fluid dynamics, which help improve comparisons between different methods in this domain. The proposed framework can produce high-quality samples for stochastic systems in the mesh space.

2 Background

2.1 Problem definition

Let's consider a general partial differential equation (PDE) defined on the d -dimensional space and one-dimensional time domain,

where Ω ⊂ R d is the spatial domain, T end is the endpoint of time, ι : ∂ Ω × [0 , T ] is the random parameter over time for stochastic systems (e.g., boundary conditions), and u : Ω × [0 , T ] is the primary solution variable (e.g., velocity and pressure of fluid flow). Here µ : D is the global physical system parameter and is time-invariant (e.g., Re number) for a given rollout. j : Ω × [0 , T ] ×D → Ω × [0 , T ] is an aggregation of the spatial terms of conservation laws (e.g., source and flux). To numerically solve the conservation law in Equation 1, let C h be a mesh of Ω , that is, C h = { C i ∈ Ω : i = 1 , . . . , N } is a collection of non-overlapping cells that cover Ω . We further use u i,t to denote the evaluation of u at the cell center of C i at time step t .

Using the mesh described above, we apply a finite volume discretization to yield the parametrized, nonlinear dynamical system. Here, the dynamical system can be computationally intensive because a fine-level mesh will lead to discretization with a large degree of freedom. For stability issues, the numerical time step also needs to be very small. Therefore, numerical simulations with traditional methods require extremely expensive computation and storage.

In this paper, we are interested in two different scenarios: deterministic dynamics with invariant parameter ι over time (e.g., fixed boundary condition ), and stochastic dynamics with random

parameter ι over time (e.g., perturbations in boundary conditions of turbulent flow). Both cases will also include the global physical parameters µ . The scope of our paper is to build a unified data-driven surrogate model for generating/predicting parametric PDE solutions.

2.2 Normalizing flow models

A normalizing flow model constructs a flexible probabilistic distribution by applying a learnable bijective mapping to a simple random variable (e.g. Gaussian distributed). Suppose the mapping is z = f ( x ) with x ∈ R d being the simple input variable, then we have the probability p ( z ) of z as follows:

Here ∂ ( f ( x )) ∂ ( x ) is the Jacobian of f at x . The function f is usually a neural network with a layered structure, with each layer being a bijective mapping. Then the determinant of ∂ ( f ( x )) ∂ ( x ) is the product of determinants of these layers' Jacobian matrices. With a special design of layer structures, their determinants can be efficiently computed. For example, a RealNVP model [23] is constructed by stacking several coupling layers, each of which has a lower triangular Jacobian matrix. With h ∈ R d as the input, a coupling layer f ℓ runs the following calculation:

Here concat concatenates its arguments as one vector, and d ′ is usually about one-half of d . In this calculation, the first half of the vector is directly copied to the output. The second half goes through an entry-wise linear transformation: the operation ⊙ is the Hadamard product, the neural network s provides scaling coefficients, and the neural network t provides biases. With K such coupling layers, we have a transformation that defines a flexible distribution p ( z ) .

Here h 0 = x , and h ℓ = f ℓ ( h ℓ -1 ) . At the same time, p ( z ) also has an efficient sampling procedure given by f .

3 Methodology

Our new predicting model has three components: an encoder that compresses a graph representation of the state into a fixed-length vector, a sequential model that predicts next-step representations in the latent space, and a decoder that decodes spatial states from graph representations. The encoder is an improved version of the GMR-GMUS encoder [11]. The sequential model is a conditional flow model. The encoder and decoder are trained via self-supervised learning as in regeneration learning.

3.1 Graph representation learning

Following GMR-GMUS, we use a graph G = ( V , E ) to encode a snapshot of a deterministic or stochastic dynamical system at time step t . Here each node i ∈ V corresponds to the mesh cell C i , and each edge ( i, j ) ∈ E represents a neighboring relationship between two cells. At each step t , the solution u i,t at cell i becomes the feature vector of the node i ∈ V at time t . Because the mesh is pre-defined and fixed, the graph representation ( G, ( u i,t , i ∈ V )) preserves the full information of the mesh C h at time t . Here we use Y t = ( u i,t : i ∈ V ) to denote a snapshot at time t .

The key task for the encoder is to encode the graph into a low-dimensional vector z t . For this task, we use an improved version of the encoder of GMR-GMUS. The GMR-GMUS encoder runs a graph neural network over the graph representation to learn node representations. Then it takes node vectors of a selection of 'pivotal' nodes and concatenates them to get z t . In our new encoder, we select a set of locations in the spatial space instead of graph nodes to aggregate spatial information: each selected location aggregates vectors of nearby nodes to get its representation. It decouples the graph representation and the aggregation operation so that a select location can encode nodes within an arbitrary distance. To improve the training stability, we also improve the graph neural network's

PbGMR

Multi-Head Attention

Layer Normalization

Multi-Head Attention

Layer Normalization

…

z

z

t

z

t

c

t

+1

μ

Multi-Head Attention

Layer Normalization

z

0

Batch

Norm

Coupling

Layer

PbGMUS

Batch

Norm

Coupling

Layer

x

∼

Yt

Yt

+1

z

t

z

t

+1

z

t

+1

∼

p

(

z

t

+1

|

μ

,

z

0

,

z

1

, …,

z

t

)

1

-1

architecture by adding residual connections between its message-passing layers. We call this new encoder Position-based Graph Mesh Reducer (PbGMR). These two modifications clearly improve the encoder's ability to preserve the state information, which will be empirically shown in the experiment section.

The architecture of PbGMR is specified as follows. For notational convenience, we omit the time from the subscript. For each graph node i ∈ V , we first extract node and edge features as follows.

Here pos( i ) is the spatial location of the cell center of C i . After that, we apply L message-passing layers.

Here N i denotes all neighbors of node i , mlp e ℓ and mlp v ℓ are two separate multi-layer perceptrons, and layernorm is the layer-normalization operation. When a perceptron accepts multiple arguments, its arguments are first concatenated into a single vector. Each layer here is similar to a GraphNet block [24] but with slight differences. The prominent ones are residual connections and layer normalization in the update of node and edge representations: they help to stabilize the training procedure [25-27].

After we have learned node representations, we need to aggregate them into a single vector to represent the entire state. We randomly select a small set S of centers in the spatial area. For each center c ∈ S , we select k nearest mesh cells knn( c ) based on spatial distance and then compute the position representation h c by interpolation [28]:

Here d ( c, j ) is the spatial distance between cell C j and the center c . Given that the calculation is scaled by the sum of w cj -s, the unit of the spatial distance does not change the calculation here. The number k of neighbors is a hyper-parameter, and we fix it to 10. Finally, we concatenate all representations of centers into a single vector z = concat( h c : c ∈ S ) as the latent for the entire graph. Note that the centers in S are fixed for a problem.

As a comparison, the encoder in GMR-GMUS uses graph nodes as centers and only considers connected neighbors of a center. Then the number of neighbors in the interpolation operation is

limited by the graph structure. As a result, each center vector from GMR-GMUS can only represent information in a short range. Our new encoder overcomes this issue by decoupling the neighbors in interpolation and the neighbors in the graph representation and then gives the interpolation operation more freedom to represent a state of the system.

3.2 Decoding and self-supervised training

We devise a decoder PbMUS to recover node features on the graph from latent representation z : ˆ Y = PbGMUS( z ) . Here we also consider the computation at a single time step and omit time indices. We first split z and get vectors at interpolation centers: ( h c : c ∈ S ) = z and then compute the initial node representation r 0 i by spatial interpolation from centers:

Here knn ′ ( i ) are k centers that are nearest to the cell center i . Then we apply L message-passing layers to compute ˆ Y = gnn( G, ( r 0 i : i ∈ V )) , with gnn representing the L network layers. These layers have the same architecture as PbGMR but use different learnable parameters.

Self-supervised training. Without considering the sequential property of the data, we first train the encoder and decoder on single steps with self-supervised training. This training method shares the same spirit of regeneration learning and improves encoder and decoder's abilities to capture spatial patterns in the data. In particular, we use the reconstruction error as the minimization objective to train the encoder-decoder pair.

3.3 Attention-based temporal conditioned generative model

Now we consider the sequence of latent representations from the encoder and devise a sequential generative model to model the sequence of latent representations. The low-dimensional latent space reduces the modeling difficulty, and the vector form of latent representations z t avoids the graph structure and allows more model choices. We first decompose the sequence as follows.

The key is to devise a model for the conditional p ( z t | µ , z 0 , z 1: t -1 ) . We take two steps to construct this conditional: we first represent the condition ( µ , z 0 , z 1: t -1 ) with a single vector c t and then adapt RealNVP [23], a normalizing flow model, to describe the conditional p ( z t | c t ) .

Calculate the conditional vector with a transformer. Because physical parameters µ control the entire system, and the initial condition z 0 contains information about the initial condition, we consider them special and use them in the prediction of z t for each t . We structure the problem as a 'translation' problem and use the transformer [29] to run the calculation. In our case, the input 'sentence' is ( µ , z 0 ) , the first few 'tokens' in the target sentence are ( z 1 , . . . , z t -1 ) , and the 'next token' to be predicted is c t .

While the computation is exactly the same as one step in a translation task, the rationale is very different. First, the translation task directly gets the z t from the transformer, but our model needs to send c t to a conditional flow model to get the prediction z t . Second, the translation task has an informative input sequence, but our model uses a less informative input and depends on the transformer to get a reasonable output sequence.

Predict z t with a conditional flow model. Once we have a vector c t representation of the condition, we can construct a conditional flow model from RealMVP. Specifically, we append the condition vector c t to the input of the scaling function s and the bias function t in equation 3 in each layer.

From these layers, we have the flow model p ( z t | c t ) . By chaining up the transformer and the conditional flow model, we have our sequential model for conditional probability p ( z t | µ , z 0 , z 1: t -1 ) .

During training, we train the sequential model on ( z 0 , . . . , z T ) that are computed from our PbGMR encoder. During inference, we can efficiently sample z t from the sequential model. The whole training and inference processes can be found in Appendix A.2 and Appendix A.3.

4 Related Work

Deep learning for physics. For a deterministic system, one-step learning simply takes the current state as the input and outputs the next-step prediction [3, 30-32]. To further improve the accuracy of long-term forecasts, dimension reduction is applied together with sequence nets, aiming to solve long dynamical systems on a regular domain [33-35]. Such works often adopt CNNs as encoders so that can not be applied to irregular mesh space. To further work on mesh data directly with dimension reduction, the GNN encoder/decoder was initially introduced by [11]. Various GNN architectures were also proposed to facilitate the learning of physics. [13, 15, 36-39]. For physical stochastic systems, CNN is mainly used for probabilistic prediction [18, 40]. However, it lacks researches solving stochastic system in the graph space.

Regeneration learning and generative modeling. Regeneration learning is a new learning paradigm for data generation, which first generates a latent representation for the data. Then the generation process happens on the latent space. There are many recently popular generative models for images[41-43], videos[44], speeches[45], and music [46] are built on this paradigm. We notice that it should be further investigated for graph generation tasks.

5 Experiments

5.1 Deterministic dynamics

We benchmark our proposed model with three datasets from three flows: flows over the cylinder, high-speed flows over the moving edge, and vascular flows [11]. Note that for test cases, we feed the model physical parameters µ and the snapshot at the beginning time to generate the whole trajectory. Detailed description can be found in Appendix A.5.

We compare our new method against five SOTA models for fluid dynamics, including two variants of MeshGraphNet[3] and three variants of the GMR-GMUS[11]. We use relative mean square error (RMSE) as the evaluation metric: RMSE = ∑ (ˆ u prediction i -ˆ u truth i ) 2 ∑ (ˆ u prediction i ) 2 . More details can be found in the Appendix A.2. For cylinder and vascular flows, we calculate the RMSE with respect to velocity variables u , v , and the pressure variable p . For high-speed flows over the moving edge, we also calculated the RMSE for the temperature T .

The results on prediction errors. Table 2 shows the comparison of different models. Our model outperforms the baseline models in all the scenarios. For the cylinder flow case, our model has improvements of 22% , 17% , and 47% for u , v , and p . For the sonic flow case, our model has improvements of 61% , 70% for the u and v variables, and improvements of 82% and 50% respectively for the p and T variables. For the vascular

flow case, our model achieves improvements of 52% , 45% , and 95% for u , v , and p . The ablation

Re

= 307

Re

= 993

t= 150

t=295

t= 119

t=331

Sample

T

(0) = 201

T

(0) = 299

t= 30

t=40

t= 30

t=40

Sample

study in the Appendix section (Table 9) indicates that both the new encoder and the conditional normalizing flow are essential to improve prediction accuracy.

The error in the time axis is shown in Figure 3. The performance of our model is shown with the solid black line. The model has small initial errors in all three tasks. Then its error accumulates over time, but the accumulation is much slower than that of competing models, so our model consistently has the lowest error in the final time step.

To better understand our model, we have also done an extensive ablation study reported in the appendix. Here we only show a comparison of our model with GMR-GMUS in the encodingdecoding tasks. The results in Table. 1 show that the new model has clearly better encoding and decoding capabilities. In one encoding and decoding case, the reconstruction error of our model is less than one fifth of that of the GMR-GMUS model in the appendix. The ablation study also shows that both residual connections and encoding centers enhance the encoding-decoding accuracy (please see Table 8 in the Appendix section).

Contour analysis. As a probabilistic model, we plotted the contour of the predicted sample and the standard deviation (std) for all three datasets. Figure 2 visualizes the velocity predictions from two datasets. In each subplot, the top row is one prediction (e.g. one sample from our model), and the bottom row visualizes the standard deviation of predictions. To clearly compare different predictions, we use the same color bar within the same dataset. With visual inspection, the sample predictions are reasonable and stable for all three datasets.

The standard deviations also show reasonable spatial-temporal patterns consistent with our expectations. For cylinder flow with Re = 307 , the standard deviation is larger around the vortex-shedding region, indicating the model is less certain about the fast-changing dynamics. Moreover, the standard deviation tends to grow with time, reflecting the error accumulation behavior for long-time rollouts. Moreover, the standard deviation is smaller when Re is higher, indicating that the model is more certain about the prediction. The same trends can also be seen in other datasets. Therefore, our proposed model can accurately predict the dynamics for deterministic systems while providing a spatial-temporal uncertainty estimate, potentially improving the interpretability of deep learning systems. More contours can be found in Appendix A.6

5.2 Stochastic dynamics

Dataset. We apply the proposed method to solve a stochastic dynamical system governed by the unsteady-state incompressible Navier-Stokes equations. We also compare our method to the unsteady Reynolds-averaged Navier-Stokes equations (URANS), the most applied method in the industry due to the balance of accuracy and computational efficiency. Let Ω be the channel with a backward-facing

0

100

200

300

400

10

-

3

10

-

2

10

-

1

10

0

Time Step

RMSE

0

10

20

30

40

10

-

4

10

-

3

10

-

2

Time Step

0

100

250

10

-

3

10

-

2

10

-

1

10

0

Time Step

step shown in Figure 5. The goal of the problem is to generate the temporal unsteady velocity field close to the ones simulated by large eddy simulation (LES) with the inflow velocity along the inlet boundary subject to uncertainty. The inflow at every time step is uncertain and sampled from a 60-dimensional stochastic space subject to a uniform distribution as { u inlet 1 , ..., u inlet 60 | u inlet i = 10 + u ′ , u ′ ∼ U (0 , 1) } .

Analysis of fluid motion. To visually study the generated results, we plot several time steps of the stream-wise velocity ( u ) from a collection of model samples in Figure 4. The proposed method clearly generates diverse turbulent flow samples. Specifically, the multiscale structure of the vortex can be seen in the contour plots. To quantitatively examine the model performance on turbulent statistics, we plot and compare the mean and variance of the velocity profiles from the URANS, the proposed model, and LES results in Figure 5. The mean velocity profile predicted by the URANS model (gray lines) has significant discrepancies compared to the LES result (cyan lines), particularly in the right half of the domain where the backward-facing step's geometry is less important and small-scale turbulent features are dominant. As for the velocity variance, the URANS model underestimates the fluctuation in the whole domain. On the contrary, both the LES velocity mean and variance can be captured well by our generated samples (purple lines).

Comprehensive evaluations. To evaluate the temporal energy transport, we plot the power spectral density using Welch's method [47] in Table 3 ( left ). The high wave numbers reflect the rapid changes of fluid motion (e.g., instantaneous fluctuations), and the low wave numbers reflect the slow changes, such as the re-circulation introduced by the backward-facing step. By leveraging the sequence models,

¯

-

1

0

1

y

-

5

0

1

2

3

4

5

6

7

8

9

10

-

1

0

1

x

y

σ

2

(

u

)

u

the multiscale problem in the time domain can be automatically learned. However, URANS directly calculate an ensemble average over infinite experiments, and it can not capture stochastic behavior. Therefore, the energy ( E ( k ) ) of high-wave-number ( k ) signals is underpredicted by URANS while the proposed model has a consistent energy spectrum pattern with the LES simulations. Moreover, the turbulent kinetic energy (TKE) is computed to evaluate the quality of generated samples. Physically, the TKE is characterized by measured root-mean-square velocity fluctuations. By applying the decoder, the point-wise instantaneous flow can be recovered from the latent vector to the physical solution reasonably well. The TKE metric can be defined as a scalar:

It indicates the preservation of the TKE benchmarked by the LES simulations. Our model can generate a flow field preserving 99% of the energy, whereas URANS can only do less than 20% . As for the temporal mean field, although URANS aims to solve it directly, it is inevitable to introduce the bias from the LES result due to the ergodicity assumption of RANS. Instead, we directly formulate the probabilistic problem to learn the distribution of the spatiotemporal field. We calculate the error of the temporal mean field and find that although URANS leverages the conservation laws, it still obtains a much larger error than our generated samples. Finally, we further propose to measure the quality of predicted rollouts using evaluation metrics of video generation in computer vision. In particular, we consider two metrics: the continuous ranked probability score (CRPS) [48, 49] and Frechet Video Distance (FVD) [50]. We use CRPS to assess the respective accuracy of our probabilistic forecasting and LES models. Since our model considers the distribution of the forecasts as a whole, it outperforms URANS, which only focuses on the mean of the distribution. Our model is also better evaluated in the FVD metric. This indicates the learned distribution by our model is close to the samples from LES.

We have also experimented with two deep learning methods, MeshGraphNet and GMR-GMUS, which are designed for deterministic systems. Their predictions are both far from LES simulations, not to mention that they can only make deterministic predictions. More results can be found in Appendix A.13.

6 Conclusion

We propose a new learning model to predict/generate deterministic and stochastic fluid dynamics over unstructured mesh in a uniform way. With an integration of a novel graph auto-encoder, a transformer, and a normalizing flow model, the new model decomposes temporal and spatial correlations in a dynamical system. It outperforms competitive baseline models for deterministic systems while providing a reasonable spatial-temporal pattern of forward uncertainty estimations. The samples

URANS (

), LES (

)

Quantity of interest

URANS

Ours

10

0

10

1

10

2

10

-

14

10

-

7

10

0

k

E

(

k

)

CRPS (

C

u

,

↓

)

3.24

1.28

CRPS (

C

v

,

↓

)

2.0

1.08

FVD (

d

u

,

↓

)

228112

1262

FVD (

d

v

,

↓

)

137860

397

mean flow error (

e

u

,

↓

)

0.31

0.0176

mean flow error (

e

v

,

↓

)

0.94

0.176

turbulent energy (TKE,

↑

)

0.192

0.99

from the model trained on stochastic systems capture the rich physical patterns of expensive LES simulations. The current model still has one limitation: it can not accurately model stochastic variables close to boundary areas, which is shown in Appendix A.12. In future work, we will design new learning architectures to overcome this issue.

References

Fadl Moukalled, Luca Mangani, Marwan Darwish, F Moukalled, L Mangani, and M Darwish. The finite volume method . Springer, 2016.

Stephen B Pope and Stephen B Pope. Turbulent flows . Cambridge university press, 2000.

Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning meshbased simulation with graph networks. In International Conference on Learning Representations , 2021.

Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physics , page 111902, 2023.

Yarin Gal, Petros Koumoutsakos, Francois Lanusse, Gilles Louppe, and Costas Papadimitriou. Bayesian uncertainty quantification for machine-learned models in physics. Nature Reviews Physics , 4(9):573-577, 2022.

Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems , 32, 2019.

Jayaraman Thiagarajan, Rushil Anirudh, Vivek Sivaraman Narayanaswamy, and Timo Bremer. Single model uncertainty estimation via stochastic data centering. Advances in Neural Information Processing Systems , 35:8662-8674, 2022.

James M Dolezal, Andrew Srisuwananukorn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Brittany Cody, Aaron S Mansfield, Sagar Rakshit, Radhika Bansal, Melanie C Bois, et al. Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. Nature communications , 13(1):6572, 2022.

Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-informed deeponets. Journal of Computational Physics , 475:111855, 2023.

Zhong Yi Wan, Leonardo Zepeda-Núñez, Anudhyan Boral, and Fei Sha. Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems. arXiv preprint arXiv:2301.10391 , 2023.

Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.09113 , 2022.

Rishikesh Ranade, Chris Hill, Lalit Ghule, and Jay Pathak. A composable machine-learning approach for steady-state simulations on high-resolution grids. arXiv preprint arXiv:2210.05837 , 2022.

Jiayang Xu, Aniruddhe Pradhan, and Karthikeyan Duraisamy. Conditionally parameterized, discretization-aware neural networks for mesh-based modeling of physical systems. Advances in Neural Information Processing Systems , 34:1634-1645, 2021.

Dmitrii Kochkov, Tobias Pfaff, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Bryan K Clark. Learning ground states of quantum hamiltonians with graph networks. arXiv preprint arXiv:2110.06390 , 2021.

Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning , pages 8459-8468. PMLR, 2020.

Nicholas Geneva and Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows. arXiv preprint arXiv:2006.04731 , 2020.

Michael McCabe and Jed Brown. Learning to assimilate in chaotic dynamical systems. Advances in Neural Information Processing Systems , 34:12237-12250, 2021.

Mustafa Z Yousif, Linqi Yu, and HeeChang Lim. Physics-guided deep learning for generating turbulent inflow conditions. Journal of Fluid Mechanics , 936:A21, 2022.

Deniz A Bezgin and Nikolaus A Adams. Normalizing flows as a novel pdf turbulence model. arXiv preprint arXiv:2101.03590 , 2021.

Mengyuan Yang, Xiaolin Zheng, Qianqiao Liang, Bing Han, and Mengying Zhu. A smart trader for portfolio management based on normalizing flows. IJCAI, 2022.

Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods in numerical dynamics , volume 1000, pages 1-20, 2007.

Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, and Yoshua Bengio. Regeneration learning: A learning paradigm for data generation. arXiv preprint arXiv:2301.08846 , 2023.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 , 2016.

Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning , pages 4470-4479. PMLR, 2018.

David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning , pages 342-350. PMLR, 2017.

Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems , 34:22614-22627, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro SanchezGonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In Conference on Robot Learning , pages 1157-1167. PMLR, 2023.

Meire Fortunato, Tobias Pfaff, Peter Wirnsberger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets. arXiv preprint arXiv:2210.00612 , 2022.

Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.00728 , 2022.

Han Gao, Jian-Xun Wang, and Matthew J Zahr. Non-intrusive model reduction of largescale, nonlinear dynamical systems using deep learning. Physica D: Nonlinear Phenomena , 412:132614, 2020.

Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Neural Networks , 146:272-289, 2022.

Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems , 35:2240-2253, 2022.

Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems , 29, 2016.

Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566 , 2018.

Marco Maurizi, Chao Gao, and Filippo Berto. Predicting stress, strain and deformation fields in materials and structures with graph neural networks. Scientific Reports , 12(1):21834, 2022.

Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. Advances in Neural Information Processing Systems , 35:26256-26268, 2022.

Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physicsinformed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1457-1466, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 4779-4783. IEEE, 2018.

Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, and Tie-Yan Liu. Telemelody: Lyric-to-melody generation with a template-based two-stage method. arXiv preprint arXiv:2109.09617 , 2021.

Peter Welch. The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics , 15(2):70-73, 1967.

James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science , 22(10):1087-1096, 1976.

Christopher AT Ferro, David S Richardson, and Andreas P Weigel. On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorological Applications: A journal of forecasting, practical applications, training techniques and modelling , 15(1):19-24, 2008.

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems , 30, 2017.

A Appendix

A.1 Proof

The Jacobian of the transformation in Equation (4) can be written as:

fwhere diag ( exp [ s ( f i -1 concat(( x ) 1: d ′ , c ))] is the diagonal matrix whose diagonal elements correspond to the vector exp [ s ( f i -1 concat(( x ) 1: d ′ , c ) )] . Since the concatenation operation still keeps the matrix triangular, the determinant can be computed as:

A.2 Training

As a regeneration learning method, there are two stages during the training. The first stage is to train a graph auto-encoder, PbGMR-GMUS, in this case, through self-supervised learning. This training process happens over all time steps and sequences in the training set. And for a given sequence [ Y 0 , . . . , Y T ] , we compute ˆ Y t = PbGMUS( G , PbGMR( G , Y)) at each step and minimize the reconstruction loss:

In the second stage, the parameters in PbGMR-GMUS will be fixed. And the attention-based sequence model, as well as the conditional flow model, will be trained. We maximize the log-likelihood in the latent space:

We don't set a time window so that the whole trajectory will be trained. The entire training process can be found in Algorithm 1. Particularly in the second stage, such a teacher forcing-like strategy allows a faster training process compared with the previous work [11], which adopts a rollout strategy during the training.

A.3 Inference

For inference we first encode the first two snapshots into latent space [ z 0 , z 1 ] . The condition vector c t is computed from the attention-based sequence model. By sampling a noise vector x from an isotropic Gaussian, and putting it going backward through the flow, we can obtain a new sample from p ϕ ( z t | c t ) as latent representation at each step. We describe the procedure of training in Algorithm 2.

A.4 Latent analysis

Figure 6 plots the latent z t and conditional vectors c t for the stochastic BFS cases. For visualization purposes, we plot the first two modes of the PCA result. Moreover, we plot two different realizations of the stochastic process with clearly distinctive trajectories, as shown in the c t plot (right). Since the cases are inherently from the same distribution, the sampled latent trajectories z t plot (left) have a lot of overlaps after the few initial development steps. The plots of different cases visually conform to the physical intuition and indicate that our model can learn the relation between different cases well.

Algorithm 1 Training Process

Input: Domain graph G , Node features over time [ Y 0 , . . . , Y T ] , PbGMR θ , PbGMUS ϕ , Attention encoder MHA ω , Attention decoder Masked-MHA λ , conditional flow model p ψ ( . | ) , physical condition parameter µ

Algorithm 2 Inference Process

Input: Domain graph G , The firt two snapshot of node features [ Y 0 , Y 1 ] , PbGMR θ , PbGMUS ϕ , Attention encoder MHA ω , Attention decoder Masked-MHA λ , conditional flow model p ψ ( . | ) , physical condition parameter µ , sample length T

Output: A trajectory sample [ Y 2 , . . . , Y T ] [ z 0 , z 1 ] = PbGMR θ ([ Y 0 , Y 1 ] , G ) ˆ µ , ˆ z 0 = MHA ω ( µ , z 0 ) for t ∈ [2 , . . . , T ] do c t = Masked-MHA λ (( z 1 , . . . , z t -1 ) , I = (ˆ µ , ˆ z 0 ))[ -1] Sample latent representation from conditional flow z t ∼ p ϕ ( z t | c t ) end for Recover physical parameters on the original space [ Y 2 , . . . , Y T ] = PbGMUS ϕ ([ z 2 , . . . , z T ])

-

30

-

25

-

20

-

20

0

20

C

0

C

1

(a)

-

5

0

5

-

10

0

C

0

C

1

(b)

A.5 Dataset

Four fluid simulation datasets, cylinder flow, sonic flow, vascular flow and stochastic backward-facing step (BFS) are used in the experiments. The labeled data are generated by solving the Navier-Stokes equation by OpenFOAM [21], a commonly used finite volume method-based open-source solver for fluid simulation. Cylinder and sonic flow consist of 50 cases in the training dataset and 50 cases in the test dataset. The vascular flow cases have 10 training and 10 test cases. The stochastic BFS case has 5 training cases. Unlike the first three deterministic cases, the stochastic BFS case is a stochastic case. The simulation details can be found in Table 4. Moreover, the details input and output of the PbGMR-GMUS are listed in Table 5.

The governing equation for the first three cases is the same as the one listed in [11]. For the last stochastic BFS case, the governing equation is listed in Equation (19). One significant difference in this case is that it introduces stochasticity by perturbing the inlet u velocity at every time step.

subject to u inlet i = 10 + u ′ , u ′ ∼ U (0 , 1) at every time step.

A.6 Additional visualization results

Apart from the contour shown in Figure 2, additional contour plots for the cylinder, sonic flow and vascular flow are shown in this section. Figure 7 shows the sampled velocity and std of the velocity for vascular flow cases. Figure 8 presents the sampled pressure for cylinder, sonic and vascular flow. Additionally, Figure 9 shows the sampled temperature variable in sonic flow.

R

= 0

.

31

(Vascular flow)

R

= 0

.

49

(Vascular flow)

t= 180

t=230

t= 150

t=200

Sample

Re

= 307

Re

= 993

t= 150

t=295

t= 119

t=331

Sample

T

(0) = 201

T

(0) = 299

t= 30

t=40

t= 30

t=40

Sample

R

= 0

.

31

(Vascular flow)

R

= 0

.

49

(Vascular flow)

t= 180

t=230

t= 150

t=200

Sample

T

(0) = 201

T

(0) = 299

t= 30

t=40

t= 30

t=40

Sample

A.7 Multi-head attention

Transformer [29] has been proven successful in the NLP field. The design of the multi-head attention (MHA) layer is based on the attention mechanism with Query-Key-Value (QKV). Given the packed matrix representations of queries Q , keys K , and values V , the scaled dot-product attention used by Transformer is given by:

where D k represents the dimensions of queries and keys.

The multi-head attention applies H heads of attention, allowing a model to attend to different types of information.

A.8 Additional details for experimental setups

We described the details of the experiments of PbGMR-GMUS and attention-based conditional flow model. We provide the hyperparameters used in the experiments in Table 6.

1e-2

1.0

u

error

error

0.8

p

error

E

S

0.6

0.4

0.2

0.0

50

100

150

200

250

300

350

400

# pivotal positions

A.9 Performance impact of substituting Transformer with LSTM

In this research, we conduct an ablation study in which we replace the proposed attention-based sequence model in our framework with a Long Short-Term Memory (LSTM) [53] architecture. The aim is to evaluate the impact of the underlying sequence modeling on the performance of our system. The result is listed in Table 7. Following this change, we observe that there is a notable decrease in the performance metrics, suggesting a less optimal fit for the task at hand compared to the Transformer-based model. This outcome underscores the usefulness of attention-based sequence models, like the Transformer, which appears to capture dependencies in the data more effectively than the LSTM. The attention mechanism inherent to Transformers allows the model to focus on different parts of the input sequence dynamically, which may explain the observed performance superiority.

A.10 Performance impact of number of centers

We conduct an ablation study to investigate the effect of the number of centers on the RMSE of our framework's predictions for cylinder flow recovery. The result is in Figure 10. As we progressively increased the number of centers, the recovery RMSE exhibited a decreasing trend, demonstrating an enhancement in prediction accuracy. Interestingly, upon reaching 256 centers, the rate of RMSE decrease started to decelerate significantly. This suggests that the optimal number of centers for cylinder flow is around 256. Beyond this point, further increments do not contribute as much to the improvement in encoding-decoding accuracy, while simultaneously increasing the computational cost. Hence, a count of 256 centers appears to be the sweet spot, offering a good trade-off between recovery accuracy and computational efficiency.

-

1

0

1

x

y

A.11 Centers

The centers distribution for Stochastic BFS can be found in Figure 11.

A.12 Analysis for stochastic BFS

Backward-facing step flow refers to a fluid flow configuration characterized by a sudden contraction followed by an expansion in a channel or pipe. This flow configuration is commonly encountered in various engineering applications, such as in heat exchangers, combustion chambers, and aerodynamic systems. In backward-facing step flow, the fluid initially enters the channel or pipe through an inlet, where it encounters a step or abrupt contraction. As the fluid passes over the step, its velocity increases, and the flow separates, forming a recirculation zone downstream. This recirculation zone is often referred to as the "backflow region." The backward-facing step flow has been extensively studied due to its complex flow characteristics and its relevance in understanding phenomena like separation, reattachment, and flow control.

The simulations and generated samples exhibit remarkable similarity in the contours of both streamwise velocity and wall-normal velocity (Figure 4, 12, 13, 14). This close resemblance demonstrates the high fidelity of the simulation results and the accuracy of the generated samples. The level of detail captured in these contours highlights the effectiveness of the proposed generative framework.

9

8.9

We also randomly choose nine spatial locations to analyze the learning performance, and the coordinates of these points are plotted in Figure 15. Figure 16 and 17 presented the comparison of u , v velocity between the LES simulation, URANS simulation and our model. Generally, the pointwise velocity temporal signals obtained from the generated sample provide a comprehensive representation of the flow characteristics and look visually similar to those obtained from LES simulations. However, the traditional URANS modeling approach cannot capture the intricate fluctuations in the flow field. Note that we also found the generated velocity time signals at points 1 and 2 have a relatively large discrepancy from the LES simulation, and it is explainable. Point 1 is very close to the inlet boundary; therefore, the u velocity here is very random because of our stochastic velocity inlet condition. Point 2 locates in a region where the BFS flow is not fully developed yet and in this region the v velocity

-

1

0

1

1

2

3

4

5

6

7

8

9

x

y

0

100

200

10

10

.

2

10

.

4

10

.

6

10

.

8

u

0

100

200

10

.

2

10

.

4

10

.

6

10

.

8

0

100

200

-

10

0

10

0

100

200

-

1

0

u

0

100

200

5

10

15

0

100

200

8

10

12

14

0

100

200

0

10

t

u

0

100

200

0

5

10

15

20

t

0

100

200

-

10

0

10

t

0

100

200

-

0

.

2

-

0

.

1

0

0

.

1

0

.

2

v

0

100

200

-

5

0

5

·

10

-

2

0

100

200

-

5

0

5

0

100

200

0

10

v

0

100

200

-

10

0

0

100

200

-

2

0

2

0

100

200

-

5

0

5

t

v

0

100

200

-

10

0

10

t

0

100

200

-

10

0

10

t

magnitude is very small. These conditions make it hard to capture an accurate distribution of the velocity at these 2 points. Finally, we also compare the velocity distribution between our generated samples and the LES simulation results, as shown in Figure 18, 19. Except for points 1 and 2, the distribution of our generated samples aligns very well with the LES simulations.

Ours

Ours

LES

LES

0.08

LES

0.06

00

0.02

10.1

10.2

10.3

10.4

10.5

10.6

10.7

10.8

10.4

10.5

10.6

10.7

10.8

00°0

-20

15

-10

0.14

1.4

Ours

0.12

Ours

0.20

1.2

LES

LES

0.10

1.0

0.15

800

Arisua

0.6

0.10

0.4

0.04

Ours

SO'0

0.2

0.02

LES

0'0

00'0

00°0

1.5

1.0

0.5

0°0

0.5

1.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

10

12

14

0.175

0Z0

Ours

0.175

Ours

Ours

LES

0.150

LES

0.150

LES

0.15

0.125

0.125

0.100

Den

0.075

0.075

0.050

OS00

0.05

0.025

0.025

0.00

0.000

0.000

10

15

20

10

15

20

-10

8

Ours

60

Ours

Ours

LES

50

LES

0.20

LES

40

0.15

Density

0.10

20

0.05

10

0.2

0.1

0.0

0.1

0.2

0.06

0°0-

0.02

0.00

0.02

0.04

0.06

000

0.16

Ours

Ours

Ours

0.10

0.14

0.5

LES

LES

LES

0.12

0.08

0.4

0.10

.6

isuan

0.3

80°0

0.04

900

0.2

00

0.02

0.1

0.02

0.00

000

0.0

10

-5

0

10

15

15

-10

-5

10

3

-1

SE'0

0.16

0.14

Ours

Ours

Ours

0.30

0.14

0.12

LES

LES

LES

0.25

0.12

0.10

0.10

800

0.15

900

90°0

0.10

00

00

0.05

0.02

0.02

0.00

-10.0

7.5

5.0

2.5

2.5

5.0

7.5

10.0

00°0

-10

15

000

15

10

-5

10

A.13 Ablation study

Comprehensive ablation studies are included in this section to demonstrate the effectiveness of each parts. The velocity reconstruction error on the Backward-facing step flow dataset with or without residual connection and centers is shown in Table 8. The result indicates that each component can improve the graph reconstruction accuracy. We also test the influence of PbGMR-GMUS and the attention conditional flow model to the final performance of the deterministic task, as shown in Table 9. The result shows that both the encoder-decoder part and the flow model are necessary for the success of the whole framework.

To efficiently incorporate the temporal information, we conceive a temporal conditional normalizing flows to account for the long-time temporal dependency. The original RealNVP is only a probabilistic model, and it is not straighforward to include temporal information. To achieve this, we propose an encoding-decoding transformer structure to incorporate fixed physical parameters and all previous steps into condition vector c . In this ablation study, we remove the transformer structures and replace condition vector c by the current step latent vector z t -1 when predicts next step z t . This variant can be treated as a probabilistic 1-step prediction method. The result in Table 10 shows that the proposed structure can significantly improve the accuracy compared with such 1-step method, while still easy to calculate the probability of a coupling layer if we concatenate condition vector at every coupling layer, as shown in Appendix A.1.

Our model can benefit from longer time dependencies, since we take all previous steps in the experiment setting. Also, we can use the moving window to reduce the cost further. We did one additional experiment of reducing the number of inputs during the inference time, as shown in Table 11. When reducing the window size from 400 to 150, the degradation in accuracy is not significant. So the model is also applicable to cases with thousands or even longer steps with the sliding window.

Though we only tested RealNVP in the experiment part, other normalizing flows are also feasible here. In Table 12, we test another normalizing flow model MAF [54], and find the result is comparable to RealNVP. It indicates that the proposed framework is flexible.

As a probabilistic model, the proposed PbGMR-GMUS + conditional flow model is trying to fit such a stochastic process and get various samples with the same physical statistics. To demonstrate these, we test MGN and GMR-GMUS on the stochastic dataset. In Table 13, we can see that the performance of the proposed model is better than other learning-based models on all metrics. In Figure 20, we visualize the mean and variance of velocity from each model, and find the MGN can not capture the real distribution at all. The result of GMR-GMUS model, though looks reasonable in statistics, can only produce the same output given the same input condition, which doesn't reflect the stochasticity of the last dataset. In Figure 21, we get two samples from GMR-GMUS, and find the two samples are exactly the same.

Peer reviews:
comments: SUMMARY:
The paper proposes PbGMR-GMUS to encode the graphs of physics systems into low-dimensional features, which are further reconstructed into the desire graph representations. An attention-based model is integrated into flow-based generative models to predict the future dynamics in the latent space. The proposed method is able to handle both deterministic and stochastic fluid dynamics thanks to the probabilistic model and achieve superior performance over previous methods.

STRENGTHS:
* An effective model to learn the graph representations in low-dimensional space.
* The simulation of dynamics is achieved by a probabilistic method using generative models, which rollout vivid predictions for long-term predictions.
* The method is able to solve both deterministic and stochastic fluid dynamics.

WEAKNESSES:
* Miss citation at L62 for GMR-GMUS.
* On top right of Figure 1, is it "Coupling layer" instead of "Couling layer"?
* Unclear caption for Figure 5. Which one is the variance?

QUESTIONS:
* How to prove the statement about the first benefits at L163-164? Is there any further explanation?
* Is the main difference to between simulating deterministic and stochastic fluid dynamics that the $\mu$ in Equation 14 would change with time for stochastic dynamics?
* Any video demos for the predictions of both deterministic and stochastic fluid dynamics? The temporal consistency can be better illustrated by videos.

LIMITATIONS:
The proposed PbGMR-GMUS is technically limited, which is a variant of GMR-GMUS with simple modifications. Video demos are missing to verify the temporal consistency of the predictions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel mesh-based machine-learning approach for modeling stochastic fluid dynamics. Similar to [11], the state transitions are modeled in a compact latent space (referred to as the mesh-reduced space) rather than the high-dimensional mesh space. Compared with [11], this paper makes several technical contributions below. \n1. It proposes to use virtual pivotal positions instead of selecting pivotal nodes from the mesh topology.\n2. It employs a deep generative model based on a normalizing flow approach, RealNVP, to model the stochastic dynamics. The model also incorporates multi-head attention over sequential latent states. \n\nThe experimental results demonstrate that the proposed model significantly outperforms [11] in the deterministic setup, while in the stochastic setup, the model is compared against an off-the-shelf CFD method named URANS, rather than learning-based models.\n\nSTRENGTHS:\nOriginality: To the best of my knowledge, this paper introduces the first learning-based probabilistic model that simulates fluid systems upon the mesh topology. The key innovation lies in integrating a flow method for stochastic temporal modeling into the GMR-GMUS [11] framework, which presents a reasonable and creative combination of existing approaches.\n\nQuality and clarity: Overall, the paper is clear and easy to follow.\n\nSignificance: The proposed model is shown to be effective in deterministic setups, which consistently outperforms MeshGraphNet and GMR-GMUS [11] by substantial margins.\n\nWEAKNESSES:\nMain concerns:\n\n1. The proposed model appears to be a straightforward extension of GMR-GMUS [11] combined with RealNVP [25]. \n2. Ablation studies for using virtual pivotal positions, residual connections, and the conditional normalizing flow are lacking, which makes it difficult to determine the specific contributions of these components to the performance improvement shown in Table 1. It is suggested that the authors address this issue and provide a more comprehensive analysis.\n3. Although the paper addresses the new problem of probabilistic modeling of fluids, the lack of comparison with existing learning-based models such as GMR-GMUS and MeshGraphNet in Section 5.2 raises doubts about the true challenges of the prediction task for stochastic fluid systems and the superiority of the proposed approach.\n\nMinor concerns:\n\n1. Accurate citations with corresponding publication sources should be provided. For example, GMR-GMUS [11] was published at ICLR 2022. his will enhance the credibility and traceability of the referenced papers.\n2. It would be valuable to include a brief discussion regarding the limitations of the research to provide readers with a broader perspective on the potential areas for improvement or expansion.\n3. In Line 62, the authors introduce GMR-GMUS without relevant sources, which made me confused until I came to Section 3.1.\n4. In Lines 113-121, it would be good if the authors could describe the learning method by referring to specific components or steps in Figure 1, so the readers can better grasp the concepts and processes discussed in the text.\n\nQUESTIONS:\n1. In Line 118, the claim that the proposed model enables the enlargement of the detection distance compared to GMR-GMUS lacks detailed explanations. Further elaboration on this point would be beneficial.\n2. In Line 87, it is mentioned that the stochastic dynamic system includes a random parameter $\\mu$ that varies over time, such as perturbations in the boundary conditions of turbulent flow. However, in Line 161, $\\mu$ is referred to as the global physical system parameter and seems to be time-invariant. It would be helpful for the authors to clarify whether $\\mu$ is time-varying across the input and output sequences.\n3. Regarding $\\mu$, as well as the positions $p(i)$ of the mesh cells, it is not clear from the provided context whether they are given in the future prediction time horizon. \n4. I am not very familiar with RealNVP, but based on the information provided, should it be written as $P(z_{1:T} \\mid \\mu, z_0, x; \\theta)$ in Equation (10) instead of $P(z_{1:T} \\mid \\mu, z_0; \\theta)$?\n\nLIMITATIONS:\nNo discussions on limitations or broader societal impacts are presented in the current text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a unified framework for solving deterministic and stochastic physical dynamical systems on high-dimensional mesh space. Instead of updating values at each discretized mesh, the paper introduces an approach that evolves states in a low dimensional latent space through encoding states and physical parameters. This encoding is done by a message passing graph neural network and multi-head attention (MHA) model, which encodes physics parameters into a conditional vector. The encoded state is then evolved using a conditional normalizing flow, dependent on the encoded conditional vector. The effectiveness of this method is shown through experiments, in which it outperforms other mesh-based ML models in terms of accumulation error in deterministic problems, with applications in stochastic problems also being demonstrated.\n\nSTRENGTHS:\nThe proposed method is simple and applicable to wide range of PDE-simulation problems with discretized domain. Wide variety of experiments are conducted, and both quantitative and qualitative evaluations are provided.\n\nWEAKNESSES:\nAbout the novelty, it is still unclear that what components of the proposed model enables solving both deterministic and stochastic  systems efficiently. Are there any limitations that prevent existing models from being applied to both systems?\n\nIt is concerning that the proposed method take all the previous (latent) vectors as input to predict the next state, as opposed to fixed small number of input vectors of MeshGraphNets reported in Section 5, that may result in unfair advantages over other methods. Also, although in Table 1 the reconstruction error of PbGMR-GMUS are compared against GMR-GMUS, it is still unclear how the proposed model would compare against an ablation model whose encoder is GMR-GMUS.\n\nThe proposed model is compared against baselines which evolve states in original space. There is a possibly missing reference [1], and the motivation of using latent evolution is similar to the presented one. How does this work compare against it? Is the proposed method also applicable to grid space?\n\n[1] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution (NeurIPS 2022)\n\n\n**Minor comments**\n\nIn Appendix A.10, Figure 11 is cited, but seemingly wrong.\n\nQUESTIONS:\nPlease have a look at the weakness mentioned in the strength and weakness section and address these. Overall the idea seems interesting, the authors need to substantiate their claims in light of existing literature and possibly more experiments.\n\nLIMITATIONS:\nYes, the limitations are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new approach for modeling fluid dynamics. The approach involves using GNNs to derive global latent space representations and construct a transformer-based conditional generative models for the dynamics. The resulting model is able to generate stochastic predictions from given initial conditions and system parameters at inference time. The proposed framework is compared against some competitive baselines on a few deterministic and stochastic benchmarks.\n\nSTRENGTHS:\n*  This work seeks to perform probabilistic modeling of high-dimensional dynamical systems originating from parametric PDEs. This is a very important problem as traditional methods are typically quite expensive and lacking ways to quantify uncertainties. The method proposed here is a nice attempt that addresses both challenges.\n* The method presented is very flexible and easily adapts to various problems and configurations. Because of the complexity of the cases, it has the potential to serve as a fast surrogate for modeling turbulence useful for inner-loop applications.\n* The introduced regeneration learning framework is quite novel as it effectively integrates different blocks (graph representation learning, attention-based model, normalizing flows). Such formulation offers an efficient and attractive alternative to directing learning long-time spatiotemporal fields on the original mesh. Moreover, the framework can handle deterministic/stochastic systems in a unified way.\n* The presentation is very clear, logical and easy to understand in general. The plots are of high quality and informative.\n* Claims are backed by strong and convincing empirical evidence. The examples in the numerical experiments section demonstrate sufficient complexity and variety to help justify the value of the proposed method.\n\nWEAKNESSES:\n* A number of minor typos in texts and figures (see section below).\n* More discussions on the limitations would strengthen the paper further.\n\nQUESTIONS:\nQuestions\n* How much of the improvements can be attributed to using *pivotal positions* and how much to using a different architecture for the latent dynamical model? It would be nice to have some ablation studies\n* What is the advantage of using the RealNVP vs other generative models (e.g. there is a number of normalizing flows [here](https://github.com/VincentStimper/normalizing-flows))?\n\nComments\n* Ln 168 - I believe it’s inaccurate to say the geometry of the graph is not included in the latent vector - it’s really embedded implicitly in the representation.\n* Ln 243 - “.. potentially improving the interpretability of deep learning systems” - UQ and interpretability are different concepts.\n\nTypos\n* Equation (1), $j$ is not defined in text. Based on what follows, it can be a stochastic operator as well?\n* Ln 95 - functions that *are* parameterized by neural networks\n* Equation (5) - consider not using $p$ for position to avoid confusion with probability densities?\n* Typeset errors on ln 219 and 289?\n* Figure 3 - axis labels are missing\n* Table 3 - what is referred to by “Flow” is your model? I don’t believe this is referenced elsewhere in the text.\n* In appendix, Ln 589 - the results are in Figure 10 not 11\n\nLIMITATIONS:\nThe authors do present ablation studies showing that a certain number of *pivotal positions* are required. Other potential areas to discuss are scalability, applicability to more complex systems, sampling requirements and parameter extrapolations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}