From Zero to Turbulence: Generative Modeling for 3D Flow Simulation
Authors: Marten Lienen
David Lüdke
Jan Hansen-Palmus
Stephan Günnemann
Venue: ICLR 2024 poster (2024)
Keywords: pde
generative
diffusion
navier-stokes
cfd
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Generative models are a promising alternative to time series methods for neural surrogate turbulence solvers

Abstract: Simulations of turbulent flows in 3D are one of the most expensive simulations in computational fluid dynamics (CFD). Many works have been written on surrogate models to replace numerical solvers for fluid flows with faster, learned, autoregressive models. However, the intricacies of turbulence in three dimensions necessitate training these models with very small time steps, while generating realistic flow states requires either long roll-outs with many steps and significant error accumulation or starting from a known, realistic flow state—something we aimed to avoid in the first place. Instead, we propose to approach turbulent flow simulation as a generative task directly learning the manifold of all possible turbulent flow states without relying on any initial flow state. For our experiments, we introduce a challenging 3D turbulence dataset of high-resolution flows and detailed vortex structures caused by various objects and derive two novel sample evaluation metrics for turbulent flows. On this dataset, we show that our generative model captures the distribution of turbulent flows caused by unseen objects and generates high-quality, realistic samples amenable for downstream applications without access to any initial state.

Published as a conference paper at ICLR 2024
FROM ZERO TO TURBULENCE: GENERATIVE MODELING FOR 3D FLOW SIMULATION
Marten Lienen, David Lüdke, Jan Hansen-Palmuse Stephan Günnemann Department of Informatics & Munich Data Science Institute
Technical University of Munich, Germany
[m. minei,,en, d. luecke, j wwhasnn-pppa 1 m s. guennemann ) (ttuu.d de
ABSTRACT
Simulations of turbulent flows in 3D are one of the most expensive simulations in computational fluid dynamics (CFD). Many works have been written surrogate models to replace numerical solvers for fluid flows with faster, learned, autoregres- sive models. However, the intricacies of turbulence in three dimensions necessitate training these models with very small time steps, while generating realistic flow states requires either long roll-outs with many steps and significant error accumula- tion or starting from a known, realistic flow state- someteiin we aimed to avoid in the first place. Instead, we propose to approach turbulent flow simulation as a gener- ative task directly learning the manifold of all possible turbulent flow states without relying on any initial flow state. For our experiments, we introduce a challenging 3D turbulence dataset of high- nigh-resolution flows and detailed vortex structures caused by various objects and derive two novel sample evaluation metrics for turbulent flows. On this dataset, we show that our generative model captures the distribution of turbulent flows caused by unseen objects and generates high-quality, realistic samples amenable for downstream applications without access any initial state.
INTRODUCTION
CFD is integral component of engineering today, and significant computing resources are spent it every day at scales small and large. Engineers simulate fluid flows to maximize the throughput in chemical plants, optimize the energy yield of wind turbines, improve the efficiency of aircraft engines (Reguly et al., 2016). The widespread use of these simulations makes their acceleration with machine learning highly impactful (Raissi al., 2019; Rackauckas et al., 2021; Kochkov et al., 2021; Li et al., 2021).
CFD simulation consists of a discretized geometry represented as grid or mesh, boundary conditions that specify, for example, the position and behavior of walls and inlets, and initial conditions that provide a known state of the flow. Since the turbulent flow's behavior is unknown, these initial conditions are usually specified as constants or smooth approximations of the expected flow. To produce realistic turbulent states, a numerical solver solves the Navier-Stokes equation
tu Øt
1 Vp
2 u
(u V)u
P
forward in time until the flow transitions from the simplified initial conditions fully developed turbulence. The equation describes the relationship between velocity u and pressure for viscous fluid with kinematic viscosity y and constant density p. For liquids, Eq. (1) is often supplemented with the incompressibility assumption V u 0, and we do so, too.
While numerical solvers can simulate most kinds of flows to high precision, large simulations can be very slow, especially in three dimensions (3D). A prominent approach to accelerate these simulations to emulate the numerical solver with autoregressive models parameterized by neural networks. Many works have demonstrated formidable speedups in two dimensions (2D) for scenarios of varying complexity and all kinds models of this class (Horie & Mitsume, 2022; Pfaff et al., 2021; Shu et al.,
Find code and data https: : // cit. tum wtw.tttm.ddddeeeee de/ddd/ddlllaa /generat vie turbul ence.

---

Published as a conference paper at ICLR 2024
2023; Yang & Sommer, 2023; Janny et al., 2023; Obiols-Sales et al., 2020; Zhao et al., 2022; Kochkov et al., 2021; Wang et al., 2020; Li et al., 2021; Brandstetter et al., 2022) and several benchmark datasets for the 2D Navier-Stokes equation have been published in recent years (Bonnet et al., 2022; Gupta & Brandstetter, 2022; Takamoto et al., 2022; Otness et al., 2021). In comparison, 3D results much sparser in terms of models (Stachenfeld et al., 2022) well datasets (Takamoto et al., 2022; Li et al., 2008), even though 3D simulations offer the highest acceleration potential because of their enormous computational costs.
Autoregressive models usually outperform numerical solvers by taking larger time steps without diverging. However, in 3D, turbulent flows exhibit a stark difference to their 2D counterparts hampering this strategy. Because short-lived, small-scale vortices drive the chaotic turbulence in 3D to much higher degree, their behavior needs to modeled, which limits the size of the time step of an autoregressive model. At the same time, the complete replacement of the numerical solver requires the model to evolve the flow from the solver's initial state until turbulent flow state has fully developed, which can require hundreds of steps and lead to catastrophic error accumulation.
However, in many applications, practitioners use numerical simulations not to explore one specific solution trajectory through time but as proxy to explore the distribution of possible flow states. For example, design optimization, an engineer might be interested in the possible size, length, and energy of vortices caused by object, such particular vortex. In other applications, the formation of certain patterns, such as jets, or the location of stagnation points the boundary of object, where the flow velocity is zero, might be important. We conclude that many use cases can be solved by independent snapshots of possible flow states just as well with a classical numerical simulation.
airplane wing, but not in the exact evolution of one
In this paper, we take an entirely different direction to autoregressive models and propose a generative approach to turbulent flow simulation by directly learning the manifold of all possible turbulent flow states, This way, we overcome the problem of long roll-outs of autoregressive models and skip the initial transition phase as well all the intermediate states that numerical solver has generate to produce diverse set of flow states.
Our contributions can be summarized as follows:
We present a novel 3D turbulence dataset with challenging geometries and systematically investi- gate the specific challenges of forecasting 3D turbulent flows compared 2D.
We propose to approach 3D) turbulence simulation generative task to overcome the roll-out dilemma of autoregressive models. In doing so, we derive an appropriate generative diffusion model and evaluation metrics.
We verify experimentally that our model captures the distribution of turbulent flows, generalizes to unseen geometries and generates high-quality, realistic samples amenable for downstream application, ultimately eliminating the reliance on numerical solvers.
TURBULENCE IN TWO AND THREE DIMENSIONS
For a complete introduction turbulent flows, we re- fer the reader to (Pope, 2000) and (Mathicu & Scott, 2000). (Ouellette, 2012) gives a great overview of the differences between 2D and 3D turbulence.
3D
In principle, one can solve the incompressible Navier- Stokes-Eq. (1) in two spatial dimensions just as well as in three. However, when we compare the resulting flow fields of two simulations in Fig. I that only differ in that one simulation domain extends into the third dimension, it becomes obvious that some effect must be exclusive to 3D fluid flows. The 3D data exhibits many small-scale features, while the two-dimensional flow field is dominated by relatively smooth and orderly large-scale structures. But why is that?
2D
Figure 1: The same simulation exhibits vastly different qualities when the solver operates in three dimensions compared to two.
2

---

Published as a conference paper at ICLR 2024
One of the defining features of turbulence in 3D is the energy cascade, a process that moves kinetic energy from large-scale vortices to ever smaller scales until it is finally directly converted to thermal energy at the molecular scale. The main drivers of this process and strain self-amplification (Carbone & Bragg, 2020). Together, they are responsible for the continued creation of vortex structures all scales and the high-frequency features characteristic of 3D turbulence.
two effects called vortex stretching
In two dimensions, both of those effects vanish mathematically, ultimately due to the fact that the vorticity w V x u, which describes the local rotationality of the flow, is orthogonal to the velocity gradient in 2D. As a consequence, the energy cascade is only driven by weaker interaction effect between vorticity and strain, which inverts the energy cascade (Johnson, 2021). The inverted energy cascade, in contrast to 3D, transports energy from the small to the large scales and creates more homogeneous, long-lived structures.
Temporally, the lifetime of vortex structures depends on their size, with smaller structures decaying faster than larger ones (Lozano-Durke & Jiménez, 2014). From machine learning perspective, this means that forecasting the future trajectory of a fluid simulation in 3D requires smaller time sleps than for 2D data to resolve the nonlinear behavior of small-scale features under the Navier-Stokes equation.
3 AUTOREGRESSIVE FORECASTING IN 3D
Autoregressive models are popular surrogate models for numerical solvers of time-dependent partial differential equations (PDEs) and, in particular, the Navier-Stokes equation, as they promise faster runtimes than full simulations. They emulate numerical solvers by learning predict the next state from the current state and, optionally, a set of previous states, which enables approximating fluid dynamics over time via an iterative roll-out. While autoregressive models offer advantages, i.e. faster evaluation, case of parallelization, and the ability to use larger time steps, their possible runtime improvement directly limited by how many solver steps the model can compress into one model evaluation.
Gaussian smoothing kernel width a 8
Trained stride [ms]
24
32
0.1
0.8
1.6
2.4
3.2
Normalized MSE (u)
200
B ull?
100
Nee, stride t
Smoothing
0
0++
0.1
3.2
6.4
9.6
12.8
0.1
10
Step size At [ms]
Roll out time [ms]
Figure 2: 1-step forecast MSE of u increases rate comparable to Gaussian smoothing of states to remove small-scale features.
Figure 3: Error accumulates quickly for longer roll-outs even when the time step is sufficiently small.
As described in Section 2, 3D turbulent flows contain many small-scale features that evolve nonlinearly on short timescales. In our experiment shown in Fig. 2, we have trained a Dilated ResNet (DilResNet) on one-step-ahest prediction with varying step size At. While the model achieves very small error for the smallest time step. the one-step forecasting error grows as we increase the time step that we train the model for. Notably, the error curve aligns closely with the curve resulting from smoothing of the target with a Gaussian kernel of increasing width. By smoothing the velocity field with Gaussian kernel of variance o², we remove features of scale approximately a and below. The alignment of the curve then indicates that the model trained to predict At ahead fails to predict features below cut-off scale a that scales linearly with At. In fact, visual inspection of the forecasts reveals that the model makes increasingly smooth predictions the model's 's time step increases. Conversely, this means that the model needs to trained and evaluated with a very small time step to even have a chance forecast fully resolved future states of 3D turbulent flows multiple steps.
3

---

Published as a conference paper at ICLR 2024
Ultimately, modeling 3D turbulent flow simulations poses an inescapable dilemma for autoregressive models. On the one hand, turbulent flows exhibit significant small-scale behavior that is important to the overall behavior of the flow because of backscattering effects from small to large scales, which requires models to be trained with a small step size. On the other hand, these simulations need to be forecast for hundreds of steps into the future traverse the transition phase from initial simulation state to fully developed turbulent flow, which necessitates long roll-outs. However, autoregressive models trained with smaller step sizes and, consequently, more roll-out steps suffer from more error accumulation in their forecasts, cf. Fig. 3. Conversely, models trained with large time steps accumulate less error but start with large amounts of error from the first step.
GENERATIVE MODELING FOR 3D TURBULENT FLOWS
We have that forecasting 3D turbulent flows with their chaotic nature and many small-scale, short-lived features is challenging, in particular for autoregressive models. Given their chaotic nature and sensitivity infinitesimal perturbations. turbulent flows can be understood as stochastic processes (Pope. 2000), i.c. the solution trajectory {(u, P)TTER++ given some initial conditions follows the distribution p( ((, p)ERERR u4: Po). Yet, many applications do not require an exact trajectory but rather selection of turbulent flow states that represent the set of all possible flows well. Thus, we propose to circumvent the roll-out dilemma autoregressive models by modeling the marginal distribution E. [p((1,, p). uo. po, f)] p((u,p) Po) directly, capturing the distribution of turbulent flows with a generative model.
4.1 GENERATIVE TURBULENCE SIMULATION
Consider simulation domain Ω C R3 with boundary conditions B and initial conditions ()) : Ω + R3, (0 Ω + R for velocity and pressure, where B is chosen such that the flow becomes turbulent. The fields Usol IR+ Ω - R& and Paol I Ω + R solving the Navier- Stokes-Eq. (1) describe the resulting, turbulent flow. In the following, we define X = (u p) to denote the concatenation of velocity and pressure. Now we define the probability distribution over all attained flow states after time I,
P2 X
X (0) B)
lim EreRe [6x,a((t)]
(2)
T
If the initial state (0) is not turbulent, there is transition phase of length turb depending on the domain geometry and (0) until turbulence is fully developed. Beyond this point, the distribution
becomes, remarkably, independent of the initial state, i.e. P2taga (X X B) P2 (X B), because of the often assumed ergodicity of turbulence (Galanti & Tsinober, 2004), which says that a turbulent flow will visit all possible flow states. Based this insight, define the task of generative turbulence simulation as finding model Pø such that
Po X
B3)
P2tuub
X B).
Importantly, such a model does not have access to an initial turbulent state
P2teub (X B))
4.2 DISCRETIZATION
We discretize the simulation domain 52 into a regular grid of cells Ω4 = [W] [M] x [D] where [N] denotes the set of integers 1 through N. In the following, we will denote cell indices (i, j. k) E Qr as i. Each cell has one of K types, T; [K], marking cell as, for example, wall or inlet, which discretizes the location and type of boundary conditions B. To discretize solution fields (t) and
(t) at time we define the tensor X; (Maat (1, axi) || Pool (1, æ:)) E I where 2; is location of
the center of cell i.
See Appendix for more details on how we construct X and T.
4.3 GENERATIVE MODEL
We base our model TurbDiff denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020; et al., 2015), which have been adapted to other domains (Kollovich et al., 2023; Lüdke al., 2023). DDPMs are latent-variable models that learn generative model Po reverse fixed
4

---

Published as a conference paper at ICLR 2024
Markov chain q(z,, 1n-1), which gradually transforms data sample zy over N steps into Gaussian noise until no information remains, i.c. q('N zo) = =N I). Training occurs by minimizing the KL-divergence between Po - x..)
and q [D x,,,, that Po approximates the true pos- terior. The reverse process then constitutes a powerful sampling mechanism that transforms sampled noise æ (O, I) into data sample by iteratively sampling from Po (#n-1 x.n) and, the same time, formally defines our generative model as Po (xo) = p((N)) I== Po x .N.
We adapt DDPM and include Dirichlet boundary conditions to guide the diffusion process similar to the inpainting method proposed by (Lugmayr et al., 2022). Dirichlet conditions fix the value of velocity or pressure a specific location a given value, e.g. u = the walls or the given inflow velocity at the inlet. Therefore, we sample the values at these locations from fixed distributions instead of the learned distribution Pg- This ensures that the values at the boundaries converge towards their prescribed values throughout the sampling process, guiding the sample whole towards the data distribution.
In particular, we begin by defining mask Mi = -T free that selects all cells free cells, i.e. those that have no boundary conditions or any other kind of special role. Let Xo ~ P>taag (X B) and we want to sample from its distribution by transforming a sample XN ~ p(0, I). Then, we define our model as
[Po(X 1-1 X,,T), T) (9(X,-1 Xo, Xn);
1f
p (XX, 12
X,, T);
otherwise.
In effect, we denoise the input only in cells that contain actual velocity and pressure data. For other cells, we sample their values from the true posterior q( q(x.-1 Xo. X²). which is possible because for these cells Xo.i is predetermined from the boundary conditions and therefore always available. Since we also use the same case distinction during training, our model is only trained on actual data cells and not on boundary values.
4.4 PARAMETRIZING POSTERIOR AND DIFFUSION PROCESS
Since we discretize the data domain diren-dimensional, regular grid, we base our model archi- tecture on a U-Net (Ronneberger et al., 2015) extended to 3D (Çiçek et al., 2016). After 4 levels of downsampling, we apply transformer (Vaswani et al., 2017) with pre-group normalization (Xiong et al., 2020; Wu & He, 2020) to ensure global communication within the model across the whole simulation domain. Within the transformer, we flash attention to reduce the memory requirements of the transformer from (n²) to (()) (Dao et al., 2022).
For the forward diffusion process, choose N 500 steps and noise schedule that scales the noise so that the log-signal-to-noise -to-noise ratio of the noise grows linearly (Kingma et al., 2021). This schedule ensures that several diffusion steps are spent in the high-signal regime let the model denoise small details. but at the same time, reaches large enough noise levels to ensure that q xo) = IO, despite the long tails of velocity and pressure values in turbulent flows.
For more details on the components and hyperparameters, see Appendix D.
5 RELATED WORK
The works of (Drygala et al., 2022) and (Drygala et al., 2023) are most closely related to our research. They prove that generative adversarial networks (GANs) sample from the correct distribution for ergodic systems such as turbulent flows (Galanti & Tsinober, 2004) and train GAN to generate 2D slices of high-resolution 3D turbulent flow data of the velocity field orthogonal to the slice. Compared to their work, we allow arbitrary geometries by encoding all types of boundary conditions via learned cell type embeddings and we propose a metric that measures the quality of samples as a whole, whereas Drygala et al. examine particular sections of their samples individually.
Several other works have been published on machine learning for 3D fluid flows. (Xie et al., 2017) super-resolve low-resolution turbulent flows with a GAN in temporally coherent manner. (Matsuo et al., 2021) learn to reconstruct 3D flows from 2D slices for channel flows around a cylinder. (Stachenfeld et al., 2022) propose model to forecast non-forced, decaying turbulence at coarser resolution to achieve speedups over numerical solvers. (Kim et al., 2019) predict velocity fields one step ahead for 3D flows in computer graphics applications. They report that they need to train with
5

---

Published as a conference paper at ICLR 2024
small time step for their latent-space autoregressive model track small details and generate accurate roll- It is important to note that these works refine, expand, or evolve turbulent flow fields given as imput to the model. In contrast, sample turbulent flows from scratch based solely on the domain geometry and boundary conditions.
EXPERIMENTS
6.1 DATASET
To evaluate the viability of generative modeling for 3D turbu- lent flows, have generated a challenging new dataset. The dataset consists of 45 simulations of an incompressible flow through 0.4 m x 0.1 x 0.1 channel at 20ms=!, ! which results in Reynold's number of Re 2 x 10 well into the turbulent regime. The channel is discretized into 192 48 x 48 regular grid cells, which balances the spatial resolution to resolve small-scale behavior adequately with memory and storage requirements.
U
For each simulation, we place a distinct object into the flow, each of which causes characteristic flow pattern. Fig. 4 shows a subset of the objects and how the object's shape in- fluences the flow. We split the simulations into 27 for training, 9 for validation, and 9 for testing to verify that models can generalize to unseen objects in the flow.
Figure 4: A subset of the objects in our turbulent flow dataset with iso- surfaces of the vorticity magnitude.
We run all simulations for 0.5% s of physical time with Open- FOAM (Weller et al., 1998), industrial-grade CFD solver, in large eddy simulation (LES) mode and save flow state every 0.1 ms. The resulting dataset contains 5000 flow states for each simulation and requires 2.2 TB of storage after preprocessing.
See Appendix A for more details on our datasets and data generation pipeline.
6.2 BASELINES
In our experiments, we compare our generative model against two autoregressive models specifically developed for turbulent flows, Turbulent Flow Net (TF-Net) and DilResNet. TF-Net takes inspiration from hybrid RANS-LES modeling of turbulent flows and learns spatial and temporal filtering with an architecture of multiple U-Nets (Wang et al., 2020). DilResNet a general-purpose purpose architecture that combines stacks of dilated convolutions with residual connections into expressive convolutional model with large receptive field (Stachenfeld et al., 2022). Both models take a set of recent observations X (t-(h-1):0) and map it to prediction of the system state X ((+1) step ahead.
For both forecasting models, we compare two different ways to turn them into generative models. For the first, we take a sample (()) Ditrach (X T) and then apply the model autoregressively 22 times to generate new sample X (1+22) We chose 22 steps because that is the distance between two steps in our dataset which they become approximately uncorrelated. Therefore, we consider the model to have transformed (*) into new sample X (1-22), Obviously, the model could not replace the numerical solver in this way since we need the solver to generate the input to the generative model in the first place, but it serves us very strong baseline because it already starts with sample from the distribution that we want to sample from. We denote these models by DilResNet-22 and TF-Net-22, respectively.
For the second way to turn our baselines into generative models, we add a small amount of noise to the initial state of the numerical solver and then roll out the models for 200 steps. We chose 200 steps because that is the minimum number of steps the flow needs to develop turbulence for any simulation in our dataset. The small amount of noise leads independent samples because turbulent flows are chaotic and therefore sensitive to initial conditions, i.c. even an infinitesimal change initial conditions will produce completely different trajectory. Of course, these generative variants need

---

Published as a conference paper at ICLR 2024
far more roll-out steps, but it completely forgoes the use of a numerical solver, thereby offering the largest speed-up. We denote these models by DilResNet-init and Te-Net-init, respectively.
We list hyperparameters for these baselines in Appendix C.
6.3 METRICS
Comparing the quality of turbulent flow samples is a non-trivial task. Unlike in the one-step-aleee -ahedd prediction of autoregressive models, we do not have a ground truth state but must derive new metries to quantify the difference between the samples of distributions. We quantify the sample quality of TurbDiff and the baselines by their Wasserstein-2 (W2) distance to samples from the numerical solver. The W2 distance between two sets of samples is the minimal average distance achievable for any matching between the two sample sets w.r.t. some underlying distance d. Formally, let μ4, v/ be two empirical distributions over the sample sets we want to compare and I' be the space of all possible joint distributions that have μ and v respectively as marginal distributions. Then the W2 distance between μ and y is defined
((2))) $
W2(u,v) min (pT()))))
In addition to being the basis of the ubiquitous Fréchet inception distance (FII) in generative image modeling (Heusel et al., 2017), W2 also used in video (Unterthiner et al., 2019) and molecule generation (Preuer et al., 2018). However, these metrics rely on learned embeddings from pre-trained models that are widely accepted to capture the important characteristics of individual samples.
It remains specify the underlying distance function d. The Euclidean distance would be an obvious but lousy choice for curious reason. Because of the chaotic nature and strong, small-scale fluctuations of turbulent flows, it is highly unlikely for two independent samples of turbulent flow to be close under this distance. In fact, in our dataset, the smooth mean flow is closer to all samples than any two uncorrelated turbulent flow samples This, however, would be antithetical to our goal to define a metric that rewards turbulent samples and discourages smoothing and mean flow samples.
each other with respect the Euclidean distance.
Turbulent kinetic energy (TKE) Our first choice of distance compares samples by their TKE spectra. The TKE each cell is defined as the quadratic velocity deviation from the mean velocity, F = 1| üū 112. To get the spectrum, i.e. the spectral intensity as a function of the wavenumber, we take the 3D fast Fourier transform (FFT) of E over a cuboid subset of the domain and integrate the squared frequency magnitude spherically over all 3D wavenumbers [[ ky kz)]l = k:
F(K) = 1 Joca F(x))e ((z))e-(kk))- i(ku)) m dæ dv
(6)
Then we define the distance between two samples as the L2 distance between their log energy spectra, d(x (2)) = log Exa - log E Excell2.
We denote this distance by W2T
Distributional distance TKE spectra capture global patterns in the flow velocity by measuring the energy contained in the various spatial scales. However, the energy spectra neglect the pressure and are invariant wrong positioning or orientations of flow patterns. To remedy this shortcoming, we introduce a second metric that measures if two samples contain similar velocity and pressure values similar locations.
If we regard a turbulent flow as a stochastic process as is common, the velocities and pressures in each cell follow a marginal distribution p(u,, Pi). Then, we can call two samples close if their values in each cell follow similar marginal distributions. However, comparing cell values pointwise directly would make it highly unlikely for two samples from the same flow to be close, as explained above. Instead, we propose group cells with similar marginal distributions together and interpret the values in each cell group as samples from the same marginal. Then we compare these sample sets by their W2 distance and average over all cell groups
7

---

Published as a conference paper at ICLR 2024
(a) Samples generated with TurbDiff
(b) Data sample for the same object
Figure 5: Example flows for a test object from our dataset showing isosurfaces of the vorticity w.
Table TurbDiff generates samples from scratch similar quality in W2, & W2. distance to samples generated by the strongest regression baseline which takes data sample as initial state. In addition, our model much faster to evaluate because it eschews the 10 minutes needed by the numerical solver generate a realistic initial state for Net-esNet-22. For models marked with (*), we have excluded runs that diverged due to error accumulation.
W 2,TKE
W2R
RMSE a max TKE
Runtime [s]
TF-Net-init
1.834
TF-Net-22
189
493
108
602 + 0.23
DilResNee-init
60 47
4.6 10
4
12.82
DilResNet-22
2.15 + 0.06
1.240 + 0.001
2.5 1.2
602 + 1.58
TurbDiff (ours)
0.4
1.38 + 0.04
5.9 1.8
20.63
In particular, for each simulation, we approximate the marginal velocity distribution p(u) of each cell with an isotropic Gaussian ((4, diag(o?))) Then, we cluster the cells into regions 'R with similar marginal distributions via k-means with the W2 distance between isotropic Gaussians, which has closed-form solution. Finally, we define our distance between two samples X (1) and X (2) as
d( d(x()),
Σ
R
W
((())
(1) U
(8)
R'
R
where ( R
E R is the set of normalized velocity, vorticity and pressure at each cell in homogeneous region. This metric includes the generated pressure well as the estimated vorticity, which means that it also takes into account the velocities in neighboring cells. All components are normalized by their standard deviation over the training set to account for their different scales. We choose the W2 distance with the 2-norm to compare the sample sets ( to emphasize modeling the extreme values which are important to represent the characteristic intermittency of turbulent flows, i.e. the heavy tails of their velocity and vorticity distributions (Jiménez, 2006).
In addition to velocity and pressure, also include the estimated vorticity wi because vorticity a defining aspect of turbulence, as we have seen in Section 2. Furthermore, since we estimate the vorticity from the velocity via finite differences, including it in the distance makes it more discriminative because it includes distance from the multi- poinit-point marginal velocity distribution of cell i and its neighbors.
In our results, we denote this distance by W2,R. See Appendix E for more details on how exactly we compute each metric.
8

---

Published as a conference paper at ICLR 2024
6.4 SAMPLE QUALITY
We train all models on the 27 training simulations in our dataset and then evaluate them unseen test simulations with unseen geometries. For each simulation, we generate 16 samples and evaluate our sample metrics against 16 true samples chosen equidistantly from the second half of the simulation. Choosing the data samples from the second half of each simulation ensures that they represent fully-developed turbulence, and picking them equidistantly means that the samples are completely uncorrelated. Figs, 5a and 5b show samples generated by TurbDiff and from the dataset for the same object. See Table I for the complete results.
First, we observe that our generative model generates samples very close to the data distribution in terms of W2,TKE and W2.R distance. Only the DilResNet-22 baseline achieves better scores. However, this baseline needs a true sample from our target distribution P2truc (X T) generated via a numerical solver as initial state and then evolves it forward in time for 22 steps or 2.2 ms. As trained the regression baselines the dynamics of all scales accurately, see Section 2, and 22 roll-out steps still have tolerable error accumulation, the baseline is able preserve the turbulent characteristics of its input state and achieve slightly better scores. If we evaluate the baselines in the more realistic setting denoted by the iiii suffix without any support from numerical solver and generate samples by unrolling from the pre-turbulent initial state of the numerical solver, the increased eπror accumulation of the longer roll-out decreases the sample quality immensely. However, this is the only setting that achieves the original goal of surrogate models to replace costly numerical solvers.
the smallest timestep in data to allow them to resolve
These results support our observation from Sections 3 and that autoregressive models are severely challenged by the long roll-outs required to fully replace numerical solvers for 3D turbulence and that a generative approach sidesteps these difficulties elegantly.
Runtime Regarding runtime, TurbDiff outperforms the numerical solver by a factor of 30, as reported in Table 1. The solver needs roughly 10 minutes to transition from its initial state to fully-developed turbulence, whereas our model can generate a sample in about 20 seconds. While both baselines can be evaluated quicker if we consider just the model itself, the complete pipeline as a surrogate model requires the generation of realistic initial state for the model, which takes 10 minutes to generate with a numerical solver. In the iinit setting, the models avoid this additional runtime but suffer from significant error accumulation.
All model times represent the minimum achieved time on an NVIDIA A100, and we measured the solver time with 16-core parallelism on an Intel Xeon E5-2630.
Estimating turbulence properties In applications, samples from a generative turbulence simulator could be used measure, for example, the effect of an object's 's shape on the turbulence patterns causes, e.g. their size or energy. As an example, we estimate how far behind the object in our flow simulations the mean TKE takes its maximum, i.e. where the turbulent vortices caused by the object have built up the most kinetic energy. This location depends strongly the shape of the object and requires accurate samples to evaluate. The root-mean-Se -eaa---uuarr----eeeeeee our samples reported in Table I is 5.9 cells or just 1.2 cm in channel of 40 cm in length. This is significantly lower than the variability of the true location and provides an actual useable signal to practitioners compared to the baselines in the full-surrogate setting.
(RMSE) of the estimate from
7 CONCLUSION
We have shown that generative modeling lets us access the full acceleration potential of neural surrogates for 3D turbulent flows by circumventing the need for long roll-outs or initial states from numerical solvers. Our dataset is an important building block for further research into surrogate models for engineering applications that challenges models to learn the turbulent dynamics of high-velocity flows and how objects influence the resulting vortex structures. Our new metrics for generative turbulence models measure succinetly if a model's samples follow the expected velocity and pressure distributions as well as if the turbulent kinetic energy is distributed correctly across spatial scales.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was funded by the Bavarian State Ministry for Science and the Arts within the framework of the Geothermal Alliance Bavaria project.
REFERENCES
Radovan Bast. Numgrid: Numerical integration grid for molecules, January 2021.
Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, and Stephan Günne- mann. Neural Flows: Efficient Alternative to Neural ODEs. In Neural Information Processing Systems, 2021. doi: 10.485500an2:..22110.11 13040.
Florent Bonnet, Ahmed Jocelyn Mazari, Paola Cinnella, and Patrick Gallinari. AirRRANS: High Fidelity Computational Fluid Dynamies Dataset for Approximating Reynolds-Averaged Navier- Stokes Solutions. In Neural Information Processing Systems, 2022. doi: 10.48550/arXiv.2212. 07564.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message Passing Neural PDE Solvers. In International Conference on Learning Representations, 2022.
John Burkardt. LEBEDEV RULE: Quadrature rules for the unit sphere, 2010.
LEEBDDEVRRULE:
Maurizio Carbone and Andrew D. Bragg. Is vortex stretching the main cause of the turbulent energy cascade? Journal of Fluid Mechanics, 883:R2, January 2020. ISSN 0022-1120, 1469-7645. doi:
10.1017/jfm.21999999
Özgün Çięek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U-Net:
Learning Dense Volumetric Segmentation from Sparse Annotation. In Sebastien Ourselin, Leo Joskowicz, Mert R. Sabuneu, Gozde Unal, and William Wells (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2016, Lecture Notes in Computer Science, pp. 424 432, Cham, 2016. Springer International Publishing. ISBN 978- 978-3-319-46723-8 doi: 10. 10017/978-3-319-46722
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with I0-Awareness. In Neural Information Processing Systems, 2022. doi: 10445500aaXx..2255..333
Claudia Drygala, Benjamin Winhart, Francesca di Mare, and Hanno Gottschalk. Generative Modeling of Turbulence. Physics of Fluids, 34(3):035114, March 2022. ISSN 1070-6631, 1089-7666. doi: 10.1063/5.0082562
Claudia Drygala, Francesca di Mare, and Hanno Gottschalk. Generalization capabilities of conditional GAN for turbulent flow under changes geometry, February 2023.
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. In Proceedings of the 37th International Conference Machine Learning, pp. 3165-3176. PMLR, November 2020.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. POT: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021.
B. Galanti and A. Tsinober. Is turbulence ergodic? Physics Letters 330(3):173-180, September 2004. ISSN 0375-9601. doi: w...01/...pyyllta.20000000
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Graph Neural Networks for Molecules. In Neural Information Processing Systems, volume 34, pp. 6790-6802. Curran Associates, Inc., 2021.
Universal Directional
10

---

Published as a conference paper at ICLR 2024
Jorge Gomes, Emanuele Bagnaschi, Isabel Campos, Mario David, Luís Alves, João Martins, João Pina, Alvaro Sópez-Gareiaa and Pablo Orviz. Enabling rootless Limux Containers in multi-user environments: The udocker tool. Computer Physics Communications, 232:84 97, November 2018.
ISSN 00104655. doi: 10. 10. wttp://ccp.....005555
Jayesh K. Gupta and Johannes Brandstetter. Towards Mulli-spatiooeeporaa-acaca Generalized PDE Modeling, September 2022.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 385(78255:357-366 September 2020. doi:
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Neural Information Processing Systems, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Neural Information Processing Systems, 2020. doi: 10:48550/arX1:22006.11 1239..
Masanobu Horie and Naoto Mitsume. Desiid-Embeddde Neural Networks: Graph Neural PDE
Solvers with Mixed Boundary Conditions. In Neural Information Processing Systems, 2022.
J. D. Hunter, Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007. doi: 10. 109/CSS..000..55.
Steeven Janny, Aurélien Béneteau, Madiha Nadri, Julie Digne, Nicolas Thome, and Christian Wolf. EAGLE: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformers, In International Conference on Learning Representations, 2023. doi: 10448500arxX22230000000...
J. Jiménez. Intermittency in Turbulence. In Jean-Pierree Françoise, Gregory L. Naber, and Tsou Sheung Tsun (eds.), Encyclopedia of Mathematical Physics, pp. 144-151. Academic Press, Oxford, January 2006. ISBN 978-0- 12-512666-3. doi: 10.1016/B0- 001511166666006666
Perry I.. Johnson. On the role of vorticity stretching and strain self-amplification in the turbulence energy cascade. Journal of Fluid Mechanics, 922:A3, September 2021. ISSN 0022-1 1120-, 1469- 7645. doi: 00.1017/jfm.2021.4999
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep Fluids: Generative Network for Parameterized Fluid Simulations. Computer Graphics Forum, 38(2):59-70, 2019, ISSN 1467-8659. doi: 10.111 111111//cf.13619
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. In Neural Information Processing Systems, 2021.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic. Kyle Kelley, Jessica Hamrick. Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter development team. Jupyter Notebooks publishing format for reproducible computational workflows. In Fernando Loizides and Birgit Scmidt (eds.), Positioning and Power in Academic Publishing: Players, Agents and Agendas, pp. 87-90, Netherlands, 2016. IOS Press.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang. Michael P. Brenner, and Stephan Hoyer. Machine learning-acceleratenn accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021. ISSN (007784244, 1091-6490. doi: 10. w1w.ppan.....11..... 18.
11

---

Published as a conference paper at ICLR 2024
Marcel Kollovich, Abdul Fatir Ansari, Michael and Yuyang Wang. Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting, In Neural Information Processing Systems. arXiv, 2023, doi: 10.48550/ arXiv.2307.1 1494.
Jasper Zschiegner, Hao Wang,
V. Lebedev and D. Laikov. A Quadrature Formula for the Sphere of the 131st Algebraic Order of Accuracy. Doklady Mathematics, 1999.
Yi Li, Eric Perlman, Minping Wan, Yunke Yang, Charles Meneveau, Randal Burns, Shiyi Chen, Alexander Szalay, and Gregory Eyink. A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. Journal of Turbulence, 9:N31, January 2008. doi: 10. 00.10000116652 46652408003366388,
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations. In International Conference on Learning Representations. arXiv, 2021. doi: 10.48550/ arXiv.2010.08895.
Marten Lienen and Stephan Günnemann. Learning the Dynamics of Physical Systems from Sparse Observations with Finite Element Networks. In International Conference on Learning Representa- tions, 2022.
Adrián Lozano- -Drrnn and Javier Jiménez. Time-resolved evolution of coherent structures in turbulent channels: Characterization of eddies and cascades. Journal of Fluid Mechanics, 759:432-471, November 2014. ISSN 0022- 1120, 1469-7645.
doi: 10.1017//fm..2144.57.
David Lüdke, Marin Biloš, Oleksandr Shchur, Marten Lienen, and Stephan Günnemann. Add and Thin: Diffusion for Temporal Point Processes. In Neural Information Processing Systems. arXiv, 2023. doi: 10.445500/rXiv.2311
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In /EEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
Jean Mathieu and Julian Scott. Introduction to Turbulent Flow. Cambridge University Press, Cambridge, 2000. doi: 10.
Mitsuaki Matsuo, Taichi Nakamura, Masaki Morimoto, Kai Fukami, and Koji Fukagata. Supervised convolutional network for three-dimensional fluid data reconstruction from sectional flow fields with adaptive super-resolution assistance, March 2021.
Alex Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. In Interna- lional Conference on Machine Learning, 2021. doi: (0-485500axXxx22122.9922..
Octavi Obiols-Sales Abhinav Vishnu, Nicholas Malaya, and Aparna Chandramowlishwaran. CFD- Net: A deep learning-based accelerator for fluid simulations. In International Conference Supercomputing, pp. 1-12, June 2020. doi: 10.1 (0.1145/33277773339777
Karl Otness, Arvi Gjoka, Joan Bruna, Daniele Panozzo, Benjamin Peherstorfer, Teseo Schneider, and Denis Zorin. An Extensible Benchmark Suite for Learning to Simulate Physical Systems. In Neural Information Processing Systems, 2021. doi: 10.48500karxxv2200..77999
Nicholas T. Ouellette. Turbulence two dimensions. Physics Today, 65(5):68-88) 69, May 2012. ISSN 0031-9228, doi: 10.1063/PT.3.1570.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, PerghhhPrfooacce Deep Learning Library, December 2019.
12

---

Published as a conference paper at ICLR 2024
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzaler and Peter W. Battaglia. Learning Mesh- Based Simulation with Graph Networks. In International Conference on Learning Representations, number urXiv:2010.03409, 2021.
Stephen B. Pope. Turbulent Flows. Cambridge University Press, 2000.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Günter Klambauer. Fri Frh'echet ChemNet Distance: A metric for generative models for molecules drug discovery, August 2018.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ramadhan, and Alan Edelman. Universal Differential Equations for Scientific Machine Learning, November 2021.
M. Raissi, Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 00219991. doi: 10.1016/ j.jep.2018.10004 10.044.
István Z. Reguly, Gihan R. Mudalige, Carlo Bertolli, Michael Giles, Adam Betts, Paul H.J. Kelly, and David Radford. Acceleration of Full-Scale Industrial CFD Application with OP2. IEEE Transactions on Parallel and Distributed Systems, 1765):1265-1278, May 2016. ISSN 1558-2183. doi: 1010///DD..201.245555
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein- notation. In International Conference Learning Representations, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10674-10685, June 2022. doi: 10.11099CVPS52688.0222 01042.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention, Lecture Notes in Computer Science, pp. 234 241, Cham, 2015. Springer International Publishing. ISBN 978-3-319- 24574-4. doi: 10. 111100/978-3311-245777777
Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Networks. In Proceedings of the 38th International Conference on Machine Learning, pp. 9323- 9332. PMLR, July 2021.
Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for fideliiyy flow field reconstruction. Journal of Computational Physics, 478, April 2023. ISSN 0021-9991. doi: 10. jcp.......
Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamies. In International Conference on Machine Learning, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference Learning Representations, January January 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models, March 2023.
Kimberly Stachenfeld, Drummond B. Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez. Gonzalezz Learned Coarse Models for Efficient Turbulence Simulation. In International Conference Learning Representations, 2022.
Bane Sullivan and Alexander Kaszynski. Py Vista: 3D plotting and mesh analysis through stream- lined interface for the Visualization Toolkit (VTK), Journal of Open Source Software, 4(37): 1450, May 2019. doi: 10.21 10.21105/joss.0144 01450..
13

---

Published as a conference paper at ICLR 2024
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: An Extensive Benchmark for Scientific Machine Learning. In Neural Information Processing Systems, 2022.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards Accurate Generative Models of Video: A New Metric & Challenges, March 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Neural Information Processing Systems, 2017.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Ker,, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261 2020. doi:
Rui Wang. Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards Physics- informed Deep Learning for Turbulent Flow Prediction. In SIGKDD International Conference Knowledge Discovery and Data Mining, 2020.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
H. G. Weller, G. Tabor, H. Jasak, and C. Fureby. A tensorial approach to computational continuum mechanies using object-oriented techniques. Computer Physics, 12(6):620-631, November 1998. ISSN 0894-1866. doi: 10.1063/1.16874.
Yuxin Wu and Kaiming He. Group Normalization. International Journal of Computer Vision, 128(3): 742-755, March 2020. ISSN 1573-1405. doi: 10. 10.1000//11223-0199 198-w. w.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A Temporally Coherent, Volumet- ric GAN for Super-resolution Fluid Flow. ACM Transactions on Graphics, 36(4):1-14, August 2017. ISSN 0730-0301, 1557-7368. doi: 10.1 02.114//377299.3073666
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Yan Liu. On Layer Normalization the Transformer Architec- ture. In International Conference on Machine Learning, 2020. doi: 10448550arxi..2222.47755
Omry Yadan. Hydra- A framework for elegantly configuring complex applications. Github, 2019.
Gefan Yang and Stefan Sommer. A Denoising Diffusion Model for Fluid Field Prediction, January 2023.
Qingqing Zhao, David B. Lindell, and Gordon Wetzstein. Learning to Solve PDE-constrained Inverse Problems with Graph Networks, In International Conference on Machine Learning, pp. 26895-26910. PMLR, June 2022.
14

---

Published as a conference paper at ICLR 2024
A DATASET DETAILS
Our dataset consists of 45 simulations, each with a separate object in the flow that causes distinct turbulence patterns. See Fig. for visualization of all objects. Each simulation is in channel of size 0.4 x0.1 x0.1 cm, which is discretized into 192x48x48 cells. The inflow velocity is 20 meters per second, and we simulate the flow for 0.5 with a snapshot saved every 0.1 ms. This results in 5,000 flow states in total per simulation.
For the simulations, we use OpenFOAM, an industrial grade CFD solver, LES mode. LES is an established method to produce realistic turbulent flow patterns in contrast to simpler Reynold's s- averaged Navier-Stokes methods. resolves the vortices above cut-off scale and uses a simplified model represent the effect of sub-grid scales so that the flow does not have to be spatially resolved all the way to the Kolmogorov scale, which is infeasible computationally as well as spatially in most situations and also for our dataset.
We choose a low viscosity of v = 1 x 10-5, which leads to an average Reynold's number of | 2 x 10a and guarantees that fully-developed turbulence will occur.
Re
Turbulent flow simulations need significant time to build up the turbulent flow state from the smooth initial state, i.c. initiate turbulent structures at all scales and break any symmetry that the simulation geometry has. The exact time depends on the boundary and initial conditions. the flow. For our dataset, this initial transition lasts for roughly 22 ms to 45 ms in simulation time, which corresponds to about 2,000 to 4,000 solver steps or 10 to 20 minutes of solver time on 16 CPU cores.
also the object in
For training our generative model, we discard the first 25 ms so that it only learns the distribution of turbulent states and not the distribution of the initial transition phase. For regression models, we train on the full sequence to ensure they also learn the initial transition for the full roll-out experiments.
B DATA PREPROCESSING
OpenFOAM generates data within the simula- tion cells. However, for the baselines and our model, which rely on convolutions, we have to embed the data into 3D grids. To also repre- sent the boundary, inlet, outlet, and inside of the geometries, we choose the following repre- sentation. The out-of-domain cells, e.g. in the objects, are represented with zeros. The outer boundary around the whole domain represents the Dirichlet boundary conditions and bound- ary layer types such as wall or inlet. During the construction of our grid-base representation, we keep track of the type of each cell to construct the cell type tensor T. See Fig. for an illustration.
Inlet
Outlet
Wall
1O
Outside
Channel
Figure 7: An illustrative 2D slice of our grid-base representation of the simulation data.
C BASELINE DETAILS
We implemented the two baselines TF-Netand DilResNetas described the original papers and replaced all 2D convolutions with 3D convolutions. Further, to stay comparable include the cell type information T into the model by learning 8-dimensional embedding vectors c(T) and concatenating them to the flow state (X c(T)). We choose all hyperparameters (Stachenfeld et al., 2022), i.e. DilResNet takes only the most recent step X and predicts the delta to the next step X = x - (() and uses dilated convolutional blocks of 7 layers with residual connections. TF-Net takes the 6 most recent (t-3it) and predicts the next state X (++1) directly.
model, we
in
D MODEL DETAILS
DDPM The DDPM framework described in Section 4.3 has two hyperparameters, the number of steps N and the noise schedule 3, that controls how quickly the data samples are transformed into
15

---

Published as a conference paper at ICLR 2024
Gaussian noise. We chose N = 500 and the log-linear signal-to- to-ioise ratio (SNR) schedule from (Kingma et al., 2021) that scales S, such that the log SNg of the data falls linearly from 1 x 10 to I x 10 We have found that this schedule is especially suitable for turbulence data, because it leaves very little signal in the noisy training samples for large N. This counteracts the large extreme values in turbulence data and ensures that the distribution q(=N æo) converges sufficiently closely to (0, I) to make the reverse process work well. See (Ho et al., 2020) for the full derivation of the DDPM framework and definition of Bn.
Architecture We need to parametrize (T x..) such that it produces an estimate of q(an-1 (o, ). ) In particular, we use U-Net with 4 levels of downsampling and apply a trans- former at the coarsest scale. First, we encode the noise level n into a 32 dimensional vector with a sinusoidal positional encoding (Vaswani et al., 2017) and then apply a 2-layer MLP to it. Then, each cell's velocity and pressure p features are concatenated with a 4-dimensional, learned cell type embedding and mapped linearly onto 64 dimensional latent vector. Both, the latent feature vector tensor and the noise level embedding are then passed into U-Net. The U-Net downsamples the spatial resolution 4 times by a factor of 2, applying a ResNet style block at each level (He et al., 2015). After the final downsampling step, apply full attention between all downsampled cells. Finally, the resulting representation is upsampled again 4 times to the original resolution with an equivalent ResNet-style block applied each level the upsampled data concatenated with skip connection from the downsampling, The final feature tensor is then mapped from the 64-dimensional latent space the (u. p) data space via final ResNet-style block.
We trained TurbDiff from 3 different seeds for 10 epochs with a batch size of 6. The optimizer is RAdam with a learning rate of x 10 and and exponential learning rate decay to 1 x 10 over those epochs.
Training
Normalization The data for all models is normalized with the maximum norm of u for the velocity and the maximum of [p] for pressure. We choose to normalize by the maxima of each feature to handle the long tails of the pressure and velocity distributions, which show values of up to 10 away from the mean. Lastly, for our diffusion model, we do not apply noise the cell-type embeddings but concatenate them without noise at each diffusion step.
E METRICS DETAILS
The TKE computes an FFT over the flow field. Since we integrate the spectral intensity over all spatial frequencies of the same magnitude, we would incur boundary effects we take the FFT over a rectangular domain because higher frequencies could be computed along the long dimension of the domain than the shorter ones. To avoid this problem, we compute the the TKE distance separately over 48x48x48 cubes behind the object, in the middle of the flow and at the end of the channel and then compute the individual distances
dTKEE X,, Xb)
derRE X Xa,block Xt,hlocks i)..
(9)
We estimate the L2 distance between two log TKE spectra Ex and (k)
23 log E, - log Es 12 1 (log (kk - log (k)) di
(10)
via Gauss-Legendre integration to get a highly accurate estimate. To evaluate log F((k) for non-integer k (kgy ky, k,), we interpolate the values linearly from the discrete FFT onto the points k. This let's us get a highly accurate estimate of the total spectral intensity at a frequency i estimating the spherical integral via the Lebedev integration method (Lebedev & Laikov, 1999).
For the distributional distance, we need to take care that no region of cells with approximately the same marginal distribution has too many cells, because compute the exact W2 distance within each region R and the runtime of W, scales with (nn3) where n is the number of cells within each region. To avoid this problem, we split regions larger than 512 cells into groups of at most 512 cells
16

---

Published as a conference paper at ICLR 2024
arbitrarily. We can split them arbitrarily because all the cells in the region behave in very similar ways anyway.
F LIMITATIONS
Performance While we have shown that our approach reduces the time to first sample significantly compared to CFD solver, a solution based on 3D grids will necessarily scale cubically in resolution per dimension. This could be alleviated by representing the data in different ways such as frequencies (Li et al., 2021) hierarchical models such as the U-Net could be parallelized over multiple devices. Another aspect that generative diffusion is known to produce high-quality samples but also for being expensive. This could be improved by faster sampling routines (Nichol & Dhariwal, 2021; Rombach et al., 2022; Song et al.. 2021) or replacing the sampling routine altogether with model (Biloš al., 2021; Song et al., 2023).
general function spaces (Lienen & Günnemann, 2022). Alternatively,
Equivariance Since TurbDiff is based on a basic 3D U-Net, it does not exploit any the symmetries that turbulence physical process obeys, which could improve sample efficiency and generalization (Pope, 2000). This includes translation, rotation and reflection equivariance (Satorras et al., 2021; Finzi et al., 2020; Weiler al., 2018; Gasteiger et al., 2021) but also the turbulence-specifie Reynolds number similarity.
Geometry With our grid embedding, we can represent any geometry given that the resolution is fine enough, however only to Oth order approximation, i.e. as step functions due to the grid structure. Since sharp corners have strong effect on turbulent flows, sample quality for smooth geometries would likely benefit from 1t order approximations with meshes (Lienen & Günnemann, 2022) or even smoother representations based on signed distance functions (Park et al., 2019). Mesh-based models could furthermore posses certain invariance with respect to the mesh density as was demonstrated by (Lienen Günnemann, 2022).
17

---

Published as a conference paper at ICLR 2024
step higher
slep lower
comer
corners sam
cornery agen
arighbor
pillar
wide pallar
offset- pillar
pillar
opp-pillar
double
wide teath
offiet teeth
elhow
wide elbow
elbow-asymm
elbow--nug
open-elbow
donut
tr
U
H
disjoint- T
minus
H
T
square
square-large
square-offset
2x2-large
3x.3
3x33-ime
proct
crossaaide
cosss
H
platform
step-low
high-platform
step-high
altar
Figure 6: All objects in our dataset.
18


Reviews:
[{"review_id": "jcMr55NA2D", "reviewer": "Reviewer_DVzo", "summary": "The paper proposes a novel modeling method of turbulence flow based on generative models. It consists in learning the distribution of turbulent flow states without the need to be tied to the initial flow state. The proposed model is augmented with Dirichlet boundary conditions to learn physically meaningful turbulent flow representations.", "questions": "1/ Have you tested the proposed model on unstructured data like graph-meshes and cloud of points ? to what extent, it is applicable ? turbulent flow are irregular, regular grids can hide some patterns\n\n2/ What are the capabilities of the proposed model at the boundary layers ? do you have any metrics on the surface of the geometry ? (most challenging part)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1/ Originality and significance:\nTurbulence modelling is a crucial task in many engineering applications. However, the problem is still open and challenging. In the litterature several methods have been proposed to tackle the problem from different points of views but the proposed solutions are either expensive/ intractable in practice or are very complex to set. In this work, the originality of the work is in its simplicity of usability while providing good turbulent flow results in a raisonnable time. Hence, the proposed model is scalable.\nMoreover, it is based on a generative model (diffusion model  DDPM) that has achieved significant leaps in a wide spectrum of applications. Augmenting DDPM with physics constraints like Dirichlet enables generating 3D turbulent flows of good quality. The strength of the proposed model is related to the fact it learns a manifold of turbulent flow states without being tied to the initial flow state.\n\n2/ Quality:\nThe methodology of the proposed work is rigorous. A targeted 3D turbulent flow dataset is generated specifically to assess the capabilities of the proposed generative turbulent flow model in the appropriate conditions. Moreover, data and task dependent metrics are introduced to evaluate to the model in physical way.\n\n3/ Clarity:\nThe paper is very clear and simple to follow. It is well structured.", "weaknesses": "The proposed work lacks of ablation studies and baselines both quantitative and qualitative analysis. Diving into the different blocks of the architecture to understand their impact/contribution is of great importance and can give more insights. Especially that the proposed model is a combination of sophisticated blocks: DDPM, U-NET, transformers. One would like to understand how these blocks cohabitate to generate good quality of turbulent flow.\n\nFigure and the details on the architecture are not sufficient.\n\nInvariance to meshing / discretization scheme has not been discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLEUrcnIdq", "reviewer": "Reviewer_oPBx", "summary": "This work proposes a generative model based on a diffusion process for one-shot generation of solutions to turbolent flows.\nThe approach is remarkable in his simplicity and potential applicability.", "questions": "Q1: plans to release code and database\nQ2: plans to scale up experiments\nQ3: why does it generalize in such a small data regime? any thoughts?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The approach of using a one-shot generation for solutions to turbolent flow is quite remarkable, as well as the fact that results seem to be rather impressive on such a small data regime.\nThe experiments show training on 27 airfoils, 9 for val and 9 for testing. It is indeed surprising the model can generalize in such a small regime, perhaps indicative of the fact that the model can pool information across all cells/volumes, thus perhaps capturing somehow the intrinsic physics of the problem.", "weaknesses": "There is no mention of releasing the data-set and the source-code. This is a major issue for reproducibility.\nThe size of the data-set is also surprisingly small. While it's quite remarkable the model seem to capture the essence of the problem is such a small data-regime, I am also quite puzzled by that.\nIt would be great to scale to deep-learning type of sample size (10^5 samples - of course what's a sample in the CFD case is a bit up to interpretation).\n\nI also agree the limitations of A#F are palpable: especially the lack of equivariance and the geometry limitations are quite important.\n\nOf course this also means the architecture has much to grow, once such limitations are addressed, but this also points to the fact this is likely an initial work, and possibly not ready yet for primetime publishing in top venue like iClear.\n\nFinally, the speedup is somewhat disappointing: the nominal 30x achieved is really not so important when considering that A100 vs Xeon-2630 is probably already 50x faster (depending on workloads).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooPangZUTk", "reviewer": "Reviewer_JYNa", "summary": "Methods\n- The paper focuses on the topic of 3D turbulence generation, by modeling it as a generative task to overcome the roll-out dilemma of autoregressive models.\n- The paper derives an appropriate generative diffusion model and the discretization on 3d domain for the 3D turbulence generation. Dirichlet conditions are handled in the model. Cell types are handled via learned cell type embeddings.\n- The network is based on denoising diffusion probabilistic models (DDPMs). The network architecture employed in the diffusion model extends the U-Net framework into the 3D domain, incorporates a transformer (after 4 level of down-sampling) to enable global communication across the simulation domain.\n\nExperiments\n- The paper provides a new 3D turbulence dataset, consisting of 45 simulations of an incompressible flow in 192 × 48 × 48 grid cells.\n- 9 out of 45 samples from the dataset are used to validate the method. From the results, the proposed method is able to generate realistic 3D-flow samples that can capture the turbulent flows caused by unseen objects.\n- The paper proposes metrics for evaluating the quality of the generated flows by utilizing properties of fluids, including turbulent kinetic energy, marginal velocity, vorticity and pressure distribution, and the location of the strongest turbulent vortices.\n- The method is compared with two autoregressive models (Turbulent Flow Net and DilResNet) and each with two different settings to convert them to generative models, showing good performance compared with them.", "questions": "- In Fig5 and Fig6, what’s the value that is visualized? the norm of velocity? pressure?\n- Would it help to pre-processing the data, e.g. rotations, flips to augment the dataset?\n- Is it relatively easy or hard to apply this method to more complex turbulent generation, e.g., with fluid–solid interaction? \n- The grid size used for generating dataset is 192 × 48 × 48. In fluid simulations, larger grid size are often used to achieve higher accuracy. Is this grid size chosen due to performance limitation of the CFD solver?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper leverages generative diffusion models, which are more commonly associated with image and video tasks, for turbulence simulations, providing a new approach for 3D turbulence generation.\n- In the experiment section, the metrics are well designed to capture the characteristics of fluids, and the proposed method shows good performance in terms of both accuracy and runtime.\n- The paper gives a good discussion regarding the challenges of employing autoregressive models for 3D turbulence generation.\n- In addition to the method, the paper introduces a new 3D turbulence dataset that can benefit further research in the field of turbulence generation.", "weaknesses": "- It would be good (but not necessary in this paper) to see how the method performs on learning and generating more complex fluid phenomena, such as fluid interactions with air, and generating classical fluid phenomena like the Karman vortex street.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SeOVnWHVmU", "reviewer": "Reviewer_MYNP", "summary": "The paper concerns simulation of 3D turbulent flows and replacing direct numerical solvers with autoregressive, learned models. The authors argue that turbulent flows are substantially harder to model properly in 3D than in the 2D case which many papers previously have targeted. They then propose a generative model for flow simulation that is independent of the initial state but still able to capture the distribution of fluid states with high-resolution in complicated settings with various objects affecting the flow.", "questions": "- I think it is a little unclear that you write \"Thus, we propose to circumvent the roll-out dilemma of autoregressive models by modeling the marginal distribution E_t[p((u, p)t | u0, p0)] directly, capturing the distribution of turbulent flows with a generative model.\", but then in the next paragraph remove the dependence on the initial state and remove the expectation. As far as I can see, the stochastic process perspective has been used previously (e.g. ref Yang and Sommer), and what is done here is removing the condition on the initial state. I'm note sure where the expectation comes in.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- well-written and clearly presented paper\n- important problem and method that can potentially have impact\n- targets 3D instead of 2D and argues that flow simulations in this case is substantially harder\n- works in challenging situations with boundary effects and objects influencing the flow", "weaknesses": "- while the paper presents a new method that is different from the generative flow methods in the literature, it is quite related to previous approaches and thus provide a somewhat incremental - but still important - contribution\n\nMinor points:\n- some parts need proofreading, e.g. \"We conclude that many use cases can be solved by independent snapshots of possible flow states as well as by a classical numerical simulation. We conclude that many use cases can be solved by independent snapshots of possible flow states just as well as with a classical numerical simulation.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)