Learned Simulators for Turbulence
Authors: Kim Stachenfeld
Drummond Buschman Fielding
Dmitrii Kochkov
Miles Cranmer
Tobias Pfaff
Jonathan Godwin
Can Cui
Shirley Ho
Peter Battaglia
Alvaro Sanchez-Gonzalez
Venue: ICLR 2022 Poster (2022)
Keywords: learned simulation
turbulence

Abstract: Turbulence simulation with classical numerical solvers requires  high-resolution grids to accurately resolve dynamics. Here we train learned simulators at low spatial and temporal resolutions to capture turbulent dynamics generated at high resolution. We show that our proposed model can simulate turbulent dynamics more accurately than classical numerical solvers at the comparably low resolutions across various scientifically relevant metrics. Our model is trained end-to-end from data and is capable of learning a range of challenging chaotic and turbulent dynamics at low resolution, including trajectories generated by the state-of-the-art Athena++ engine. We show that our simpler, general-purpose architecture outperforms various more specialized, turbulence-specific architectures from the learned turbulence simulation literature. In general, we see that learned simulators yield unstable trajectories; however, we show that tuning training noise and temporal downsampling solves this problem. We also find that while generalization beyond the training distribution is a challenge for learned models, training noise, added loss constraints, and dataset augmentation can help. Broadly, we conclude that our learned simulator outperforms traditional solvers run on coarser grids, and emphasize that simple design choices can offer stability and robust generalization.

Published as a conference paper at ICLR 2022
LEARNED COARSE MODELS FOR EFFICIENT TURBULENCE SIMULATION
Kimberly Stachenfeld,'
Drummond B. Fielding. Dmitrii Kochkov," 3
Miles Cranmer,*
Tobias Pfaff, Jonathan Godwin, Can Cui.
Shirley Ho,
Peter Battaglia, ! Alvaro Sanchez-Gonzalez'
DeepMind, London, UK CCenter for Computational Astrophysics, Flatiron Institute, New York, NY GGogle Research, Cambridge, MA "Princeton University, Princeton, NJ
a va warosg@goggl...ooo com, stachenfel d geggoole..... le..le... com
ABSTRACT
Turbulence simulation with classical numerical solvers requires high-resolution grids to accurately resolve dynamics. Here we train learned simulators at low spatial and temporal resolutions to capture turbulent dynamics generated at high resolution. We show that our proposed model can simulate turbulent dynamics more accurately than classical numerical solvers at the comparably low resolutions across various scientifically relevant metries. Our model is trained end-to-end from data and is capable of learning range of challenging chaotic and turbulent dynamics at low resolution, including trajectories generated by the state-of-the-arr Athena+ engine. We show that our simpler, reneral-purpose architecture outper- forms various more specialized, turbulence-specifee architectures from the learned turbulence simulation literature. In general, unstable trajectories; however, we show that tuning training noise and temporal downsampling solves this problem. We also find that while generalization beyond the training distribution is a challenge for learned models, training noise, added loss constraints, and dataset augmentation can help. Broadly, we conelude that our learned simulator outperforms traditional solvers run on coarser grids, and emphasize that simple design choices can offer stability and robust generalization.
that learned simulators yield
INTRODUCTION
Turbulent fluid dynamies are ubiquitous throughout the natural world, and many scientific and engineering disciplines depend high quality turbulence simulations. Examples range from design problems aeronautics (Rhie, 1983) and medicine (Sallam Hwang, 1984) to scientific problems of understanding molecular dynamics (Smith, 2015), weather (Beljaars, 2003), and vast galactic systems (Canuto & Christensen- Dalienssenn-aaaardd.... 1998). Predicting turbulent dynamics challenging, and finding a general solution to the governing equations of fluid dynamics, the Navier-Stokes Equation, a famously open problem in Mathematical Physics (Jaffe, 2006). The challenge is due in large part the fact that turbulence is chaotic: small-scale changes the initial conditions can lead large difference in the outcome, Nonetheless, over the past several decades, high-quality numerical solvers have been engincered that integrate the governing partial differential equations (PDEs) and can maintain accuracy over long integration periods even for complex dynamics. These solvers often must operate over high-rese nigh-resolution grids and therefore require substantial computational resources. Otherwise, high-frequency information is lost due to hard-to-model "numerical viscosity", and the simulated dynamics can further diverge from the underlying equations due the chaotic dynamics.
Learned simulation represents promising avenue for efficient fluid simulation because learned models can potentially adapt to capture the large-scale dynamics on coarse grids. Learned simulators vary in the extent to which they incorporate components from classical solvers, like those that learn closures and subgrid discretization (Duraisamy et al., 2019; Freund et al., 2019; Pathak et al., 2020; Um et al., 2021; Kochkov al., 2021), versus adopting pure Machine Learning (ML) approach (Kim et al., 2018; Lusch et al., 2018; Sanchez-Gon Sonealzz et al., 2020; Wang et al., 2020; Li et al., 2020c;a; Pfaff et al., 2021). Advantages of the pure MI. route that the same model can be used

---

Published as a conference paper at ICLR 2022
without specialized knowledge of the domain and that they do not involve the challenge of interfacing with PDE solvers. However, fully learned simulators often work well only when conditions are similar to the training distribution, which can limit their stability over time and their generalization capabilities.
Our primary contribution is demonstrate that fully learned simulators can learn predict turbulent dynamics on coarse grids more accurately than classical solvers. This is true over a range of performance metries, but the learned simulators particularly excel in their ability preserve high frequency information. While learned simulators can be prohibitively unstable, we show that the simple practice of tuning training noise and the temporal downsampling factor can reliably produce stable rollouts. We introduce simple model that combines neural networks architectures for grids (Dilated CNNs (Yu & Koltun, 2016), ResNets (He et al., 2015)) with recent advancements on general purpose learned simulation on graphs sencode-process-deee 2020; Pfaff et al., 2021) trained with noise /Sanclez-Gonzalez et al., 2018; 2020; Pfaff et al., 2021)). We implement this well as set of learned models from the literature that have been proposed for modeling turbulence, and show that Dilated ResNet (Dil-ResNet) outperforms the often more complicated, more heavily parameterized models. Rather than looking at single problem, evaluate our models over a variety of challenging turbulent and chaotic domains, showing that Dil-ResNet is generally able capture variety of turbulent dynamics out-of-the-box. This includes two environments generated by the new Athena++ solver(Stone et al., 2020), a state of the art (magneto-) (magneto-)hydroddynamm Jhydrodynamies simulator used for astrophysics, where extremely high-resolution and large-scale simulations are often required. Finally, we evaluate the ability of the learned models to generalize initial conditions, rollout durations, and environment sizes outside of the training distribution. Generalizing out of distribution is a known challenge for learned models, and we document where our models fail and what model and training choices can improve generalization.
models Saachez--Gnzzaler et al.,
2 RELATED WORK
Turbulence simulation is a crucial step in problems across disciplines. These include forecasting problems, like predicting the spread of a wildfire (Lconardi et al., 2019) or the evolution of a hurricane (Zhang et al., 2020), engineering challenges, like designing more aerodynamic machinery (Rhie, 1983), and scientific questions, like understanding the physics of insect flight (Dickinson et al., 1999) or dynamics of celestial objects (Fielding et al., 2018). Across applications, a variety of statistical measures are relevant for quantifying physical variables under turbulence (Velocity Autocorrelation, variable histograms, Cooling, Energy spectra). The primary challenge for simulating turbulence is accurately capturing small- and large-scale flow dynamics, and their interplay, simultancously.
Traditional approaches to computational fluid dynamics integrate the governing PDEs describing the fluid flow, given initial and boundary conditions. State-of-the-art turbulence simulators, such as Athena++, are built upon efficient algorithms that prioritize the preservation of high-order statistics. Such simulators are used frequently in scientific applications to numerically compare model predictions with experimental observations, but can be extremely-often prohibitivell-exxeeiive to run. Thus, there is significant practical interest in developing more efficient simulators.
Deep learning models for predicting physical dynamics have traditionally been used for modeling now-dimensional dynamical systems with graph-like (Battaglia al., 2016; Sanchez-Gonzalez et al., 2018) and grid-like specifications (Raissi et al., 2017: Chen et al., 2018), but there is a growing body of work for fine-grained modeling of complex dynamics, as in particle-based liquids and soft-body physics (Li et al., 2019; Ummenhofer et al., 2020; Sanchez- are often used in computer graphics to speed up (Wiewel et al., 2019; Um et al., 2018: Ladický et al., 2015; Holden et al., 2019) or super-resolve (Xie et al., 2018) simulations. A number of papers predict the steady-state flow around 2D airfoil wings, using CNN (Thuerey et al., 2020; Bhatnagar et al., 2019) GCN (Belbute-Pene
et al., 2020). Learned models
et al., 2020) architectures, or the transient dynamics of various low Reynolds number flows using mesh representations (Pfaff et al., 2021).
In recent years, number of models have been introduced specifically for learning turbulent dynamics. Many of these use neural networks to model some part of an otherwise classical, theoretically motivated numerical integration setup (Duraisamy et al., 2019; Freund et al., 2019; Pathak et al., 2020: Um et al., 2021; Kochkov et al., 2021; Tompson et al., 2016). PDE formulation (Raissi et al., 2017; Champion al., 2019), or embedding space (Lusch et al., 2018). Other models incorporate
2

---

Published as a conference paper at ICLR 2022
a
X
X
Learned simulator
Xe+
XT
1D KS Equation (KS-1D)
Xx;
Prediction
festur
Ground Trath
Processor
dCNN,
xN
X-
CNN
CNN Ae
AX
50
100
150
Time
2D Incompressible Decaying
d
3D Compressible Decaylig
3D Mixing Layer
Turbulence (Incomp-2D)
Turbulence (CompDecay-55)
(CoolMixLayer-
Initial
Prediction
Ground truth
Initial
Prediction
Ground truth
Init
Pred GT
Figure 1: (a) Learned simulator framework. The model is trained to predict the difference between the current and next state. Gaussian noise can be added the inputs during training produce models that are robust to small perturbations. Shown below is the schematic for the Dilated ResNet model (Dil-ResNet), (b-e) Frames predicted by rollouts from the learned simulator model compared to ground truth frames. (b) KS- The model follows the ground truth closely for the first 150 steps (t < 75) and remains plausible thereafter. (c) INCOMP- 2D: The model remains accurate after 119 model steps (91,392 solver steps). (d) COMPDECAY-31 The model remains accurate after 31 model steps (1984 solver steps). (e) COOLMIXLAVER
The model remains qualitatively accurate after 59 model steps (59,000 solver steps) of box size of (L 0.75). Videos available at sites.googlecom wwwp///wwwwweeeeeddderbbeeeeeeee
some information about turbulence simulation into the architecture, but like ours, are learned end- to-end in supervised way. Wang et al. (Wang al., 2020) implemented fully learned, neural network "Turbulent Flow Network" (TF-Net) that applies physically inspired decomposition with learned parameters and processes the components with U-Nets for fluid simulation. and colleagues introduced "Fourier Neural Operators" for turbulence simulation (Li et al., 2020a), which apply neural networks that convolution parameters in the Fourier domain. This model is part of a broader family of work (Portwood et al., 2019; Li et al., 2020b) which combines basis transforms (Fourier, Multipole, etc) with neural operators that process the transformed input. We implement models from these papers baselines to compare our learned simulator.
3 MODEL
3.1 LEARNED SIMULATION FRAMEWORK ON GRIDS
We train a simulation model s X + X' that maps X₁ to X1+AI, where is a space of n+1 dimensional tensors which represent the state variables for each point grid with spatial dimensions and one feature dimension at time I. The true physical dynamics applied over K time steps yields trajectory of states which denote (Xtor... Xtx), The "rollout" trajectory produced by iteratively applying for steps is denoted (X X Xtx) where X Xt, represents initial conditions given as input. Our learnable simulators (Fig. la) implement s using neural networks denoted NN (with weights 0), and predict the next state as the difference from the current state, X
s( () X, NN(X;;0).
3.2 MODELS
In this work study "fully learned" models, which learn the physical dynamics end-to-end without any hard-coded solver components. We implemented a novel Dilated ResNet model, as well as several baseline models. Detailed model parameters be found in the Appendix.

---

Published as a conference paper at ICLR 2022
Dilated ResNet Sracodd-Process-Deeee (Dil-ResNet) We designed this architecture to combine the encode-process proces--decode paradigm SSanchez-Gonzalee et al., 2018; 2020) with dilated convolutional networks (Fig. la). The encoder and decoder each consist of single linear convolutional layer. The processor consists of N = 4 dilated CNN blocks (dCNN,)) connected in series with residual connections (Yu & Koltun, 2016; He et al., 2015). Each block consists of 7 dilated CNN layers with dilation rates of (1, 2, 4, 8, 4, 2, 1). A dilation rate of N, indicates each pixel is convolved with pixels that are multiples of N pixels away (N=1 reduces regular convolution). This model is not specialized for turbulence modeling per se, and its components are general general-purpose purppse tools for improving performance in CNNs. Residual connections help avoid vanishing gradients. and dilations allow long-range communication while preserving local structure. All individual CNNs use a kernel size of 3 and have 48 output channels (except the decoder, which has an output channel for each feature). Each individual CNN layer in the processor is immediately followed by a rectified linear unit (RcLU) activation function. The Encoder CNN and the Decoder CNNs do not use activations.
simplification of the TF-Net baseline explained below. It uses a simple linear CNN encoder and decoder (same as Dil-ResNet) and standard U-Net block (Ronneberger et al., 2015). As in (Ronneberger et al., 2015), each CNN stack is made of 3 CNNs, followed by activations. Other hyperparameters of the U-Net also matched those used in (Ronneberger al., 2015).
Encode-U- U-Net-Decood (U-Net) We designed this architecture
Turbulent Flow Net (TF-Net) (Wang et al., 2020) TF-Net uses a domain-specific variation of U- Nets model turbulence, along with other architectural elements inspired by RANS-LES Hybrid Coupling in the encoder. Hyperparameters matched those used in (Ronneberger et al., 2015; Wang al., 2020).
Constrained Turbulent Flow Net (Con-TF-Net) (Wang et al., 2020) Wang et al. also explored a constraint term the loss which penalizes the divergence, which was constrained to be zero everywhere for the PDE in their incompressible turbulence experiments. We implemented a similar model, adapting the constrained function match constraints or preserved quantities in our datasets. For KS- 1D, the mean is constrained to be O; for INCOMP-2D, the divergence is constrained to be O, and for COMPDECAY-31 the total energy constrained to be the same as input's s total energy.
Constrained Dilated ResNet (Con-Dile (essee))-eesNet) This model is Dil-ResNet (described above) with the additional constraint terms from (Con-TF-Net) added to the loss function. This allows us to examine the effect of the loss term separately from the other aspects of the TF-Net architecture.
Fourier Neural Operator (FNO) (Li et al., 2020a) This model uses the neural operator formalism introduced in carlier work (Li et al., 2020c;b), applied in the Fourier domain. We implemented FNO for ID and 2D spatial domains as in (Li et al., 2020a) Hyperparameters are from (Li et al., 2020a).
CNN padding We use periodic padding for spatial axes to implement periodic boundary conditions. For the COOLMIXLAYRR-3D which has a fixed boundary condition along the vertical axis, we mask out the value at the boundary when computing training loss and set it to the ground truth at each step of the rollout. We also augment the input state with a binary feature to distinguish boundary pixels.
3.3 TRAINING
Loss All models' are trained to predict AX. We use mean square error loss E(Y,, X1+ MSE(NN(X., X,; 0), X to optimize parameters 0. Input X, and target AX = X(+ - X, features normalized to be zero-mean and unit variance, which seemed to improve training speed. The additional mean square error loss components for constraint preservation in Con- Non-Dil-ResNet and Res--ill-eeet are added to the main loss with a relative weight of 1.
Training noise Optionally, we trained with Gaussian random noise with fixed standard deviation added to the input X, and subtracted from the target AX, this has been shown to improve stability of rollouts and prevent error accumulation by training the model to correct for small errors (Sanalez-Gonzalez al., 2018: 2020; Pfaff al., 2021). See Appendix for more details.
In the original papers, TetN Con-TF-Net, and FNO directly predict X but we did not find substantive differences from predicting AX By having all models predict A as target, this allows fairer comparison by reusing the same target normalization.
4

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTAL TURBULENT DOMAINS
We use four PDEs exhibiting chaotic or turbulent dynamics representative of a wide range of problems across science and engineering. (Fig. Ib-e). These environments range from the classic problems (KS Equations, Incompressible Turbulence) to more challenging environments that incorporate compressibility and non-uniform initial conditions. The training data is generated at high resolution Spatial and temporal downsampling factors and additional details about the environments are shown in the Appendix.
ID Kuramoto-Sivashinsk. Equation (KS-1D): A PDE that generates unstable, chaotic dynamics in ID, solved using Fourier spectral method (Kuramoto, 1978; Sivashinsky, 1977). The state consists of scalar velocity v per ID grid point. While KS- ID is not technically turbulent, it is well-studied chaotic equation that is useful for assessing the ability of our learned models to capture dynamics that are highly unstable and nonlinear.
2D Incompressible Decaying Turbulence (INCOMP-2D): Fluid flow modeled by Navier-Stokes in which small-scale eddies decay into large-scale structures due the inverse energy cascade, solved using direct numerical simulation (Kochkov et al., 2021). The state consists of a vector velocity U2: My per 2D grid point. These simulations are relevant to open questions in atmospheric dynamics (Boffetta & Ecke, 2012a).
3D Compressible Decaying Turbulence (COMPDECAY-3D)) Decaying transonic turbulent flow under Navier-Stokes, assuming adiabatic equation of state with adiabatic index = 5/3. The state consists of scalar density p. vector velocity U2, Py. Ug.. and scalar pressure P per 3D grid point. Energy each grid point is F 1/2po² + 3/2P. Simulations were carried out with Athena- (Stone et al., 2020), a sfate-of-the-art simulator used in astrophysics. All real fluids are somewhat compressible, and this is particularly non-negligible when modeling gases (Pope, 2000). In astrophysics problems, for which the At hthhaa++ + was developed, understanding the properties of these flows plays a crucial role regulating planet, star, black hole, and galaxy formation (Brandenburg & Nordlund, 2011).
3D Compressible Turbulence with Radiative Cooling Mixing Layer (CoOLMIXLAVER-3D)
Turbulent mixing resulting from the Kelvin-Helmholtz instability, caused by velocity differences across the interface between fluids of different densities, solved with Athena++. The state is specified in the same way as in COMPDECAY-3D. Mixing involves strong cooling, leading to net flow from the low-density phase into the mixing layer. This environment was additionally challenging because of its open boundaries and non-uniform initial conditions. This process is common in atmosheric flows well as many aspects of galaxy formation (Fielding et al., 2020).
RESULTS
We evaluate the learned simulator models in terms of stability, performance, efficiency, and general- ization. Unlike numerical solvers, which are PDE specific, learned simulator models can be designed to be reusable across domains. In Figure 1, we show that the same general seneral-purpose architecture (Dil-ResNet) and loss, trained on each of the domains, learns to capture a range of qualitatively diverse turbulent dynamics across domains (Fig. 1b-e, videos: sites. wwressgoolle.coooo com/view/learneduuue ttrneeeeceuuueeeeeee. simulators). All results are evaluated on held-out test trajectories sampled from the same distribution used for training (except for generalization sections). All units (except time) are normalized and therefore dimensionless.
Spatial coarsening Numerical solvers are known to lose information, particularly high-frequency information, when applied coarse grids. We compared our Dil-ResNet learned simulator to Athena++ run on spatially coarsened grids (see Section 5 for comparison with learned models). & We chose the energy field = pp2 + 2 P as the quantitative metric of 3D turbulence for the main text, as it summarizes performance on all state variables. We show results for Athena++ ++ run at resolutions of 32 (the same resolution as the learned models) and at 64 (higher resolution than the learned models, lower than the ground truth). The "ground truth" data is from Athena+ ++ run at 1283. All grids are downsampled to 323 for comparison. We compared models in terms of the RMSE
Note the reason that error rises and falls for this model is that COMPDECAY-3D is decaying turbulence. As time passes, the distribution hecomes more uniform in space, resulting in lower error across models.
5

---

Published as a conference paper at ICLR 2022
Error
Energy Fields (t-1)
Energy Field
Field
Athena++S2
Athera
ONCE
Energe Fiedd IMSEE
Dil-ResNeet
Athona+
Athenare 64
Athengee 128
Saaning
camer
Ground Truth
Energy Spectnim
Energy Spectrum
Energy Spectrum
Soecium WHis
10
10
d
Erengy
fneegy
DRN
A2h61
Time
Tregunnry [170s]
Figure 2: Comparison between learned 32 Dil-ResNet and same-resolution 32 htna +++ interme- diate resolution 64 Athena+t, and ground truth high resolution 128 Athena++ in COMPDECAY- 3D. (a) Energy Field RMSE (y-axis) for Dil-ResNet (323 resolution (blue)) and Athena+ (643 (dark gray), and 323 (light gray)) on a test trajectory the window of times seen during training. Dil-ResNet RMSE lower than that of the comparably coarse Athena+ 32 but higher than Athena++ 64 (b) Energy Field RMSE (y-axis) as function of rollout duration (ι-axis) over the training window (white background) and generalizing beyond (pink background)", The learned simulator's 's сπтог grows faster starting around 3x the training window (at z 3000). Each blue line corresponds different seed. Ground truth (red) error at 0 and hidden by ar-axis. (c) Energy Spectrum RMSE (y-axis). Dil-ResNet has lower error than both comparable and higher resolution 32 and 64 Athenat- + models. (d) Energy Spectrum RMSE (y-axis) over rollout. Dil-ResNet's rollouts maintain lower Energy Spectrum error for up 4x the training duration. (e) Energy Spectrum for the learned and Athenat ++ simulators at rollout As 1 (the end of the training window). The æ-axis represents spatial frequency, and the y-axis represents spectral power. Compared to Dil-ResNet, the coarse Athena++ models lose power in the high frequency range. (f-i) Sample energy Fields at 1 for Dil-ResNet and the three Athenatt resolutions. States from coarse Athena ++ resolutions lose high frequency detail with respect to the ground truth, which the coarse learned model captures.
of the predicted Energy Field and the Log Energy Spectrum (See Appendix for definition). We used the energy field as the main variable as is a combination of all five state variables, but see Videos and Appendix for var-vari results.
The learned simulator outperforms the comparable resolution Athena+ + 32 across a variety of metrics, despite having no built-in specializations for turbulent dynamics. In Figure 2a-b, we show RMSE for the Energy Field for t < 1, which corresponds to the initial phase of the turbulence decay seen during training (white window in Fig 2b). Energy Field is given as E px PP. and implicitly summarizes performance on all state variables (other metrics are shown in the Appendix). The learned simulator outperforms both the same- and higher-resolution Athena+ rollouts in terms of the Log Energy Spectrum (Fig. 2c,d), the Athena++ simulators lose high frequency components that the learned simulators preserve (Figure 2c-i). Log Energy Spectrum is computed by (I) taking the 3-D Fourier transform, (2) computing the amplitude of each component, (3) taking the histogram of the 3-D Fourier transform over frequency amplitude (/22 k2 + k?) to get a 1-D PSD, and (4) then taking the log. We looked at range of other physically relevant metrics and find that learned simulator outperforms the comparably coarse 32 Athena++ but not 64 Athena++ predicting feature histograms, the phase histograms (pressure v. density and entropy v. pressure), and mean squared error for each feature. The learned simulator outperformed both 32 and 64 Athena++ ++ simulators on higher order velocity autocorrelations, as well as spectrum error. Since different scientific questions rely on different metrics, the tradcoff's of learned versus physics based simulators may vary across applications. These metrics are defined in the Appendix.
Stability and training noise While scientific simulators are typically designed to be stable over rollouts time, common failure mode in learned models is that small errors accumulate prediction and lead to a domain shift. One reason for this is that, as the model is fed its most recent back in input for predicting future steps, its distribution of input states begins deviate from

---

Published as a conference paper at ICLR 2022
CompDecay-3B 3D Energy RMSE
CompDecay-3D Energy RMSE Time
No noise
Noise 0.01
step
1.5e-2
No noise
Noise 0.01
step
100
1e-2
10-1
10-2
5e-3
10
10
Rollout
Rollout
108
100
10
0.0
0.2
0.4
0.6
0.8
1.0
Hi
Time [4] (AI-10')
Model Timestep At (x10-)
Figure 3: Effects of noise and temporal downsampling on rollout stability. (a) One step errors are larger for models trained with noise. Note the error spikes very small and not model-related general artifacts, but specific to particular frames of this test trajectory. (b) However, models trained without noise can yield unstable rollouts, especially when using very small time steps, which is not problem for models trained with noise. (c, e) One-step model error rises monotonically with coarser temporal downsampling. (d, f) Rollout error has U-shaped curve over temporal downsampling factors, for a trajectory of the same time duration, with minimum error around At = 0.032.
that experienced at training, where it fails to generalize and can make arbitrarily poor predictions. We found that adding Gaussian noise o 0.01 to the inputs X, during training led to less accurate one-step predictions (Fig. 3a), but more stable trajectories (Fig. 3b). This of particular importance for models that take very large number of small steps. This is presumably because the training distribution has broader support and the model is optimized to map deviant inputs back to the training distribution.
Temporal coarsening An advantage of learned simulators is that they can exploit a much larger step size than the numerical solver, they discover efficient updates that capture relevant dynamics on larger timescale. This allows for faster simulation. See Videos and Appendix for qualitative examples of Dil-ResNet trained on large range of exponentially increasing coarse timesteps in KS-ID, INCOMP-24 2D and COMPDECAY-3D. which the model can adapt to (note that separate model trained for each At). Quantitatively, though the one-step error is at its lowest when using smaller time steps (Fig. 3c), the rollout error has an optimal time step at around 0,032 (Fig. 3d). This demonstrates the tradeoff between large and small time steps. Large time steps (> 0.032) cause predicting the next state to become more challenging. Small time steps (< 0.008), which require more simulator steps for the same duration, often yield unstable models because they provide more opportunities for errors to accumulate (Fig. 3d) (e.g. for some At, 1s/ At steps required). However, they can still be stabilized to some extent with training noise (Fig. 3f).
KS-1D
Incomp-2D
CompDecay-30
Vellocity RMSE
Velocity RMSE
Energy Field RMSE
Velsoty RASE
Valadty IMSE
0.02
TE
17
Energy AMSE
Dii-ResNet
Con-Dil-RedNet U-Net in-1)
0.01
0.5
TF-Net
Con- TT
0.00
FNO (kmBB
nolse
noise
nolse
noise
nolse
noise
FNO (full rank)
Figure 4: Comparison across learned models, contrasting noise and no-noise training conditions, across the three primary tasks (a) KS-ID, (b) INCOMP-2D, and (c) COMPDECAY-3D. With a few exceptions, the various learned models had comparable performance, though the Dilated ResNets (Dil-ResNet, Con-Dil Son-Dil-ResNef ResNet) consistently have the lowest error. In KS- ID (a), the noise harmed performance; in INCOMP-2D (b) the noise particularly benefits the FNO rollouts; COMPDECAY-3D (c) the noise mainly stabilized rollouts.
7

---

Published as a conference paper at ICLR 2022
Comparison across learned models In Figure 4 we compare the learned models in terms of the performance across the different primary turbulence datasets. We find that Dilated ResNets (Dil-ResNet, -on-Dil-ResNet) are the top performing models, numerically outperforming more specialized architectures. However, we find that all learned models (ours and the baselines) produce qualitatively good rollouts compared to A Athena++ 323, suggesting learned simulators are generally a good approach to turbulence simulation on coarse grids (see Videos), We also find training noise improves stability across learned models. All models shown in Figure are evaluated such that the first predicted step is the 7th step of the ground truth trajectory. This is to allow comparison with TF-Net, which takes C' context states as input. We show in further detail the predictions of the different models in the Appendix.
Running time Learned simulators can accelerate simulations not only by coarsening in time and space, but also by running off-the-shelf on specialized GPU hardware, unlike e.g., Athena + which is currently CPU specific. By downsampling in time (Al 0.0005 - 0.032) and space (128 - 32 the Dil-ResNet model trained on COMPDECAY-3D rumning on a single GPU speeds up the wall-clock simulation time by 1000x compared to Athena++ running on an 8-core CPU (further details in Appendix). GPU solvers can be developed for many applications and can greatly accelerate simulation; however, these can require lot of expertise and time to develop, and will still require high spatial and temporal resolution grids.
Constraint satisfaction and preserved quantities Traditional solvers often implement constraints to preserve conserved quantities. Learned simulators do not necessarily learn such constraints and may fail to capture these (see Appendix). We found that training with noise prevents this in some cases: it helps Dil-ResNet keep the mean value of the KS- 1D velocity, and the INCOMP-2D divergence bounded near zero. However, in COMPDECAY-3D using noise does not seem to help preventing the model from predicting drifts in total mass, vector momentum and energy, even though they are fixed across the whole dataset. We speculate this is because these preserved quantities only account for 5 degrees freedom out of a 3D state that is made of 163840 (= 5 5323) output variables.
Generalization to longer rollouts Dil-ResNer can stably unroll KS-1D for much longer than the trajectories seen in training. This is expected, as the distribution of states is stationary in time (see Appendix). By contrast, the distribution of states in COMPDECAY-3D is not stationary in time, as the flow decays progressively into smaller and smaller structures. We can use this to construct more challenging generalization setting by training Dil-ResNet on early stages of the decay only (t < 1), and testing on longer durations in which the turbulence develops further than observed during training. (1 < t < 4 in Fig. 2b, d). We find Dil-ResNet remains competitive with 323 At Athena+t for turbulence that has decayed for up to twice longer than observed during training.
Generalization to different initial conditions We varied the ratio of solenoidal to compressive components in the initial velocity field in COMPDECAY-3D (Fig. 5a) in order to test the models' ability to generalize to novel initial conditions. Compared to Athena++ at 323 and 643, we found that the models generalize well to more solenoidal but not to more compressive initial conditions, possibly due to faster turbulence decay under compressive conditions We find that Dil-ResNet (with or without noise) has particular difficulty generalizing to the more compressive components. This is somewhat ameliorated with the added constraint to the loss function (Con-Dilates Sillated-ResNet), Results for other models are similar (see Appendix).
Generalization to larger boxes We tested the generalization capability of Dil-ResNet to CoolMiXLAYERS boxes with different size I in the x and direction (the length along the verti- cal 2 axis perpendicular to the mixing layer remained constant). We quantified the predicted cooling velocity (small laminar flow in the low-density phase perpendicular to the interface, see Appendix for more details) which is of scientific relevance and known to depend on the box size. Specifically, for MOOMMIXLYYER-3
the dynamics undergo transition when the box width increases relative to the cooling length, which is difficult to study because simulations for these widths essentially prohibitive. We found that unless Dil-ResNet is trained on range of sizes, the predicted cooling velocity does not follow the right trend, and even when trained on a range of sizes, generalization beyond the training
Note the baseline error is expected decrease for more compressive components due to faster decay.
8

---

Published as a conference paper at ICLR 2022
Generalization Different Initial Conditions CompDecay-3D 3D
Generalization Different Sizes CoolMixLayer-3D
Traihing value
iin low density phase)
Alferar 64
-1.0
DilReeNNet
Cooling velocity
Ground Truth Dil-Reshlet Train LH0.75
Dil-ResNet
22-0
L-00.0075.11225
0.25
0.50
0.75
1.00
0.5
1.0
1.5
2.0
more compressive
more solenoidal
length
Figure 5: Generalization outside of the training distribution. (a) Generalization to different initial conditions. We vary the ratio of compressive and solenoidal components the initial velocity field. Solid markers indicated the training region. We find that, compared to coarse At hena++ generalization to more compressive initial states challenging for learned models and inconsistent across seeds, but that loss constraints (Con-Dil-ResNet) ameliorate this to some extent. (b) Cooling velocity generalization as function of the box size the mixing layer. The black line indicates the ground truth cooling velocity averaged across time for 8 test trajectories with different box sizes, with one standard deviation represented as the shaded region. None of seeds of Dil-ResNet trained single length of T. 0.75 (blue) generalize outside the training range. Dil-ResNet trained on multiple lengths (orange) shows better generalization performance. However, generalization far beyond training box sizes remains a challenge.
range is not reliable across seeds (Fig. 5b). We speculate that achieving this form of generalization would require either stronger inductive biases, more sophisticated dataset engineering, both.
CONCLUSIONS
Learned simulation techniques have been advancing steadily in recent years. Our results demonstrate that learned simulators can outperform comparably coarse solvers challenging turbulence settings on range of scientifically relevant statistical measures, suggesting that state-of-the-art learned simulators can be useful for efficient, accurate simulation even in chaotic domains. Learned simulators perform particularly well in terms of preserving high frequency information. The Dilated ResNet model had lower log Energy Spectrum RMSE than both 32 At Athena++ and the higher resolutions 64 Athena++, and the RMSE remained lower even beyond the duration seen during training. In our experiments, we found that using training noise and temporal downsampling improved the stability and accuracy of extended rollouts.
distif-distiibution generalization is a known challenge for learned simulation (Duraisamy et al., 2019) and for deep learning methods generally, and we find that the coarse Athena++ models outperform the leamedd models on more extreme generalization conditions. However, certain training choices can make learned models more robust. In particular, physically-infommed regularization (e.g., Con-Dil-BR Dis-Di---eNet and Con-TF-Nett improved generalization to different initial conditions on COMPDECAY-3D and for COOLMIXLAYER-
augmenting the training data with larger range of box sizes improved generalization performance on an even larger range. Future work should explore methods from other areas of deep learning for improving adversarial robustness and domain generalization, such as feature denoising (Xie et al., 2019) and discriminative objectives (Tzeng et al., 2017: Lample et al., 2017).
More broadly, we conclude that key potential role for learned simulators in the near term is distilling expensive, high- high-resolution engineered simulators into low-resolution algorithms to strike better performance-veree eftrsuusefficiecyy tradeoffs. Our approach can be applied effectively to diverse and challenging environments, which suggests that even generic CNN-based models may apply well to low-resolution simulation wide range of physical domains which can be represented as grids.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like thank Meire Fortunato, Matthew Grimes, and Lyuba Chumakova for helpful discussions and comments on the manuscript.
REFERENCES
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interac- tion networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00222, 2016.
Filipe de Avila Belbute-Peres, Thomas D. Economon, and J. Zico Kolter. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. In Proceedings of the 37th International Conference on Machine Learning ICML 2020, 2020.
A. Beljaars. Parameterization of physical processes turbulence and mixing. In James R. Holton (ed.), Encyclopedia of Atmospheric Sciences, pp. 1705-1717. Academic Press, Oxford, 2003. ISBN 978-0- 978-0-11-227000-11 doi: /ttmm//cceennc/rarielee/epaaaaep /000:///000002200000000
URL https:
www. sci sciencediiiiccccccccc rect. com/ sci
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Predic- tion of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, (4(2):525-545, 2019.
Guido Boffetta and Robert E. Ecke. Two-dimensional turbulence. Annual Review of Fluid Mechanics, 44(1))427-451, 2012a. doi: 10. / 102000011122000 URL htw.t:tp//// /estts://do1: /ooi......./// org/ 10. fluiluuuuo d- 10--111710-011110
Guido Boffetta and Robert E Ecke. Two-dimensional turbulence. Anmual Review of Fluid Mechanics, 2012b. 2012
Axel Brandenburg and Åke Nordlund. Astrophysical turbulence modeling, Reports on Progress in Physics, 74(4):046901, April 2011. doi: 10. 10.088/003-4885744444660000
V. M. Canuto and Christensen-Dalsgaard. Turbulence in astrophysics: Stars. Annual Review of Fluid Mechanics, 30(1):167-198, 1998. doi: 10.1 1444///mrre..fuii....11111
Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116
(45):22445-2245) 2019. ISSN 0027-8424. doi: 10.1073/ppax.11069951116 URL https:
// wwww.pnas org/ cowttet / /5///5/22445
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint 1rXiv:18866.77366, 2018.
Michael H. Dickinson, Fritz-Olaf Lehmann, and Sanjay P. Sane. Wing rotation and the aerodynamic basis of insect flight. Science, 284(5422):1954 1999. doi: 10. (0.1126/cceeeee2288.5777 1954. URL hwtps:tt///.......m..t http:///wwww./. wwwp:www............ http://wwwcceeeeee.oooooo/aaaa/a http:///co....aa......////eeeeee 1954.
284. 5422.
Karthik Duraisamy, Gianluca laccarino, and Heng Xiao. Turbulence modeling in the age of data. Annual Review of Fluid Mechanies. 51(1):357-377, Jan 2019. ISSN 1545- 4479. doi: Sttp://mmmure--uui--00005555 8-005477 URL /ttp:///ddoo...o//........... annurev- wwiesflluid--iio C1010 800000777807777..
Christoph Federrath and Ralf S. Klessen. The Star Formation Rate of Turbulent Magnetized Clouds: Comparing Theory, Simulations, and Observations. The Astrophysical Journal, (2):1). December 2012. doi: 00.1088/0004-63777771777
Drummond Fielding, Eliot Quataert, and Davide Martizzi. Clustered supemovae drive powerful galactic winds after superbubble breakout. Monthly Notices of the Royal Astronomical Society, 481(3)33355-337, December 2018. doi: 10.1093 19330/mas/////2446.
10

---

Published as a conference paper at ICLR 2022
Drummond B. Fielding, Eve C. Ostriker, Greg L. Bryan, and Adam S. Jermyn. Multiphase gas and the fractal nature of radiative turbulent mixing layers. The Astrophysical Journal, 894(2):L24, May 2020. ISSN 2041-8213. doi: -1044472241-8211//33333 3847 204472201-10 -221 -211188dd2b ab8d2c.
URL http: /tt.///ddd.oo.//1.1.////
Drummond B. Fielding, Stephanie Tonnesen, Daniel Defillppiis Miao Li, Kung-Yl Su, Greg L. Bryan, Chang-Goo Kim, John C. Forbes, Rachel Somerville, Nicholas Battaglia, Evan E. Schneider, Yuan Li, Ena Choi, Christopher C. Hayward, and Lars Hernquist. First Results from SMAUG: Uncovering the Origin the Multiphase Circumgalactic Medium with a Comparative Analysis of Idcalized and Cosmological Simulations. The Astrophysical Journal, 903(1):32, November 2020. doi: 10.3847/1538-4455//m0 4357/abbc6d.
Jonathan B. Freund, Jonathan F. MacArt, and Justin Sirigmano. DPM: A deep learning PDE augmen- tation method (with application to large-eddy simulation), 2019.
Uriel Frisch. Turbulence. The legacy of A.N. Kolmogorov. Cambridge University Press, 1995.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Daniel Holden, Bang Chi Duong, Sayantan Datta, and Derek Nowrouzeezahrai. Subspace neu- ral physics: Fast data-driven interactive simulation. In Proceedings of the 18th annual ACM STGGGAAPH/Eurogupphic Symposium on Computer Animalion, pp. 1-12, 2019.
Arthur M. Jaffe. The millennium grand challenge in mathematics. Notices of the AMS, (36))666666526660 2006.
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus H. Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. CoRR, abs/1806.0200 2018. URL http: / arxiv. org/ hbap//b///uie 1806. 02071.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics, 2021.
A. Kolmogorov. The Local Structure of Turbulence in Incompressible Viscous Fluid for Very Large Reynolds' Numbers. Akademiia Nauk SSSR Doklady, 30:301-305, January 1941.
Yoshiki Kuramoto. Diffusion-Induced Chaos in Reaction Systems. Progress of Theoretical Physics Supplement, 64:346-367, 02 1978. ISSN 0375-9687. doi: 0..144//TPS.644346 URL https: /////// wwt.oogg///.oooo 11..1....1.1111.. 346.
L' L'ubor Ladicky, SoHyeon Jeong, Barbara Solenthaler, Mare Pollefeys, and Markus Gross. Data- driven fluid simulations using regression forests. ACM Trans. Graph., 34(6), 2015.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. arXiv preprint arXiv: 00409, 2017.
Stefano Leonardi, Martand Mayukh Garimella, and Umberto Ciri. Effect of turbulence wildfire propagation. In APS Division of Fluid Dynamics Meeting Abstracts, APS Meeting Abstracts, pp. $18.006, November 2019.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Fourier neural operator for parametric partial differential equations. 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations, arXiv preprint 295555. 2020b.

---

Published as a conference paper at ICLR 2022
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, 2020c.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communicationss 9(1):4950, 2018. ISSN 2041-1723. doi: 10.1038 141466-018-072166 URL wwwph:tt///t....m oeaddii..rrr.oooo / 1038/ 8 007667008--777
Steven A. Orszag and G. S. Patterson. Numerical Simulation of sirre-Dimensional Homo- geneous Isotropic Turbulence. Physical Review Letters, 28(2):76-79, January 1972. doi: 10.1 10.1105PhhssReveeet2.....
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, and Marcus Day. Using machine learning to augment coarse-gie curse-grid computational fluid dynamics simulations, 2020.
Tobias Pfaff, Meire Fortunato, Alvaro Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021. /openrevieww net/ / forum? id=roNgYLO__G XP.
URL wwtptttpp///m.
Stephen B. Pope. Turbulent Flows. Cambridge Univ. Press, 2000.
Gavin D Portwood, Peetak P Mitra, Mateus Dias Ribeiro, Tan Minh Nguyen, Balasubramanya T Nadiga, Juan Saenz, Michael Chertkov, Animesh Garg, Anima Anandkumar, Andreas Dengel, et al. Turbulence forecasting via neural ode. arXiv preprint arXiv:1911.05110 2019.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.1061,, 2017.
Li Rhic, C Chow. Numerical study the turbulent flow past an airfoil with trailing edge separation. AIAA Journal, 21(11):1525-1532, 1983.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241 Springer, 2015.
Ahmed M. Sallam and Ned H. C. Hwang. Human red blood cell hemolysis in a turbulent shear flow: Contribution of reynolds shear stresses. Biorheology, 21(6):783-777, 1984.
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470-4479, PMLR, 2018
Alvaro Gonzzlez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. International Conference on Machine Learning, pp. 1459-8468. PMLR, 2020.
G.I. Sivashinsky. Nonlinear analysis of hydrodynamic instability in laminar flames- i. derivation of basic equations. Acta Astromautica, 4(1 4(11):1 177- 1977. ISSN 0094-5765. doi: https:/doi..rg// 10110160094-57666779900666 URL httph:t//////.. hwtp://wwwwwe//e sci encedi r sci ence/
arriciee/11/00345677770000
E. R. Smith. A molecular dynamics simulation of the turbulent couette minimal flow unit. Physics of Fluids, 27(11):115105, 2015, doi: 10.1063/1.4935213.
James M. Stone, Kengo Tomida, Christopher J. White, and Kyle G. Felker. The athena++ adaptive mesh refinement framework: Design and magnetohyyrodynamic solvers. The Astrophysical Journal Supplement Series, 249(1):4, Jun 2020. ISSN 1538-4365. doi: 10.38771538-4335//52299
URL http:: //pp:///xxxo/ doi orgg /10.000000 3847 /1538-138--11 4365/3365551 abb22...
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-36, 2020.
12

---

Published as a conference paper at ICLR 2022
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. CoRR, abs/1607.033977, 2016. URL http://
stxiv.org// httgg/abs/16000 035977.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. In Computer Graphics Forum, volume 37, pp. 171-182. Wiley Online Library, 2018.
Kiwon Um, Robert Brand, Yun, Fei, Philipp Holl, and Nils Thuerey. Solver-in- Solver-in-the-Lane -et-tthh----p--- Learning from differentiable physics to interact with iterative PDE-solvers, 2021.
Benjamin Ummenhofer, Lukas Prantl, Nils Thürey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2020.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction, 2020.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. Computer Graphics Forum, pp. 71-82. Wiley Online Library, 2019.
Cihang Xie. Yuxin Wu, Laurens der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 501-509, 2019.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. Tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Trans. Graph., 37(4), July 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions, 2016.
Jun A. Zhang, Evan A. Kalina, Mrinal K. Biswas, Robert F. Rogers, Ping Zhu, and Frank D. Marks. A review and evaluation of planetary boundary layer parameterizations in hurricane weather research and forecasting model using idealized simulations and observations. Atmosphere. 11 (10), 2020, ISSN 2073-4433. doi: 1 URL ht tttpp:: // //www//wwt////////////////////// mdpi. com/ 2073-443/11/0//D9
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
Hyperparameter
Dil-ResNet
U-Net
TF-Net
FNO
Kernel size Latent size Activation
3
3
3
48
48
ReLU
ReLU MSE
ReLU MSE
ReLU MSE
Loss
MSE
Dilated block depth Dilated block dilations
7 (1, 2, 4, 8, 4, 2.2
# processor blocks N Shared processors?
4
No
U-Net layers
7
4 (x3 encoders) + 3 (single decoder)
Stride CNN stack depth Base latent size
2
2
3
64
64
Spatial downsample by layer Latent sizes
(1, 2, 4, 8, 4, 2, 1) (64, 128, 256, 512,
(1, 2, 4, 8, 4, 2, I) (64x3, 128x3,
256, 128, 64)
256x3, 512x3
256, 128, 64)
Spatial filter kernel size Length of input sequence C
5
1
6
# modes layers
8 or all
4
Constraint weight
I
1
Table A.I: Model hyperparameters for the four basic models used. Con-Dil-ResNet and Con-TF-Net have the same architectural parameters as Dil-ResNet and TF-Net, respectively, with the added constraint weighted by the constraint weight. Note that spatial filter kernel for TF-Net is different from the kernel size in the first row: this spatial filter is applied during the decomposition in the TF-net model.
B ADDITIONAL MODEL DETAILS
Encode-U--Ne-- Decode (U-Net) We designed this architecture simplification of the TF-Net baseline explained below. It uses a simple linear CNN encoder and decoder (same as Dil-ResNet) and standard U-Net block (Ronneberger et al., 2015). The standard U-Net block consists of 7 layers. The first 3 layers each consist of a CNN stack followed by factor-of-2 downsampling via striding the input. The fourth layer of the U-Net another CNN stack, which this point operates over grid that is 8 times smaller than the input along each spatial axis. The last layers each consist of factor-of-2 upsampling using nearest neighbors, concatenation with the output of one of the first three layers (in reverse), and another CNN stack. This downsamplinguupsappll long-range communication similar the Dilated ResNet, while also preserving local structure. As in (Ronneberger et al., 2015), each CNN stack is made of 3 CNNs, followed by activations. Other hyperparameters of the U-Net also matched those used in (Ronneberger et al., 2015).
process allows
Turbulent Flow Net (TF-Net) (Wang et al., 2020) TF-Net uses a domain-specific variation of U-Nets to model turbulence, along with other architectural domain-specific elements inspired by RANS-LES Hybrid Coupling in the encoder. The model implements custom encoder by taking sequence of the most recent C' states as inputs, and uses this to build three separate input arrays, by applying specific combinations of learned spatial filters, learned temporal filters, and differences
14

---

Published as a conference paper at ICLR 2022
of those. Each of these components are processed by separate U-Net encoder (first 4 layers of the U-Net), then the latent outputs of each layer are concatenated and fed into a shared U-Net decoder (final 3 layers). Hyperparameters also matched those used in (Ronneberger et al., 2015; Wang et al., 2020).
Constrained Turbulent Flow Net (Con-TF-Net) (Wang et al., 2020) Wang et al. also explored a constraint term in the loss which penalizes the divergence, which was constrained to be zero everywhere for the PDE in their incompressible turbulence experiments. We implemented a similar model, adapting the constrained function to match either constraints or preserved quantities in our datasets. For KS-ID, ID, the mean is constrained be for INCOMP-2D the divergence is constrained to be 0, and for COMPDECAY-3D the total energy is constrained to be the same as the total energy of the input.
Constrained Dilated ResNet (Con-Dil-ResNet) This model is Dil-ResNet (described above) with the additional constraint terms from (Co--TF-Net added to the loss function. This allows us to examine the effect of the loss term separately from the other aspects of the TF-Net architecture.
Fourier Neural Operator (FNO) (Li al., 2020a) This model uses the neural operator formalism introduced in earlier work (Li et al., 2020c;b), applied in the Fourier domain. This model applies matrix multiplications in the Fourier-transfommer grid with complex weights learned independently for each component, as well linear updates The model combines pixel-wise learned linear embeddings in the spatial domain The model combines learned linear layers, applied to each point of the spatial the Fourier-transformed grid using learned weights.
grid separately, with matrix multiplication unique matrix of weights is learned for point in the Fourier grid. Low-pass filtering is optionally implemented by truncating all but K modes along each dimension in the Dourier-transformee grid. The authors explored using a version of the model that takes a stack of past states as input. Since the state is fully-Masn -Mrkko,,, we opted for the version of the model that takes single state as input as it is more comparable to the majority of the other models. Unlike previous models, which rely on spatially local convolutions and depth for long-range communication, this model applies long-range communication at every step in the Fourier domain. We implemented this model for 1D and 2D) spatial domains as in (Li et al., 2020a).
We found difficulties running this model on the 3D datasets due to compute and memory constraints. We speculate this is due to time and space complexity of the Fourier transform (when taking into account the backwards pass) which would probably require a custom gradient re-materialization strategy. This may be specific to TensorFlow which required us to reimplement the non-differentiable tf. signal rfft 3d sequence of tf.
f..ts.isngill signal. rfft2d and tf. signal.ffe
B.1 ADDITIONAL DATASET DETAILS
B. 1.1 ID SORRAMOTO-SIVASHING (KS) EQUATION
This is well-studied ID PDE that generates unstable, chaotic dynamics in dimension (Kuramoto, 1978; Sivashinsky, 1977) with periodic boundaries. The ground truth simulations are computed using the Fourier Spectral Method. Initial condition was set to cos (wi + (1) (w2z + oo)), where ranges from 0 to 2n,, di are sampled uniformly from [0, 2T)). and wi are integers sampled uniformly from [1, 12). We perform a warmup from this initial condition for 75 simulation time units.
B.1.2 2D INCOMPRESSIBLE DECAYING TURBULENCE
This models fluid flow described by the Navier-Stokes equations in which small-scale eddies decay into the large-scale structures due to the inverse energy cascade (Boffetta & Ecke, 2012b). The underlying simulations were performed by solving the incompressible Navier-Stokes equations using a direct numerical simulation (DNS) finite-volume solver. Boundaries along both dimensions are periodic, and the initial conditions consist of random velocity fields with small-scale variation. Initial conditions for different trajectories were obtained by sampling high-resolution velocity field from log- normall distribution of amplitude and wavenumber & = 10. We perform warmup for 500 simulation time units, this is done to discard the transient flow that is heavily influenced by the underlying numerical scheme.
15

---

Published as a conference paper at ICLR 2022
Compressible Radiative Cooling
KS Equation
Incompressible Decaying
Compressible Decaying
Mixing Layer
Numerical Solver
Fourier Method
DNS
Athenat+
Athena++
Spatial dims
2
3 5 D Uny Bys U2y P
# Features
1
2
5
Features
v
Cas My
Di Uny Bys U2y
P
Box size
Le
2x n/a
2m 2m
0.25 to 2
Ly
0.25 to 2 3
L
n/a
n/a
Grid element size
Solver
2x / 256
2m / 576
1 1/128
1 1/128
Learned model (relative to solver)
2T 64
2 / /48
1/32
1/32
4x
12x
4x
4x
Warm-up duration
75
500
0.05
2.32
Trajectory duration
181
400
7.226
Time step Solver
0.5
0.00436
0.0005
0.00012 to 0.00014
Learned model
0.5
3.35
0.032
0.12 1000x to 875x
(relative to solver)
1x
768x
64x
# Trajectories Training
1000
190
27
20 L, = 0.75 5 if Lg * 0.75 per L
Validation
100
10
4
Test
100
10
per Lx
Training details Early stopping?
No
No
No
Yes (4 multisize)
Batch size
32
8
Noise
le-2
le-4 divergence
1c-2 total energy
1c-3
Constrained
mean V
1e-3
Table B.1: Dataset details. P refers density, P to pressure, and Ux; by U2 to velocity components. Warm-up up refers to the initial transient from initial conditions which is influenced by the underlying numerical scheme and discarded from evaluation and training. Figures are dimensionless.
# Parameters
ID
2D
3D
Dil-ResNet
195361
584162
1757477
U-Net
3916993
11739202
35072581
TF-Net
17358540
31239137
93712968
FNO (K=8)
288433
4159250
62222693
FNO (K=full)
1191601
42479378
603994469
Table B.2: Number of parameters per model.
B.1.3 3D COMPRESSIBLE DECAYING TURBULENCE
This models decaying transonic Navier-Stokes turbulent flow in a 3D cubic box with periodic boundary conditions (e.g., Orszag & Patterson, 1972). These simulations adopt adiabatic equation of state with a constant adiabatic index 7 = 5/3. Simulations were carried out with At Athena++ (Stone al., 2020). The initial turbulence is driven scales > L, the size of the box. The turbulent
16

---

Published as a conference paper at ICLR 2022
driving pattern in the initial condition is split into its compressive and solenoidal components using a Helmholtz decomposition. The relative strength of these two components is varied from purely compressive to purely solenoidal. The initial driving pattern results in wott-men-squaree velocity of /2c,, where c² (P/))(P/p) is the sound speed of the fluid. The initial conditions are varied across trajectories by randomizing the phase of the spectral components, leading to different pattern in real space. We perform a warmup for 0.05 simulation time units. Compressible turbulence is ubiquitous across problems in science: all fluids are somewhat compressible, and this is usually non- negligible when modeling gases (Pope, 2000). In astrophysics problems, for which the Athena+ was specifically developed, understanding the dynamics and properties of these flows on small and large scales plays crucial role in regulating planet, star, black hole, and galaxy formation (Brandenburg & Nordlund, 2011).
B.1.4 3D COMPRESSIBLE RADIATIVE COOLING MIXING LAYER DYNAMICS
These simulations model the interplay of radiative cooling and mixing that results from turbulence driven by the the interface between two fluids of different densities. The simulations are set up as boundary problem initialized with a low-density fluid (p = 0.01) on the top half of the domain (z > 0) moving in the positive x direction (v-- = 2.04, Uyy = 0== = 0), and high-density fluid (p = 1.) moving in
instability, which can arise when there is velocity difference across
the negative x direction in the bottom half of the domain (Ug -2.44, F. = U2 0). The initial pressure is set to 1. The boundary conditions are periodic in both x and y, and fixed in z. To break the symmetry, small perturbations v. added along the boundary. These perturbations are changed across trajectories by randomizing the phase of the spectral components, leading to different patterns in real space. We perform warmup for 2.32 simulation time units. Simulations were carried out with Athena++ (Stone et al., 2020).
This data presents few unique challenges compared to the others. First, it the only domain with fixed boundary conditions. Second, because turbulent dynamics are limited to the vicinity of the mixing layer, and because the behavior of the fluid above, at, and below the mixing layer is markedly different, there is less data representative of each type of fluid dynamic behavior.
B.2 ADDITIONAL MODEL DETAILS
The learned simulator consisted of CNN encoder, dilated CNN processor with residual skip connections, and a CNN decoder. The parameters of the model are listed in Table A.1.
CNN parameters All individual CNNs use a kernel size of and have 48 output channels (except the decoder, which has an output channel for each feature). Each individual CNN layer in the processor is immediately followed by rectified linear unit (RcLU) activation function. The Encoder CNN and the Decoder CNNs do not use activations.
Dilated connections The dilation rate in the CNN filter introduces space between each element in the filter. Whereas a standard CNN filter with kernel 3 would operate over 3 adjacent pixels, dilated CNN filter with kernel 3 and dilation 2 will operate over 3 pixels spaced at intervals of 2 in each dimension, spanning a field 5 pixels wide. This increases the scale of the CNN kernels while preserving the number of parameters and the resolution.
CNN padding For each dimension in X' with periodic boundary conditions, we implemented periodic padding. For dimensions with fixed boundary condition, we forced the boundary value rather than letting the model predict it, and masked the loss so the model was not trained to predict boundary conditions. The tensor was padded with repetitions of the boundary value. We also augmented the input state with a feature that indicated fixed-boundary versus non-boundary states using a one-hot vector, so the model could distinguish them.
B.3 ADDITIONAL TRAINING DETAILS
Training noise In some cases, we trained with Gaussian random noise with fixed variance or added to the input X, of the loss function. Note that this not only affects the input to the neural network, but
17

---

Published as a conference paper at ICLR 2022
also slightly modifies the target AX X1+11 X₃ (and also impacts the variance used to normalize targets).
Loss At training time we sample pairs of input-output states (separated by the model time step) from the trajectories, and perform gradient updates based single step of the model. We do not rollout the model during training.
Optimization We optimized the loss using an Adam optimizer. We trained the models for up to le in the first 6M steps.
10M steps, with exponential learning rate decay annealed from Models usually reached convergence at around 5M steps. Training took up to week on an NVIDIA V100 GPU.
Hyper-parameter optimization We did not perform exhaustive hyper- hyper-parameter optimization, except the scale of the noise (which we scanned for each domain) and some informal tuning of the depth of the dilated blocks and latent size (to make sure the network had enough capacity for the most challenging domain). Note that achieving optimal performance on trajectories from the training distribution was not part of the scope of this work, and there are likely hyperparameters that would improve performance over our results.
Validation All of the research was performed by looking performance on the validation sets. The test set was completely held-out until final evaluation prior to writing the paper.
Early stopping For the Mixing Layer Turbulence (CooL.M which more prone to overfitting, we used early stopping based on the validation performance. For all other models we simply evaluated the model as it was the end of training.
B.4 ADDITIONAL RESULTS
Three instances of each model were trained times using 3 different initialization seeds. Bar plots indicate median performance and bars indicate min-max performance. We chose the energy field 2
E po2 + 3 P as the quantitative metric of 3D turbulence for the main text, as it summarizes performance on all state variables.
Figures and videos for 3D environments (COOLMIXLAYER- and COMPDECAY-3D) show a single slice of the 3D grid y 0, displaying the and coordinates in the horizontal and vertical axis, respectively.
Downsampling in time Fig. B.6 as well as Videos show models for KS- ID, INCOMP-2D, and COMPDECAY-3D models trained and working well on a wide range of time step sizes. Note that for fair comparison in Fig. 3d,f performance is averaged across only time steps that are predicted for all models (e.g. multiples the largest time step, 256).
Downsampling in space All comparisons across spatial resolutions are always obtained by first downsampling the data into common 32 grid, using approach that preserves mass, momentum, and energy (Same approach used by Athena++) Fig. B.2 (top row) shows the downsampling comparisons equivalent to Fig. 2g-i for each of the state variables independently.
Running time For simulation time unit of COMPDECAY-3D, the CPU runtime for Athena+ with 8 CPU processors is 44s, ~~00s, and 10000s for 32", 643, and 128 resolutions, respectively (quartic scaling, as the time step is also scaled with the resolution to compensate for numerical viscosity). In comparison, the learned model's runtime is 1s on an NVIDIA V100 GPU, and 20-30s on 8-core CPU. Note that the learned model runs at reduced spatial and temporal resolution, but preserves the dynamics of the high resolution 128 Athena+ simulation. Simulations in Athena++ may be faster implemented for GPUs. However, because scientific simulators like Athena++ specialized, each simulator's GPU implementation requires specialized engineering effort, whereas learned models can take advantage of methods designed more generally for deep learning.
18

---

Published as a conference paper at ICLR 2022
Cooling velocity Having reliable models that generalize to larger boxes increases the applicability of learned models to scientific domains by enabling experiments in regions of parameter space that would otherwise be prohibitively expensive to simulate. For example, for COOLMIXLAYER 3D, the dynamics undergo a transition when the box width increases relative to the cooling length (Pturb/caol), which is difficult to study because simulations for these widths essentially prohibitive. Thus, we want to understand what affects the learned simulator's ' ability to generalize to range of box widths (the length Le and width Ly of the input tensor X) not previously seen in the training data. For COOLMIXLAYER- flowing velocity at the low density fluid boundary that develops as a means to resupply the energy that has been radiated away in the mixing layer. Cooling velocity is useful metric because it a scalar quantity that depends on the box width. Furthermore, understanding how turbulent dynamics at mixing layer pull heat from the surroundings is scientifically relevant for questions in astrophysics (Fielding et al., 2020). We evaluate generalization to different box sizes (Lgg = Ly e [0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0 (Fig. 5b). We find that the model trained on box length Lg = I = 0.75 (orange) is not able to produce trajectories with the correct cooling velocity for other box sizes. However,
we can look at the cooling velocity, the average in-
also find that augmenting the dataset with data from range of box sizes (Lz Ly E [0.5.0.75, 1.0, 1.25)) improves accuracy of the cooling velocity estimate for the unseen box lengths, although the generalization outside the training domain remains imperfect and unreliable across seeds. We speculate that achieving this form of generalization would require stronger inductive biases and/or more sophisticated dataset engineering.
The choice of metrics can impact the utility of learned simulators for scientific applications. Here we describe few physically motivated metrics we use to evaluate the learned models.
Other Statistical Metrics
We use the power spectral density, which we quantify for the total energy in Figure 2. quantify how well the different models are preserving information at different resolutions. Simulators at coarse resolution are known to lose nigh-frreuency information, and we are specifically interested understanding how learned and physics-based simulators compare in terms of preserving detailed, high-frreuencyy components. To quantify the power spectral density, we n-- Fourier transform, take the magnitude at each point of the Fourier transformed signal to get the n-D PSD, and convert this to a ID spectrum by taking the mean magnitude over points in the n-D frequency grid with the same frequency vector length to get ID power spectral density PSD(k) over scalar frequency k = VE 122 for spatial dimensions the environment along dimension) are cutoff, leaving only frequencies that fall within the sphere of radius 1. We plot log PSD and compute spectrum error over these frequencies. The spectra for different state variables are shown in Figure B.1 for the coarsened Athhen++ + models and Dil- ResNet, and in Figure B.3 for the learned models. The error over time for the spectra is shown in Figures B.2 and B.2.
Frequencies with a vector of length > (the length of
The probability distribution functions (PDFs) of the fluid quantities provides important basis for understanding turbulent flows. Due to the inherently chaotic nature of turbulence, statistical measures are often the most robust measures of their properties. The shape and widths of PDF' are common metrics for characterizing turbulent flows(Federrath & Klessen, 2012). Moreover, joint PDFs of fluid or pressure-entropy) are used understand more complex flow behavior and to characterize which flow components are responsible for observed behavior (Fielding et al., 2020). Joint distribution functions are often especially useful when certain physical processes only take place in specific region of phase space (e.g., combustion radiative cooling). PDFs for different state variables and joint distributions (entropy v. pressure, density V. pressure) are shown in Figure B. for the coarsened Athenat models and Dil-ResNet, and in Figure B.3 for the learned models. The error over time for these statistics is shown in Figures B.2 and B.2.
quantities (c.g., reloccty-temperautee
Another measure of interest are spatial autocorrelation measures, particularly velocity autocorrelation. Like the histograms, these capture statistical structure and provide a metric that is more invariant to the chaotic behavior of turbulence. The 7thh order autocorrelation is closely related to the velocity structure function, (2) = Er [ - v=ll"), for velocity v, defined at a position vector r (e.g., v,), with unit vector u multiplied by scalar é. Seminal self-similarity arguments that underpin turbulent theory lead to the prediction that Sp(e) o go/ (Kolmogorov, 1941; Frisch, 1995). Structure and autocorrelation functions, therefore, provide important measures on the ability of learned simulators capture subtle, but scientifically essential properties of turbulent flows. First-
19

---

Published as a conference paper at ICLR 2022
and second-order autocorrelation functions for different state variables shown in Figure B.1 for the coarsened At athena++ ++ + models and the Dil-ResNet model and Figure B.3 for models trained on COMPDECAY-3D. The emror for each of these functions time shown in Figures B.2 and B.4.
Model comparison Fig. B.7 shows predicted rollouts for KS-ID of different learned models presented in Figure 4a. Fig. B.3 shows statistics of the final frame of the predicted rollouts for different learned models presented in Figure 4. The error of each of these statistics over time is shown in Fig. B.4. Videos show the rolled out predictions for different models for INCOMP-2D and COMPDECAY-3D.
KS-1D generalization Figs. B.8 and B.9 show generalization to larger domains and longer trajectories. While the learned simulations do not perfectly capture the ground truth, that qualitative features of turbulence are preserved across the rollout.
COMPDECAV-3D generalization Figs. B.8 and B.9 show quantitative generalization performance of the Dil-ResNet model to larger domains and longer trajectories on KS-1D. While the learned simulations do not perfectly capture the ground truth, we see that qualitative features of turbulence are preserved across the rollout. Figures B.2 and B.4 show different metrics for different variables across rollout (pink background indicates the region not seen during training). Videos show the predicted rollouts when generalizing to different initial conditions and to longer rollouts.
20

---

Published as a conference paper at ICLR 2022
Energy Field
Pressure
11
Log Spociium
Reslett
Atherae 64
11-
10
Atheras- 32
Athenae-
10
16mannd Truih)
0.25
0.50
0.75
1.00
0.25
0.50 Frequency
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Frequarcy
Frequency
Frecuency
1.0
1.0
1.0
1.0
Adboar ((=1)
0.5
0.5
0.0
0.0
10
20
Autocor
0.2
0.2
0.05
0.2
0.00
0.0
20
10
20
10
10
20
Dislance
Diviance
Distance
Div.arce amc.arce
1500
3000
1500
Hecooram Woos Etr
1000
1000
500
500
500
Energy
Density
Velecty
Dil-ResNet (seed 1)
Dil-ResNet (seed 2)
Dil-ResNet (seed 3)
Athena + 64 Athena 32
Ground Truth
Pressure
2
1
2
2
1
Entropy
Entropy
Entropy
Entropy
Entropy
Entropy
Pressure
2.
Density
Density
Density
Density
Density
Density
Figure B.1: COMPDECAV-3D Comparison of learned Dil-ResNet model trained with noise=0.01 to Athena+ ++ at resolutions of 323, 643, and 128 (ground truth) for variety of different metrics at = 0.99 (the end a rollout that lasts the duration seen during training).
21

---

Published as a conference paper at ICLR 2022
3.5
3.5
3.5
2.00
2.00
11
200
Entropy
Demalty Pressure
Figure B.2: COMPDECAY-3D Model metrics time across learned Dil-ResNet model trained with noise=0.01 and Athena++ coarsened to different levels of resolution. Except for the last row, which shows the error in the Entropy Pressure and Density v. Pressure 2D histograms, each column is state variable (Energy Field, Density, Velocity Components, and Pressure) and each row is the error for different function over the data. Note rollouts start have elapsed, because Te-Net takes in 6 frames and all models should start at the same point. Same color scaling is used across each 2D histogram row, Background pink for times not shown during training.
0.16, after 5 model steps would
22

---

Published as a conference paper at ICLR 2022
Energy
Density
Velocity
Pressure
Di-ResNec
Log Spectrum
12.0
Uett
TF-Net (T
Con-TF-Nee (T=5)
9
11.0
0.25
0.50
0.75 1.00
0.25
0.50
0.75 1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
3.0
Frequeniy
Frequency
Frequency
Frequency
Altocor (bw1)
0.4
0.4
Autcorr
0.2
0.05
0.0
0.0
0.00
0.0
10
10
20
1.0
20
20
Osllance
Distance
Distance
Distance
4000
6000
Wlos Err
2000
2000
1000
2000
10
15
-2
Energy
Densky
Velccity
Pressure
ResNet
Con ResNet
U Net
TF. Net
Con Net
ENC (k-D)
ENO (k-32))
Grourd Truth
Pessun
Rossure
Figure B.3: COMPDECAY-3D Comparison of learned models trained with noise=0.01 for a variety of
different metries at 0.99 (the end of rollout that lasts the duration seen during training).
23

---

Published as a conference paper at ICLR 2022
0.75
0.5
0.-
Caso Mhasled
0.00
00
TRD 8o-1
Tine
Tyee
Thee
14'
100
1978
1989
1903
0.7
0.5
Tira
Tima
-05-
107
1.00
Entrosy Pressure
Deriitt v. Prespure
Figure B.4: COMPDECAY-3D Model metrics over time across different learned models trained with noise=0.01. Except for the last row, which shows the error in the Entropy v. Pressure and Density v. Pressure 2D histograms, each column a state variable (Energy Field, Density, Velocity Components, and Pressure) and each row is the error for different function the data. Background is pink for times not shown during training. One seed shown per model (quantitative results for multiple seeds shown in Figure B.11)
Compressible Decaying seeds)
128
1.0
Allanua et 24
RMSE in preserved quantities (3 seeds)
(lu ro4a). (MAtad BiC2)
No moloe Notu
KS-10 Meah
0.05
800
(F:00 Sheegatte
Wereenm
0.1
100
200
300
600
Time
0.
0.5
0.5
3.0
3.5
2.0
3.0
4.0
Figure B.5: (left) KS equation (KS-1D) net velocity error, and Incompressible Turbulence (INCOMP- 2D) divergence error as function of model step. Training with noise helps keeping the values bounded to be close to O. (right) Preservation of the 5 conserved quantities in 3D Compressible Turbulence (COMPDECAV-3ID) as a function of time. In this case, training with noise does not completely prevent drift of the conserved quantities.
24

---

Published as a conference paper at ICLR 2022
Ground Truhh
Example
Example
Example
Example
Thedican Prod Teddiin Dradi medician Praditan Reaician
10
Dowmssmple
focter
50
100
150
50
1.00
150
50
100
150
50
100
150
Time
Time
Tim
Time
Figure B.6: (top) Sample trajectories from the model trained on the KS-1D dataset at different temporal downsampling factors. (bottom) MSE and correlation performance of ML models for different downsampling factors in time function of simulation steps.
Example
Example
Example
Example
Tush
20
NA
50
100
150
0
50
100
150
50
100
150
50
100
350
Time
Time
Time
Time
Figure B.7: KS-1D Rollouts for models trained with noise=0.01.
25

---

Published as a conference paper at ICLR 2022
Example
Example
Example
Example
Groundt Truth Spocial
Peed on
Figure B.8: Dil-ResNet model generalizing to larger spatial domains on KS-ID (trained on domains 2x wide).
THE
So
3o
Figure B.9: Dil-ResNet model generalizing longer trajectories KS- ID (trained trajectories with 181 simulation time units).
Larger Space Generalization
Longer Rollout Generalization
5.6
2.0
2.1e115
b.1e.10
Dii-ResNet
Velecity RMSE
1.5
Velecity RMSEE
1.5
Con-Dil ResNet
TF- Net (T-6)
1.0
1.0
Con-TF-Net (T=6) (K=8)
FNO (full rank)
noise
noise
noise
noise
Figure B.10: Model comparison for KS- 1D generalization to larger spatial domains (left) and longer rollouts (right).
26

---

Published as a conference paper at ICLR 2022
CompDescy Intial Condition Gevraliztion
CompDesay 3D Longer Rollout Generalization
50
ConDil Reshit
UkMed (pri)
Enirgy deSE
2,0
Con TF Net
100
RND (o-f)
rank)
1.0
Figure B.11: Model Comparison for Generalization. (left) Generalization different ratios of solenoidal to compressive components in the initial conditions for learned models (trained with noise=0.01). 0=more compressive, ==more solenoidal components in initial conditions. The gray line marks the value the models were trained on (0.66). (right) Generalization to longer rollouts. Added noise during training improves stability and therefore leads to lower error on longer rollouts. Cutoff point for TF-Net with no noise has RMSE 2.99, and cutoff bar for Dil-ResNet goes to Inf (numerical error). As shown Figure 5, we see that adding the constraint to the loss function makes the model generalize more reliably to the more compressive regime, and that training noise promotes stability for longer rollouts.. Note that these results are numerically different from Figure because rollouts started from the sixth timestep allow comparison with Te-Net, which takes input sequence of length 6. 6.
27


Reviews:
[{"review_id": "sO9-q7mFxJ", "reviewer": "Reviewer_1vW7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would like to applaud the authors' effort to perform extensive experiments on different systems with different models. I think the experimental results well support a general claim that a learned simulator is indeed promising for some turbulence phenomena.\n\nThat being said, I feel that the main claim of the paper is somewhat vague. Is it on the goodness of the proposed architecture, called dilated ResNets? If it is, then I would like to know more about the difference from the baseline methods. Since such a comparison result is shown only in a small subsection, as the authors also acknowledged (\"... although most learned models were able to provide a good qualitative match ...\"), I could hardly understand the overly notable superiority of the dilated ResNets. Figures B.3 and B.4 also provide some comparison, but to me, they were not really clear, either. Maybe it would be helpful if the authors can add some more explanations about the quantitative and qualitative differences between the architectures if any. If the authors think that the inter-architecture difference is not necessarily significant, then the main claim of the paper should be emphasized in a different way.\n\nA small question on the training set. In the last two datasets, are the training sets generated with Athena++ 128 (and then downscaled)? If so, maybe it should be stated more clearly in the main text (sorry if there's already such a statement).\n\nIt would be kind if the semantics of \"predicted Energy Field and the Log Energy Spectrum\" are presented somewhere in the main text. I think many machine learners are not familiar with such notions.\n\nOn page 8, \"We found that training with prevents this in some cases: ...\" --> with what? Maybe a word is dropped.", "summary_of_the_paper": "The authors evaluate some different architectures of neural nets for learning turbulence. They showcase the results for four different systems and in terms of several metrics. The main conclusion is that a learned simulator can indeed be good even for turbulence and possibly improved with additional mechanisms such as additional noise and constraints.", "summary_of_the_review": "The work seems to be valuable in the sense that it conducts extensive experiments on simulator learning for turbulence. Meanwhile, the main claim of the paper is somewhat vague because the difference between the proposed architecture and the baseline architectures is not explained in full detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cypnkhuopQY", "reviewer": "Reviewer_HsnX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The demonstration doesn’t clearly show the benefits of this method. What’s the difference between the three blue lines in figs 2b and 2d? Where is the red line that is supposed to denote the ground truth? Isn’t the error is larger in Dil-ResNet in fig 2b? What’s the point of t=1000 for a simulation that is already imprecise at t=1? Is it t=1 or t=1000 in figs 2f-2i? There are similar issues in other figs, like figs 4 and 5. It’s hard to show significant advantages with these low-accuracy simulations.  \n\nAs I said before, it would be more convincing if the authors can demonstrate some standard examples, such as Taylor Vortex, Karman Vortex, Frog-leaping, etc. For some turbulent flows that are chaotic themselves, the solutions at the resolution of 32 are meaningless, and the results are even worse than some large-eddy simulations. \n", "summary_of_the_paper": "This paper present a  neural network-based model by training learned simulators at low spatial and temporal resolutions to capture turbulent dynamics generated at high resolution. It also shows that the proposed model can simulate turbulent dynamics more accurately than classical numerical solvers at the same low resolutions across various scientifically relevant metrics. Finally, a series of numerical examples including one-dimensional, two-dimensional, and three-dimensional nonlinear equations are used to demonstrate the feasibility of the algorithm. I reviewed this paper in NeurIPS 2021, which was titled “Learned Simulators for Astrophysical Turbulence”. The authors revised the paper based on previous reviews. Thanks for the efforts the authors have made.", "summary_of_the_review": "The authors do make efforts on this paper, however, only some statistical presentation of some low-precision turbulence simulation data is not convincing enough to make it attractive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RzuKKZY3CR", "reviewer": "Reviewer_YTSP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors compare a variety of models to the proposed dResNet, with a simpler Unet, the TF net, and a Fourier-Op network. It was, e.g., interesting to see that the latter does fairly badly in some of the tests. They also evaluate how introducing noise in the predictions helps to stabilize the rollouts. \n\nThe results contain a nice variety, and it is impressive to see predictions of full 3D turbulence simulations. The resolutions are not huge, with 32^3, but nonetheless contain complex dynamics.\n\nThe paper also contains an interesting evaluation of generalization for different initial conditions and simulation domain sizes. The longer rollouts are also an important point to study in this context. Here I found the presentation of the paper unclear: how many steps did the NN actually perform? The captions often give conflicting information: e.g., in Fig. 2, the time axis goes from 0 to 4, and the caption talks about t=992. It would be important to specify these durations more clearly, such that readers know how many steps the models predicted independently.\n\nMinor, but for divergence constraints I'd recommend to cite the Tompson et al. 2018 paper \"Accelerating eulerian fluid simulation with convolutional networks\". Also, the title of the website and on OpenReview differ. \"Efficient turbulence\" sounds very generic to me, being more specific in the title is a good idea.\n\nHere are a few additional points which I hope the authors can clarify in the rebuttal:\n\n- For the proposed dResNet architecture, what is the role of the encoder decoder? If it's just a single linear layer, why is it so important? Are there comparisons showing the usefulness of the encoding-decoding step?\n\n- The graphs in Fig. 3 (and others) are confusing - time is shown as 0 to 1, but the caption claims time steps of \\Delta t = 1, and on the right up to 128. What are the real sizes, and how many time steps were evaluated?\n\n- In Table A1, no model sizes are given. How many parameters did the different models have?\n", "summary_of_the_paper": "This paper proposes a method to predict turbulent dynamics of coarse simulations. To achieve this, a modified ResNet architecture is proposed, which the authors call Dilated ResNet (dResNet). This dResNet is combined with an encoder/decoder model inspired by graph networks. Unfortunately, the details of this architecture are not made very clear, but I hope this is something the authors can adress in future revisions of their work. Beyond the proposed architecture, the paper presents an interesting range of test cases from 1D KS to 3D compressible turbulence.\n", "summary_of_the_review": "In summary, the paper represents an interesting read, and contains a variety of very interesting results. There are a few open points, but, giving the authors the benefit of doubt, I hope that these can be cleared up during the rebuttal. Given my current understanding of the work I would argue for accepting the paper to ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OSXLJlc6phi", "reviewer": "Reviewer_zinf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper uses the existing modeling architecture on predicting turbulent flows. This paper shows several advantages of the proposed model: Compared with the classical solvers, using the same small resolution, the proposed model can produce more accurate results. It can also produce as accurate (or more accurate) results when using larger timesteps. The smaller grid, larger timestep, and the ability to utilize GPUs make this model more efficient than the classical solver. \nThis paper also compares different network architectures, which shows the best result is from using Dilated ResNet.\nThere are also certain discussions of the results, including the experiments of tuning training noise and temporal downsampling to solve the unstable issues, using different timesteps, evaluating longer rollouts, using different initial conditions, and using different box sizes.\n\nOther questions:  \nFor the CNN padding, can it deal with the scenarios of non-periodic and non-fixed boundary conditions?  \nWhen /Delta t changed, does the model need to be re-trained?  \nUsing classical solvers, as the resolution increases, the results would gradually converge to the ground-truth value. Does this property hold true for the proposed model?  \nIf we want to get more accurate results (for example, as accurate as using 256^3 Athena++, can the proposed model still out-performs the classical solver?  ", "summary_of_the_paper": "This paper aims to learn a simulator that predicts large-scale turbulent dynamics of a known system. As shown in the results, on coarse grids, the proposed method predicts more accurately than the classical solvers, especially on preserving the high-frequency information. ", "summary_of_the_review": "Although using some existing learning models, there are certain contributions of this paper in applying the ml models on predicting the turbulent flows, which shows some advantages over the classical solvers.\n\n===============\nUpdates: I thank the authors for the response. The response addresses my concern about the contributions of the paper. I would agree that  the empirical work of this paper is impressive. I decide to change my score towards acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper compared different architectures of deep neural nets for learning full 3D turbulence simulations.  On coarse grids, the proposed method predicts more accurately than the classical solvers, especially on preserving the high-frequency information.  The reviews think the paper is clearly written with strong experiments. Pls include the suggested references in the final version.
Decision: Accept (Poster)