SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\int_ {t _ {0}} ^ {t _ {1}} d t \int f (u; v (t)) \bar {p} (u; v (t)) d u \approx \int d w f _ {w} (w; v) p _ {w} (w) \approx \frac {1}{K} \sum_ {i = 1} ^ {K} \mathcal {D} \left(Z _ {1} ^ {(i)}\right), \tag {16}
$$
where  $w$  is the spatio-temporal lifted version of  $u$ , i.e., the space of DNS trajectories in space and time, and  $f_{w}$  absorbs both  $f$  in the closure Eq. (14) and the implicit conditioning in Eq. (13), and  $p_{w}$  is a prior for the trajectories. This notational change allows us to approximate the integral directly by a Monte-Carlo approximation on the lifted variables, i.e., the distribution of trajectories (as shown in Fig. 2) which are modeled using the NSDE in Eq. (15). See Algo. 1 for the calculation steps.
We stress that while we motivate our design via mimicking DNS trajectories,  $Z$  is low-dimensional and is not replicating the real DNS field  $u$ . However, it is possible that with enough training data,  $Z$  might discover the low-dimensional solution manifold in  $u$ .
Training objective The NSDE is differentiable. Thus, with the data-driven closure model Eq. (16), we can apply end-to-end learning to match the dynamics with the closure model to ground-truths. Concretely, let  $v(t_0)$  be  $\bar{u}(t_0)$  and we numerically integrate Eq. (11)
$$
v \left(t _ {1}\right) \approx v \left(t _ {0}\right) + \int_ {t _ {0}} ^ {t _ {1}} \mathcal {R} _ {c} ^ {\mathrm {N S}} (v (t)) d t + \mathcal {M} \left(v \left(t _ {0}\right)\right) \left(t _ {1} - t _ {0}\right) \tag {17}
$$
This gives rise to the likelihood of the (observed) data in NSDE. Specifically, following the VAE setup of [52], we have
$$
- \log p (\bar {u} (t _ {1}) | Z) = \mathcal {L} ^ {\text {r e c o n}} \left(v \left(t _ {1}\right), \bar {u} \left(t _ {1}\right)\right) = \left(2 \sigma^ {2}\right) ^ {- 1} \| v \left(t _ {1}\right) - \bar {u} \left(t _ {1}\right) \| ^ {2} \tag {18}
$$
where  $\sigma$  is a scalar posterior scale term. The training objective  $\mathcal{L}(v,u) = \mathcal{L}^{\mathrm{recon}}(v,u) + \mathrm{KL}^{\mathrm{NSDE}}$  includes a KL divergence term associated with the neural SDE. See Appendix D for more details on the additional term.
To endow more stability to the training, following [81] the training loss incorporates  $S$  time steps  $(S > 1)$  of rollouts of the LES state:
$$
\mathcal {L} ^ {(S)} (v, \bar {u}) = \sum_ {k = 1} ^ {S} \mathcal {L} (v (t _ {0} + k \Delta t), \bar {u} (t _ {0} + k \Delta t)) \tag {19}
$$
For a dataset with multiple trajectories, we just sum the loss for each one of them and optimize through stochastic gradient descent.
3.2 Implementation details
We describe details in the Appendix G for the Transformer-based encoder and decoder and Appendix D for implementing NSDE with a Transformer parameterizing the drift and diffusion terms.
We showcase the advantage of our approach using two instances of a chaotic Navier-Stokes flow: 2D Kolmogorov flow, [47] at a Reynolds number of 20,000 and flow past a circular cylinder at a Reynolds number of 500.
Dataset generation The reference data for Kolmogorov flow consists of an ensemble of trajectories generated by randomly perturbing an initial condition. The flow past cylinder dataset is generated as a single long, chaotic trajectory, split up into 14 equal segments with an initial subsegment thrown away to avoid correlation between the segments. Each trajectory is generated by solving the NS equations directly using a high-order spectral finite element discretization in space and time-stepping is done via a 3rd order backward differentiation formula (BDF3) [23], subject to the appropriate Courant-Friedrichs-Lewy (CFL) condition [21]. These DNS calculations were performed on a mesh with 2304 elements for Kolmogorov flow and 292 elements for the cylinder flow with each element having a polynomial order of 8 and 13 respectively. The DNS trajectories were then sampled to a coarse grid with order 4 and 5 respectively for the two datasets. For more details see Appendix B.
Benchmark methods For all LES methods, including niLES, we use a  $10 \times$  larger time step than the DNS simulation. We compare our method against a classical implicit LES at polynomial order 4 using a high order spectral element solver using the filtering procedure of [28]. As reported in [14], the implicit LES for high order spectral methods is comparable to Smagorinsky subgrid-scale eddy-viscosity models. We also train an encoder-decoder deterministic NN-based closure model using a similar transformer backend as our niLES model. This follows the procedure from prior works [56, 81, 47, 24] in training deterministic NN-based closure models. Additionally, we train a deterministic encoder-processor-decoder network with a neural ODE processor [19] with strictly more parameters than the niLES model.
Metrics For measuring the short-term accuracy we used root-mean-squared-error (RMSE) after unrolling the LES forward. Due to chaotic divergence, the measure becomes meaningless after about 1000 steps. For assessing the long-term ability to capture the statistics of DNS we use turbulent kinetic energy (TKE) spectra. The TKE spectra is obtained by taking the 2D Fourier transform of the velocity field, after interpolating it to a uniform grid. Then we obtain the kinetic energy in the Fourier domain and we integrate it in concentric annuli along the 2D wavenumber  $k$ .
Long-term turbulent statistics We summarize in Figs. 3 and 5. In the short-term regime, both niLES and deterministic methods achieve higher accuracy than the implicit LES, even when unrolling for hundreds of steps. Beyond this time frame, the chaotic divergence leads to exponential accumulation of pointwise errors, until the LES states are fully decorrelated with the DNS trajectory. At this point, we must resort to statistical measures to gauge the quality of the rollout.
When considering reconstruction errors from very short-term rollouts, such as when the model is unrolled only 8 times, identical to the training setup, the deterministic approaches and niLES yield similar reconstruction errors. Beyond this regime, and going up to 1000s of rollout steps, the probabilistic approach of niLES provides superior generalization ability even when measuring a deterministic reconstruction loss.
Furthermore, niLES captures long-term turbulent statistics significantly better than the other two approaches, particularly in the high wavenumber regime. The deterministic NN is not stable for long term rollouts due to energy buildup in the small scales, which eventually leads the simulation to blow up.
Inference costs The cost of the inference for our method as well as the baselines are summarized in Table 1. niLES uses four SDE samples for both training and inference, and each SDE sample is resolved using 16 uniformly-spaced time steps corresponding to a single LES time step. The inference cost scales linearly with both the number of samples and the temporal resolution. However, niLES achieves much lower inference cost than the DNS solver while having similarly accurate turbulent statistics for the resolved field. The deterministic NN model has slightly lower inference cost than niLES, since it can forego the SDE solves. While the implicit LES is the fastest method and is long-term stable, it cannot capture the statistics especially in the high wavenumber regime.
Limitations A drawback of our current approach stems from using transformers for the encoder-decoder phase of our model, which might result in poor scaling with increasing number of mesh
Figure 3: Root mean squared error (RMSE) over the first 1000 steps (first two columns) and the turbulent kinetic energy (TKE) spectrum  $E(k)$  averaged over the first 2500 steps (right two columns) of two independent test trajectories unseen during training or validation. niLES has an improved ability to capture the long term statistics accurately compared to both implicit LES and deterministic NN. The energy buildup in the small scales (large wavenumber) in the deterministic NN model eventually leads to unstable trajectories.
(c)
Figure 4: Comparison between rollout predictions after 800 LES steps on a held-out trajectory. Velocities in the  $x$  (top row) and  $y$  (bottom row) directions respectively. Snapshots of filtered DNS (reference) (a), niLES (b), implicit LES (c) and deterministic NN models (d). The niLES captures several finer scale features of the flow consistent with the reference filtered DNS trajectory. The implicit LES has an overall smoothing effect and some turbulent structures are not captured. The deterministic NN LES shows artifacts which indicate instability.
elements. Alternative architectures which can still handle interacting mesh elements should be explored. Additionally, the expressibility of the latent space in which the solution manifold needs to be embedded can affect the performance of our algorithm, and requires further study.
The relevant literature on closure modeling is extensive as it is one of the most classical topics in computational methods for science and engineering [68]. In recent years, there has also been an explosion of machine learning methods for turbulence modeling [9, 25] and multi-scaled systems in general. We loosely divide the related works into four categories, placing particular emphasis on the treatment of effects caused by unresolved (typically small-scaled) variables.
Classical turbulence methods primarily relies on phenomenological arguments to derive an eddy viscosity term [48], which is added to the physical viscosity and accounts for the dissipation of energy from large to small scales. The term may be static [4], time-dependent [76, 31] or multi-scale [39, 40].
Data-driven surrogates often do not model the closure in an explicit way. However, by learning the dynamics directly from data at finite resolution, the effects of unresolved variables and scales are expected to be captured implicitly and embedded in the machine learning models. A variety of architectures have been explored, including ones based on multi-scaled convolutional neural networks [70, 82, 77], transformers [11], graph neural networks [73, 49] and operator learning [65].
Figure 5: Average RMSE over 10 testcases on the cylinder wake dataset. Shaded regions depict the spread (minimum and maximum) among the testcases.
Figure 6: Comparison between rollout predictions of the horizontal component of velocity in the cylinder wake dataset after 100 LES steps on a held-out testcase. From top to bottom: snapshots of (a) filtered DNS (reference), (b) niLES, and (c) ILES. The niLES captures correct shapes of the vortices when compared to ILES. The latter has an excessive smoothing effect and the vortices appear more rounded.
Table 1: Inference times in wall clock seconds to evolve the state for one second of simulation time for the Kolmogorov dataset
Method	DNS	Implicit LES	Deterministic NN	niLES (Ours)
Inference time [s]	15600	8.85	11.4	23.8
Hybrid physics-ML contains a rich set of recent methods to combine classical numerical schemes and deep learning models [61, 6, 47, 56, 24, 81, 58, 35]. The former is expected to provide a reasonable baseline, while the latter specializes in capturing the interactions between modeled and unmodeled variables that accurately represent high-resolution data. This yields cost-effective, low-resolution methods that achieve comparable accuracy to more expensive simulations.
Probabilistic turbulence modeling seeks to represent small-scaled turbulence as stochastic processes [1, 32, 33, 20, 36]. Compared to their deterministic counterparts, these models are better equipped to model the backscattering of energy from small to large scales (i.e. opposite to the direction of energy flow in eddy viscosity), which is rare in occurrence but often associated with events of great practical interest and importance (e.g. rogue waves, extreme wildfires, etc.).
Our proposed method is inspired by methods in the last two categories. The closure model we seek includes the coarse solver in the loop while using a probabilistic generative model to emulate the stochastic process underlying the turbulent flow fields. This gives rise to long-term statistics that are accurate as the inexpensive neural SDE provides a good surrogate for the ensemble of the flows.
Due to chaotic divergence, it is infeasible to predict the state of the system with pointwise accuracy. Fortunately, in many systems of interest, the presence of an attractor allows a much lower-dimensional model to capture the essential statistics of the system. However, the time evolution of the attractor is not known, which makes building effective models challenging.
In this work we have argued that taking a probabilistic viewpoint is useful when modeling chaotic systems over long time windows. Our work has shown modeling the evolution with a neural SDE is beneficial in preserving long-term statistics and this line of thinking is likely fruitful.