12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19–22, 2022

DEEP REINFORCEMENT LEARNING FOR LARGE-EDDY SIMULATION
OF WALL-BOUNDED TURBULENCE

Junhyuk Kim
Hyojin Kim
Department of Mechanical Engineering
Department of Mechanical Engineering
Yonsei University
Yonsei University
50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea
junhyuk6@yonsei.ac.kr
gywls072@yonsei.ac.kr

Jiyeon Kim
Changhoon Lee
School of Mathematics and Computing
Department of Mechanical Engineering
Yonsei UniversitySchool of Mathematics and Computing
50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea
Yonsei University
jykim3994@yonsei.ac.kr
50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea
clee@yonsei.ac.kr

ABSTRACT

pioneering approach, Smagorinsky (1963) suggested a modelthat the anisotropic part of SGS stress tensor τi j is represented
by the resolved strain-rate tensor S̄i j and eddy viscosity νt .
For an effort to apply LES to complex flows, some modifications such as dynamic Smagorinsky model (DSM) (Germanoet al., 1991; Lilly, 1992) and Vreman model (Vreman, 2004)
were suggested. While those existing SGS models continue
to evolve and are successfully applied to practical problems,
there is certainly room for significant improvement in termsof accuracy and cost. For example, coarse-grained LES with
DSM is quite inaccurate in canonical flows.
In recent years machine learning models using filtered
DNS data for training are challenging to the conventional SGSmodels. In turbulent channel flow Gamahara & Hattori (2017)and Park & Choi (2021) developed neural network (NN)based model with the classical supervised learning framework,
which is composed of four steps; data collection, training, apriori test and a posteriori test. However, due to the significant
mismatch between a priori test and a posteriori test, it is difficult to develop a reliable model. It is most likely because themodel was trained only at the equilibrium state and the effect(reaction) of modeled SGS stress in actual LES was not taken
into account in the training process. Another major drawback
is that the classical supervised learning requires high-fidelity
data for training. In real-world problems, such data is usuallynot available, and only some partial or statistical data can becollected. As an alternative algorithm, we are expecting thatdeep reinforcement learning (DRL) would help overcome thedisadvantages of classical supervised learning. Very recently,
Novati et al. (2021) proposed the DRL-LES framework for developing a SGS model in forced homogeneous isotropic turbulence, which could predict energy spectrum quite well. But thelearning is not successful in the case that actions are fully collocated in all grids without interpolation. To overcome the previous limitations, we proposed a physics-constrained DRL algorithm that perfectly guarantees reflectional equivariance andboundary conditions of the SGS model. Through this, we extend DRL to LES modeling in wall-bounded turbulence, which
remains as a challenging problem due to its inhomogeneity.

Large-eddy simulation (LES) is actively carried out for
many scientific and engineering problems. Recently supervised learning approaches using high-fidelity flow data in
training process are applied to LES modeling, however, there
are significant disadvantages. For example, the trained model
is not effective in actual LES, and such high-fidelity data is
usually not available in real-world problems. To overcome
these, we employed deep reinforcement learning (DRL) where
actual LES and training of subgrid-scale (SGS) model are carried out simultaneously using only target statistics as given information. We additionally applied physical constraints suchas reflectional invariance and wall boundary conditions on
DRL for reducing the training cost. Through this, we are challenging to find a reliable SGS model for three-dimensional
LES of wall-bounded turbulence. The DRL model that produces the local SGS stress based on the local velocity gradientwere trained, as a result, we found that in various training environments DRL could discover models that make mean velocity
and mean Reynolds shear stress of actual LES be consistent
with the target, while the conventional SGS models usuallymispredict them. We conclude that DRL would be a effective
tool for turbulence modeling in practical problems.

INTRODUCTION
For many problems involving turbulent flows, precise
simulation is important, but in real-world problems the cost ofdirect numerical simulation (DNS) that represents the smallest
scale is still too high despite of the rapid development of computational hardware and numerical algorithm. For the purposeof improving the trade-off between cost and accuracy of simulation, several types of turbulence models have been developed.
As one of them, we focus on the large-eddy simulation(LES) where instantaneous vortical motions are represented
and the subgrid-scale (SGS) stress should be modeled by theresolved scale information. Many algebraic SGS models basedon statistical and physical analysis have been proposed. As a112th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19–22, 2022

Figure 1. Illustration of present work. A DRL-LES framework for developing a subgrid-scale (SGS) model in wall-bounded turbulence is proposed. Running of three-dimensional (3D) LES and learning of deep neural network (DNN) that produces the SGS stressfrom the resolved velocity gradient, are carried out simultaneously. Target statistics for training are the mean viscous stress and themean Reynolds shear stress, used for reward calculation.
At this time, the target to maximize is the long-term reward
i−t r with 0 < γ < 1) that the instantaneous re(Rt = ∑∞ii=t γMETHODOLOGYward rt coming out after the action is accumulated for a time
The DRL-LES framework consists of a LES solver andhorizon. In our LES SGS modeling problem, the state and aca reinforcement learning algorithm. LES of turbulent channeltion are the resolved flow variables and modeled SGS stresses,
flow with a NN-based SGS model and its training using therespectively, and the reward can be defined as the statistical
data generated from the LES are carried out simultaneously,
distance between the target (filtered DNS) and the LES. In theas shown in figure 1.
training process, the DRL algorithm optimizes the DNN-based
SGS model in the direction of making the statistical accuracyof the LES precise. In this study, one of DRL algorithms, deep
Numerical method
deterministic policy gradient (DDPG) is used.
The governing equations of LES are filtered incompressible Navier–Stokes equations with a spatial filter operation (¯·),∂ τi j∂ ūi ∂ (ūi ū j )∂ p̄
1 ∂ 2 ūi∂ ūi= 0,+=−+−(1)
∂ xi
∂t∂xj
∂ xi Reτ ∂ x j ∂ x j
∂xj

"Eπω∞

∑γ

#i−tri ≈ Qθ (st , at )

(3)

i=t

The equation is non-dimensionalized by the channel half width(δ ) and the friction velocity (uτ ). x1 (x), x2 (y), and x3 (z) mean
the streamwise, wall-normal, and spanwise directions, and thecorresponding velocity components are ui (= u, v, w), respectively. The simulation parameter is the friction Reynolds number Reτ = uντ δ where ν is the kinematic viscosity. The SGSstress τi j is modeled by DNN πω with trainable parameters ω.
The ultimate goal of present work can be defined as below.

Here, training of Qθ with deterministic policy is mostly depending on recursive relation of Bellman equation.
Using critic DNN that predicts the long-term reward, we
can train the actor DNN πω in the direction of increasing thecritic value. Its objective function to maximize is as follows :argmaxω Est [Qθ (st , πω (st ))]argminω ||Starget − SLES(πω ) ||(4)(2)
Our purpose through DRL is to discover an optimal SGS
model πω that could make the statistics of LES precise. πω isfully connected neural network that produces SGS stress basedon the resolved velocity gradient and the local grid size. This
local framework can be extended to more complex flow easily. And critic DNN is convolutional network as a surrogate
model predicting the future state of flow based on the presentstate. And the instantaneous reward is defined by two statistical quantities, mean viscous stress and mean Reynolds stress,
as follows.

where S is statistics of fDNS and LES. Although the goal of
DRL is the same as developing a DNN-based SGS model withhigh statistical accuracy, its results can be highly dependent
on the choice of state and action, the definition of reward, andsome important techniques of the algorithm. In the sense that
performance of a posteriori test is guaranteed if only the training is successful, the construction of the robust learning algorithm is important.

Deep reinforcement learning2

DRL is an algorithm that finds the optimal action at to receive the maximum reward in a given state st of environment.

rt (y) = ∑ −ci SifDNS (y) − SiLES (t + ∆t DRL , y)i=121/2,(5)12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19–22, 2022

Figure 2. The effect of physical constraints on DRL performance. r and b in the caption denote reflectional invarianceand boundary constraints, respectively. (a) is the accuracy ofhd ū/dyi and hū0 v̄0 i and (b) is the accuracy of hd w̄/dyi.
Figure 3. Test for (∆x+ , ∆z+ ) = (70.7, 35.3). (a), (b) and (c)are the streamwise mean velocity, the mean Reynolds shear
stress and the mean SGS shear stress, respectively.

whereS1 =


1 d ū
, S2 = ū0 v̄0 .
Reτ dyas follows:(6)
NAi =Here, ∆t DRL is the period of data collection and learning, andwe set it as 30 LES time steps, which corresponds to 5.4 walltime units. And, We carried out DRL for learning steps of
10, 000 – 15, 000 to find the optimal SGS model. And, moredetails related to DRL algorithms are presented in Kim et al.
(2022).

1/21 y∑ − SifDNS (y j ) − SiLES* (y j ) .
Ny j=1(7)Here, Ai is the accuracy of statistics. First, the accuracy ofmean viscous stress and mean Reynolds stress used in reward
is presented in figure 2(a). It is observed that DRLb,r with two
physical constraints has better speed and convergence accuracy than DRL pure without physical constraint. Also, as shownin figure 2(b), DRL without reflectional invariance could generate statistically anti-symmetric flows that are unphysical inchannel flow. It indicates that applying physical constraints on
DRL is useful and necessary.
Next, time-averaged results of actual LES with the trainedmodel are presented in figure 3 with the conventional SGS
models including DSM and no model (τi j = 0). LES withthe trained DRL model could predict the target statistics ofRESULTS
DRL was mainly carried out for Reτ = 180 and filter(grid) size (∆x+ , ∆z+ ) = (70.7, 35.3). We applied two kindsof physical constraints, boundary condition and reflectional invariance to DNN and observed their effect on training performance. The statistical accuracy used for the quantification is3

12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19–22, 2022

Figure 5. (a) and (b) are the mean SGS dissipation and SGS
transport, respectively.
−τi j S̄i j /Reτ ) and SGS transport (TSGS = Re−1τ ∂ (τi j ūi )/∂ x j )in figure 5. We found that to accurately predict the streamwisemean velocity and mean Reynolds shear stress, accurate representation of mean SGS transport is required, but the accuracyof mean SGS dissipation and backscatter are not essential requirements. Accurate prediction of SGS energy transfer might
be relevant with the high order statistics or temporal behaviorof resolved variables.
Finally, we trained the DRL model in a new environment
with more coarse grid size (∆x+ , ∆z+ ) = (94.2, 47.1). Even
though the optimal mean SGS shear stress is two times larger
than that of the previous case, DRL could find the optimal SGS
model well. Its test results in actual LES are given in figure 6.
At this condition, the prediction of DSM highly mismatcheswith the target statistics, while our DRL model successfully
predicts the learning target. It indicates once again that ourDRL algorithm could discover a successful model in the actual
LES test.

Figure 4. One-dimensional energy spectrum at y+ ≈ 15. (a)and (b) is streamwise and spanwise directions, respectively.

mean velocity and mean Reynolds shear stress almost perfectly, while DSM overpredicts the mean velocity and the magnitude of mean Reynolds shear stress and the No model highly
mispredict them. By the total shear stress equation, the DRL
model produced mean SGS shear stress −τxy very well compared to DSM. Here, we want to briefly mention the results ofclassical supervised learning. The supervised learning modelwith the same input information quite underpredicted the magnitude of hu0 v0 i in the actual LES. In some cases, the actual
LES diverged in the model using more input information. This
indicates that even when fDNS data are available, there arelimitations in developing successful models through the classical supervised learning and that it is important for the development of a successful model to reflect the reaction of the SGS
model through the online learning algorithm.
Also, the one-dimensional energy spectrum also supports
the results (figure 4). No model significantly overestimatedthe overall distribution of the energy, while DSM and DRL
predict them reasonably. But, it is noticeable that DRL shows
better prediction than DSM at low wavenumbers with high energy, although at high wavenumbers some inaccuracies are observed. Actually, (not shown here) LES with the trained DRL
model well represents the vortical structures, unlike the Nomodel.
We also present the statistics of SGS dissipation (εSGS =CONCLUSIONS
It is clear that the DRL framework is a promising algorithm for developing the SGS model of LES. And with theincrease of the available computational resources, the performance of model will naturally increase. Although wehave shown successful applicability only in channel flow, thisframework could be extended to various turbulence modelingproblems in complex geometries. For the global goal of successful predictions in various complex flows, construction ofdiverse flow solvers and learning them simultaneously should
be carried out. In the near future, we hope to develop the turbulence model that shows practical utility in real-world problems.
Although it was possible to find an optimal model through4

12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19–22, 2022
line learning algorithms, the DRL and the differentiable PDE
method, would be interesting as future work.

ACKNOWLEDGMENTS
This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government(MSIP) (2017R1E1A1A03070282, 2022R1A2C2005538).

REFERENCES
Gamahara, M. & Hattori, Y. 2017 Searching for turbulencemodels by artificial neural network. Phys. Rev. Fluids 2 (5),054604.
Germano, M., Piomelli, U., Moin, P. & Cabot, W. H. 1991 A
dynamic subgrid-scale eddy viscosity model. Phys. Fluids
A 3 (7), 1760–1765.
Kim, J., Kim, H., Kim, J. & Lee, C. 2022 Deep reinforcement learning for large-eddy simulation modeling in wallbounded turbulence .
Kim, J. & Lee, C. 2020 Prediction of turbulent heat transfer
using convolutional neural networks. J. Fluid Mech. 882,
A18.
Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner,M. P. & Hoyer, S. 2021 Machine learning–accelerated computational fluid dynamics. Proc. Natl. Acad. Sci. 118 (21),e2101784118.
Lilly, D. K. 1992 A proposed modification of the germano
subgrid-scale closure method. Phys. Fluids A 4 (3), 633–635.
Lu, P. Y., Kim, S. & Soljačić, M. 2020 Extracting interpretablephysical parameters from spatiotemporal systems using unsupervised learning. Phys. Rev. X 10, 031056.
Novati, G., de Laroussilhe, H.L. & Koumoutsakos, P. 2021
Automating turbulence modelling by multi-agent reinforcement learning. Nat. Mach. Intell. 3 (1), 87–96.
Park, J. & Choi, H. 2021 Toward neural-network-based large
eddy simulation: application to turbulent channel flow. J.
Fluid Mech. 914, A16.
Sirignano, J., MacArt, J. F. & Freund, J. B. 2020 DPM: A
deep learning PDE augmentation method with application
to large-eddy simulation. J. Comput. Phys. 423, 109811.
Smagorinsky, J. 1963 General circulation experiments withprimitive equations. Mon. Weath. Rev. 91 (3), 99–164.
Vreman, A. W. 2004 An eddy-viscosity subgrid-scale modelfor turbulent shear flow: Algebraic theory and applications.
Phys. Fluids 16 (10), 3670–3681.

Figure 6. Test for (∆x+ , ∆z+ ) = (94.2, 47.1). (a) and (b) are
the streamwise mean velocity and the mean Reynolds shear
stresss, respectively.

DRL, the understanding of the working principle of reinforcement learning in relation to turbulence physics, understandingof characteristics of the trained network, and guidance of hyperparameters is poor. In fact, it has been reported that thephysical properties inherent in data can be identified throughthe characteristic analysis of the trained network (Kim & Lee,2020; Lu et al., 2020), showing its importance. Clearly, we expect that the effort of fundamental understanding would help
improve the robustness and generalization of DRL.
Also, in a parallel point of view, a differentiable PDE
method (Sirignano et al., 2020; Kochkov et al., 2021) has
emerged as a promising algorithm. Although this methodshould build a differentiable numerical solver and requires
huge cost and memory for each learning iteration, it has shown
the robust a posteriori test results compared to the classicalsupervised learning. Therefore, a comparative research of on-

5