12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19-22, 2022

REINFORCEMENT LEARNING FOR REDUCTION OF SKIN FRICTION DRAG IN
A FULLY DEVELOPED TURBULENT CHANNEL FLOW
Takahiro Sonoda
Department of Mechanical Engineering
The University of Tokyo
Komaba 4-6-1, Meguro-ku, Tokyo, Japan
taka1030@iis.u-tokyo.ac.jp

Zhuchen Liu
Department of Mechanical Engineering
The University of Tokyo
Komaba 4-6-1, Meguro-ku, Tokyo, Japanzhuchen@iis.u-tokyo.ac.jp

Toshitaka Itoh
Institute of Industrial Science
The University of Tokyo
Komaba 4-6-1, Meguro-ku, Tokyo, Japantitoh@iis.u-tokyo.ac.jp

Yosuke Hasegawa
Institute of Industrial Science
The University of Tokyo
Komaba 4-6-1, Meguro-ku, Tokyo, Japan
ysk@iis.u-tokyo.ac.jp

reinforcement learning is effective for the control of complex
turbulent flows with high-dimensional control inputs.
The objective of this study is to establish a framework for
optimizing a control policy for turbulent flow. Specifically, we
attempt to optimize the spatio-temporal distribution of wall
blowing and suction in a fully developed turbulent channel
flow using reinforcement learning. The number of degrees offreedom of the control input corresponds to the total numberof computational grid points on the wall, and therefore ranges
from several thousands to several tens of thousands. Also, thecontrol input is allowed to be changed in time. By applying
the current framework, it will be shown that the reinforcement
learning successfully finds an effective control policy which
achieves a higher drag reduction rate than that obtained by theopposition control (Choi et al, 1994).

ABSTRACT
Recently, reinforcement learning has attracted muchattention in flow control because of its unique advantage forobtaining a long-term optimal policy. So far, it has been
applied only to low-dimensional flows with simple actuationsof limited degrees of freedom. In this study, we apply
reinforcement learning to wall turbulence control in order todetermine the complex spatio-temporal distribution of wall
blowing and suction for reducing skin friction drag based onsensing information at a certain distance from the wall. It isdemonstrated that reinforcement learning successfully finds
an effective control policy showing non-linear relationshipbetween the sensing information and the control input, andachieves better drag reduction performance than that obtainedby the well-known opposition control.

NUMERICAL SETUP
We consider a fully developed turbulent channel flow asshown in Figure 1. Wall blowing and suction with zero-netmass-flux are given as a control input in order to reduce theskin friction drag. Direct numerical simulation (DNS) isconducted under a constant flow rate condition with the bulk"# $Reynolds number of 𝑅𝑒! = %! = 4646.72, where 𝑈! is theBACKGROUND
While turbulence has the advantages of significantlyenhancing heat and mass transfer, it also increases friction
drag and noise. Therefore, smart control of turbulence is a key
technology in engineering so as to promote its advantages withmitigating its drawbacks. Since turbulence is an extremelycomplex physical phenomenon with strong non-linearity andmulti-scale nature, however, it is quite challenging to develop
an effective control law to achieve various objectives
encountered in engineering flows (Brunton et al, 2015).
With the rapid development of neural networks in recentyears, reinforcement learning, which maximizes the long-term
reward sum based on rewards obtained from short-term trials,
has been attracting attention as a novel and effective approach
to find new control strategies. Recently, promising results ofreinforcement learning have been reported in a wide range offields such as robot control (Kober et al, 2013) and Go (Silveret al, 2016), and its applications to flow control have also beenattempted. For example, Rabault et al. (2019) applied
reinforcement learning to flow control around a cylinder byoptimizing wall blowing and suction on the surface of acylinder. Fan et al. (2020) also considered a similar
configuration with rotating small cylinders. Although large
drag reduction rates have been confirmed in both the studies,
their flow configurations are still simple two-dimensionalflows and only control inputs with limited degrees of freedom
are considered. Therefore, it remains unclear whether

bulk mean velocity of the flow, ℎ is the channel half-width,and 𝜈 is the kinematic viscosity of the fluid. The consideredbulk Reynolds number corresponds to the friction Reynolds' $number of 𝑅𝑒& = %" = 180 in the uncontrolled flow, where
𝑢& is the friction velocity.

Figure 1. Flow configuration and coordinate systemin the present study.

1

12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19-22, 2022

Figure 2. Schematic diagram of the DDPG algorithm employed in the present study
obtained at 𝑦,+ = 15 for 𝑅𝑒& = 180 (Hammond et al, 1998).
It should be noted that, in the previous studies, it is commonly
assumed that the relationship between the sensing informationand the control input is linear, and only the proportional
constant is optimized so as to maximize the drag reduction rate.
In the present study, however, the control input can be acomplex non-linear function of the sensing data, since they areconnected through neural network as explained below.
Our aim is to find the effective control policy whichconnects the state 𝑠 and the control input 𝑎. For this purpose,
we use the Deep Deterministic Policy Gradient (DDPG)
algorithm (Lillcrap et al, 2015), in which an action 𝑎 is
obtained from a deterministic policy. DDPG consists of twoneural networks called actor and critic, where actor isresponsible for deciding actions and critic is responsible for
evaluating actor. The schematic of the entire networks isshown in Figure 2.
By applying the action (or control input) 𝑎 , theinstantaneous state 𝑠 is changed to the next state 𝑠′, and theagent receives a reward 𝑟 , which is a short-term(instantaneous) friction coefficient 𝐶- obtained from DNS:

In order to reduce the computational cost for the training
the neural networks employed in the present reinforcement
learning, we consider the minimal channel (Jimenez and Moin,1991), which is the smallest domain size for maintaining
turbulence. The resultant control policy (control law) is thenassessed in a larger domain to verify its control performance.
Hereafter, the latter larger domain is referred to a full channel.
The domain sizes of the minimum and full channels employedin the present study are 1𝐿( , 𝐿) , 𝐿* 4 = (2.67, 2.0, 0.8) and(2.5𝜋, 2.0, 𝜋) , respectively, where 𝐿( , 𝐿) and 𝐿* are thedomain sizes normalized by the channel half height ℎ in the𝑥, 𝑦 and z directions.
The governing equations for an incompressible fluid, i.e.,the continuity and Navier-Stokes equations are solved by apseudo-spectral method where Fourier transform is applied in
the x and z directions, while Chebyshev polynomials are usedin the wall-normal direction y. The number of modes used in
the minimum and full channels are 1𝑁( , 𝑁) , 𝑁* 4 =(16, 65, 16) and (64, 65, 64), respectively. The 3/2 rule isused to remove the aliasing errors, so that non-linear terms areevaluated on 1.5 times finer physical mesh.
For time advancement, a fractional step method is used todecouple the pressure term from the Navier-Stokes equation.
The second-order Adams-Bashforth and Euler implicit
methods are used for the convection and viscous terms,respectively. The time step is set as Δ𝑡 + = 0.06 and 0.03 forthe minimal and full channels. Here, the superscript of +indicates a quantity in the wall unit for the uncontrolled flow.
These values ensure that the Courant number is less than unity
even with the presence of the control input, i.e., wall blowingand suction considered in the present study.

&....

𝑟 = −𝐶- = − $ # %.
/#!

%(1)

where 𝝉IIII
𝒘 is the spatial average of the wall shear stress and 𝝆
is the density of the fluid. We put a negative sign in Eq. (1),so that maximizing reward corresponds to minimizing the drag.
By repeating the above interactions with the environment, theagent learns the optimal policy 𝜇∗ (𝑎|𝑠) so as to maximize thelong-term drag reduction effects.
Critic estimates the long-term drag reduction based on the
instantaneous reward 𝑟 obtained by applying the control input
determined by actor. It is achieved by minimizing thefollowing squared residual of the Bellman equation:
𝐿234546 = {𝑟(𝑠, 𝑎) + 𝛾𝑄7 (𝑠 8 , 𝑎′) − 𝑄7 (𝑠, 𝑎)}" ,(2)where 𝑄7 (𝑠, 𝑎) is the expected total reward when a certain
action 𝑎 is taken under a certain state 𝑠. Meanwhile, actor
network is trained so as to maximize the expected total reward
𝑄7 (𝑠, 𝑎). During the training, the two networks, i.e., actor andcritic, are trained alternatively, so that both networks will beoptimized after a number of trials.
In this study, the network is trained by repeating DNS with
one episode of 𝑡 + = 600, where the superscript + denotes adimensionless quantity in wall units of the uncontrolled flow.

REINFORCEMENT LEARNING
In reinforcement learning, the agent (learner) receives astate 𝑠 from the environment (control target) and outputs anaction 𝑎 based on a policy 𝜇(𝑎|𝑠). In the present study, 𝑎 is acontrol input, i.e., wall blowing and suction. Its spatial
distribution is determined by the state 𝑠 , which is theinstantaneous streamwise and wall-normal velocity
fluctuations u' and v' at 𝑦,+ = 15. This sensing plane height isdetermined by following the opposition control (Choi et al,1994), whose maximum drag reduction rate of 25% is2

12th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19-22, 2022
Each episode starts with the same initial field, and the control
input determined by the policy is applied to obtain its shortterm reward. In this study, the agent obtains the state of the+flow field every 𝛥𝑡9654:;= 0.6, and the control input between
the updates is determined by linear interpolation. At the same+time, the network is trained every 𝛥𝑡5394;4;<= 0.6 based on
the short-term reward. The network has 1 hidden layer
consisting of 8 nodes. ReLU is used as the activation function,
while that of the output layer is tanh. These hyper-parameters
are found to be optimal in a preliminary survey. The training
is repeated until the control policy converges.

on only the wall-normal velocity fluctuation at the sensing
plane. It should be emphasized that the neural network
employed for actor allows to find such a non-linear
relationship between the sensing information and the actuation
as shown Figure 4 (a). More specifically, the best policy
applies blowing (𝜑 > 0) when the high-speed fluid
approaches the wall (𝑣 8 < 0, 𝑢8 > 0), while suction (𝜑 < 0)for the low-speed fluid moves away from the wall(𝑣 8 > 0, 𝑢8 < 0). When this policy is applied to a full-size
channel, about 31% drag reduction is achieved. Consideringthat the drag reduction rate of the opposition control (Choi etal, 1994) is about 23% in the present flow configuration, it canbe concluded that a more effective control policy than theopposition control can be obtained by the present
reinforcement learning.

RESULTS
Figure 3 shows the time average of 𝐶- in the final periodof each episode, i.e., 500 ≤ 𝑡 + ≤ 600. It can be seen that 𝐶gradually decreases with increasing the episode number. It
should also be noted that the present 𝐶- is lower than that of
the opposition control shown by the red line in Figure 3. This
indicates that an effective control strategy is found from thepresent reinforcement learning (RL).

(a)

Figure 3. 𝐶- obtained by the present reinforcementlearning (RL) as a function of episode number(b)

(a)

(b)

Figure 5. Instantaneous visualizations of the velocity field andthe control input applied at the bottom wall at (a) t+ = 0.6 and(b) t+ = 20.4 after applying the best control policy obtained in
the present reinforcement learning. Write contours show isosurfaces of the second invariant of the deformation tensor. Red
to blue colors on the bottom wall indicate wall blowing andsuction, respectively.

Figure 4. Control policies obtained by the present
reinforcement learning (a) and the opposition control (b)Figure 4 (a) shows the best control policy obtained inepisode 89. Specifically, the control input 𝜙 at the bottom wallis plotted as a function of the state, i.e., u' and v' on the sensing
plane at 𝑦,+ = 15. It is found that the obtained control policy
shows rapid change from blowing (blue) to suction (red). It isin contrast to the control policy of the opposition control
shown in Figure (b), where the control input linearly depends

312th International Symposium on Turbulence and Shear Flow Phenomena (TSFP12)
Osaka, Japan, July 19-22, 2022
CONCLUSIONS
Reinforcement learning was applied to a turbulent channel
flow in order to find effective control policies for reducing
skin friction drag for the first time. The obtained control policy
based on the instantaneous streamwise and wall-normal
velocity fluctuations at a sensing plane of 𝑦,+ = 15 achieved
31% drag reduction rate, which is higher than that obtained by
the opposition control. The obtained contro policy ischaracterized by a sharp change from wall blowing andsuction, and finding such a complex and non-linear control
strategy became possible by a systematic learning framework
leveraged by neural networks. The present results demonstrate
the effectiveness of reinforcement learning for developing
new strategies for turbulence control.

The instantaneous flow fields at t+ = 0.6 and 20.4 after the
onset of the control with the best policy obtained in the present
reinforcement learning are shown in Figure 5 (a) and (b),respectively. Just after applying the control, at t+ = 0.6, it canbe seen that the control input has an elongated structure in thestreamwise direction. This corresponds to the near-wallcoherent structures. Interestingly, with time passes, the control
input transits to a coherent wave-like input as shown in Figure
5 (b), which is almost uniform in the spanwise direction andits streamwise wavelength is equal to the streamwise domainsize.
In order to extract a coherent component from the control
input, we take the spatial average of the instantaneous control
input in the spanwise direction. The resultant spanwiseV+ is shown as a function of theaveraged control input 𝜙streamwise coordinate and the time in Figure 6. It can be seenthat the wall blowing and suction switches to the other at ahigh frequency, while its wave nodes slowly move to upstream.
Since the present control policy shown in Figure 4 (a) rapidly
switches from strong wall blowing to suction depending on thestate u’ and v’ at the detection plane, it induces a strong
perturbation at the detection plane. This in turn affects thecontrol input in the next time step. Such a feedback between
the control input and the flow state at the detection plane
should yields the wave-like coherent control input shown inFigure 6.

REFERENCES
Brunton, S. L. and Noack, B. R., 2015, "Closed-loopturbulence control: Progress and challenges.", Applied
Mechanics Reviews, 67.5.
Choi, H., Moin, P. and Kim, J., 1994, "Active turbulencecontrol for drag reduction in wall-bounded flows.", Journal of
Fluid Mechanics, 262, 75-110.
Fan, D., Yang, L., Wang, Z., Triantafyllou, M. S. andKamiadakis, G. E., 2019, "Reinforcement learning for bluff
body active flow control in experiments and simulations.",
Proceedings of the National Academy of Sciences, 117.42,26091-26098.
Hammond, E. P., Bewley, T. R., & Moin, P., 1998,“Observed mechanisms for turbulence attenuation andenhancement in opposition-controlled wall-bounded flows.”,
Physics of Fluids, 10.9, 2421-2423.
Jiménez, J. and Moin, P., 1991, "The minimal flow unit innear-wall turbulence." Journal of Fluid Mechanics, 225, 213240.
Kober, J., Bagnell, J. A. and Peters, J., 2013,"Reinforcement learning in robotics: A survey.", The
International Journal of Robotics Research, 32.11, 1238-1274.
Lieu, B. K., Marref, R. and Jovanovic, M. R., 2010,“Controlling the onset of turbulence by streamwise travellingwaves. part 2. direct numerical simulation” Journal of Fluid
Mechanics, 663, 100–119.
Lillicrap, T. P., Hunt, L. J., Pritzel, A., Heess, N., Erez, T.,Tassa, Y., Silver, D. and Wierstra, D.,2015, "Continuouscontrol with deep reinforcement learning." arXiv preprint
arXiv:1509.02971.
Min, T., Kang, S. M., Speyer, J. L. and Kim, J., 2006,"Sustained sub-laminar drag in a fully developed channel
flow." Journal of Fluid Mechanics, 558, 309-318.
Rabault, J., Kuchta, M., Jensen, A., Reglade, U. andCerardi, N., 2019, "Artificial neural networks trained throughdeep reinforcement learning discover control strategies for
active flow control.", Journal of Fluid Mechanics, 865, 281302.
Silver, D, Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D.,
Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach,M., Kavukcuoglu, K., Graepel, T. and Hassabis, D.,2016,"Mastering the game of Go with deep neural networks and tree
search.", Nature, 529.7587, 484-489.

Figure 6. Spanwise-averaged control input as a function of thestreamwise coordinate x and the time t.
It has been reported that drag reduction can be achieved byapplying a traveling wave-like wall blowing and suction (forexample, Ming et al., 2006, Lieu et al., 2010). Therefore, it isof interest to clarify whether the current drag reduction effects
are caused by the coherent control input. For this purpose, weconducted additional computation where only the coherent
control input shown in Figure 6, which is uniform in the
spanwise direction, is applied. It should be noted that, in thiscase, the applied control should be regarded as apredetermined control rather than a feedback control, since thecontrol input no longer depends on the instantaneous flow
state. It is found that applying the coherent control input only
does not lead to drag reduction effects. This indicates that thepresent control obtained by the reinforcement learning isessentially a feedback control.

4