# Actor-Critic methods for model-free control of spatially evolving flows

Amine Saibi, Lionel Mathelin, Onofrio Semeraro

## ► To cite this version:

Amine Saibi, Lionel Mathelin, Onofrio Semeraro. Actor-Critic methods for model-free control of spatially evolving flows. Joint event Euromech Colloquium on Data-Driven Fluid Dynamics/2nd ER-COFTAC Workshop on Machine Learning for Fluid Dynamics, Apr 2025, London, United Kingdom. hal-05379604

HAL Id: hal-05379604

https://hal.science/hal-05379604v1

Submitted on 24 Nov 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Actor-Critic methods for model-free control of spatially evolving flows

Amine SAIBI¹,², Lionel Mathelin², Onofrio Semeraro²

¹ Inst. Jean Le Rond d'Alembert, Sorbonne Université, Paris, France,

² Lab. interdisciplinaire des sciences du numérique (LISN), CNRS, Univ. Paris-Saclay, Orsay, France

Corresponding author: Amine SAIBI (amine.saibi@upmc.fr)

Data-driven reinforcement learning (RL) methods, such as actor-critic algorithms, are increasingly employed in flow control to handle complex conditions including time delays, partial observability, and other plant limitations. Yet, ensuring stable and reliable learning remains challenging due to the difficulty of coping with incomplete information and stochastic dynamics. We propose an approach that incorporates an optimistic dynamic programming perspective into actor-critic RL methods for controlling spatially distributed flows and we provide an adapted algorithm to handle single constant time-delay flows from input-output data alone. We derive insights to discuss plant limitations and how they affect learning efficiency and provide theoretical elements of practical guidance in parameter selection and monitoring error bounds. This leads to more efficient, stable, and robust learning outcomes, which we also discuss with statistical evidence. We test our approach on canonical fluid dynamics problems such as the complex Ginzburg-Landau equation and the 1D-Kuramoto-Sivashinsky to demonstrate that we achieve rapid and reliable convergence, including in more complex scenarios such as the two-dimensional boundary-layer flow developing over a flat plate with a minimal number of sensors and actuators. These results highlight the potential of an actor-critic framework for scalable and data-efficient flow control, making RL-based strategies more practical and trustworthy under real-world constraints.

<figure><img src="image_3.png" /><figcaption>Figure 1: On the *left* the complex Ginzburg-Landau equation in supercritical regime. On the *right* the kuramoto-Sivashinsky equation with convective instability. The evolution in time and space is shown and the color bars represent the modulus of the solution.</figcaption></figure>

## References

[1] B. Recht. A tour of reinforcement learning: The view from continuous control. *Annu. Rev. control* *robot. auton. syst.*, 2:253–279, 2019.

[2] J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. *J. Fluid Mech.*, 865:281–302, 2019.