Action-sparsity-seeking algorithms for active flow control using deep reinforcement learning
Creators: name:  R. PARIS 
name:   S. BENEDDINE 
name:   J. DANDOIS 
Publisher: name: Proceedings of  the 9th European Conference for Aerospace Sciences. Lille, France, 27 June - 1 July, 2022
Year: 2022
Subjects: FLOW CONTROL
AEROSPACE

description: Deep reinforcement learning (DRL) has recently shown tremendous performances on simple test environments, with the ability to discover effective, robust and energy efficient control laws. DRL considers an agent interacting with the system to control (called the environment), through control actions and partial state observations. The agent is tasked with both embodying the control law (using a neural network structure) and implementing the learning algorithm that fine-tunes the parameters of the control law (also known as the policy) using data drawn from past interactions between the agent and its environment. The reward quantifies the current performance of the environment state with respect to the pre-defined control goal. The training aims at maximising the expected sum of reward. Most of the commonly used synthesis methods for flow control, such as H2 or Hinf synthesis, rely on linear reduced-order modelling of the notoriously non-linear Navier-Stokes equations driving the flow. Their range a validity and overall efficiency often suffer from this approximation. Compared to these methods, the ability of DRL to handle and optimise highly non-linear control laws in a model-free paradigm, fosters a strong interest for these learning algorithms. Unlike most of DRL seminal fields of application (e.g. board games, robotics, video games), where sample cost is cheap and the dynamics take place in low-dimensional manifolds, fluids mechanics requires more resources whether is in a numerical or experimental context and often behave highly non-linearly. Considered within a global movement to scale-up AI techniques to reality-like cases, the application of DRL-driven control in fluid mechanics meets significant challenges that need to be overcome in order to bring these methods to industrial applications. Among these issues, one can cite sample efficiency and exploration strategy, resilience to environmental noise, measurement and actuation constraints both in terms of location or response delay. The latter, is not specific to AI control techniques, but we believe that the DRL paradigm offers an appropriate angle of attack on the issue, through the use of various stochastic estimators. We thus develop methods, on top of training algorithms, that aim at reducing the number of observed variables as well as actuators, while preserving performance as much as possible. These algorithms exploit pre-trained control policies using all the sensors or actuators as starting points and implement sparsity-seeking methods leveraging neural-network based estimators. These algorithms are tested on increasingly complex academic flows ranging from one-dimensional partial differential equations (e.g. the chaotic Kuramoto-Savishinsky equation) to simple transitional bi-dimensional flows (e.g. cylinder, airfoil or cavity flows). The performances of the obtained policies are compared between one another and with more systematic studies whenever possible. In the long run, these methods aim at providing an optimised trade-off between hardware needs and control performance. Acknowledgements This work is funded by ONERA and the French Agency for Innovation and Defence (AID) via a PhD scholarship. Their support is gratefully acknowledged.