# Deep reinforcement learning for optimizing control law parameters on a smart beam with distributed piezoelectric transducers

Maryne Febvre, Jonathan Rodriguez, Simon Chesne, Manuel Collet

► To cite this version:

Maryne Febvre, Jonathan Rodriguez, Simon Chesne, Manuel Collet. Deep reinforcement learning for optimizing control law parameters on a smart beam with distributed piezoelectric transducers. Smart Materials and Structures, 2025, 34 (10), pp.105001. 10.1088/1361-665X/ae0903. hal-05311293

HAL Id: hal-05311293

https://hal.science/hal-05311293v1

Submitted on 18 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
PAPER • OPEN ACCESS

Deep reinforcement learning for optimizing control
law parameters on a smart beam with distributed
piezoelectric transducers

To cite this article: Maryne Febvre et al 2025 *Smart Mater. Struct.* 34 105001

View the [article online](#) for updates and enhancements.

You may also like

- Community resilience to the health and wellbeing impacts of environmental and climate change: a review
  Zoe Alexandra Becvarik, Zoe Leviston, Erin Ingrid Walsh et al.

- Predicting lymphocyte dose and surviving fraction for VMAT and IMPT treatments with a dynamic lymphocyte flow model for locally advanced cervical cancer
  Sander Cyril Kuipers, Marianne van Tuyll van Serooskerken, Danny Lathouwers et al.

- Fertilizing potential of nitrogen in human excreta across diverse countries and food systems
  Thomas Starck and Fabien Esculier

<figure><img src="image_2.png" /></figure>
# Deep reinforcement learning for optimizing control law parameters on a smart beam with distributed piezoelectric transducers

Maryne Febvre<fnref n="1" />,<fnref n="2" />,* ID, Jonathan Rodriguez<fnref n="2" /> ID, Simon Chesne<fnref n="2" /> ID and Manuel Collet<fnref n="1" /> ID

E-mail: maryne.febvre@insa-lyon.fr

Received 13 June 2025, revised 9 September 2025

Accepted for publication 18 September 2025

Published 30 September 2025

<figure><img src="image_3.png" /></figure>

CrossMark

## Abstract

Recent research has focused on the design of adaptive structures with distributed transducers for efficient vibration mitigation. This study explored the optimization of a control strategy to enhance vibration mitigation in structures with multiple piezoelectric transducers. A key challenge lies in tuning control parameters while maintaining stability and robustness against environmental variations. Artificial intelligence, particularly neural networks, offers a promising approach for handling complex multi-input, multi-output control problems. In this work, deep reinforcement learning (DRL) is applied to autonomously tune positive position feedback controllers for a smart beam equipped with three pairs of collocated piezoelectric transducers. The training process prioritizes minimizing vibrations of the third bending mode, which is one of the most dominant. A pole-zero model, derived from experimental measurements, enables numerical learning process while preventing potential damage to the test bench. A comparative analysis is conducted between the DRL strategy and a simplex optimization method, both using the same objective function. A statistical analysis evaluates the reliability and effectiveness of both approaches, highlighting their advantages and limitations in achieving robust control.

Keywords: piezoelectric transducers, deep reinforcement learning, active control, distributive systems

## 1. Introduction

Smart structures go a step further than traditional mechanical properties by incorporating functions such as vibration isolation, noise reduction, shape adaptation, and structural

health monitoring while maintaining minimal mass and power consumption [1–3]. These advanced functionalities are achieved using smart materials that integrate sensing and actuation at the material level. Among them, piezoelectric transducers [4, 5] are widely used due to their ability to convert mechanical energy into electrical energy and vice versa.

Recent research on adaptive structures has explored the use of distributed transducers to create new functional materials such as mechanical lenses for energy focusing [6], mechanical diodes [7], and mechanical insulators [8, 9]. Active vibration control in such systems has traditionally been achieved using classical control laws, including PID [10], LQR [11], LQG [12], $H_2$ [13], $H_\infty$ [14], and positive position feedback (PPF)

<figure><img src="image_4.png" /></figure>

Original Content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

<footnote n="1">¹ CNRS, Ecole Centrale de Lyon, ENTPE, LTDS, UMR5513, 69134 Ecully, France</footnote>

<footnote n="2">² INSA Lyon, CNRS, LaMCoS, UMR5259, 69621 Villeurbanne, France</footnote>

<footnote>* Author to whom any correspondence should be addressed.</footnote>
controllers [15]. These methods are efficient for controlling piezoelectric structures [16], but their scalability is a challenge as the number of transducers increases. Multi-input, multi-output (MIMO) systems require careful tuning of a significant number of control law parameters, making optimization increasingly complex, balancing efficiency and stability, especially in the presence of physical uncertainties. Completely decentralized approaches are difficult to optimize, even if they can be used to limit delay and guarantee greater margins of stability.

To address this challenge, researchers have explored intelligent control methods [17], taking advantage of advances in computer hardware and artificial intelligence. Neural networks (NNs) have been investigated for structural control with piezoelectric transducers since the 1990s to create neuro-controllers [18–23] or to be used as predictive models for feed-forward control [24–26]. Although these studies demonstrated promising results, most relied on supervised learning and required extensive datasets for training, making them sensitive to its bias. Unlike supervised learning, reinforcement learning (RL) [27] eliminates the need for predefined training data sets by continuously learning from interactions with the system and can be used to control systems. RL-based controllers have been successfully implemented for unmanned aerial vehicle (UAV) trajectory tracking [28] and for controlling the vibration of piezoelectric MIMO structures in simulation [29]. However, experimental implementations on test benches face challenges such as long training times, sensitivity to noise, and stability concerns [30, 31].

To overcome these limitations, researchers have explored RL-based tuning of classical controllers. For example, Khalatbarisoltani [32] works with an active mass drive system on structures stressed with seismic events. RL is used with online tuning to determine the gains of a fuzzy PD controller. Experimental tests allow to prove successful amplitude attenuation of the structural response amplitude in response to seismic perturbations. In 2024, Panda [33] propose to use RL algorithm with policy gradient based method to tune a P and a PI controller. The method's efficiency is illustrated on two numerical cases considering harmonic excitation: a quarter car model with active suspension system and an 8-story benchmark building.

In rotating machinery such as UAVs, the excitation is broadband with a fundamental frequency tied to the motor's rotational speed and its harmonics [34]. In such cases, the fundamental frequency may lie well above the structure's first natural mode, which raises new challenges for vibration control strategies. The proposed DRL-based tuning framework offers a promising path toward addressing such scenarios in future applications.

This approach offers the potential for improved control efficiency while exploiting the interpretability of classical control laws with stability and robustness quantification. However, its application to smart MIMO structures is still at an early stage, especially in experimental test bench scenarios. The current challenges lie in adapting RL tools to the context of vibrations,

and examining how robust the methods are in the face of system variations and perturbations.

This paper presents an autonomous method for tuning the parameters of a PPF control law in a MIMO smart beam. The proposed approach uses deep RL (DRL) with the trust region policy optimization (TRPO) [35–37] algorithm to minimize vibrations within a specified target frequency range. A downhill simplex algorithm [38] is chosen as a reference optimization strategy to provide a quantitative comparison with the proposed method.

The proposed methodology is experimentally validated on an academic structure for clarity and reproducibility. Nevertheless, it is directly transferable to practical applications, including aircraft turbines, wind turbines, and UAVs. The experimental setup employed in this study consists of a beam instrumented with three pairs of distributed piezoelectric transducers.

To enable DRL training, a transfer function model is identified from test bench measurements. The DRL strategy introduces randomness in both NN initialization and training, while the Simplex method is influenced by its starting point, particularly in non-convex optimization problems. This study investigates how initialization impacts the reliability and effectiveness of control law parameters tuning for both approaches through a statistical analysis conducted on the smart beam. Initially, a first reward function is defined to assess statistical outcomes and identify the most successful NN initializations. The reward function is then improved to enhance control stability, ensuring that the final controllers can be safely implemented on the experimental setup.

Efficient agents are subsequently selected to continue training with the improved reward function. The tuned controllers are then evaluated on the test bench, and the overall effectiveness of the proposed tuning methodology is discussed.

The main contributions of this paper are: (i) the development of an autonomous DRL-based strategy for tuning PPF controllers in a MIMO smart structure, (ii) a comparative analysis with a classical optimization method (downhill simplex), and (iii) an experimental validation demonstrating the robustness and effectiveness of the proposed approach.

## 2. Problem formulation

### 2.1. *Structure definition*

The smart structure studied in figure 1 consists of a steel beam of length $L_b$, width $b_b$ and thickness $h_b$. Three identical pairs of piezoelectric transducers (PZT-PIC255) of length $L_p$, width $b_p$ and thickness $h_p$ are bonded to the beam. The top transducers $Pa_1$, $Pa_2$ and $Pa_3$ are used for actuation and $Ps_1$, $Ps_2$ and $Ps_3$ for sensing. All transducers are centered alongside of the beam main axis. The positions of the pairs of transducers are defined by the distances $e_{p1}$, $e_{p2}$ and $e_{p3}$. The dimensions of the beam
<figure><img src="image_5.png" /><figcaption>Figure 1. Beam dimensions diagram.</figcaption></figure>

<table><caption>Table 1. Setup dimensions.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Dimensions</th><th rowspan="2">Materials</th></tr><tr><th>Name</th><th>Value</th><th>Unit</th></tr></thead><tbody><tr><td rowspan="3">Beam</td><td>L<sub>b</sub></td><td>400</td><td>(mm)</td><td rowspan="3">Steel</td></tr><tr><td>b<sub>b</sub></td><td>60</td><td>(mm)</td></tr><tr><td>h<sub>b</sub></td><td>2</td><td>(mm)</td></tr><tr><td rowspan="6">Piezoelectric Patch</td><td>L<sub>p</sub></td><td>61</td><td>(mm)</td><td>PI DuraAct</td></tr><tr><td>b<sub>p</sub></td><td>35</td><td>(mm)</td><td>Patch transducer</td></tr><tr><td>h<sub>p</sub></td><td>1</td><td>(mm)</td><td>P-876.A15</td></tr><tr><td>e<sub>p1</sub></td><td>70</td><td>(mm)</td><td></td></tr><tr><td>e<sub>p2</sub></td><td>169</td><td>(mm)</td><td></td></tr><tr><td>e<sub>p3</sub></td><td>269</td><td>(mm)</td><td></td></tr></tbody></table>

are given in table 1. Appendix A provides material properties of the smart beam.

## 2.2. Control law definition

A feedback control law $C(s)$ is applied to the structure defined by the system $G(s)$, as shown in the block diagram figure 2. Here $r=0$ is the target signal, $e$ the error, $u_d = [u_{d1} \ u_{d2} \ u_{d3}]^T$ the disturbances, $u_c = [u_{c1} \ u_{c2} \ u_{c3}]^T$ the control signals, $u = [u_1 \ u_2 \ u_3]^T$ the input drive signals ($Pa_1$, $Pa_2$ and $Pa_3$ voltages) of the system $G(s)$ and $y = [y_1 \ y_2 \ y_3]^T$ the respective system outputs ($Ps_1$, $Ps_2$ and $Ps_3$ voltages).

One channel $u$ is used to apply both the input disturbance and the control signals to the system, thought the piezoelectric actuators. The signal $u_d(t)$ is a random noise $Rnd(t)$ with a sampling frequency $F_s$ in Hz and filtrated by a low pass second order filter with a cutoff frequency $F_c = 10000$ Hz, such as:

$$ \begin{bmatrix} u_{d1}(t) \\ u_{d2}(t) \\ u_{d3}(t) \end{bmatrix} = \begin{bmatrix} \text{Rnd}(t) \ast \mathcal{L}^{-1} \left\{ \left( \frac{2\pi F_c}{s+2\pi F_c} \right)^2 \right\} \\ 0 \\ 0 \end{bmatrix} \quad (1) $$

where $\mathcal{L}^{-1}$ is the inverse Laplace transform and $\ast$ being the temporal convolution.

The feedback control laws are implemented between each pair of collocated piezoelectric elements of the beam. Three PPF controllers [39] are used such as:

$$ C(s) = \begin{bmatrix} C_1(s) \\ C_2(s) \\ C_3(s) \end{bmatrix} = \begin{bmatrix} \frac{1}{s^2+2\xi_1\omega_1s+\omega_1^2} \\ \frac{K_{c1}}{s^2+2\xi_1\omega_1s+\omega_1^2} \\ \frac{K_{c2}}{s^2+2\xi_2\omega_2s+\omega_2^2} \\ \frac{K_{c3}}{s^2+2\xi_3\omega_3s+\omega_3^2} \end{bmatrix} \quad (2) $$

where $s$ is the Laplace variable of the transfer functions, $K_{c1}$ is the gain, $\omega_i$ is the filter frequency adapted to the target mode in $[\text{rad}\cdot\text{s}^{-1}]$ and $\xi_i$ is the damping ratio of the control loop $i = \{1; 2; 3\}$.

In this work, PPF controllers are adopted due to their well-established advantages for vibration control of smart structures in damping structural oscillations [40]. PPF ensures effective damping of targeted frequency modes without destabilizing higher modes, while requiring the adjustment of only three parameters. This offers a favorable balance between simplicity and parameterization compared to conventional controllers, such as derivative feedback. Moreover, with collocated piezoelectric transducers, PPF controllers can be tuned manually following standard approaches from the
<figure><img src="image_6.png" /><figcaption>Figure 2. Control loop block diagram.</figcaption></figure>

literature [41]. However, as the number of collocated transducers increases, manual tuning becomes impractical due to the mutual influence of controllers targeting the same frequency bandwidth. This motivates the development of an autonomous optimization strategy, as investigated in this study.

## 2.3. Test bench setup

To satisfy the free boundary condition on the testbench, the beam is suspended with soft rubber bands at its four corners as shown in figure 3. The main support is not shown in the picture and the electrical wires have been hidden for clarity.

The data acquisition setup implemented on the piezoelectric beam is shown in figure 4. Here, $y_1, y_2$ and $y_3$ are the electric voltages measured at the piezoelectric transducers $Ps_1, Ps_2$ and $Ps_3$, respectively, and $u_1, u_2$ and $u_3$ are the electric voltages applied to the piezoelectric transducers $Pa_1, Pa_2$ and $Pa_3$, respectively. All devices used in the following experiments are listed in table 2.

## 2.4. System identification

A model is built from measurements performed on the test bench. The input of the identification process is a band-limited noise $v(t)$. The noise power sent to the amplifier is equal to 0.1 with a sampling frequency $F_s = 10$ kHz, such as:

$$v(t) = \text{Rnd}(t). \quad (3)$$

Considering the MIMO system, the excitation is applied to the actuators one by one, while the others are short-circuited. The measured signals $y_1, y_2$ and $y_3$ are used to identify nine transfer functions $G_{i,j}$, where $i = \{1;2;3\}$ and $j = \{1;2;3\}$, defined by:

$$\begin{bmatrix} Y_1(s) \\ Y_2(s) \\ Y_3(s) \end{bmatrix} = \begin{bmatrix} G_{1,1}(s) & G_{2,1}(s) & G_{3,1}(s) \\ G_{1,2}(s) & G_{2,2}(s) & G_{3,2}(s) \\ G_{1,3}(s) & G_{2,3}(s) & G_{3,3}(s) \end{bmatrix} \begin{bmatrix} U_1(s) \\ U_2(s) \\ U_3(s) \end{bmatrix} \quad (4)$$

where $Y_1(s), Y_2(s), Y_3(s)$ and $U_1(s), U_2(s), U_3(s)$ are the Laplace transforms of $y_1, y_2, y_3$ and $u_1, u_2, u_3$ respectively.

Pole and zero estimation is used to build the experimental model from the measured frequency response functions between 10 and 1000 Hz, such as:

$$G_{i=\{1;2;3\},j=\{1;2;3\}}(s) = \frac{\kappa_{ij} \prod_{k=1}^{n_z} (1 - z_k^{-1}s)}{s^{n_0} \prod_{l=1}^{n_p} (1 - p_l^{-1}s)} \quad (5)$$

where $\kappa_{ij} \in \mathbb{R}$ is a constant gain, $n_0 \in \mathbb{Z}$ is the number of null poles (integrator if $n_0 > 0$, derivative if $n_0 < 0$), $z_k \in \mathbb{C}$ zeros and $p_l \in \mathbb{C}$ poles of the system, $n_z \in \mathbb{N}$ the number of zeros and $n_p \in \mathbb{N}$ the number of poles, $G_{i,j}$ must be proper ($n_p + n_0 \ge n_z$).

Figure 5 shows measurements made with successive actuation for some selected transfer functions. Numerical transfer functions have 13 poles and 12 zeros, except for $G_{2,1}$ which has 14 poles and 13 zeros as needed to estimate the zeros between 450 and 700 Hz. The number of poles and zeros is not a multiple of the number of modes, but they are used to introduce phase shift corrections at low and high frequencies. The first mode has a low dynamic considering the transfer functions $G_{1,1}$ (figure 5(a)) and $G_{3,3}$ (figure 5(f)) and is not estimated in the model. A zero amplitude measured at 0 Hz is explained by the use of piezoelectric sensors which do not measure the continuous component. A significant drop in coherence between non-collocated actuators and sensors occurs between 600 and 800 Hz (figures 5(d) and (g)). In this frequency range, the model estimates zeros that are unreliable. Higher modes exist above the frequency of interest and may introduce control instabilities not considered in the model. Outside the considered frequency range, the sixth mode is estimated in $G_{1,3}$ (figure 5(b)) and $G_{3,1}$ (figure 5(e)). This allows phase fitting until 1000 Hz.

The model also fits in the time domain as shown in figure 6, which illustrates the system response to a band-limited noise between 0 and 1000 Hz applied to $Pa_2$. The model fits the real system, with an error over the frequency or time domain indicator estimated from 0.2% to 7%. These differences can be explained by the non-estimation of the first frequency in the $G_{1,1}$ transfer function and the amplitude estimation of the dynamics due to the post-experiment processing of the data.

Despite the differences with the test bench, the model is suitable for use with the optimization algorithms.
<figure><img src="image_7.png" /><figcaption>**Figure 3.** Experimental setup: beam with piezoelectric transducers.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 4.** Experimental setup: data acquisition systems.</figcaption></figure>

<table><caption>**Table 2.** Experimental devices.</caption><thead><tr><th>Device</th><th>Brand</th><th>Reference</th><th>Sensitivity</th><th>Measuring range</th></tr></thead><tbody><tr><td>Acquisition system</td><td>DSPACE</td><td>DS1202</td><td>—</td><td>±/–50 V</td></tr><tr><td>HMI</td><td>DSPACE</td><td>ControlDesk</td><td>—</td><td>—</td></tr><tr><td>Piezo transducers</td><td>PI</td><td>P-876.A15</td><td>0.64 µm m<sup>-1</sup> V<sup>-1</sup></td><td>–250 to 1000 V</td></tr><tr><td>Power amplifier</td><td>PI</td><td>DuraAct piezo amplifier</td><td>&lt;10 mV</td><td>Input voltage: –4 to 10 V<br>Output voltage: –100 to 250 V</td></tr></tbody></table>

## 3. Method

This study explores two approaches for tuning PPF controller parameters on a smart beam: one based on DRL and the other on a simplex optimization algorithm. Both methods are applied to the structure, with the objective of minimizing vibrations within a target frequency band centered around the third bending mode.

### 3.1. Simplex

The simplex method, also known as the Nelder-Mead method [38], is a nonlinear optimizer for multidimensional space functions $f(p)$, where $p \in \mathbb{R}^P$ are the points and $P$ is the dimension. The name simplex comes from the geometry field, where it denotes the simplest possible object for the given dimension: a line in one-dimension, a triangle in two-dimension and a tetrahedron in two-dimension.
<figure><img src="image_9.png" /><figcaption>**Figure 5.** Bode and coherence plot with transfer function estimated from measurements (blue line) and the identified model (red line) between 0 and 1500 [Hz].</figcaption></figure>

The Simplex algorithm evolves with an objective function to minimize, represented by the negative value of the reward $R_1$ computed from the environment in this case. Figure 7 shows a simplified scheme of the algorithm in this application case. Note that $m \in \mathbb{N}$ is a function evaluations in an iteration $n \in \mathbb{N}$. The algorithm starts with an initial point, defined here by a set of controller parameters $A_{\text{init}}$. After initializing the

environment (control off), the initial controller can be applied to the structure to compute the associated reward. Then the simplex optimization algorithm is executed until the termination state is reached. It can be terminated by one of two conditions, defined here with default values: the number of solver iterations $n$ exceeds $N = 1800$, or the tolerances on the objective function and parameters are below $\epsilon = 10^{(k-4)}$. During
<figure><img src="image_10.png" /><figcaption>**Figure 6.** Time measurements from the experimental setup (blue line) and identified model (red line) during 30 s (a)–(c)–(e) and between s 6.0 and 6.02 (b)–(d)–(f).</figcaption></figure>

one iteration, the algorithm can evaluate the objective function
only 1800 times. The performance of the algorithm is sup-
posed to be highly dependent on the starting point, so several
must be tested.

3.2. DRL

The DRL algorithm is based on iterative improvement without
the need of a database, it defines a goal to be achieved,
materialized by a mathematical function to be maximized,

which is representative of the dynamics of the system under
study: the reward function. The updating method used is called
TRPO and consists of an agent whose formal equations are
detailed by Schulman [37].

The agent is composed of two NNs, the actor and the critic. It interacts multiple times with the environment to update his policy according with the reward R₁ increases. The actor and the critic NNs are initialized with random weights and biases. The NNs properties and the algorithm hyper-parameters are given in appendix B and C.
<figure><img src="image_11.png" /><figcaption>**Figure 7.** Simplex algorithm scheme applied on the studied smart system. (0)-Simplex initialization with P + 1 points defined in dimension P (here P = 9), (m1)-simplex point coordinate to test on the environment, (m2)-environment running: the beam response to a white noise is simulated with the control on set with PPF control law parameters defined by the simplex point location to test from (m1), (m4)-reward computation, (simplex representation form Cheng et al [42]). Reprinted from [42], Copyright (2015), with permission from Elsevier.</figcaption></figure>

The main steps during an episode of the training applied to the system under study are shown in figure 8.

The training is divided into N episodes. An episode n starts with the initialization of the environment (control off). The initialization states are used as input to the NN to estimate the expected reward R<sub>V</sub> for the critic and the actions An (controller parameters) of the actor. The selected controller is evaluated within the specified environment to compute the corresponding episode reward R<sub>1n</sub>. The reward function is calculated using the expected and actual rewards (R<sub>V</sub> and R<sub>1</sub>) in order to update the biases and weights of the actor and critic NNs. Training can end under two conditions: the computed reward reaches a target value Rt defined by the user, or the maximum number of episodes is reached. In this example, the maximum number of episodes is set to 5000 and the maximum reward is set to 20. This value is high for this application and cannot be reached given the stability constraint in the reward definition computed from the model. This way, the training will be stopped by the number of episodes condition. The number of episodes is chosen to be relatively small, but the point is to observe the state of the grounded solution at a given training episode. Divergences may occur depending on the NN initialization, so several must be tested.

### 3.3. Environment definition

The environment is defined by the previously estimated model from equation (4). It consists of observations, actions and

rewards. These are calculated from the metrics computed from the model.

#### 3.3.1. Observations.
Three observations are defined as the root mean square of the time signal measured by the sensors during 30 s, considering a given disturbance u<sub>d</sub> such as:

$$S = \begin{bmatrix} S_1 \\ S_2 \\ S_3 \end{bmatrix}_{u_d} = \begin{bmatrix} \text{rms}(y_1) \\ \text{rms}(y_2) \\ \text{rms}(y_3) \end{bmatrix}_{u_d} \quad (6)$$

This metric is relevant considering the noise excitation of the beam and can vary according to the frequency range of the excitation due to the modal distribution of the beam. Note that these states values are normalized between -1 and 1 to avoid having large values as NN input, which can degrade it is training. The environment does not change over time, so the agent only interacts with it once before adjusting its policy. Thus, the observation definition plays a minor role in this context, since the trained network is only adapted to one configuration of the beam and takes only one set of observations as input.

#### 3.3.2. Action.
Nine actions have to be tuned according to the three PPF controllers of equation (2) implemented on the structure. They are grouped in an action variable A ∈ ℝ<sup>3×3</sup> such as:
<figure><img src="image_12.png" /><figcaption>**Figure 8.** Deep reinforcement learning (TRPO) Scheme applied on the studied smart system. (0)-Agent initialization: the actor and the critic neural networks are initialized with random weight and bias. ● (n1)-Environment initialization: the beam response to a white noise is simulated with the control off state. ● (n2)-Observation computation: the observations are calculated using chosen functions to convert the time domain data to representative values of the simulated environment. ● (3)-Agent decision process: the agent estimates an expected reward with the critic and an action value with the actor according with the input observations. ● (n4)-Environment running: the beam response to a white noise is simulated with the control on set with PPF control law parameters chosen by the actor during step (n3). ● (n5)-Reward computation: the defined reward function is calculated using step (n1) and step (n4) simulations. ● (n6)-Agent update: the actor and critic weights and biases are updated using TRPO algorithms [36], considering the expected reward from step (n3) and the reward from step (n5).</figcaption></figure>

$$A = \begin{bmatrix} K_{c1} & \xi_1 & \omega_1 \\ K_{c2} & \xi_2 & \omega_2 \\ K_{c3} & \xi_3 & \omega_3 \end{bmatrix}. \quad (7)$$

The limits of the damping parameters are set by the controller definition proposed by Goh and Caughey [39] with $\xi_i \in$

[0; 1]. The filter frequency parameter is limited by the model frequency range with $\omega_i \in [0; 6283]$. The controller gain limits are set in a wide range of values, such as: $K_{c_i} \in [-1 \times 10^0; -7 \times 10^{10}]$. This range of parameters includes both stable and unstable controllers. To prevent a significant discrepancy in the number of NN outputs, all action values are normalized. A scaling function is applied to calculate the real values to be tested on the structure. Power-of-10 functions are used for the controller gains $K_{c_i}$, while linear functions are used for $\xi_i$ and $\omega_i$.

### 3.3.3. Reward.
The reward definition is designed with two objectives: firstly, to minimize the vibrations of the beam at 380 Hz and, secondly, to ensure the stability of the controlled structure. It is defined using a frequency domain integration between $f_1 = 350$ and $f_2 = 388$ Hz to consider only the dynamics of the poles close to the third bending mode at 380 Hz. It contains the transfer functions $G_{1,1}$, $G_{2,2}$ and $G_{3,3}$ in a ratio where the control is off and on, such as:

$$R_1 = 20 \times \log_{10} \sum_{i=1}^{3} \frac{\int_{f_1}^{f_2} G_{i,i}^{\text{off}} df}{\int_{f_1}^{f_2} G_{i,i}^{\text{on}} df} \quad (8)$$

This equation defines the reward for a stable controlled system where gain margins ($Gm$) are above 10 dB. For systems operating within a limited stability range ($Gm < 10$ dB), a stability constraint is introduced into the reward definition with the gain margin, such as:

$$R_1 = 20 \times \log_{10} \sum_{i=1}^{3} \frac{\int_{f_1}^{f_2} G_{i,i}^{\text{off}} df}{\int_{f_1}^{f_2} G_{i,i}^{\text{on}} df} \times \left| \prod_{i=1}^{3} \frac{Gm_{\text{dB}}^i (0 < Gm_{\text{dB}}^i < 10)}{10} \right| \quad (9)$$

where $Gm_{\text{dB}}^i$ is the gain margin of the loop $i = \{1, 2, 3\}$ when the other two controllers are applied to the system. In case of an unstable system, the reward is defined such as:

$$R_1 = -1. \quad (10)$$

The conditional reward definition permits a transition from the stable to the unstable region into the algorithmic process, thus avoiding discontinuity in the region of the instability.

* Figure 9(a) shows the reward variation with gain $K_c$ for different damping coefficients $\xi_{1/2/3}$ and a fixed frequency $\omega_{1/2/3} = 380 \times 2\pi = 2.39 \times 10^3$ (rad s$^{-1}$). A numerical artifact can be observed for $\xi_{1/2/3} = 0.25$ and $\log_{10}(Kc_{1/2/3}) = 6.6$ with a drop in reward (indicator (5) in figure 9(a).

* Figure 9(b) shows the reward variation with gain $Kc_{1/2/3}$ for different frequencies $\omega_{1/2/3}$ and a fixed damping coefficient $\xi_{1/2/3} = 0.55$.

* Figure 9(c) shows the reward variation with gain $\xi_{1/2/3}$ for different frequencies $\omega_{1/2/3}$ and a fixed gain $Kc_{1/2/3} = 10^6$.

* Figure 9(d) shows the reward variation with gain $\omega_{1/2/3}$ for different frequencies $\xi_{1/2/3}$ and a fixed gain $Kc_{1/2/3} = 10^6$.
<figure><img src="image_13.png" /><figcaption>Figure 9. Reward R₁ variation with PPF control law parameters set as identical for all control loops.</figcaption></figure>

In these examples, the controllers are identical for all three loops to provide a two-dimensional visual representation. The reward increases with the parameter value ($Kc_{1/2/3}$, $\omega_{1/2/3}$ or $\xi_{1/2/3}$) (indicator (1) in figure 9) and then gradually decreases as the gain margin decreases (indicator (2) in figure 9). High gain values lead to an unstable system (indicator (3) in figure 9). In addition, high values of frequencies $\omega_{1/2/3}$ and damping ratio $\xi_{1/2/3}$ lead to an unstable controlled system (indicator (4) in figure 9).

To consider the minimization process of the Simplex algorithm, the objective function is chosen with a negative sign of the conditional reward defined in equations (8)–(10).

## 4. Results and discussions

Using the DRL formalism, DRL and simplex methods are applied successively to the same environment defined with actions to be optimized according to the reward. A statistical study was conducted to compare their effectiveness and reliability. The initial point is the same for both methods for each independent test, as shown in figure 10. The random seed initializes a DRL agent by setting the weights and biases of the NNs randomly and estimating the initial actions $A_{init}$. At the end of the optimization process, the simplex outputs

<figure><img src="image_14.png" /><figcaption>Figure 10. Flowchart: DRL and simplex with a single initialization.</figcaption></figure>

the estimated optimal actions $A_{opt}$ and the DRL outputs the trained NNs with their associated estimated optimal actions $A_{opt}$.
<figure><img src="image_15.png" /><figcaption>**Figure 11.** Statistical distribution of final rewards obtained from 85 interdependent initializations for Simplex and DRL optimization methods.</figcaption></figure>

## 4.1. Statistical results

The parameters initialization step is an input condition on both methods. The aim of this numerical experiment is to determine the success rate without prior knowledge of the optimal results using random initialization. The convergence speed and control efficiency of the two methods are thus studied using a statistical approach. 85 agents are randomly initialized with different seeds to ensure numerical randomness also in the DRL training process. The starting points used for the Simplex optimization are actions $A_{init}$ given by the initialized agents from DRL trainings.

Figure 11 shows the distribution of Simplex and DRL methods according to the reward obtained at the end of the optimization.

A reward of 9.5 corresponds to the structure when control is off. For simplex, 52% of the tested cases lead to a controlled system that is worse than the uncontrolled one, compared to 30% for DRL. If the controller is effective (reward above 9.5), cases are divided into three categories:

* $9.5 < R_1 < 13$: Moderate control efficiency

* $13 < R_1 < 17$: Expected control efficiency

* $R_1$: Excellent control efficiency.

This discrimination is arbitrary and is based on preliminary measurements obtained on the test bench. These results need to be analyzed in the light of the number of environmental assessments required to achieve the lower reward of the category available in figure 12.

These figures highlight the convergence speed of the DRL method compared to the Simplex method. For the Simplex, the number of environment evaluations can reach 3.24 million (according to the options defined for the termination conditions). For all categories, the Simplex stopped because of the tolerance conditions. It can be deduced that Simplex leads to

<figure><img src="image_16.png" /><figcaption>**Figure 12.** Environmental evaluations (training episodes) per final reward (mean and standard deviation) across 85 interdependent initializations for Simplex and DRL.</figcaption></figure>

find a local optimum as a solution. All DRL algorithms are trained on 5000 episodes, but not all episodes are necessary to reach the different categories.

Figure 13 shows the reward variations with episode during DRL training, divided into the above categories. Even after 5000 episodes, only 6% of the tested agents manage to reach the assumed optimal reward value ($R_1 > 17$). This can lead to three main hypotheses.

* First, other agents need more episodes to reach the same reward. As can be seen in figures 13(a) and (b), the reward has an increasing tendency and does not seem to have reached a ceiling.

* Second, this maximum reward value is reached with a discontinuity according to the parameters as observed in figure 9.

* Thirdly, the agent diverges especially for categories that give an unstable control system ($R_1 < 0$).

It has been observed that some agents estimate action values outside the defined range, increasing with episodes and leading to infinite values.

The DRL method is more efficient than the Simplex method in that it finds a proportionately higher number of stable cases with less environmental evaluations. Both methods were run on the same computing server using a single thread of an Intel Xeon Gold 5318Y processor. Computational performance indicators are given in table 3. Although simplex is faster than DRL method, it interacts with the environment 700 times more than DRL. These differences in computation time are explained by the DRL method's need to update the weights and biases of the NNs and initialize the environment for each episode. However, the maximum memory usage for both methods is similar, with a difference of only 20%. The NN size is small, requiring similar computational resources to classical optimization methods such as Simplex.
<figure><img src="image_17.png" /><figcaption>**Figure 13.** DRL reward variation with episodes grouped by final reward value achieved: between 9 and 13 (a), between 13 and 17 (b), and over 17 (c) (the figures do not represent simplex variations with an episode; it only represents the terminal state).</figcaption></figure>

<table><caption>**Table 3.** Simplex and DRL computation performances comparison: example of one training.</caption><thead><tr><th>Method</th><th>Memory peak (GB)</th><th>Run time</th><th>Number of environment evaluations</th></tr></thead><tbody><tr><td>DRL</td><td>1.68</td><td>2 h 19 min and 16 sec</td><td>5000</td></tr><tr><td>Simplex</td><td>1.38</td><td>1 h 22 min 27 sec</td><td>1223 606</td></tr></tbody></table>

## 4.2. Control performances on the model

The Simplex and DRL methods resulted in the generation of tuned controllers exhibiting a range of degrees of efficiency, as quantified by the reward value. The controllers are tested on the model and their influence on the dynamics of the system is shown in figure 14 and grouped according to the above range of reward categories. The disturbance is introduced through the second actuator $Pa_2$.

* Figures 14(a)–(c) show moderate effects of the control law on the target mode. The gain reduction is about 5 dB for the two tested methods.

* Figures 14(d)–(f) show the expected effects of the control law on the target mode. The gain reduction is between 8 and 10 dB for DRL and between 5 and 10 dB for simplex within the target frequency band. The selected controls cause an amplitude split between one side and the other of the primary target frequency.

* Figures 14(g)–(i) show aberrant effects of the control law on the target mode. The gain reduction is between 10 and 30 dB for DRL and between 10 and 40 dB for Simplex at 380 Hz. This controller performance is unexpected. Further investigation reveals that these chosen controllers are unstable when applied to the test-bench. In fact, the phase margins are all negative (as shown in appendix D, figure 20). These controllers tuning exploit the limits of the reward definition: the performances exceed the gain margin stability constraint penalty and the numerical computation was not able to identify the system instabilities.

Also for most of the above tuned controllers it is not possible to perform measurements on the experimental setup due to unstable tuned controllers. The modeling does not consider experimental uncertainties, and the reward definition is unable
<figure><img src="image_18.png" /><figcaption>Figure 14. Bode plot for different controllers applied to the model: final reward between 9 and 13 (a)–(b)–(c), 13 and 17 (d)–(e)–(f) and over 17 (g)–(h)–(i).</figcaption></figure>

to compensate for these because phase margin stability was not imposed in the algorithm.

### 4.3. Reward improvement

The reward gain margin is not enough to ensure stability within the test bench environment. To address this issue, the phase margin $Pm_{deg}^i$, for each control loop $i = \{1;2;3\}$ in degree, is used in the reward definition for equations (8)–(10), such as:

* Stable case: $R_4 = 20 \times \log_{10} \sum_{i=1}^{i=3} \frac{\int_{f_1}^{f_2} G_{i,i}^{\text{off}} df}{\int_{f_1}^{f_2} G_{i,i}^{\text{on}} df}$

* Limit of stability case:

$$R_4 = 20 \times \log_{10} \left( \sum_{i=1}^{i=3} \frac{\int_{f_1}^{f_2} G_{i,i}^{\text{off}} df}{\int_{f_1}^{f_2} G_{i,i}^{\text{on}} df} \right) \times \left| \prod_{i=1}^{i=3} \frac{Gm_{\text{dB}}^i (0 < Gm_{\text{dB}}^i < 10)}{10} \right| \times \left| \prod_{i=1}^{i=3} \frac{Pm_{\text{deg}}^i (0 < Pm_{\text{deg}}^i < 180)}{180} \right|$$

* Unstable case: $R_4 = -1$

Figure 15 shows the new defined reward variations with controllers parameters. Including the phase in the definition constrains the reward value when a controller is selected near the instability region. As can be seen on figure 15(b), the reward drops off faster than with the previous definition. This includes

more penalty in the reward for being close to the instability range.

Previous results are used as a baseline so that only the tests that provide significant results are reused, excluding unstable controlled model cases. Only the NNs coming from the 20 best trainings are selected. This choice gives the DRL method an advantage over the simplex method, as the best initialization for DRL is not the same as the best initialization for the simplex method. Nevertheless, the aim is to facilitate a comparison between the two methods in terms of their convergence rate and capacity to adapt to changes while starting from the same point. To adjust the controller according to the new reward definition $R_4$, DRL and simplex methods are investigated as introduced in figure 16:

* The first strategy uses the initialized agents and restarts training with the goal of maximizing $R_4$.

* The second approach uses the final agent trained on $R_1$ as the initial state to continue training on $R_4$.

Figure 16 shows the different training strategies and figure 17 the control efficiency on the modeled structure.
<figure><img src="image_19.png" /><figcaption>Figure 15. Reward R₄ variation with control law parameters set as identical for all control loops.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Figure 16. Flowchart: restarting or continuing the training strategy.</figcaption></figure>

* Figure 17(a)–(c)–(e): A comparison of the training after 5000 episodes reveals that the restart strategy fails to reach the target control efficiency, in contrast to the continuous strategy.

* Figure 17(a): The mean reward is biased by the high reward values, which can reach over 17 with reward definition R₁ even for unstable controlled systems, which is not the case for R₄.

* Figures 17(d) illustrate similar controller performance on the system for the two methods. The influence of the stability criteria on the system response occurs at low frequency (71 Hz) with a decrease in the first mode amplitude gain with R₄ optimization (visible in appendix D on figure 21).

* Figures 17(f): Simplex tuning methods are still less efficient than DRL with a difference of 2 dB on the magnitude at the target frequency mode (380 Hz).

The point here is to show the efficiency of the method in achieving a reference target, not to exceed it. This would require more episodes if better performances were achievable.

### 4.4. Control performances on the test bench

The efficiency of the DRL tuned controllers is evaluated on the experimental test bench. A quantitative analysis is presented which evaluates the damping effect on the target mode. A
<figure><img src="image_21.png" /><figcaption>(a) Initial training with $R_1$</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(b) From $Pa_2$ to $Ps_2$ (Initial training with $R_1$)</figcaption></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /><figcaption>(c) Restart training with $R_4$</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(d) From $Pa_2$ to $Ps_2$ (Restart training with $R_4$)</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(e) Continue training with $R_4$</figcaption></figure>

<caption>(f) From $Pa_2$ to $Ps_2$ (Continue training with $R_4$)</caption>

<caption>**Figure 17.** Reward variation with episodes (a)–(c)–(e) and Bode plots (b)–(d)–(f) for the 20 best tuned controllers with DRL and simplex methods applied to the model for different strategies and reward definitions.</caption>

reference controller is adjusted by hand following the method given by Preumont [41] ensuring a gain margin above 10 dB: to tune the PPF controller, first set $\omega_i$ to the mode frequency to be controlled, then set $\xi_i$ usually between 0.5 and 0.7, and adjust the gain $K_{Ci}$ with the root locus to ensure stability.

Some DRL or Simplex tuned controllers have been observed to exhibit experimental instability, with only 10 out of 20 measurements yielding valid results. The reward definition is adapted to the experimental measurement by considering transfer functions from $Pa_2$ to $Ps_1$, $Ps_2$ and $Ps_3$ as shown in figure 18. Examples of tested controllers are listed in table 4.

* Table 4—Tuned values: the parameters tuned by the controllers are different from each other. Some gains are null, resulting in unrealistic values for $\xi$ and $\omega$ that have no effect on the control of the structure. The tuned references

were chosen to be identical for each loop. The DRL tuned controllers highlight the possibility of using only two controllers over three to achieve the same control objective. This phenomenon is interesting from a design point of view to reduce the energy and utility resources needed to build and control the structure.

* Table 4—Reward: The performance of the DRL-tuned controllers varies around the reference value.

* Figure 18: The damping effect of the reference controller is included into the performance standard deviation of the DRL-tuned controllers.

These results show the efficiency of the DRL proposed method to achieve a target control performance. It also provides additional information on controlling the structure, highlighting how the same objective can be achieved with only two control loops rather than three by identifying redundancies.
<figure><img src="image_27.png" /><figcaption>**Figure 18.** Bode plots for 10 tuned controllers with DRL measured on the experimental setup.</figcaption></figure>

<table><caption>**Table 4.** Control law values and reward computed from measurements on the experimental setup (selected examples DRL 1, 3, 5 and 10 out of 10 training sessions).</caption><thead><tr><th rowspan="2">Tuning method</th><th colspan="8">Tuned values</th><th rowspan="2">Reward<br>R<sub>4</sub></th></tr><tr><th>K<sub>c1</sub></th><th>K<sub>c2</sub></th><th>K<sub>c3</sub></th><th>ξ<sub>1</sub></th><th>ξ<sub>2</sub></th><th>ξ<sub>3</sub></th><th>ω<sub>1</sub></th><th>ω<sub>2</sub></th><th>ω<sub>3</sub></th></tr></thead><tbody><tr><td>Crtl off</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>9.54</td></tr><tr><td>Reference</td><td>-4 × 10<sup>-5</sup></td><td>-4 × 10<sup>-5</sup></td><td>-4 × 10<sup>-5</sup></td><td>0.23</td><td>0.23</td><td>0.23</td><td>2387</td><td>2387</td><td>2387</td><td>14.8</td></tr><tr><td>DRL 3</td><td>0</td><td>-1,75 × 10<sup>-6</sup></td><td>-7,00 × 10<sup>-5</sup></td><td>0,80</td><td>0,45</td><td>0,19</td><td>6273</td><td>2869</td><td>2142</td><td>10.62</td></tr><tr><td>DRL 1</td><td>-1,07 × 10<sup>-6</sup></td><td>0</td><td>-6,09 × 10<sup>-5</sup></td><td>0,32</td><td>0,78</td><td>0,11</td><td>2255</td><td>1479</td><td>2152</td><td>10.71</td></tr><tr><td>DRL 5</td><td>-1,99 × 10<sup>-6</sup></td><td>0</td><td>-5,73 × 10<sup>-5</sup></td><td>0,42</td><td>0,41</td><td>0,11</td><td>2881</td><td>3143</td><td>2169</td><td>21.97</td></tr><tr><td>DRL 10</td><td>-6,82 × 10<sup>-5</sup></td><td>-9,40 × 10<sup>-5</sup></td><td>0</td><td>0,21</td><td>0,11</td><td>0,91</td><td>2265</td><td>2198</td><td>108</td><td>11.18</td></tr></tbody></table>

**5. Discussion**

Several aspects of this study deserve further discussion regarding limitations, methodological choices, and perspectives for future work.

*Model limitations:* To enable multiple training runs, a transfer function model was identified from experimental measurements. While this approach avoids the risks and time consumption associated with online training on the test bench, it also introduces discrepancies between the model and the

physical system. These differences become critical when feedback control laws are considered, particularly near stability limits. Improving the accuracy of the model, for example through adaptive modeling or digital twins, could help address control instabilities that are not estimated by the current model.

*Reward function design:* The performance of DRL depends on the definition of the reward function. In this work, stability constraints were incorporated through gain and phase margins to ensure controllers could be safely implemented on the experimental setup. Nevertheless, designing an effective reward requires prior knowledge of the controlled structure in
order to avoid unrealistic results and to constrain the optimiz-
ation appropriately (e.g. action range, stability margins, effi-
ciency criteria).

* **Optimization strategies:** Interestingly, training multiple agents provides access to diverse yet equally effective solutions. This diversity can be advantageous in practice, particularly in cases such as the present study, where not all transducers are required to achieve effective vibration mitigation. In such situations, DRL can identify control strategies that reduce the number of active transducers, thereby lowering material use and potentially minimizing power consumption.

* **Perspectives for control extension:** Beyond the single-mode application demonstrated here, PPF controllers can be implemented in parallel for multimodal damping. Moreover, approaches such as time-switching or adaptive control could extend applicability to systems with high modal density or varying operating conditions. Such extensions, combined with autonomous DRL-based tuning, open promising avenues for future research in smart structures.

While the present work focuses on demonstrating the feas-
ibility of autonomous PPF tuning with DRL, it also paves the
way toward self-adaptive strategies, where controllers could
adjust in real time to evolving system conditions. Although
this capability is not implemented here, it represents a prom-
ising direction for future research.

**6. Conclusion**

This study demonstrates the efficiency of using DRL to tune
multiple control laws parameters in smart structures with
transducer networks. The main findings are summarized as fol-
lows:

* **Controller performance:** DRL-tuned PPF controllers achieved vibration attenuation at the target frequency of 380 Hz comparable to or even exceeding that of manually tuned and simplex-based controllers, with up to 10 dB reduction in amplitude.

* **Statistical reliability:** While both methods depend on ini-
tialization, DRL demonstrated greater robustness, with 70% of
training runs converging to stable solutions within 5000 epis-
odes, compared to 48% for the simplex method.

* **Improved reward function:** Incorporating gain and phase margin constraints into the reward definition ensured stable controllers, enabling safe implementation on the experimental setup.

* **Training strategies:** Continuing training with the improved reward led to faster convergence and better performance than restarting from the beginning, highlighting the adaptability of DRL agents.

* **Multiple solutions:** Because the optimization function is non-convex, testing several DRL initializations provides access to diverse yet effective control strategies. In this study, three distinct control laws were identified, each achieving comparable damping performance.

* **Improved tuning:** Compared with manual tuning, the DRL approach identified a solution in which only two out of three actuators were required. This highlights the potential of the method not only for vibration mitigation but also for optimal control aimed at minimizing external power consumption and actuator requirement.

The proposed approach can be adapted for use in more
complex scenarios, such as multimodal damping with multiple
controllers, different control definitions, and larger distrib-
uted transducer networks. Application to rotating machinery
or non-stationary excitation would require investigating the
algorithm's ability to adapt to frequency or time domain vari-
ations while ensuring both accuracy and rapid response. Future
work could also examine the impact of noise during training by
incorporating stochastic disturbances directly into the model
response, thereby assessing the robustness of the method under
more realistic conditions.

**Data availability statement**

The data cannot be made publicly available upon publication
because they are not available in a format that is sufficiently
accessible or reusable by other researchers. The data that sup-
port the findings of this study are available upon reasonable
request from the authors.

**Acknowledgments**

This work was supported by the LABEX CeLyA (ANR-10-LABX-0060) of Université de Lyon, within the program ‘Investissements d’Avenir’ operated by the French National Research Agency (ANR).

**Supplementary data**

The experimental the codes used to generate the results in this
study are publicly available on GitHub: https://github.com/
mfebvre/PiezoBeam-OptDRL.git

**Conflicts of interest**

The authors declare that they have no conflicts of interest.
# Appendix A. Smart beam material properties

The smart beam material properties are given in table 5.

<table><caption>**Table 5.** Smart beam material properties.</caption><thead><tr><th rowspan="2">Variable</th><th colspan="2">Beam</th><th colspan="2">Piezoelectric transducer (PZT-PIC255)</th><th rowspan="2">Unit</th></tr><tr><th>Name</th><th>Value</th><th>Name</th><th>Value</th></tr></thead><tbody><tr><td rowspan="6">Young modulus</td><td rowspan="6">E<sub>b</sub></td><td rowspan="6">205 × 10<sup>-9</sup></td><td>c<sup>S</sup><sub>11</sub></td><td>1.20 × 10<sup>-11</sup></td><td>[Pa]</td></tr><tr><td>c<sup>S</sup><sub>12</sub></td><td>7.52 × 10<sup>-10</sup></td><td>[Pa]</td></tr><tr><td>c<sup>S</sup><sub>13</sub></td><td>7.50 × 10<sup>-10</sup></td><td>[Pa]</td></tr><tr><td>c<sup>S</sup><sub>33</sub></td><td>1.11 × 10<sup>-11</sup></td><td>[Pa]</td></tr><tr><td>c<sup>S</sup><sub>44</sub> = c<sup>S</sup><sub>55</sub></td><td>2.11 × 10<sup>-10</sup></td><td>[Pa]</td></tr><tr><td>c<sup>S</sup><sub>66</sub></td><td>2.58 × 10<sup>-10</sup></td><td>[Pa]</td></tr><tr><td rowspan="4">Density Coupling constant</td><td rowspan="4">ρ<sub>b</sub></td><td rowspan="4">7850</td><td>ρ<sub>p</sub></td><td>7750</td><td>[kg·m<sup>-3</sup>]</td></tr><tr><td>e<sub>31</sub></td><td>-5.35</td><td>[C·m<sup>-2</sup>]</td></tr><tr><td>e<sub>33</sub></td><td>15.78</td><td>[C·m<sup>-2</sup>]</td></tr><tr><td>e<sub>15</sub></td><td>12.29</td><td>[C·m<sup>-2</sup>]</td></tr><tr><td rowspan="2">Relative permittivity</td><td rowspan="2">—</td><td rowspan="2">—</td><td>e<sup>S</sup><sub>11</sub></td><td>919.1</td><td>[—]</td></tr><tr><td>e<sup>S</sup><sub>33</sub></td><td>826.6</td><td>[—]</td></tr></tbody></table>

# Appendix B. Mode shape of the smart beam

A finite element model was used to position the piezoelectric transducers so to maximize the coupling coefficient for the bending modes between 0 and 1000 Hz. The optimization and the model are not presented in this study but figure 19 shows the modal shapes, excluding the six rigid body modes.

<figure><img src="image_28.png" /><figcaption>**Figure 19.** Mode shape visualization.</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>**Figure 19.** (Continued.)</figcaption></figure>

## Appendix C. Hyperparameters and NN size

The hyperparameters of the TRPO [37] algorithm govern the training process. Default parameters are used in this study since they are efficient enough, but tuning them will influence the learning efficiency of the actor and critic NNs. Table 6 summarizes the most important hyperparameter values and their effect on training.

<table><caption>**Table 6.** Hyperparameter definition for TRPO algorithms.</caption><thead><tr><td>Hyperparameter</td><td>Name</td><td>Default value</td><td>Range values</td><td>Effect</td></tr></thead><tbody><tr><td>Entropy loss weight</td><td>w ∈ ℝ</td><td>0.01</td><td>{0; 1}</td><td>It enhances agent exploration with high entropy loss.</td></tr><tr><td>Learning rate</td><td>α ∈ ℝ</td><td>0.01</td><td>{0; ∞}</td><td>Training may lead to sub-optimal results or divergence if the learning rate is too high. If too low, training will take a long time to complete.</td></tr><tr><td>Discount factor</td><td>γ ∈ ℝ</td><td>0.99</td><td>{0; 1}</td><td>In the learning process, it reduces the influence of the expected reward calculated by the critic.</td></tr></tbody></table>

The Critic and Actor NNs dimensions used for this case are specified in table 7. The dimensions of the NNs have been adjusted according to previous observations and common guidelines. This type of application requires small networks, as the number of inputs and outputs is relatively low.
<table><caption>**Table 7.** Actor and critic neural network definition.</caption><thead><tr><th rowspan="2">Layer</th><th rowspan="2">Name</th><th colspan="2">Number of neurons</th></tr><tr><th>Actor</th><th>Critic</th></tr></thead><tbody><tr><td>Input</td><td>N<sub>i</sub></td><td>3</td><td>3</td></tr><tr><td>Hidden layer 1</td><td>N<sub>hu</sub></td><td>25</td><td>25</td></tr><tr><td>Hidden layer 2</td><td>N<sub>hu</sub></td><td>25</td><td>25</td></tr><tr><td>Hidden layer 3</td><td>N<sub>hu</sub></td><td>25</td><td>25</td></tr><tr><td>Hidden layer 4</td><td>N<sub>hu</sub></td><td>25</td><td>25</td></tr><tr><td>Output</td><td>No</td><td>9</td><td>1</td></tr></tbody></table>

## Appendix D. System controlled response

Figure 20 shows the estimated gain and phase margin of the second transfer function loop. The other controllers are applied to the remaining two loops. The tuned controllers give a reward value of $R_1 = 17.61$, and is supposed to give an excellent vibration control efficiency. The gain margin is close to 10 dB, which is the upper limit of the stability definition. However, the phase margin is negative, which leads to control instability. Including only the gain margin in the reward is insufficient to inform the algorithm about system stability; the phase margin should also be included so the improved reward $R_4 = 0.19$.

<figure><img src="image_30.png" /><figcaption>**Figure 20.** Bode model response: from $P_{a2}$ to $P_{s2}$ gain and phase margin estimation for tuned values leading to $R_1 = 17.61$ and $R_4 = 0.19$, control off (black dash line), control on (blue line).</figcaption></figure>
Figure 21 shows a wide frequency band of the model response, comparing the free response with the controlled response with the DRL and Simplex tuning methods using the restart strategy.

<figure><img src="image_31.png" /><figcaption>**Figure 21.** Bode response for the 20 best tuned controllers with DRL and Simplex methods applied to the model: from $Pa_2$ to $Ps_2$ (Restart training with $R_4$).</figcaption></figure>

## ORCID iDs

Maryne Febvre ID 0009-0003-9793-0000

Jonathan Rodriguez ID 0000-0001-8872-8968

Simon Chesne ID 0000-0003-1306-7231

Manuel Collet ID 0000-0003-2049-0644

## References

[1] Hassani S, Mousavi M and Sharif-Khodaei Z 2022 *Smart Bridge Monitoring* (Butterworth-Heinemann) ch 14, pp 343–72

[2] Hurlebaus S and Gaul L 2006 *Smart structure dynamics* *Mech. Syst. Signal Process.* **20** 255–81

[3] Giurgiutiu V 2001 *Actuators and Smart Structures* (Elsevier) pp 58–81

[4] Curie P 1984 *Oeuvres de Pierre Curie / Publ. par les Soins de la Société Française de Physique* (Bibliothéque nationale de France)

[5] Lippmann W 1881 *Annales de Chimie et de Physique* (Bibliothéque nationale de France)

[6] Yi K, Monteil M, Collet M and Chesne S 2017 *Smart metacomposite-based systems for transient elastic wave energy harvesting* *Smart Mater. Struct.* **26** 035040

[7] Zheng Y, Wu Z, Zhang X and Wang K W 2019 A piezo-metastructure with bistable circuit shunts for adaptive

nonreciprocal wave transmission *Smart Mater. Struct.* **28** 045005

[8] Rodriguez J, Collet M and Chesné S 2022 *Active vibration control on a smart composite structure using modal-shaped sliding mode control* *J. Vib. Acoust.* **144** 021013

[9] Cudney H, Inman D and Oshman Y 1990 *Distributed structural control using multilayered piezoelectric actuators* 1989 *American Control Conference*

[10] Galeazzi C, Morganti F, Arduini C and Gaudenzi P 1996 *Analysis and control of microvibrations on ARTEMIS satellite* *J. Intell. Mater. Syst. Struct.* **7** 216–26

[11] Butler R and Rao V 1994 *Optimal control of infinite-order smart composite structural systems using distributed sensors* *Compos. Eng.* **4** 577–89

[12] Vasques C and Rodrigues J D 2006 *Active vibration control of smart piezoelectric beams: comparison of classical and optimal feedback control strategies* *Comput. Struct.* **84** 1402–14

[13] Stavroulakis G E, Foutsitzi G, Hadjigeorgiou E, Marinova D and Baniotopoulos C C 2005 *Design and robust optimal control of smart beams with application on vibrations suppression* *Adv. Eng. Softw.* **36** 806–13

[14] Tanaka M, Chonan S and Jiang Z W 1996 *Force control of a flexible finger with distributed sensors and piezoelectric actuators* *J. Intell. Mater. Syst. Struct.* **7** 301–6

[15] Han J-H and Lee I 1999 *Optimal placement of piezoelectric sensors and actuators for vibration control of a composite
plate using genetic algorithms *Smart Mater. Struct.*
**8** 257

[16] Aridogan U and Basdogan I 2015 A review of active vibration and noise suppression of plate-like structures with piezoelectric transducers *J. Intell. Mater. Syst. Struct.* **26** 1455–76

[17] Li L, Song G, and Ou J 2009 Adaptive fuzzy sliding mode based active vibration control of a smart beam with mass uncertainty, *Struct. Control Health Monit.* n/a–n/a

[18] Snyder S D and Tanaka N 1993 A neural network for feedforward controlled smart structures IEEE **4** 373–8

[19] Lee G-S 1996 System identification and control of smart structures using neural networks *Acta Astronaut.* **38** 269–76

[20] Smyser C P and Chandrashekhara K 1997 Robust vibration control of composite beams using piezoelectric devices and neural networks *Smart Mater. Struct.* **6** 178

[21] Spencer M G, Sanner R M and Chopra I 1998 An adaptive neurocontroller for vibration suppression and shape control of a flexible beam *J. Intell. Mater. Syst. Struct.* **9** 160–70

[22] Valoor M T, Chandrashekhara K and Agarwal S 2001 Self-adaptive vibration control of smart composite beams using recurrent neural architecture *Int. J. Solids Struct.* **38** 7857–74

[23] Jha R and Rower J 2002 Experimental investigation of active vibration control using neural networks and piezoelectric actuators *SMS* **11** 115–21

[24] Snyder S D and Tanaka N 1995 Active control of vibration using a neural network *IEEE* **6** 819–28

[25] Liang Y, Xu S, Hong K, Wang G and Zeng T 2019 Neural network modeling and single-neuron proportional-integral-derivative control for hysteresis in piezoelectric actuators *Meas. Control* **52** 1362–70

[26] Mohit, Chhabra D and Kumar S 2015 Active vibration control of the smart plate using artificial neural network controller *Adv. Aerosp. Eng.* **2015** 1–20

[27] Sutton R S and Barto A G 1992 Reinforcement learning: an introduction (MIT Press)

[28] Koch W, Mancuso R, West R and Bestavros A 2019 Reinforcement learning for UAV attitude control *ACM Trans. Cyber-Phys. Syst.* **3** 1–21

[29] Qiu Z-c, Chen G-h and Zhang X-m 2021 Reinforcement learning vibration control for a flexible hinged plate *Aerosp. Sci. Technol.* **118** 107056

[30] Guerra-Langan A, Estrada S A and Windsor S 2022 Reinforcement learning to control lift coefficient using

distributed sensors on a wind tunnel model *AIAA Scitech 2022 Forum* (American Institute of Aeronautics and Astronautics)

[31] Haughn K P T, Harvey C and Inman D J 2024 Deep learning reduces sensor requirements for gust rejection on a small uncrewed aerial vehicle morphing wing *Commun. Eng.* **3** 53

[32] Khalatbarisoltani A Soleymani M Khodadadi M 2019 Online control of an active seismic system via reinforcement learning *Struct. Control Health Monit.* **26** e2298

[33] Panda J, Chopra M, Matsagar V and Chakraborty S 2024 An iterative gradient descent-based reinforcement learning policy for active control of structural vibrations *Comput. Struct.* **290** 107183

[34] Perez M, Billon K, Gerges T, Capsal J-F, Cabrera M, Chesné S and Jean-Mistral C 2022 Vibration energy harvesting on a drone quadcopter based on piezoelectric structures *Mech. Ind.* **23** 20

[35] Febvre M, Rodriguez J, Chesne S and Collet M 2023 Artificial intelligence for active vibration control optimization on smart structures *ASME 2023 Conf. on Smart Materials, Adaptive Structures and Intelligent Systems*

[36] Febvre M, Rodriguez J, Chesne S and Collet M 2024 Deep reinforcement learning for tuning active vibration control on a smart piezoelectric beam *J. Intell. Mater. Syst. Struct.* **35** 1149–65

[37] Schulman J, Levine S, Abbeel P, Jordan M and Moritz P 2015 Trust region policy optimization *Proc. 32nd Int. Conf. on Machine Learning Proc. Machine Learning Research* vol 37, ed F Bach and D Blei (PMLR) pp 1889–97

[38] Nelder J A and Mead R 1965 A simplex method for function minimization *Comput. J.* **7** 308–13

[39] Goh C J and CAUGHEY T K 1985 On the stability problem caused by finite actuator dynamics in the collocated control of large space structures *Int. J. Control* **41** 787–802

[40] Li G, Miao X-Y, Hu Y-T and Wang J 2013 Analysis of smart beams with piezoelectric elements using impedance matrix and inverse Laplace transform *Smart Mater. Struct.* **22** 115001

[41] Preumont A 2018 *Vibration Control of Active Structures: An Introduction* 4th edn (Springer)

[42] Cheng J Y and Mailund T 2015 Ancestral population genomics using coalescence hidden Markov models and heuristic optimisation algorithms *Comput. Biol. Chem.* **57** 80–92