# Damage detection in a semi-active structural control system based on reinforcement learning

Aleksandra Jedlińska, Dominik Pisarski, Grzegorz Mikulowski, Bartlomiej Blachowski, Łukasz Jankowski

<figure><img src="image_1.png" /></figure>

## To cite this version:

Aleksandra Jedlińska, Dominik Pisarski, Grzegorz Mikulowski, Bartlomiej Blachowski, Łukasz Jankowski. Damage detection in a semi-active structural control system based on reinforcement learning. International Conference on Noise and Vibration Engineering (ISMA 2024), KU Leuven, Sep 2024, Leuven, Belgium. hal-04702122

HAL Id: hal-04702122

https://hal.science/hal-04702122v1

Submitted on 19 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Damage detection in a semi-active structural control system based on reinforcement learning

A. Jedlińska, D. Pisarski, G. Mikułowski, B. Błachowski, Ł. Jankowski

Institute of Fundamental Technological Research Polish Academy of Sciences, Poland
Pawińskiego 5B, 02-106, Warsaw, Poland

e-mail: ajedlins@ippt.pan.pl

## Abstract

This contribution applies the machine learning technique of reinforcement learning for simultaneous damage detection and control of structures. The proposed system consists of two components. The control component is responsible for semi-active mitigation of vibrations. The control law is determined experimentally in a trial-and-error interaction with a simulated environment. The process is data-driven: the control agent iteratively improves its control law based on the observed results of past control actions. The robustness relies on the accuracy of the structural model used for training. The control efficiency can decrease if the physical structure is damaged and diverges from the model, that is, when effective control may be most required. Thus, the second component of the proposed system monitors the structure to detect damages and inform the control component. The approach is tested in a numerical experiment of a shear-building under random seismic-type excitation. A semi-active tuned mass damper (TMD) is used as an actuator, and a classical TMD serves as a reference.

## 1 Introduction

Structural vibrations in engineering are very common - one of the examples where engineers need to pay special attention during the design process is when trying to prevent the effects of earthquakes on engineering structures. The most important negative effects include: the total destruction of the building, the loss of life of those in the vicinity of the building, and the high cost of reconstruction. Accordingly, various techniques for structural control and mitigation of structural vibrations have been investigated [1]. In general, they form a full spectrum of approaches, from fully passive to semi-active to active. Various techniques are also used to isolate the structure from the ground, which has the effect of reducing the impact of seismic forcing [2].

Active control techniques are frequently used, well-studied, and highly effective [3]. They have been successfully applied in many branches of modern industry. It is, however, worth pointing out that active control of large civil engineering structures requires the generation of large external forces and as a result a large amount of power which is hard to provide during a major earthquake [4].

Another type of control is passive control. When used for the mitigation of vibrations, it relies on structural optimization and passive energy dissipation devices. Usually, TMA (Tuned Mass Absorber) or TMD (Tuned Mass Damper) are applied, which are mechanical devices used to suppress or absorb vibrations [5]. When installed properly on a machine or structure, they draw away vibrational energy from the structure or machine and dissipate it internally, hence reducing their vibrations. They do not require a redesign of the target structure/machine and do not interfere with the main structure/machine function. This makes them an ideal solution for retrofitting/troubleshooting of existing machines [6]. TMA and TMD have been successfully implemented on high-rise buildings, rotating machines such as large water pumps, rotary no-break systems, chemical reactors, wind turbine gearboxes, scroll compressors, etc. [7].

The compromise between the solutions mentioned above are semi-active control techniques. They usually provide dissipative control [8–10]. The techniques rely on local modification of the mechanical properties of selected components or actuators (such as viscous damping, torque capacity, etc.) rather than
on directly generated large external forces [11]. The actuators used therefore generate only dissipative forces, i.e. forces that oppose the movement of the structure. Large external power sources are thus not needed and the danger of instability is reduced. However, the control is non-linear and the resulting optimal control problems are often analytically unsolvable.

The research and conclusions described in this contribution are based on the use of a tuned mass damper (TMD) as an actuator and semi-active structural control as a control strategy to enhance the quality of the control using information about structural damage. A TMD is a device installed in structures to reduce mechanical vibrations, consisting of a mass mounted on one or more damped springs [12]. Multiple modifications of the classic TMD approach have been offered and tested [13]. The vibration control effectiveness of a TMD can be increased by controlling such parameters as: the mass ratio and the natural frequency ratio between the TMD and the main structure, as well as the damping ratio of the TMD. The fundamental concept of TMD is that its natural frequency can be tuned to a certain range, allowing the damper to resonate with the structure [14]. The relatively large amplitude of TMD's movement serves energy dissipation so that the vibrational energy of the main structure can be dissipated and its vibrations can be mitigated. The control being explored involves real-time tuning of the damping of the dashpot [15]. Technically, this type of control is bilinear, and analytical solutions for optimal control are often not available. Open-loop optimal control is usually of the bang-bang type. This type of control means switching between two extreme states. In the presented methodology, switching points are learned using a reinforcement learning (RL) machine learning approach [16-20]. The model is learned by interacting with the environment, that is, through a series of trial-and-error interactions with a simulated structure subjected to random seismic-type excitation.

Evaluation of the results is carried out by comparing the total mechanical energy of the structure (without the TMD). The results described show that the control is more effective than an optimally tuned passive TMD. The approach proposed here is based on and extends our earlier results previously reported in [19,20]. The new development with respect to the above-mentioned work is to train the RL agent using additional information related to structural damage. Each of the previous works includes the results of training on models that do not take into account structural modifications. With the additional information introduced, we can consider further applications of the model. One of these is the possibility of damage detection based on structural response measurements. Another is increasing the effectiveness of the control while the original device is working on a damaged structure. Detailed damage information can be exploited to increase the effectiveness.

## 2 Reinforcement Learning – the technique

Reinforcement learning is a branch of machine learning techniques that is designed to train an agent to be able to learn the most effective action by trial-and-error interactions with a custom environment. The feedback the agent receives (rewards or penalties), makes it capable of improving its decision-making ability. Reinforcement learning procedures encourage an AI agent to discover the optimal chain of decisions that defines a ‘correct behavior’ within a model environment. This research explores the potential of reinforcement learning (RL) in semi-active structural control and damage detection. RL, unlike supervised and unsupervised learning, allows for interaction-based learning and is well-suited for the needs of structural control. Despite its successes in other complex control tasks, RL in structural control is a rarely studied approach. In this study, an DQN RL agent is utilized, which employs dense and recurrent artificial neural networks (ANN) to learn and encode Q-functions [21].

## 3 Structure

The interaction between the agent and the environment requires continuous feedback from the structure. In order to perform the tests, a numerical environment had to be created, reflecting the basic characteristics of the structure and its properties. The tests could not be conducted on a real physical structure for safety reasons. Due to the use of a structure modeled numerically, deviations from the real physical structure are
expected. The dynamics of the modeled structure is approximate, and the model's robustness to errors and measurement noise should be considered. As shown in [20], the control approach developed here is relatively insensitive to these errors.

The structure shown in Figure 1 is used in this contribution. It models an 11-story shear building, with a single degree of freedom (DOF) per story, equipped with a TMD at the top.

<figure><img src="image_3.png" /><figcaption>Figure 1: 11-story shear building with a TMD installed at the top.</figcaption></figure>

The equation of motion is formulated in 12 DOFs. A seismic type of excitation was considered and modeled as a lateral acceleration $a(t)$. The equation takes the following form:

$$[M]\{u\} + [C(t)]\{u\} + [K]\{u\} = -[M]\{r\}a(t) \quad (1)$$

The equation incorporates the mass, damping, and stiffness matrices denoted as [M], [C(t)], and [K], respectively. The bang-bang control being considered involves switching C(t) between two extreme states that correspond to the maximum and zero damping levels of the TMD dashpot in addition to the standard building damping. The critical damping ratio for the first mode of natural vibration, in the case of a building without TMD, is set to 2%, and the proportional material damping model is used. The lateral load allocation vector r is a 12-element vector of ones. For the simulation response, the Newmark algorithm was used with a time step of 2 ms. The maximum frequency of the control switching was 25 Hz. The TMD mass equals 3% of the building mass. The masses and stiffnesses of each floor are compatible with the example structure already considered in the literature [22]. The natural frequencies range from 0.89 Hz to 14.92 Hz.

Artificial intelligence models are susceptible to being overtrained for specific data or patterns. In order for the considered model to learn to control the structure robustly, the excitation must be appropriately chosen so that it is not repetitive and covers a wide spectrum of frequencies, like real physical excitations. To avoid overfitting the learned control to a particular ground motion pattern, the seismic excitation was modeled as white noise at each time step. At all-time steps, the instantaneous ground acceleration value $a(t)$ was thus independently randomized from a uniform probability distribution centered around zero. The feedback signal consisted of the full state vector. Such a choice is inconvenient in practice because it is difficult to measure, but it corresponds to the full observation and enables the assessment of the ultimate potential of the considered control architecture and RL agent.
# 4 Undamaged structure - Reinforcement Learning Control

The numerical environment enables the agent to explore potential control actions and obtain rewards or penalties. The agent considered here receives penalties defined as the total instantaneous mechanical energy of the building structure (without the TMD) and processed using the smooth L1 loss. The training aim is to minimize the total of these penalties over a training episode.

The DQN (deep Q network) approach is used. Figure 2 shows the architecture of the artificial neural network used by the RL agent to predict the Q function and select the control actions (TMD damping on/off). The input signal to the network is the full state: a 24-element vector containing normalized values of displacements and velocities of the structure stories. The network output exists as a two-element vector corresponding to two possible control actions.

<figure><img src="image_4.png" /><figcaption>Figure 2: Architecture of the artificial neural network in the RL control agent.</figcaption></figure>

To ensure the network capacity, hyperparameters were set at the minimum level at which performance began to deteriorate. The number and type of layers, and the numbers of neurons in each layer were selected by experiment. The AdamW optimizer was used. The learning rate was initially 0.001. Training episodes lasted 20 s, corresponding to 18 basic vibration periods. The initial exploration rate was 100% and decreased to 5% at a rate of 0.994 per episode.

After training, the performance of the control was quantified using the same mechanical quantity as the reward/penalty. Accordingly, the average total mechanical energy of the building structure (without the TMD and averaged over the complete test episode) was used and compared with the reference value delivered by a classical, optimally tuned passive TMD. 2000 test episodes, each 80 seconds in length, were simulated for evaluation. Results are shown in Table 1. The RL control was more effective than the optimally tuned passive TMD: it resulted in a 20.9% reduction of the average mechanical energy of the building (average reduction per single episode was 22.3%). Additionally, the RL control reduced the standard deviation of the mean total energy to an even greater extent, that is, by 42.8%.

<table><caption>Table 1: Effectiveness of the RL control agent trained using an intact structure (Figure 2) compared to an optimally tuned passive TMD, when applied to an intact structure. Mean total mechanical energy of the building (without TMD) and energy reduction per single episode.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Undamaged</th></tr><tr><th>Mean</th><th>Std. dev.</th></tr></thead><tbody><tr><th>Optimum passive TMD [a.u.]</th><td>660,21</td><td>168,36</td></tr><tr><th>RL control [a.u.]</th><td>522,29</td><td>96,36</td></tr><tr><th>Energy reduction per episode (TMD to RL)</th><td>22,3%</td><td>11,48%</td></tr></tbody></table>
# 5 Damage identification in RL-controlled structure

RL control, as demonstrated in the previous section, is much more effective than an optimally tuned passive TMD. However, the control provided by the trained RL agent is data-driven and non-linear, and consequently, the dynamics of the RL-controlled building are also non-linear. This means that typical structural health monitoring methods cannot be applied, since they are all based on the assumption of linearity of the structural response. Thus, a machine learning approach is used here for identifying the defects in the RL-controlled structure. These structural damages were modeled as a reduction in stiffness (up to 70%) on up to two randomly selected floors of the building (but not the TMD). The stiffness reduction factor on each selected floor was randomly and independently drawn from a uniform distribution from 0 to 70%. In that manner, the damage scenario was generated from scratch for each training and test episode. To enable the RL agent's neural network architecture to identify the defects, an additional branch was created, as shown in Figure 3. For the output layer, the 11 elements correspond to the 11 floors of the building and are designed to estimate the corresponding damage reduction ratios. The input layer is the full-state feedback signal and is shared with the RL control branch.

<figure><img src="image_5.png" /><figcaption>Figure 3: RL control agent artificial neural network architecture - damage identification branch</figcaption></figure>

The control branch was used to make decisions about the control actions during training, and its weights were frozen when training the damage identification branch. To identify the damage, the mean square identification error was used as the loss. The damage identification branch was trained in interaction with the damaged structure. The network was evaluated using 2000 test episodes, each lasting 80 seconds. The average root mean square error of the identified damage ratio was equal to 2.6% only and thus was much smaller than the considered damage range of up to 70%.

# 6 Structural control using damage-aware RL agent

The control branch was frozen while training the RL agent's damage identification branch. Although the dynamic characteristics of the intact and damaged structures are different, damaged structures are controlled using the same control law as the original, undamaged structure. Such a decision had a negative impact on the effectiveness of the control. The results are shown in Table 2 (which extends the data shown in Table 1): the mean total structural energy increased by 6.59%.
<table><caption>Table 2: Effectiveness of the RL control agent trained using the undamaged structure (Figures 2 and 3), compared to optimally tuned passive TMD, when applied to undamaged and damaged structures. Mean total building mechanical energy (without TMD) and energy reduction per single episode.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Applied to undamaged structure</th><th colspan="2">Applied to damaged structure</th></tr><tr><th>Mean</th><th>Std. dev.</th><th>Mean</th><th>Std. dev.</th></tr></thead><tbody><tr><th>Optimum passive TMD [a.u.]</th><td>660,21</td><td>168,36</td><td>725,17</td><td>180,56</td></tr><tr><th>RL control [a.u.]</th><td>522,29</td><td>96,36</td><td>556,7</td><td>105,34</td></tr><tr><th>Energy reduction per episode (TMD to RL)</th><td>22,3%</td><td>11,48%</td><td>24,98%</td><td>12,81%</td></tr></tbody></table>

For a more effective RL control, two approaches were tested:

1. The control branch was re-trained in interaction with the damaged structures. Every training episode used a randomly generated damage scenario, as described above.

2. Before re-training the control branch, its head (the last 3 layers) was further provided with features related to the damage extracted in the damage identification branch, as depicted in Figure 4. This makes the control head damage-aware and allows it to take damage-related information into account when processing its control-related features.

<figure><img src="image_6.png" /><figcaption>Figure 4: RL control agent artificial neural network architecture with damage identification branch and damage-related features input to the control branch head.</figcaption></figure>

While re-training the control branch, the network weights in the damage identification branch were frozen. A noticeable increase in control performance was observed only with the second of the above approaches. Table 3 shows the results, and an improvement over these shown in Table 2 can be clearly observed. A graphical representation of all cases is presented in Figure 5.

<table><caption>Table 3: Effectiveness of the damage-aware RL control agent re-trained using damaged structures (Figure 4), compared to optimally tuned passive TMD, when applied to damaged structures. Mean total building mechanical energy (without TMD) and energy reduction per single episode.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Applied to damaged structure</th></tr><tr><th>Mean</th><th>Std. dev.</th></tr></thead><tbody><tr><th>Optimum passive TMD [a.u.]</th><td>715,25</td><td>186,01</td></tr><tr><th>RL control [a.u.]</th><td>532,59</td><td>101,12</td></tr><tr><th>Energy reduction per episode (TMD to RL)</th><td>25,10%</td><td>10,44%</td></tr></tbody></table>
Table 3 summarizes the efficiency of the fine-tuned, damage-aware RL control agent in application to damaged structures, and compares it to an optimally tuned passive TMD. The results show a significant reduction in the average total energy when using the updated, damage-aware RL control strategy, with an increase of only 1.97% in average total energy compared to the original RL control strategy (instead of 6.59% obtained for the original RL control).

<figure><img src="image_7.png" /><figcaption>Figure 5: Mean total structural energy for RL control compared to optimally tuned passive TMD. Test 1: RL agent trained and tests conducted on an undamaged structure. Test 2: RL agent trained on undamaged structure; tests conducted on damaged structure. Test 3: Agent RL was additionally tuned and tests were conducted on a damaged structure.</figcaption></figure>

## Conclusions

This study presents the application and evaluation of a multi-branch neural network for the control of a shear-type engineering structure using the reinforcement learning approach. Besides the typical mechanical response feedback, the control is additionally based on structural damage information identified in real time. The RL agent is trained using the DQN Reinforcement Learning technique.

The structural model used for the evaluation is a shear-type building with eleven degrees of freedom and a TMD. For the purposes of modeling the damage, the stiffness properties of up to two floors are randomly decreased, no more than 70% each. The structure experiences a random seismic excitation, simulated using white noise.

With respect to previous research, a new element of the presented approach is the use of damage-related features, obtained in the damage identification branch of the agent, for the purpose of structural control. The tested model was thus aware of damages and could accordingly modify its control actions. The results can be quantitatively summarized as follows:
1. The Reinforcement Learning agent, in comparison to applying only optimally tuned passive TMD, caused a reduction of the total mechanical energy of the structure per episode by 20.9% on average and standard deviations of measurements by 42.8%

2. Improvement in control effectiveness was possible due to the reuse of features associated with the damage identification branch. The average total energy increased by 6.59% when the original RL control was applied to the damaged structure. Reusing the damage features for control purposes reduced this increase to only 1.97%.

3. The damage identification branch of the RL agent was able to effectively identify the damages. The average root mean square error of the identified floor damage ratio was equal to 2.6% only, and it was thus much smaller than the considered damage range of up to 70%.

Further research may include applications with incomplete feedback information, that is, the case of limited instrumentation of the controlled and monitored structure.

## Acknowledgement

The authors gratefully acknowledge the support of the National Science Centre, Poland, granted under the grant agreement 2020/39/B/ST8/02615. For the purpose of Open Access, the authors have applied a CC-BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission.

## References

[1] Basu B., Bursi O.S., Cascati F., Cascati S., Del Grosso A.E., Domaneshi M., Faravelli L., Holnicki-Szulc J., Irshik H., Krommer M., Lepidi M., Martelli A., Ozturk B., Pozo F., Pujol G., Rakicevic Z., Rodellar J. A European Association for the Control of Structures joint perspective. Recent studies in civil structural control across Europe (2014) Structural Control and Health Monitoring, 21:1414–1436.

[2] Warn G.P., Ryan K.L. A review of seismic isolation for buildings: Historical development and research needs (2012) Buildings, 2 (3):300–325. https://doi.org/10.3390/buildings2030300

[3] Korkmaz S. A review of active structural control: Challenges for engineering informatics (2011) Computers and Structures, 89 (23–24): 2113–2132. https://doi.org/10.1016/j.compstruc.2011.07.010

[4] Nishimura I., Kobori T., Sakamoto M., Koshika N., Sasaki K., Ohrui S. Active tuned mass damper (1992) Smart Materials and Structures, 1 (4):306–311. https://doi.org/10.1088/0964-1726/1/4/005

[5] Parulekar Y.M., Reddy G.R. Passive response control systems for seismic response reduction: A state-of-the-art review (2009) International Journal of Structural Stability and Dynamics, 9 (1):151–177. https://doi.org/10.1142/S0219455409002965

[6] Gutierrez Soto M., Adeli H. Tuned Mass Dampers (2013) Archives of Computational Methods in Engineering, 20 (4):419–431. https://doi.org/10.1007/s11831-013-9091-7

[7] Frahm H. Device for damping vibrations of bodies (1911) U.S. Patent No. 989,958.

[8] Fisco N.R., Adeli H. Smart structures: Part I – Active and semi-active control (2011) Scientia Iranica, 18 (3A): 275–284. https://doi.org/10.1016/j.scient.2011.05.034

[9] Casciati F., Rodellar J., Yildirim U. Active and semi-active control of structures-theory and applications: A review of recent advances (2012) Journal of Intelligent Material Systems and Structures, 23 (11):1181–1195. https://doi.org/10.1177/1045389X12445029

[10] Symans M.D., Constantinou M.C. Semi-active control systems for seismic protection of structures: A state-of-the-art review (1999) Engineering Structures, 21 (6):469–487. https://doi.org/10.1016/S0141-0296(97)00225-3
[11] Hurlebaus S., Gaul L. Smart structure dynamics (2006) Mechanical Systems and Signal Processing, 20 (2):255-281. https://doi.org/10.1016/j.ymssp.2005.08.025

[12] Ormondroyd J., Den Hartog J.P. The Theory of the Dynamic Vibration Absorber (1928) Journal of Fluids Engineering, Transactions of the ASME, 49-50 (2):021007. https://doi.org/10.1115/1.4058553

[13] Nagarajaiah S., Jung H.-J. Smart tuned mass dampers: Recent developments (2014) Smart Structures and Systems, 13 (2):173-176. https://doi.org/10.12989/sss.2014.13.2.173

[14] Elias S., Matsagar V. Research developments in vibration control of structures using passive tuned mass dampers (2017) Annual Reviews in Control, 44:129-156, https://doi.org/10.1016/j.arcontrol.2017.09.015

[15] Pinkaew T., Fujino Y. Effectiveness of semi-active tuned mass dampers under harmonic excitation (2001) Engineering Structures, 23 (7):850-856. https://doi.org/10.1016/S0141-0296(00)00091-2

[16] Silver D., Hubert T., Schrittwieser J., Antonoglou I., Lai M., Guez A., Lanctot M., Sifre L., Kumaran D., Graepel T., Lillicrap T., Simonyan K., Hassabis D. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play (2018) Science 362(6419):1140-1144, https://doi.org/10.1126/science.aar6404

[17] Khalatbarisoltani A., Soleymani M., Khodadadi M. Online control of an active seismic system via reinforcement learning (2019) Structural Control and Health Monitoring 26(3):e2298. https://doi.org/10.1002/stc.2298

[18] Pisarski D., Jankowski Ł. Reinforcement learning-based control to suppress the transient vibration of semiactive structures subjected to unknown harmonic excitation (2023) Computer-Aided Civil and Infrastructure Engineering 38 (12):1605-1621. http://doi.org/10.1111/mice.12920

[19] Jedlińska A., Pisarski D., Mikułowski G., Błachowski B., Jankowski Ł., Semi-active structural control using viscous dampers and reinforcement learning (2023) Proceedings of the 10th ECCOMAS Thematic Conference on Smart Structures and Materials (SMART 2023), July 3-5 2023, Patras (Greece), pp. 589-596. https://doi.org/10.7712/150123.9813.444767

[20] Jedlinska A., Pisarski D., Mikulowski G., Błachowski B., Jankowski Ł. Semi-Active Control of a Shear Building Based on Reinforcement Learning: Robustness to Measurement Noise and Model Error (2023) Proceedings of the 18th Conference on Computer Science and Intelligence Systems (FedCSIS 2023), Sept 17-20 2023, Warsaw (Poland), pp. 1007-1010. https://doi.org/10.15439/2023F8946

[21] Van Hasselt H., Guez A., Silver D. Deep reinforcement learning with double Q-Learning (2016) Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI 2016), Feb 12-17 2016, Phoenix (AZ, US), pp. 2094-2100. https://doi.org/10.1609/aaai.v30i1.10295

[22] Pourzeynali S., Lavasani H.H., Modarayi A.H., Active control of high rise building structures using fuzzy logic and genetic algorithms (2007) Engineering Structures 29(3):346-357. https://doi.org/10.1016/j.engstruct.2006.04.015