# A framework for computer vision-based health monitoring of a truss structure subjected to unknown excitations

Mariusz Ostrowski, Bartlomiej Blachowski, Bartosz Wójcik, Mateusz Żarski,
Piotr Tauzowski, Łukasz Jankowski

► To cite this version:

Mariusz Ostrowski, Bartlomiej Blachowski, Bartosz Wójcik, Mateusz Żarski, Piotr Tauzowski, et al. A framework for computer vision-based health monitoring of a truss structure subjected to unknown excitations. Earthquake Engineering and Engineering Vibration, 2023, 22 (1), pp.1-17. 10.1007/s11803-023-2154-3. hal-04187441

HAL Id: hal-04187441

https://hal.science/hal-04187441v1

Submitted on 29 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Earthq Eng & Eng Vib (2023) 22: 1-17

DOI: https://doi.org/10.1007/s11803-023-2154-3

# A framework for computer vision-based health monitoring of a truss structure subjected to unknown excitations

Mariusz Ostrowski¹†, Bartlomiej Blachowski¹‡, Bartosz Wójcik²,³†, Mateusz Żarski²,³†, Piotr Tauzowski¹§
and Łukasz Jankowski¹‡

1. Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

2. Silesian University of Technology, Gliwice, Poland

3. Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland

**Abstract:** Computer vision (CV) methods for measurement of structural vibration are less expensive, and their application is more straightforward than methods based on sensors that measure physical quantities at particular points of a structure. However, CV methods produce significantly more measurement errors. Thus, computer vision-based structural health monitoring (CVSHM) requires appropriate methods of damage assessment that are robust with respect to highly contaminated measurement data. In this paper a complete CVSHM framework is proposed, and three damage assessment methods are tested. The first is the augmented inverse estimate (AIE), proposed by Peng *et al.* in 2021. This method is designed to work with highly contaminated measurement data, but it fails with a large noise provided by CV measurement. The second method, as proposed in this paper, is based on the AIE, but it introduces a weighting matrix that enhances the conditioning of the problem. The third method, also proposed in this paper, introduces additional constraints in the optimization process; these constraints ensure that the stiffness of structural elements can only decrease. Both proposed methods perform better than the original AIE. The latter of the two proposed methods gives the best results, and it is robust with respect to the selected coefficients, as required by the algorithm.

**Keywords:** computer vision; structural health monitoring; physics-based graphical models; augmented inverse estimate; model updating; non-negative least square method

## 1 Introduction

Structural health monitoring (SHM) is important due to safety reasons, and it can be helpful in optimization of maintenance schedules. However, despite the significant effort devoted to this topic in recent decades, inspection of structures in actual applications is not yet automated, and at a minimum requires human supervision. Involvement of human time and effort generates significant costs. On the other hand, the design and implementation of sensor networks for large-scale structures is still very challenging and expensive. Computer vision-based structural health monitoring (CVSHM), which makes use of cameras or unmanned aerial vehicles, seems to offer a promising solution (Żarski *et al.*, 2022).

Machine learning and image processing methods have recently become increasingly popular, and a variety of CVSHM approaches dedicated to different problems are currently under intensive development (Spencer *et al.*, 2019). CVSHM allows the monitoring of large-scale structures, or at least regions potentially exposed to damage, with the aid of one or more digital cameras.

CVSHM includes inspection of the structure (by using image analysis, which shows damage such as concrete cracks, spalling, or exposed rebar), as well as vibration-based SHM (which uses object tracking methods and analyses structural dynamics). The former group involves mainly machine learning methods. Apart from a significant computational burden, one of the disadvantages of CVSHM inspection is the need to collect significant amounts of training data required for machine learning. This requirement can be addressed by employing realistic synthetic images or videos generated, using a structural model, which in this context is called a physics-based graphical model (PBGM) (Hoskere *et al.*, 2019; Narazaki *et al.*, 2021). PBGMs also can be used for the latter group of approaches to test object tracking methods at the level of computer simulations (Narazaki *et al.*, 2021).

**Correspondence to:** Bartlomiej Blachowski, Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
Tel: +48-22-826-12-81
E-mail: bblach@ippt.pan.pl

†PhD Student; ‡Associate Professor; §Research Assistant

**Supported by:** National Science Centre, Poland Granted Through the Project 2020/39/B/ST8/02615

Received October 10, 2022; Accepted November 22, 2022

© The Author(s) 2023, corrected publication 2023
This work focuses on vibration-based CVSHM. Computer vision-based (CV) methods are usually suitable for measuring large-amplitude displacements at low frequencies, whereas low-amplitude displacements are prone to errors resulting from factors such as changes of illumination or camera vibration during measurement, as well as equipment limitations such as low resolution. At medium and high frequencies, CV-based measurements become noisy (Feng and Feng, 2018; Trainotti et al., 2022). Thus, the following two topics are crucial for the performance of CVSHM: (1) accuracy of structural displacement estimation in CVSHM and (2) selection development of damage assessment methods that are robust with respect to measurement errors. Both of these topics are discussed in this paper.

Luo et al. (2021) proposed a categorization of vibration-based object tracking according to the following criteria:

a) initialization method: detection-based tracking, detection-free tracking;

b) processing mode: online tracking, offline tracking;

c) type of input: stochastic tracking, deterministic tracking.

Detection-based tracking is suitable for applications in which the tracked object appears and disappears from the region of interest (ROI). Automated detection of the tracked object introduces additional errors, according to the used model (Bose et al., 2007). Manual selection of the tracked objects used in detection-free methods is devoid of this disadvantage. Moreover, in many tasks of vibration-based CVSHM the number of the tracked objects does not change (Hu et al., 2012; Zhang and van der Maaten, 2014).

Online tracking is suitable for control and for continuous monitoring of systems (Xiang et al., 2015). Even though offline tracking by using of batch of frames does not allow continuous CVSHM, the realization of subsequent recording-postprocessing-assessment steps in a loop does not usually take a significant amount of time compared to the timescale of damage development, which is rather slow in actual structures. Offline tracking allows for the use of more computationally demanding but more accurate algorithms than those used in online tracking (Henriques et al., 2011).

The difference between stochastic and deterministic tracking concerns the type of the output, which might be treated as a random process. It is related to optimization techniques used in the detection and tracking stages (Breitenstein et al., 2009). Deterministic tracking always provide the same output for the same initial parameters and the same batch of frames (Huang et al., 2008).

The primary methods of object tracking used in CVSHM are called template matching. Template matching finds the position of the tracked object by matching it with a template, which represents the object, and with the ROI in subsequent video frames. The position of the template inside the ROI corresponds with the tracked structural displacement. Template matching

methods can be categorized into two groups: area-based template matching and feature-based template matching (Feng and Feng, 2021). In the former case both the template and the ROI are sub-regions of the frame, and they are matched according to some error or cross-correlation function of their pixel intensities (Hii et al., 2006; Adhikari et al., 2016; Liu et al., 2021). In the latter case the template is represented by its key points, which describe some characteristic features in the image. Later, these key points are matched with the key points extracted from the ROI, according to the key point descriptors that encode information about the vicinity of the key point. An example of feature-based template matching technique is the Kanade-Lucas-Tomasi (KLT) algorithm (Tomasi and Kanade, 1991; Shi and Tomasi, 1994). This algorithm tracks corner points that can be detected by employing the Harris-Stephens algorithm (Harris and Stephens, 1988). Feature-based template matching is more robust with respect to changing light conditions or other image contaminations, and it is more effective computationally. However, area-based template matching exhibits a higher degree of accuracy; hence, these methods are still attractive for CVSHM (Blachowski et al., 2023). Feng et al. (2015) showed that area-based methods can track high-contrast artificial targets and natural targets, such as bolts or nuts and prints, with satisfactory accuracy. If natural targets are not available for a structure, artificial targets can be mounted or printed. This requires access to the structure. However this approach is still less expensive and less time-consuming than dealing with traditional sensors, such as accelerometers or lasers.

Information obtained from the structural model, made, for instance, with the aid of the finite element (FE) method, can be employed to improve object tracking accuracy. Narazaki et al. (2021) proposed an interesting model-informed method of estimation of structural displacement dedicated to CVSHM. Information from the model is used for camera calibration and for tracking nodal displacements. Later, this approach to CV measurement was enhanced using the Bayesian inference (Gomez et al., 2022). The Bayesian inference of the estimated structural response with the aid of knowledge gleaned from the FE model allowed the researchers to not only enhance the accuracy of estimation but also to estimate the response outside the region made visible by the camera. Hence, this technique seems to be promising in large-scale applications in which it is difficult to record the entire structure.

Regarding the methods of detection, localization, and assessment of damages, they can be divided into model-free and model-informed methods. The former group uses only information available in measurements (Suwała and Jankowski, 2012), whereas the latter group also employs information about the investigated structure obtained from its model. A good example of a model-free method is the method proposed by Blachowski et al. (2017). This method analyzes accelerations of nodal
points of the structure; hence, extraction of information about strains is possible. Model-free methods require accurate measurement data. Thus, attention here is paid to model-based methods. Two well-known classical approaches that use a model of the structure are the methods of the damage locating vectors and the stochastic damage locating vectors (Bernal, 2002; 2006).

An important aspect is the sensitivity of the selected criterion to the development of the damage. The damage is usually represented as the stiffness reduction of affected structural members. More recently, changes in structural damping have been recognized as a promising indicator of damage. Damage often magnifies the non-proportional damping components, which not only affects energy dissipation rates but also increases phase-shifts in complex mode shapes (Curadelli et al., 2008; Iezzi et al., 2015). However, as Curadelli et al., stated, damping-based damage identification is a new field, which therefore requires additional research. Damping parameters are difficult to identify and are subjected to higher measurement errors than are natural frequencies or mode shapes. On the other hand, modes that participate in structural motion must involve strains in damaged members (Blachowski et al., 2016). If the damaged member of the structure is not subjected to strain, the response of the structure is insensitive to damage, and thus damage is not detectable.

A broad group of model-based methods is based on model updating (Friswell and Mottershead, 1995; Mottershead et al., 2011). Such methods amount to a calibration of the model, which is performed by changing its parameters to reproduce some properties of the measured system. Model updating can generally be used in two tasks: (1) obtaining a model that closely reproduces the behavior of the investigated structure, and (2) parametric identification, i.e., reproducing—usually local—properties of an actual structure (Yuen, 2010). These two groups of model updating approaches can lead to different results while showing the same measured system output. The updated model obtained in the former case is well-suitable for simulation purposes, whereas in the latter case, parametric identification allows for a more reliable assessment of damage, treated as stiffness reduction. The differences result from the fact that a model updating procedure is affected not only by the measurement noise but also by the modeling error. An example of a method dedicated to parametric identification of stiffness properties was proposed by Yuen et al., (2006). It is based on a Bayesian probabilistic framework in which the a posteriori maximum is estimated. Such an approach includes knowledge about the model in a manner that allows one to achieve a trade-off between the measurement and expected modeling errors. A broad group of model updating methods are based on modal parameters identified from the response of potentially damaged structure. The stochastic subspace identification (SSI) methods are suitable for CV measurement, since they do not employ

measured excitation that also is not measured by the camera (Peeters, 2000). This method usually is used with automated stabilization diagrams that enhances robustness and accuracy of the identification (Li et al., 2020). Then, the model is updated until it reproduces identified modal data of the potentially damaged structure.

Truss structures usually contain a large number of structural members. Hence, one of the key disadvantages of the model updating methods is the numerical difficulty of updating the stiffness parameters of all members while simultaneously aiming at detection and assessment of damage. Thus, the potentially damaged elements must be indicated first and, later, the stiffness of the corresponding model elements can be updated in order to evaluate the damage level. This problem can be solved by pursuing additional constraints on updated stiffness parameters. Blachowski (2019) showed that if only negative increments of the updated parameters are allowed, then the damage assessment procedure is more reliable, and the damaged elements are properly selected among even numerous other elements. After reformulation of this constraint was implemented in a MATLAB® non-negative least square (NNSL) solver. Another methodology was proposed by Peng et al. (2021). This method is called the augmented inverse estimate (AIE); it is based on the augmented least square problem with a regularization based on truncated singular value decomposition (TSVD). The problem is solved in an iterative manner. In each iteration step, all stiffness parameters that increase or do not significantly change their values are considered as belonging to the nondamaged elements and subsequently are removed from the equation set. Hence, the number of unknowns is reduced. Additionally, the augmented part of the equation includes information about gross errors that could disturb the solution found with classical approaches.

This paper studies a novel framework for CVSHM that can detect and assess damage to truss structures, even for highly contaminated measurement data. The framework is designed to handle truss structures that contain numerous monitored elements. The method works with CV measurement employing area-based template matching that maximizes the zero-normalized cross-correlation function (ZNCC). Truss nodes are the natural targets that are tracked to estimate the displacement. As CV measurement does not collect any direct information about excitations, the proposed framework has been designed to work with unknown loads to maintain an easy application of digital cameras as the sole measurement devices. It has been found that the original AIE is not sufficient in the case of an ill-conditioned problem, coupled with a simultaneous presence of a high-level noise in the measurement data (for example, 50%). Thus, two approaches are proposed here and are compared with the original AIE. In the first case, the AIE is modified by weighting the rows of the augmented sensitivity matrix to improve its
conditioning. The second proposed approach introduces additional constraints into model updating. The NNSL solver is used for a weighted least square problem. The idea of removing the unknowns from the equation set that correspond to the nondamaged elements is adopted from the AIE. It has been demonstrated that the two proposed modified approaches, respectively termed the weighted AIE (WAIE) and the weighted negative least square inverse estimate (WNLSIE), outperform the original AIE. Additionally, the WNLSIE is robust with respect to the selected parameters, which in engineering practice is a crucial property. Results have been obtained using PBGM-based videos that were a part of “The 2nd International Competition for Structural Health Monitoring” (IC-SHM, 2021). The results can be assessed by comparing them with known ground truth data.

The paper is structured according to the framework shown in Fig. 1. Section 2 is devoted to the adopted method of CV-based measurement, including the selection of ROIs and scaling transforms, and it describes area-based template matching that maximizes the ZNCC. This section also describes the class of FE models suitable for model updating purposes and discusses the original AIE method and introduces its two modified counterparts. Section 3 demonstrates the performance of the proposed framework, employing PBGM-based videos, and assesses its robustness with respect to measurement errors. Finally, conclusions are stated in Section 4.

<figure><img src="image_2.png" /><figcaption>Fig. 1 Flowchart of the proposed CVSHM framework</figcaption></figure>

## 2 Methodology for the proposed framework of computer vision-based structural health monitoring

### 2.1 Preparation of data before measurement

To utilize the proposed framework, the tracked elements of the structure must be manually selected. In the case of truss structures, the nodes are usually an appropriate choice. The selected areas define the templates that are later used for template matching. Examples of the areas selected with the aid of MATLAB® software are shown in Fig. 2(a). The extracted template for Node 11 and the ROI generated by adding preselected margins are shown in Fig. 2(b).

Determination of the pixel-to-meter scaling factor, to be used for converting the pixel displacements obtained from the video to displacements expressed in meters, is determined by drawing a line through the elements whose location in the structure is known. The pixel-to-meter scaling factor is then calculated based on the structure's dimensions.

### 2.2 Extraction of nodal displacements from video

Nodal displacements are extracted by maximizing the ZNCC between the template and the ROI, which is expressed by Eq. (1).

$$ \mathrm{ZNCC}(x, y) = \frac{\sum_{\zeta, \eta} (F(\zeta + x, \eta + y) - \bar{F}_{x,y})(T(\zeta, \eta) - \bar{T})}{\sqrt{\sum_{\zeta, \eta} (F(\zeta + x, \eta + y) - \bar{F}_{x,y})^2} \sqrt{\sum_{\zeta, \eta} (T(\zeta, \eta) - \bar{T})^2}} \quad (1) $$

where x and y are the coordinates of the template inside the ROI expressed in pixels, F is the image in the preselected ROI, $\bar{F}_{x,y}$ is the mean of $F(\zeta + x, \eta + y)$ in the region under the template, T is the template and T is its mean. The plotting of the ZNCC values for the ROI from Fig. 2(b) and Template No. 11 is shown in Fig. 3. Such a function can have many local maxima, hence

<figure><img src="image_3.png" /><figcaption>Fig. 2 Manual selection of tracked objects: (a) fragment of a frame of a PBGM-based video with manually selected templates to be tracked, and (b) the template for Node 11 (top) and the corresponding ROI (bottom)</figcaption></figure>
the template position in the ROI is found by means of an exhaustive search method. Additionally, to achieve subpixel precision, the ZNCC data are interpolated with a denser mesh using spline functions. In this case, the reliability and accuracy of the results are more important than the computation time.

## 2.3 Parameterized finite element model

Both the original and the modified AIE method involve a structural model. An FE model ($M, C, K(\theta), D$) is considered, which satisfies the linear equation set:

$$ \begin{cases} M\ddot{q}(\theta,t) + C\dot{q}(\theta,t) + K(\theta)q(\theta,t) = f(t) \\ q_M(\theta,t) = Dq(\theta,t) \end{cases} \qquad (2) $$

where $M$ and $C$ are constant mass and proportional damping matrices, respectively, whereas $K(\theta)$ is the stiffness matrix that depends on the parameter vector $\theta = [\theta_1 \dots \theta_{N_\theta}]^T$, $D$ is a Boolean output matrix that indicates the measured degrees of freedom, $q(\theta,t)$ denotes the vector of structural displacements, $f(t)$ is the vector of the external excitations, and $q_M(\theta,t)$ is the measured system output. The matrices $M$ and $C$ are constant, because it is assumed that the damage does not significantly affect structural mass and damping, and $K(\theta)$ depends on $\theta$ as follows:

$$ K(\theta) = K_0 + \sum_{t=1}^{N_\theta} \theta_t K_t \qquad (3) $$

where $K_0$ is the stiffness matrix that represents the part of the structure that is not monitored, and $K_t$ is the nominal stiffness matrix of the $t$th particular element from the set of the monitored elements. If $\theta_t = 1$, then the $t$th monitored finite element is considered to not be damaged. If $\theta_t < 1$, then the stiffness of the monitored element is reduced, which represents damage. The damage level $\delta_t$ of the $t$th monitored element is defined as:

$$ \delta_t = 1 - \theta_t \qquad (4) $$

In Eqs. (3) and (4), stiffness reduction applies to whole FE. Hence, any localized damage can be identified only as a global effect within all involved FE. Moreover, Eq. (3) is a suitable model for the reduction of the effective cross section area only for structural members that operate mainly under longitudinal loads, whereas it is not suitable in the case of significant bending or torsional moments. Such parameterization of the FEs is selected here, because bending in truss members is assumed to be significantly smaller than in longitudinal strains.

## 2.4 Identification of modal data with SSI-DATA and stabilization diagrams

The damage assessment methods discussed in this paper utilize experimental eigenvalues $\hat{\lambda}^{(m)}$ (squares of the natural frequencies) and eigenvectors (mode shapes) $\hat{\phi}^{(m)} \in R^{N_0}$, where $m = 1...N_m$ indexes the modes, and $N_0$ is the number of the measured outputs. Hence, modal data must be extracted from the measured displacement time series. To this end, the data-driven stochastic subspace identification method (SSI-DATA) has been employed (Peeters, 2000), which does not require any information about the excitations. Finally, the extracted modal data are used to quantify the damage.

To estimate accurately the modal data and to reject spurious modes, stabilization diagrams are employed. The stable modes are determined using the following criteria:

$$ \left\{ \begin{array}{l} \left| \frac{\hat{f}_n^{(m)} - \hat{f}_{n-1}^{(m)}}{\hat{f}_{n-1}^{(m)}} \right| \le \alpha_f \\ 1 - MAC(\hat{\phi}_n^{(m)}, \hat{\phi}_{n-1}^{(m)}) \le \alpha_\phi \\ \left| \frac{\hat{\xi}_n^{(m)} - \hat{\xi}_{n-1}^{(m)}}{\hat{\xi}_{n-1}^{(m)}} \right| \le \alpha_\zeta \end{array} \right. \qquad (5) $$

where $\hat{f}_n^{(m)}$ is the $m$th identified natural frequency for

<figure><img src="image_4.png" /><figcaption>Fig. 3 Example of the dependence of the ZNCC function on the position of the template in the ROI</figcaption></figure>
model order n; MAC(·, ·) denotes the modal assurance criterion; $\hat{\phi}_n^{(m)}$ is the mth identified mode shape for model order n; $\hat{\zeta}_n^{(m)}$ is the mth identified modal damping factor for model order n; and $\alpha_f$, $\alpha_\phi$, $\alpha_\zeta$ are preselected thresholds with the typical values of 1%, 2% and 5%, respectively. The final modal parameters are calculated as the mean values:

$$ 
\begin{aligned}
\hat{f}^{(m)} &= \frac{1}{|S_m|} \sum_{n \in S_m} \hat{f}_n^{(m)}, & \hat{\phi}^{(m)} &= \frac{1}{|S_m|} \sum_{n \in S_m} \hat{\phi}_n^{(m)}, \\
\hat{\zeta}^{(m)} &= \frac{1}{|S_m|} \sum_{n \in S_m} \hat{\zeta}_n^{(m)}
\end{aligned}
\qquad (6) 
$$

where $S_m$ is the set of stable solutions n collected for each mode m for which the conditions in inequality Eq. (7) are satisfied.

$$ 
\left\{
\begin{array}{l}
\left| \frac{\hat{f}_n^{(m)} - \hat{f}_{n \text{min}}^{(m)}}{\hat{f}_{n \text{min}}^{(m)}} \right| \le \alpha_f \beta \\
\\
1 - \text{MAC}(\hat{\phi}_n^{(m)}, \hat{\phi}_{n \text{min}}^{(m)}) \le \alpha_\phi \beta \\
\\
\left| \frac{\hat{\zeta}_n^{(m)} - \hat{\zeta}_{n \text{min}}^{(m)}}{\hat{\zeta}_{n \text{min}}^{(m)}} \right| \le \alpha_\zeta \beta \\
\\
|S_m| \ge N_s
\end{array}
\right.
\qquad (7) 
$$

In the inequality shown above, $n_{\text{min}}$ denotes the lowest order of the model for which conditions in Eq. (5) are satisfied for the mth mode, $\beta$ is an additional preselected coefficient, and $N_s$ is the preselected minimal number of solutions. For CV measurements, $\beta$ is typically close to one, and $N_s$ ranges between 5 and 20. These parameters can be selected with the trial-and-error method. After calculation of the mode shapes according to Eq. (6), the doubled modes are detected by using the MAC criterion, i.e., by checking for any pair of modes $m_1$ and $m_2$ with the criterion $\text{MAC}(\hat{\phi}_n^{(m_1)}, \hat{\phi}_n^{(m_2)}) \ge \gamma_{\text{MAC}}$. The modal data of the doubled modes are then calculated analogously to Eq. (6) but from the solutions that correspond to modes $m_1$ and $m_2$ at once.

After identification of the modal data, only those modes are considered that represent the vibration mostly in the plane perpendicular to the direction of the camera view. This is due to the higher accuracy of these modal data: the camera registers the displacements only in this plane. Remaining modes are rejected using the criterion below:

$$ S_{\text{rej}} = \left\{ k : \frac{\| \mathbf{D}_z \hat{\phi}_{\text{num}}^{(k)}(\boldsymbol{\theta}) \|_1}{\| \mathbf{D}_y \hat{\phi}_{\text{num}}^{(k)}(\boldsymbol{\theta}) \|_1} \le \varepsilon_{\text{YZ}} \right\} \qquad (8) $$

where $S_{\text{rej}}$ is the set of the rejected modes, $\|\cdot\|_1$ denotes the

11-norm and the matrices $\mathbf{D}_y, \mathbf{D}_z$ select the horizontal (parallel to the camera view) and vertical (perpendicular) degrees of freedom from the model, respectively; $\varepsilon_{\text{YZ}}$ is a preselected coefficient; and $\hat{\phi}_{\text{num}}^{(k)}(\boldsymbol{\theta})$ is the kth numerical mode matched with the experimental one according to the criterion:

$$ k_m = \arg \max_{k \in K} \left\{ \text{MAC}(\hat{\phi}^{(m)}, \mathbf{D}_y \hat{\phi}_{\text{num}}^{(k)}(\boldsymbol{\theta})) - 2 \frac{(\hat{\lambda}^{(m)} - \hat{\lambda}_{\text{num}}^{(k)}(\boldsymbol{\theta}))^2}{\hat{\lambda}^{(m)} 2} \right\} \qquad (9) $$

$K$, where $|K| > N_m$, is the search set, and $\hat{\lambda}_{\text{num}}^{(k)}$ is the kth numerical eigenvalue. In the equation above, both the MAC value and the error between the numerical and identified eigenvalues are used as mode matching criteria; however, using MAC only is also possible. The weighting coefficient of 2 is selected by employing the trial-and-error method. Mode matching between numerical and experimental mode shapes is necessary because a direct assessment of the vibration direction only from the measurement data is not possible due to the lack of a third dimension in the video data.

## 2.5 Damage assessment based on augmented inverse estimate

In this subsection, three damage assessment methods are described. They can be all classified as model updating methods for parametric identification. Each of these methods provides a way to deal with the trade-off between the accuracy of the fit between the model and the highly contaminated measurement data, and the regularization level of the solution.

In the proposed framework, it is required that the nominal FE model (before model updating) should accurately reproduce the behavior of the structure. The quality of the FE model can be checked using modal data identified from the healthy structure. If the MAC between the numerical and experimental mode shapes, or the relative error of the natural frequencies (or eigenvalues) is not satisfactory, then the nominal FE model should be initially calibrated before applying it for damage assessment. After its positive evaluation or an initial calibration, only the data identified from the monitored (potentially damaged) structure are used for damage assessment.

### 2.5.1 Background for the original AIE method

In this subsection the original AIE method is briefly introduced. More details can be found in the work of Peng et al. (2021). In this method the equation set is considered:

$$ A(\boldsymbol{\theta})x=y(\boldsymbol{\theta}) \qquad (10) $$

In Eq. (10):
$$A(\theta) = [S(\theta) E], \quad (11)$$

$$x = \begin{bmatrix} \Delta\theta_1 & \Delta\theta_2 & \cdots & \Delta\theta_{N_{\theta}} & \tilde{y}_i & \cdots & \tilde{y}_j & \cdots & \tilde{y}_k \end{bmatrix}^T \quad (12)$$

where $\Delta\theta_i$ are the increments (usually negative) of the vector $\theta$, the elements $\tilde{y}_i \cdots \tilde{y}_k$ represent the augmented part of the vector $x$, the vector $y(\theta) \in R^{N_{\theta}+N_{\text{err}}}$ contains the differences between the measured and numerical modal data,

$$y(\theta) = \hat{z} - z_{\text{num}}(\theta) \quad (13)$$

where

$$\hat{z} = \begin{bmatrix} \hat{\lambda}^{(1)} & \lambda^{(2)} & \cdots & \lambda^{(N_m)} & \phi^{(1)\text{T}} & \phi^{(2)\text{T}} & \cdots & \phi^{(N_m)\text{T}} \end{bmatrix}^T,$$

and $z_{\text{num}}(\theta)$ is defined analogously, but the elements in $y$ that are highly contaminated with measurement errors are replaced with zeros:

$$y = \begin{bmatrix} y_1 & y_2 & \cdots & y_{i-1} & 0 & y_{i+1} & \cdots & y_{j-1} & 0 & y_{j+1} & \cdots & y_{k-1} & 0 & y_{k+1} & \cdots & y_{(N_0+1)N_m} \end{bmatrix}^T \quad (14)$$

$S(\theta) \in (N_0+1)N_m \times N_\theta$ is the modal sensitivity matrix:

$$S(\theta) = \begin{bmatrix} \frac{\partial \lambda_{\text{num}}^{(k_1)}}{\partial \theta_1} & \cdots & \frac{\partial \lambda_{\text{num}}^{(k_1)}}{\partial \theta_{N_{\theta}}} \\ \vdots & \ddots & \vdots \\ \frac{\partial \lambda_{\text{num}}^{(k_{N_m})}}{\partial \theta_1} & \cdots & \frac{\partial \lambda_{\text{num}}^{(k_{N_m})}}{\partial \theta_{N_{\theta}}} \\ c_1 \mathbf{D} \frac{\partial \phi_{\text{num}}^{(k_1)}}{\partial \theta_1} & \cdots & c_1 \mathbf{D} \frac{\partial \phi_{\text{num}}^{(k_1)}}{\partial \theta_{N_{\theta}}} \\ \vdots & \ddots & \vdots \\ c_{N_m} \mathbf{D} \frac{\partial \phi_{\text{num}}^{(k_{N_m})}}{\partial \theta_1} & \cdots & c_{N_m} \mathbf{D} \frac{\partial \phi_{\text{num}}^{(k_{N_m})}}{\partial \theta_{N_{\theta}}} \end{bmatrix} \quad (15)$$

where the indices $k_1 \cdots k_{N_m}$ select the numerical modes matched according to Eq. (9), the eigenvalue derivatives $\frac{\partial \lambda_{\text{num}}^{(k)}}{\partial \theta_i}$ are calculated as shown by Fox and Kapoor (1968), and the eigenvector derivatives $\frac{\partial \phi_{\text{num}}^{(k)}}{\partial \theta_i}$ are computed as shown by Nelson (1976); $c_m$ is the modal scaling factor:

$$c_m = \frac{\hat{\phi}^{(m)\text{T}} \phi_{\text{num}}^{(k_m)}}{\| \phi_{\text{num}}^{(k_m)} \|^2} \quad (16)$$

and it is used to scale the numerical mode shapes in Vector $z_{\text{num}}(\theta)$. In Eq. (11), $E$ is a sparse matrix that contains $-1$ in positions that correspond to zeros in the

Vector $y(\theta)$ and the augmented part of vector $x$:

$$E = \begin{bmatrix} 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ -1 & 0 & \cdots & 0 \\ 0 & -1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & -1 \\ 0 & 0 & \cdots & 0 \end{bmatrix} \quad (17)$$

As can be seen in Eqs. (11)–(17), there are $N_{\text{err}}$ elements in vector $y$ that are replaced with zeros which correspond to the augmented part of matrix $A$ and vector $x$. These elements correspond to the measured data contaminated by a gross error that could be sufficiently large to disturb the solution of the system of Eq. (11) even if the equation set is overdetermined and a regularization is used. If their true values must be approximated, their estimates can be found in the augmented part of vector $x$.

The influence of errors smaller than gross errors, such as measurement noise, on the solution of Eq. (10) is reduced with the aid of regularization, based on the truncated singular value decomposition (TSVD). The regularized solution is given by the formula below:

$$\tilde{\mathbf{x}} = \left( \sum_{s=1}^{N_{\sigma}} \sigma_s^{-1} \mathbf{v}_s \mathbf{u}_s^{\text{T}} \right) \mathbf{y}(\theta), \quad \sigma_s > \sigma_{\text{tol}} \sigma_{\text{max}}(\theta) \quad (18)$$

In this equation, the symbols $\sigma_s$, $\mathbf{u}_s$, and $\mathbf{v}_s$ denote the $s$th singular value, the left singular vector, and the right singular vector of matrix $A(\theta)$, respectively; $\sigma_{\text{tol}}$ is a selected threshold; and $\sigma_{\text{max}}(\theta)$ is the maximal singular value of matrix $A(\theta)$. Only $N_\sigma$ singular values greater than the preselected threshold are considered. The regularized solution (18) is calculated in an iterative manner to update the FE model in each iteration step:

$$\theta[l+1] = \theta[l] + \kappa_{\theta} \Delta\tilde{\theta} \quad (19)$$

where $l$ denotes the $l$th iteration step, $\kappa_\theta$ is a preselected scaling factor, and the vector $\Delta\tilde{\theta}$ collects the first $N_t$ elements of vector $\tilde{x}$. After each iteration step, the elements of vector $\theta$ whose increments satisfy the condition $\Delta\tilde{\theta}_t / \max \Delta\tilde{\theta}_t > -a_\theta$ are removed and the dimension of vector $\theta$ is reduced. The corresponding columns in matrix $A$ are also removed. In other words, the elements that correspond to small changes in Vector $\theta$ are considered to be undamaged. Small changes can result from noise, or a distortion of the solution caused by the regularization procedure. Such an approach significantly enhances the selection process of the damaged elements and the quantification of the damage level, since it reduces the number of unknowns, thereby rendering the problem more overdetermined while
<figure><img src="image_5.png" /><figcaption>Fig. 4 Flowchart of the AIE operation in the proposed framework</figcaption></figure>

improving its conditioning.

Finally, the algorithm is stopped by the stop condition defined as $||\theta[l+1] - \theta[l]|| < a_{\theta tol}$. The damaged elements can be determined as those that correspond to significant changes in the parameters of the updated FE model. The AIE algorithm of damage assessment is summarized in the flowchart shown in Fig. 4.

### 2.5.2 Proposed WAIE method

In the original AIE method, the sensitivity matrix $S(\theta)$ (see Eq. (15)) includes the derivatives $\partial\lambda_{num}^{(m)} / \partial\theta_t$.

Due to the fact that $\partial\lambda_{num}^{(m)} = (2\pi f_{num}^{(m)})^2$, where $f_{num}^{(m)}$ is the mth numerical natural frequency, the values of the eigenvalues and their derivatives with respect to parameters can be large, which can degrade the conditioning of matrix $A(\theta)$. To avoid these two problems, a weighting procedure for the rows of matrix $A(\theta)$ is proposed. It consists of the weighted least square optimization problem:

$$ (P1) \quad \begin{array}{ll} \text{find} & \mathbf{x} \\ \text{to minimize} & \|A(\theta)\mathbf{x} - \mathbf{y}(\theta)\|_{W}^{2} \end{array} $$

where $\|\cdot\|_W$ is a 12-norm with respect to weighting matrix $W$. This weighting matrix is intended to prioritize minimization of the error among the elements of vectors $A(\theta)\mathbf{x}$ and $\mathbf{y}(\theta)$, so that $W$ can be reciprocal to the measurement covariance matrix $\Sigma_M$, if it is known, as in:

$$ W = \Sigma_M^{-1} \qquad (20) $$

The variances are usually near-proportional to squares of the values of the measured parameters; hence, matrix $W$,

defined in Eq. (20), also enhances the conditioning of the optimization problem (P1). If matrix $\Sigma_M$ is not known, then the selection of $W$ as shown in Eq. (21) is proposed:

$$ W = \text{diag} \left[ \left( \frac{1}{\hat{\lambda}^{(1)2}} \quad \dots \quad \frac{1}{\hat{\lambda}^{(N_m)2}} \quad \frac{w_{\phi}}{\|\hat{\phi}^{(1)}\|^2} \quad I_{\phi}^{\text{T}} \quad \dots \quad \frac{w_{\phi}}{\|\hat{\phi}^{(N_m)}\|^2} \quad I_{\phi}^{\text{T}} \right)^{\text{T}} \right] \quad (21) $$

where $I_{\phi}$ is the vector of ones having the same dimension as eigenvectors $\hat{\phi}^{(m)}$, and $w_{\phi}$ is an arbitrary selected weight. Here, it is proposed that $w_{\phi} = 0.01$, because the coefficient of variation of the measured eigenvalues is usually about ten times smaller than the coefficient of the variation of the eigenvectors.

Since matrix $W$ is usually diagonal, it is a straightforward process to find the weighting matrix $Q$, such that $Q^2 = W$. After substituting $Q^2$ with $W$, the optimization problem takes the following form:

$$ (P1) \quad \begin{array}{ll} \text{find} & \mathbf{x} \\ \text{to minimize} & \|QA(\theta)\mathbf{x} - Q\mathbf{y}(\theta)\|^2 \end{array} $$

The above optimization problem is equivalent to the least square estimate of the solution to the overdetermined equation set:

$$ QA(\theta)\mathbf{x} = Q\mathbf{y}(\theta) \qquad (22) $$

that is used instead of Eq. (10) in the AIE method, which results in the weighted AIE (WAIE) estimate. Despite the fact that matrix $QA(\theta)$ is much better conditioned than $A(\theta)$, the weighted estimate can be additionally regularized by using TSVD, as demonstrated by Eq. (18). Enhancement and the reliability of this modification is demonstrated in Section 3.
### 2.5.3 Proposed WNLSIE method

With this approach we introduce an additional constraint, namely weighted negative least square inverse estimate (WNLSIE), that the increments of $\theta_i$, $t = 1, 2, ..., N_\theta$ can only be negative. Thus, the augmented matrix $A(\theta)$ cannot be used because the constraints should not affect the augmented part of vector $x$, see Eq. (12). The estimates $\tilde{y}_i$ in vector $x$ can have both negative and positive signs. Hence, the elements of vector $y(\theta)$ and the rows of matrix $S(\theta)$ that correspond to gross error are removed instead of replacing the elements of vector $y(\theta)$ with zeros and augmenting the modal sensitivity matrix. The optimization problem can be formulated as shown below:

$$ (P2) \quad \begin{array}{ll} \text{find} & \Delta\theta \\ \text{to minimize} & \|QS(\theta)\Delta\theta - Qy(\theta)\|^2 \\ \text{subject to} & \Delta\theta_i < 0, \quad t = 1, 2, ..., N_\theta \end{array} $$

Damage assessment based on the optimization problem (P2) is similar to that of the AIE (see Fig. 4), but without replacing the erroneous data with zeros in $y(\theta)$. 0Matrix $S(\theta)$ is used instead of $A(\theta)$, and the solution of the problem is found without using TSVD-based regularization. After some modifications, the solution of the optimization problem (P2) can be found with the MATLAB® NNLS solver function lsqnonneg($QS(\theta)$, $Qy(\theta)$).

Similar to the original AIE, the unknowns corresponding to the elements considered as undamaged are removed from the equation set in the subsequent iteration steps of the model updating procedure (see Fig. 4).

## 3 Results obtained from PBGM-based videos

### 3.1 Investigated truss structure and data preparation before measurement

The investigated structure is represented by the FE model, as shown in Fig. 5. The structure has a length of 5.5118 m (0.3937 per each bay), a width of 0.3937 m, and a height of 0.4 m. Each rod is a pipe with an outer diameter of 0.01554 m and an inner diameter of 0.01087 m. The material properties of each element are Young's modulus $E = 18.5 \times 10^9$ Pa, the shear modulus $G = 79.3 \times 10^9$ Pa and the density $\delta = 8000$ kg/m$^3$. Each rod is represented by one beam finite element based on the Euler-Bernoulli beam theory and uses the cubic shape functions.

The primary measurements nodes (M1-M16) that are visible at the front of the structure in the PBGM-based videos are shown in Fig. 5, together with some additionally selected secondary measurement nodes (M17-M19). Nodes M17-M19 allow for better separation of the torsional vibration modes from the vertical bending modes during identification of modal data. They are not hidden behind nodes M10, M11 and

<figure><img src="image_6.png" /><figcaption>Fig. 5 Primary (M1-M16) and secondary (M17-M19) measurement truss nodes, and the camera view for PBGM-based videos marked on the FE model of the truss</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6 Numeration of the parametrized FEs (in green) representing monitored structural members</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 7 Examples of time series of vertical displacements extracted from the video “undamaged” compared with the corresponding ground truth data for primary measurement nodes (a) M8 and (b) M9</figcaption></figure>

<figure><img src="image_9.png" /></figure>

M15 in the image plane due to the fact that the center of the image in the PBGM-based videos is slightly below the structure, as shown in Fig. 5. The camera observes these nodes existing from an angle that ensures their visibility (compare with Fig. 2(a)). The distance between the camera and the structure is not known.

The aim is to monitor elements that connect the primary measurement nodes (M1-M16). These elements are marked in green in Fig. 6. They form a dense mesh of structural members to be identified, which often leads to numerical difficulties.

The PBGM videos show vibration of the truss excited by unknown loads. Resolution of each video is 1920 × 1080 px. The pixel-to-meter scaling factor is nearly 1.5 mm/px for the primary measurement nodes (M1-M16) and nearly 1.9 mm/px for the secondary measurement nodes (M17-M19) due to the greater distance from the camera. The framerate is 120 fps. The duration is four minutes, which corresponds to 28,800 time samples for each measured output. The videos are available on the Internet. In this paper only the videos “undamaged”, “Damage1”, “Damage2”, “Damage6” and “Damage7” are used, in which the image plane is parallel to the truss plane. All templates are selected manually, as shown in the example displayed in Fig. 2.

## 3.2 Extraction of nodal displacements from the processed videos

The ZNCC data were interpolated with an eight times denser mesh to obtain sub-pixel precision in the displacement estimation. Examples of the time histories of the measurement nodes M8 (in the middle of the structure) and M9 (close to the support) for a healthy structure, accompanied with ground truth data and the calculated error, are shown in Figs. 7(a) and 7(b), respectively. Only the vertical displacements are presented in Fig. 7, but displacements in both the vertical and horizontal directions are extracted from the videos.

It is evident that the displacement estimation for measurement node M8 is relatively accurate as opposed to M9. This is due to the fact that the truss nodes close to the support vibrate with smaller amplitudes and

<figure><img src="image_10.png" /><figcaption>Fig. 8 Measurement noise level for primary measurement nodes of the healthy structure</figcaption></figure>

show greater participation in higher-frequency modes, rendering the CV measurement less accurate.

Errors for all preliminary measurement nodes expressed as $\text{std}(q_{Mi} - q_{GTi}) / \text{std}(q_{GTi})$, where $q_{Mi}$ is the vertical displacement of the $i$th measurement node and $q_{GTi}$ is the corresponding ground truth data, for the “undamaged” video are shown in Fig. 8. For the remaining videos the results are similar. It is evident that the measurement nodes close to the support are more difficult to track. Measurement error achieves a level of 50%, which affects additional identification of modal data and the damage assessment. Additionally, the estimation of displacements of the nodes placed on top of the structure (M9-M16) seems to be more difficult. This is a result of their greater motion out of the image plane due to their location over the structural supports, and the participation of the torsional vibration modes in the structural response. It is shown in Fig. 9 that peaks in the amplitude spectrum of the displacement estimation error correspond to the out-of-plane vibration modes (more details about identification of modal data are provided in the next subsection). It also can be seen that the estimation error shown in Fig. 7(b) has a well-demonstrated period equal to the period of the first out-of-plane mode. Participation of the out-of-plane modes causes the changes to the scale and results in false in-plane displacements that are visible by the camera at nodes placed far from the image center (see Fig. 5). Thus, the M9 node is characterized by the largest measurement
<figure><img src="image_11.png" /><figcaption>Fig. 9 Amplitude spectrum of error between the estimated vertical displacement of measurement node M9 and the corresponding ground truth data $q_{M9} - q_{GT9}$, accompanied by the identified mode shapes (green dots) that correspond to the marked amplitude peaks, and compared with matched numerical modes</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 10 Stabilization diagram obtained from the video "undamaged" with the selected vertically vibrating modes marked by blue lines</figcaption></figure>

error as the result of its simultaneous location on top of the structure, close to the structural support. As shown in Fig. 9, the out-of-plane modes can be inaccurately identified due to the direction of their displacements. This confirms that the out-of-plane modes should not be involved in the damage assessment procedure, as described in Subsection 2.4.

### 3.3 Identification of the modal data

Modal data are identified with SSI-DATA and the stabilization diagrams as described in Section 2.4 for each video. To this end, both vertical and horizontal displacements are used. Systems of an order up to $n = 76$ are identified to create the stabilization diagrams. Stable solutions are obtained for the following values of coefficients: $\alpha_f = 0.01$, $\alpha_\phi = 0.02$, $\alpha_\zeta = 0.05$, $\beta = 0.7$, $N_s = 15$, and $\gamma_{MAC} = 0.92$. The vertically vibrating mode shapes are separated with $\epsilon_{YZ} = 1.3$. An example of the stabilization diagram for a healthy structure with marked frequencies of the modes selected for damage assessment is shown in Fig. 10, whereas the corresponding identified mode shapes, accompanied by their numerical counterparts, are shown in Fig. 11. Mode 6 is shown also in Fig. 9 as one affecting accuracy of the displacement estimation. Despite its out-of-plane components of motion, this mode fulfils the criterion described in Eq. (8). Error metrics between the numerical modal data of the nominal FE model and the identified modal parameters of the undamaged structure are listed in Table 1.

It can be observed that the second, fourth and sixth selected identified mode shapes differ from their numerical counterparts. These mode shapes are characterized by a relatively significant error in the longitudinal direction between the numerical and the identified mode shapes. This affects the value of the scaling factor (see Eq. (16)) and thus also the errors in the vertical degrees of freedom after mode matching.

The natural frequencies of the identified and numerical mode shapes, as well as the corresponding MAC values (Table 1) show that the nominal FE model well represents the nondamaged structure. All errors

<table><caption>Table 1 Metrics of errors between the numerical modal data of the nominal FE model and the identified modal data of the undamaged structure</caption><thead><tr><th>Identified mode <i>m</i></th><th><i>f̂</i><sup>(<i>m</i>)</sup> (Hz)</th><th><i>f</i><sub>num</sub><sup>(<i>m</i>)</sup> (Hz)</th><th><sup><i>f</i><sup>(<i>m</i>)</sup> - <i>f</i><sub>num</sub><sup>(<i>m</i>)</sup></sup>/<i>f</i><sup>(<i>m</i>)</sup> × 100%</th><th>MAC(<i>φ̂</i><sup>(<i>m</i>)</sup>, <i>D</i><sub>num</sub><sup>(<i>k</i>)</sup>)</th><th>MAC(<i>φ̂</i><sub>z</sub><sup>(<i>m</i>)</sup>, <i>D</i><sub>z</sub><sub>num</sub><sup>(<i>k</i>)</sup>)</th></tr></thead><tbody><tr><td>1</td><td>7.9221</td><td>7.9652</td><td>-0.5442</td><td>0.9994</td><td>0.9995</td></tr><tr><td>2</td><td>26.1900</td><td>26.3009</td><td>-0.4234</td><td>0.8006</td><td>0.9582</td></tr><tr><td>3</td><td>26.6574</td><td>26.7524</td><td>-0.3564</td><td>0.9957</td><td>0.9972</td></tr><tr><td>4</td><td>37.3140</td><td>37.4633</td><td>-0.4000</td><td>0.9108</td><td>0.9665</td></tr><tr><td>5</td><td>37.9672</td><td>38.1754</td><td>-0.5484</td><td>0.9653</td><td>0.9393</td></tr><tr><td>6</td><td>41.6260</td><td>41.8164</td><td>-0.4575</td><td>0.7983</td><td>0.9545</td></tr><tr><td>7</td><td>52.0019</td><td>52.2018</td><td>-0.3844</td><td>0.9933</td><td>0.9961</td></tr></tbody></table>
<figure><img src="image_13.png" /><figcaption>Fig. 11 Identified mode shapes $\hat{\phi}^{(m)}$ (green dots) compared with the numerical mode shapes $\phi_{num}^{(m)}(\theta)$ obtained from the FE model (deformed structure) for the healthy structure, accompanied with identified natural frequencies</figcaption></figure>

for the natural frequencies are below one percent and the worst MAC value is about 0.8, whereas the MAC, calculated only for the vertical displacements of the structure, is always higher than 0.9. Thus, the nominal FE model does not require any initial calibration before carrying out the damage assessment procedure.

Due to the considerations listed above, only vertical degrees of freedom are used for damage assessment, whereas both horizontal and vertical degrees of freedom are used for the stabilization diagram and for the matching of vibration modes.

The next step is the establishment of a criterion for detection of the measured data contaminated by gross errors. According to the criteria, only the elements of the vector $y(\theta)$ that are related to mode shapes are checked:

$$ (\hat{\phi}_{zi}^{(m)} - d_{zi} c_m \phi_{num}^{(m)}(\theta))^2 < \alpha_y, \quad i=1, 2, \dots, N_0, \\ m=1, 2, \dots, N_m \qquad (23) $$

where $\hat{\phi}_{zi}^{(m)}$ is ith element of the identified mode shape that contains only the vertical components and $d_{zi}$ is the ith row of matrix $D_z$. The identified mode shape is scaled in such a way that the maximal displacement is equal to one. The vector $\phi_{num}^{(m)}(\theta)$ is scaled by the modal scale factor $c_m$ as in Eq. (16) but with matrix $D_z$ instead of $D$. Depending on the method of damage assessment, the elements of $y(\theta)$ that do not satisfy condition (23) are replaced with zeros or removed. For all processed videos the value of $\alpha_y = 0.25$ is selected. Detection of the elements of vector $y(\theta)$ contaminated by the gross error is visualized in Fig. 12. There are two elements of vector $y(\theta)$ classified as gross errors.

## 3.4 Damage assessment with AIE-based approaches

In all tested damage assessment methods, the scale factor $\kappa_\theta = 0.15$ has been selected. In practice it can be selected by observation of the convergence of $\theta$. It can be carried out with the aid of a computer simulation before the application of the framework in CVSHM for an actual structure.

### 3.4.1 Augmented inverse estimate

In the case of damage assessment with the aid of the original AIE, the coefficients for the algorithm must be selected. The need to select the coefficients is a disadvantage of this method, as such a selection can be difficult in practice. The following coefficients have been selected by trial-and-error, each time using ground truth data to check the correctness of the damage evaluation: $\sigma_{tol} = 10^{-3}$ and $\alpha_\theta = 0.3$. In practice the ground truth data are not available; hence, this test only allows for an evaluation of the AIE method in CVSHM.

Damage indices (see Eq. (4)) of the monitored elements calculated with the original AIE method for processed videos are shown in Fig. 13. Despite the available ground truth data and a significant effort
<figure><img src="image_14.png" /><figcaption>Fig. 12 Error factor between the identified and numerical modal data used for an indication of the measurement data with gross error</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 13 Result of damage assessment obtained with the original AIE method compared with ground truth data</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 14 Result of damage assessment obtained with the WAIE method compared with ground truth data</figcaption></figure>
devoted to the selection of all coefficients, the results are not satisfactory. The estimated level of the damage is near ground truth data only for the “Damage6” video. Due to a significant measurement noise level (Fig. 8), many false types of damage are detected despite the selection of a relatively large $α_θ$.

### 3.4.2 Weighted augmented inverse estimate

In the case of the WAIE method, the same number of thresholds are to be selected. The weighting matrix $Q$, which corresponds to the weighting matrix $W$, defined as in Eq. (21), was employed. The value of $w_θ = 0.01$ is

selected. Similarly, the following coefficients are selected with a trial-and-error approach by comparing the results with ground truth data: $σ_{tol} = 0.05$ and $α_θ = 0.05$. The value of $σ_{tol}$ is larger than the one selected for the AIE method due to the fact that $σ_{max}$ is much smaller in this method. This results from the introduction of the weighting matrix $Q$ (see Eq. (22)).

Analogously, results for the WAIE are shown in Fig. 14 for all processed videos. It is evident that the potential of the AIE increases when the conditioning of the equation set is improved. Due to the appropriately selected weighting matrix $Q$, a smaller threshold $α_θ$ is required to reject the noisy damage indices related to the

<figure><img src="image_17.png" /><figcaption>Fig. 15 Result of damage assessment obtained with the WNLSIE method for $α_θ = 0.4$ compared with ground truth data</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 16 Result of damage assessment obtained with the WNLSIE method for $α_θ = 0$ compared with ground truth data</figcaption></figure>
measurement noise and modeling errors. The method always detects the damaged element among the three highest damage indices as opposed to the original AIE. However, the results are still not satisfactory, even with all coefficients tuned with the aid of available ground truth data.

### 3.4.3 Weighted negative least square inverse estimate

Finally, the WNLSIE method is tested. Matrix *Q* is selected analogously as in the WAIE method. The WNLSIE requires only $\alpha_{\theta}$ to be selected. In the first case $\alpha_{\theta} = 0.4$ is selected analogously to the methods discussed above to demonstrate the performance of the WNLSIE in comparison with the two remaining tested methods. In the second case $\alpha_{\theta} = 0$ is selected to demonstrate the robustness of the method and to show that the method can work without selecting additional parameters.

The results for $\alpha_{\theta} = 0.4$ and $\alpha_{\theta} = 0$ are shown in Figs. 15 and 16, respectively. In the former case the WNLSIE method significantly outperforms the AIE and WAIE methods. It detects and locates the damage correctly for all damage cases. The damage indices are also sparser, i.e., fewer cases of false damage are detected. Moreover, the damage level is relatively close to the ground truth data for each video. In the latter case, the results are not so sparse, but the method still produces satisfactory results without selecting any additional parameter, except that $\kappa_{\theta}$ can be selected with the aid of a computer simulation.

For all methods (AIE, WAIE and WNLSIE) the damage assessment procedure tends to falsely indicate the oblique members (Nos. 23-29) as damaged. The reason for this phenomenon is not precisely known. This can be due to the fact that the oblique members transmit loads resulting from torsional modes that are rejected from the measurement data prior to the damage assessment procedure being done. The effect of limited information about oblique members can be magnified by ill-conditioning of the problem in addition to a high level of measurement noise. This phenomenon is worth being investigated in future research.

## 4 Conclusions

A complete framework for the CVSHM of truss structures subjected to unknown excitations has been proposed in the present paper. It has been shown that the presented methods for the CV measurement of structural displacement and damage assessment can monitor the involved truss structure after a one-time preparation of the data, such as the manual selection of the tracked truss nodes or the selection of some thresholds for the algorithm. The CV measurement of displacements of natural targets was performed without any available high-contrast artificial targets. The proposed framework was tested using a PBGM-based video with full-HD

resolution and the frame rate of 120 fps. However, if necessary, a higher resolution and a higher frame rate can be employed in the framework.

It was shown that the CV measurement data are highly contaminated by the measurement noise—up to 50%. It was demonstrated that such errors are caused by the participation of out-of-plane and higher order modes in structural motion that cause changes in scale, additional rotation, and false in-plane motion of the tracked structural nodes in the video. Modal data have been extracted from these noisy data with the aid of SSI-DATA and automated stabilization diagrams. Only in-plane vibration modes were selected for the damage assessment procedure. It has been shown that some selected mode shapes also can be disturbed as a result of measurement noise. Only components of the mode shape that are related to higher amplitudes of motion (in this case vertical displacements) were involved in the damage assessment procedure, according to higher accuracy in estimation.

Three methods of damage assessment dedicated to modal data containing gross errors were tested and compared: (1) AIE, proposed by Peng et al.; (2) WAIE, proposed in the present paper; and (3) WNLSIE, also proposed in this paper. The two last methods are modified versions of the first analyzed method. It has been shown that the AIE does not provide satisfactory results. The WAIE can indicate the damaged elements among several elements detected as false damage. This is done by employing a weighting matrix that improves the numerical conditioning of the problem. Both AIE and WAIE require certain algorithm coefficients to be selected. In this paper, this was carried out using ground truth data about the damage available from PBGMs. However, in practice such a data is not available. Further, an appropriate selection of these coefficients can be difficult. The WNLSIE not only offers the most satisfactory results and provides the least number of detected false damage but it is also robust with respect to the selected algorithm parameters. Such properties of the WNLSIE result from the employment of the weighting matrix (as in the WAIE) and from an additional constraint imposed upon the optimization problem, which ensured that the stiffnesses of the potentially damaged elements can be only reduced. It is thus recommended to use the proposed WNLSIE method within the discussed framework.

As the damage assessment is based on the stiffness reduction of FEs of the FE model that has been updated using the measurement data obtained from the monitored structure, the proposed framework requires a sufficiently accurate FE model. The engineering design data of the monitored structure can be used as a reference, but the FE model must be initially verified or calibrated using the measurement data taken from the healthy structure. In the present research, the nominal model was sufficiently accurate without an initial calibration. The influence of the inaccuracies in the nominal model on the results
of the damage assessment procedure is an interesting
direction for future research.

All tested methods of damage assessment tend to
falsely indicate the oblique members of the structure
as damaged. Additional research is required to confirm
whether the reason for this false indication is the rejection
of the torsional vibration modes from the measurement
data, as the oblique members participate in the strains
resulting from these modes.

Acknowledgement

The authors gratefully acknowledge the support of
the National Science Centre, Poland, granted through
the Project 2020/39/B/ST8/02615.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

References

Adhikari G, Sahani SK, Chauhan MS and Das BK (2016), "Fast Real Time Object Tracking Based on Normalized Cross Correlation and Importance of Thresholding Segmentation," *2016 International Conference on Recent Trends in Information Technology (ICRTIT)*, 1-5. https://doi.org/10.1109/ICRTIT.2016.7569517

Bernal D (2002), "Load Vectors for Damage Localization," *Journal of Engineering Mechanics*, **128**(1): 7-14. https://doi.org/10.1061/(ASCE)0733-9399(2002)128:1(7)

Bernal D (2006), "Flexibility-Based Damage Localization from Stochastic Realization Results," *Journal of Engineering Mechanics*, **132**(6): 651-658. https://doi.org/10.1061/(ASCE)0733-9399(2006)132:6(651)

Blachowski B (2019), “Modal Sensitivity Based Sensor Placement for Damage Identification Under Sparsity Constraint,” *Periodica Polytechnica Civil Engineering*, **63**(2): 432–445. https://doi.org/10.3311/PPci.13888

Blachowski B, An YH, Spencer BF and Ou JP (2017), “Axial Strain Accelerations Approach for Damage Localization in Statically Determinate Truss Structures,” *Computer-Aided Civil and Infrastructure Engineering*, **32**(4): 304–318. https://doi.org/10.1111/mice.12258

Blachowski B, Ostrowski M, Zarski M, Wojcik B, Tauzowski P and Jankowski L (2023), “An Efficient Computer Vision-Based Method for Estimation of Dynamic Displacements in Spatial Truss Structures,”

in P. Rizzo and A. Milazzo (eds) *European Workshop on Structural Health Monitoring*, Cham: Springer International Publishing, 474–484. https://doi.org/10.1007/978-3-031-07258-1_49

Blachowski B, Swiercz A, Gutkiewicz P, Szelążek J and Gutkowski W (2016), “Structural Damage Detectability Using Modal and Ultrasonic Approaches,” *Measurement*, **85**: 10–221. https://doi.org/10.1016/j.measurement.2016.02.033

Bose B, Wang X and Grimson E (2007), “Multi-class Object Tracking Algorithm That Handles Fragmentation and Grouping,” *2007 IEEE Conference on Computer Vision and Pattern Recognition*, pp. 1–8. https://doi.org/10.1109/CVPR.2007.383175

Breitenstein MD, Reichlin F, Leibe B, et al. (2009), “Robust Tracking-by-Detection Using a Detector Confidence Particle Filter,” *2009 IEEE 12th International Conference on Computer Vision*, 1515–1522. https://doi.org/10.1109/ICCV.2009.5459278

Curadelli RO, Riera JD, Ambrosini D and Amani MG (2008), “Damage Detection by Means of Structural Damping Identification,” *Engineering Structures*, **30**(12): 3497–3504. https://doi.org/10.1016/j.engstruct.2008.05.024

Feng D and Feng M (2021), *Computer Vision for Structural Dynamics and Health Monitoring*. John Wiley and Sons Ltd and ASME Press. https://doi.org/10.1002/9781119566557

Feng D and Feng MQ (2018), “Computer Vision for SHM of Civil Infrastructure: From Dynamic Response Measurement to Damage Detection – A review,” *Engineering Structures*, **156**: 105–117. https://doi.org/10.1016/j.engstruct.2017.11.018

Feng D, Feng MQ, Ozer E and Fukuda Y (2015), “A Vision-Based Sensor for Noncontact Structural Displacement Measurement,” *Sensors*, **15**(7): 16557–16575. https://doi.org/10.3390/s150716557

Fox RL and Kapoor MP (1968), “Rates of Change of Eigenvalues and Eigenvectors,” *AIAA Journal*, **6**(12): 2426–2429. https://doi.org/10.2514/3.5008

Friswell M and Mottershead JE (1995), *Finite Element Model Updating in Structural Dynamics*, Springer, Dordrecht. https://doi.org/10.1007/978-94-015-8508-8

Gomez F, Narazaki Y, Hoskere V, Spencer BF and Smith MD (2022), “Bayesian Inference of Dense Structural Response Using Vision-based Measurements,” *Engineering Structures*, **256**: 113970. https://doi.org/10.1016/j.engstruct.2022.113970

Harris C and Stephens M (1988), “A Combined Corner and Edge Detector,” *Proceedings of the Alvey Vision Conference*, Alvety Vision Club, 23.1–23.6. https://doi.org/10.5244/C.2.23

Henriques JF, Caseiro R and Batista J (2011), “Globally Optimal Solution to Multi-object Tracking with Merged Measurements,” *2011 International Conference on Computer Vision*, 2470–2477. https://doi.org/10.1109/ICCV.2011.6126532

Hii A, Hann CE, Chase JG and Houten E (2006), “Fast Normalized Cross Correlation for Motion Tracking
Using Basis Functions,” *Computer Methods and Programs in Biomedicine*, **82**(2): 144–156. https://doi.org/10.1016/J.CMPB.2006.02.007

Hoskere V, Narazaki Y and Spencer BF (2019), “Learning to Detect Important Visual Changes for Structural Inspections Using Physics-based Graphics Models,” *9th International Conference on Structural Health Monitoring of Intelligent Infrastructure: Transferring Research into Practice, SHMII 2019 - Conference Proceedings*, pp. 1484–1490. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85090588897&partnerID=40&md5=ec168a32fe27ea515d23671178a4f3dc

Hu W, Li X, Luo W, Zhang X, Maybank S and Zhang Z (2012), “Single and Multiple Object Tracking Using Log-euclidean Riemannian Subspace and Block-division Appearance Model,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, **34**(12): pp. 2420–2440. https://doi.org/10.1109/TPAMI.2012.42

Huang C, Wu B and Nevatia R (2008), “Robust Object Tracking by Hierarchical Association of Detection Responses,” in D. Forsyth, P. Torr, and A. Zisserman (eds) *Computer Vision -- ECCV 2008*, Berlin, Heidelberg: Springer, Berlin, Heidelberg, pp. 788–801. https://doi.org/10.1007/978-3-540-88688-4_58

Iezzi F, Spina D and Valente C (2015), “Damage Assessment Through Changes in Mode Shapes Due to Non-proportional Damping,” *Journal of Physics: Conference Series*, **628**: 12019. https://doi.org/10.1088/1742-6596/628/1/012019

Li Shuai, Pan Jianwen, Luo Guangheng and Wang Jinting (2020), “Automatic Modal Parameter Identification of High Arch Dams: Feasibility Verification,” *Earthquake Engineering and Engineering Vibration*, **19**(4): 953–965. https://doi.org/10.1007/s11803-020-0606-6

Liu S, Liu D, Srivastava G, et al. (2021), “Overview and Methods of Correlation Filter Algorithms in Object Tracking,” *Complex & Intelligent Systems*, **7**(4): 1895–1917. https://doi.org/10.1007/s40747-020-00161-4

Luo W, Xing J, Milan A, Zhang X, Liu W and Kim TK (2021), “Multiple Object Tracking: A Literature review,” *Artificial Intelligence*, **293**: 103448. https://doi.org/10.1016/j.artint.2020.103448

Mottershead JE, Link M and Friswell MI (2011), “The Sensitivity Method in Finite Element Model Updating: A Tutorial,” *Mechanical Systems and Signal Processing*, **25**(7): 2275–2296. https://doi.org/10.1016/j.ymssp.2010.10.012

Narazaki Y, Gomez F, et al. (2021), “Efficient Development of Vision-based Dense Three-dimensional Displacement Measurement Algorithms Using Physics-based Graphics Models,” *Structural Health Monitoring*, **20**(4): 1841–1863. https://doi.org/10.1177/1475921720939522

Narazaki Y, Hoskere V, et al. (2021), “Synthetic Environments for Vision-based Structural Condition Assessment of Japanese High-speed Railway Viaducts,” *Mechanical Systems and Signal Processing*, **160**: 107850. https://doi.org/10.1016/j.ymssp.2021.107850

Nelson RB (1976), “Simplified Calculation of Eigenvector Derivatives,” *AIAA Journal*, **14**(9): 1201–1205. https://doi.org/10.2514/3.7211

Peeters B (2000), “System Identification and Damage Detection in Civil Engineering,” https://limo.libis.be/primo-explore/fulldisplay?docid=LIRIAS1725571&context=L&vid=Lirias&sea rch_scope=Lirias&tab=default_tab&lang=en_US&fromSitemap=1

Peng X, Qi FJ, Yang QW and Chen H (2021), “A Robust Estimate Method for Damage Detection of Concrete Structures Using Contaminated Data,” *Advances in Civil Engineering*, Edited by A. Palmeri, 2021, 6669958. https://doi.org/10.1155/2021/6669958

Shi J and Tomasi (1994), “Good Features to Track,” *1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition*, 593–600. https://doi.org/10.1109/CVPR.1994.323794

Spencer BF, Hoskere V and Narazaki Y (2019), “Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring,” *Engineering*, **5**(2): 199–222. https://doi.org/10.1016/j.eng.2018.11.030

Suwała G and Jankowski Ł (2012), “A Model-free Method for Identification of Mass Modifications,” *Structural Control and Health Monitoring*, **19**(2): 216–230. https://doi.org/10.1002/stc.417

Tomasi C and Kanade T (1991), *Detection and Tracking of Point Features*. https://cecas.clemson.edu/~stb/klt/tomasi-kanade-techreport-1991.pdf (Accessed: 12 January 2022)

Trainotti F, Slimak T and Rixen D (2022), “Towards Full-field Frequency Based Substructuring: An Application Case,” *Proceedings of International Conference on Noise and Vibration Engineering and International Conference on Uncertainty in Structural Dynamics (ISMA 2022 – USD 2022)*, Lauven, Belgium

Xiang Y, Alahi A and Savarese S (2015), “Learning to Track: Online Multi-object Tracking by Decision Making,” *2015 IEEE International Conference on Computer Vision (ICCV)*, 4705–4713. https://doi.org/10.1109/ICCV.2015.534

Yuen KV (2010), *Bayesian Methods for Structural Dynamics and Civil Engineering*, John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470824566

Yuen KV, Beck JL and Katafygiotis LS (2006), “Efficient Model Updating and Health Monitoring Methodology Using Incomplete Modal Data without Mode Matching,” *Structural Control and Health Monitoring*, **13**(1): 91–107. https://doi.org/10.1002/stc.144

Żarski M, Wójcik B, Miszczak JA, Blachowski B and Ostrowski M (2022), “Computer Vision Based Inspection on Post-Earthquake with UAV Synthetic Dataset,” *IEEE Access*, 10, pp. 108134–108144. https://doi.org/10.1109/ACCESS.2022.3212918

Zhang L and van der Maaten L (2014), “Preserving Structure in Model-Free Tracking,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, **36**(4): 756–769. https://doi.org/10.1109/TPAMI.2013.221