# Measurement of Bridge Vibration by UAVs Combined with CNN and KLT Optical-Flow Method
###### Abstract  
A measurement method of bridge vibration by unmanned aerial vehicles (UAVs) combined with convolutional neural networks (CNNs) and Kanade-Lucas-Tomasi (KLT) optical-flow method is proposed. In this method, the stationary reference points in the structural background are required, a UAV is used to shoot the structure video, and the KLT optical-flow method is used to track the target points on the structure and the background reference points in the video to obtain the coordinates of these points on each frame. Then, the characteristic relationship between the reference points and the target points can be learned by a CNN according to the coordinates of the reference points and the target points, so as to correct the displacement time-history curves of target points containing the false displacement caused by the UAV's egomotion. Finally, operational modal analysis (OMA) is used to extract the natural frequency of the structure from the displacement signal. In addition, the reliability of UAV measurement combined with CNN is proved by comparing the measurement results of the fixed camera and those of UAV combined with CNN, and the reliability of the KLT optical-flow method is proved by comparing the tracking results of the digital image correlation (DIC) and KLT optical-flow method in the experiment of this paper. bridge vibration; unmanned aerial vehicles; Kanade-Lucas-Tomasi optical-flow method; operational modal analysis; convolutional neural networks +
Footnote †: journal: Applied  
0000-0002-0001-9999-22000]Y. Yan  
0000-0002-0002-0002-0002-0002-0002-0002-0002-0002-0002-0002-0002-002-0002-0002-0002-0002-0002-0002-0002-0002-0002-0002-0002  
feature points of two adjacent frames to obtain the motion information of the feature points. The KLT optical-flow method is widely used in tracking the feature points of different scenes (buildings, grasslands, etc.) to validate its reliability [16] in measuring vibration of bridge models [12]. In order to adapt to various measurement environments (such as cross-river bridges, etc. ), in recent years, UAVs have been used to replace fixed cameras for bridge measurement [17; 18]. UAVs are also gradually applied to structural crack detection, displacement measurement, and damage inspection of bridges [19; 20; 21]. However, due to egomotion of UAVs during their flights, the measured displacement includes not only the displacement of the measured structure, but also the false displacement caused by UAV egomotion. The common method to eliminate the false displacement is homography transformation [22; 23], but this requires that four or more static reference points are in the same plane of the target points, which is difficult to achieve in actual measurement. The random components in the signals collected by UAVs may be suppressed by a differential filtering method [24] to obtain the structural modal parameters [25]. Although the bridge modal parameters can be extracted by this method without using any reference point, it does not obtain the real displacement time-history curves of the target points. The correction method of three-dimensional reconstruction is proposed to obtain the intrinsic and extrinsic parameters of the UAV cameras by using [PERSON]'s method [26], so as to recover the 3D world coordinates of the structure to obtain its true displacement [27]. However, this method requires that the plane of the reference points is parallel to that of the target points and the distance between the two planes is known, which is difficult to achieve in practical measurement. The correction of UAV images by neural networks has attracted much attention in recent years. The method of correcting UAV images with a radial basis function (RBF) neural network is proposed [28]. The selected control points are used by this method as the training samples of the network. The corrected images can be obtained after the UAV images are inputted into the network.