# Wireless Binocular Stereovision Measurement System Based on Improved Coarse-to-Fine Matching Algorithm

Authors: Cheng Wang; Teng Xiao; Zhiqun Gong; Shiting Yang; Dongliang Zhang; Fei Deng
Journal: Structural Control and Health Monitoring
 (2023)

---

## Abstract

To cope with the challenges some commonly used displacement sensors may face in structural health monitoring, we proposed a wireless binocular stereovision measurement system based on the improved coarse-to-fine matching algorithm. The vision measurement system can perceive multipoint three-dimensional displacement and consists of two cameras, one remote control unit, and a solar panel. The improved coarse-to-fine matching algorithm only requires a flat surface with some painted textures instead of designed markers with refined patterns (circle, square, and chessboard). Omitting the textures keeps a balance between measurement accuracy and the burden of the target installation in the actual field of structural health monitoring. The proposed coarse-to-fine matching algorithm utilizes scale-invariant feature transform (SIFT) points to generate a robust initial guess and combines improved least squares matching (LSM) algorithm for advanced corresponding point matching. A precalculated look-up table for interpolation and parallel computation is used for higher matching precision and computational efficiency. Several experiments were conducted to present the accuracy and feasibility of the proposed metering system. A simulation test, an indoor shaking table test with several commonly used displacement sensors, and an outdoor experiment consider the natural condition. Experimental results verified that the proposed matching methodology could achieve 0.2-pixel matching accuracy, 0.1 mm measuring precision in a laboratory, and millimeter-level precision in real conditions.


---

## Body

## 1. Introduction

Structural displacement is a vital evaluation index in structural health monitoring (SHM) [1]. As a principal evaluation standard of safe structural condition, deformation is directly related to the stiffness of the structure [2, 3], which can then intuitively represent the soundness of the building structure [4]. Therefore, how to efficiently estimate or measure the displacement response is widely studied in the field of SHM. For example, the acceleration-based method is commonly applied because of its advantages, including high precision and sensitivity [5]. The displacement can be obtained through noise reduction and integration process [6]. However, errors may be introduced and accumulated during the integral procedure, and the accelerometer itself is insensitive to low-frequency and uniform motion [7]. Besides, displacement can be directly measured using displacement transducers. These transducers mainly include linear variable differential transformers (LVDT, a contact-type sensor) and laser displacement meters (a non-contact-type sensor). All the displacement meters require stably fixed installation points close or attached to the structure. This prerequisite sometimes limits their applications in real conditions. Moreover, only measuring one-way displacement may not perfectly satisfy the needs of SHM [8]. Furthermore, in the field of SHM, continuous measurement is essential, and sometimes, forecasting the range of the structural deformation is difficult. Without the measuring range, it is sometimes challenging to choose the proper displacement sensors in the engineering field [9, 10]. Apart from the measurement approaches mentioned previously, global navigation satellite system (GNSS) can be a competent monitoring system. However, it faces the barriers of high cost [9, 11, 12] and limited accuracy [13]. To solve the challenges these commonly used gauging methodologies may face, we developed a wireless binocular stereovision measurement system with the coarse-to-fine matching algorithm. The proposed system can measure and remotely monitor multipoint three-dimensional displacement and feedback on the measured data from the actual site of the SHM.With the rapid development of digital imaging technology and computer vision, binocular vision measurement technology, as a vision-based measuring technology, has become one of the most critical technologies because of its high precision and stability [14]. This vision technology can sense spatial displacement and has the potential to solve the limitations of the aforementioned commonly used displacement sensors [15]. Visual metrology usually comprises several artificial targets and two cameras. The predesigned targets need to be attached to the interesting points, and the two cameras are required to be at proper sites where the targets can be captured by both cameras [16]. Many studies have been conducted on introducing these vision-based sensors to the measurement of structural displacements in the field of SHM [17]. Dong et al. [18] proposed a method for multipoint synchronous measurement of structural dynamic displacement and identified the structural model parameters by performing a fast Fourier transform on the acquired multipoint dynamic displacements. Lydon et al. [19] developed a synchronized multicamera contactless vision-based multipoint displacement measurement system. By pairing the wireless action cameras, the system can track multiple points on a lone-span bridge structure and generate multipoint displacement readings. Zhou et al. [20] investigated the effect of temperature variables on the accuracy of videogrammetric technology to examine its feasibility. Results show that temperature variations caused nonnegligible errors in the measured displacements and the errors contained not only daily fluctuation patterns but also the overall trend. Dong and Catbas [21] proposed a vision-based structural identification framework for bridges combining vision-based displacement measurement and vehicle load localization. Unit influence line (UIL) was extracted as an indicator of structural identification, and the proposed methods were validated on a footbridge at the University of Central Florida (UCF) campus. To reduce the computation time of target tracking, Wu et al. [22] proposed an improved region-matching algorithm based on prior knowledge of structural deformation and a vision-based multipoint structural displacement measurement system with surveillance and consumer cameras. Dong et al. [23] developed a vision-based multipoint dynamic monitoring framework using virtual markers selected from the specific scenario as the tracking points and optical flow as the tracking algorithm. However, all the above visual methods require specific targets on the structure as the features or tracking points. The targets are artificial markers with designed patterns [24] (circle and square) or natural keypoints [25] selected from the structure surface. However, the methods that adopt the artificial targets are more accurate than the methods based on natural features [26]. In the field of SHM, it is sometimes difficult to produce or install artificial targets with refined-designed patterns in the engineering sites, such as in remote mountain areas, on high-rise structures. Therefore, in this paper, we proposed a window-basedimage-matching algorithm. It does not require any designed patterns on the targets, such as circles [27] and squares [28]. The proposed matching algorithm can accurately match the corresponding points and generate precise displacement if the target has a flat surface with enough textures. The simplified requirements of targets can balance installation burden and measuring accuracy. A laboratory test was conducted to analyze and verify the accuracy of the proposed visual method.Binocular vision can directly measure an object’s area and distance, similar to how humans view a scene with their eyes. To be more specific, two cameras simultaneously acquire images of the same scene from different directions. Then, the 3D geometric information of the object is recovered through the difference between the left and right pictures, while the 3D position of the object is reconstructed through epipolar constraint [29]. Therefore, the accuracy of matching the same object in two different images directly determines the accuracy of the 3D coordinate reconstruction and further dominates the precision of the displacement measurement. The basic concept of image matching is to seek the corresponding points (in the second image) of the “seed” point from one reference image, referred to as the “slave” point [30]. Two main categories of image-matching methods are commonly used, area-based and feature-based methods, respectively [31–33]. The feature-based methods match the designed features detected separately from the image pairs according to the spatial structure or distance of invariant descriptor vectors [34]. The scale-invariant feature transform (SIFT) feature and its variants are the most commonly used local invariant features applied in the feature-based methods [35]. By contrast, area-based methods depend on the similarity directly calculated from the pixel intensity of the corresponding window pairs or even the entire images. Thus, they often perform better than the feature-based methods in the aspect of precision, distribution, and robustness [36]. However, the feature-based matching methods are usually more competitive than the area-based method (ABM) in looking for correspondences of image pairs with a significant geometric and radiometric difference, though featuring a lower precision in general between 0.1- and 1-pixel sizes [37]. The most prominent area-based methods are cross-correlation (widely used in the field of digital image correlation [38]) and the least squares matching (LSM). They can provide subpixel matching accuracy, 1/10 pixel, and even better in extreme cases. Several control factors, such as illumination and geometrical image shaping, may suffer great distortion in the monitoring field. The distortion can be corrected and optimized by the least squares iterations, while the cross-correlation method may not perform well in this type of situation [39]. However, a reasonable approximate initial value is the prerequisite of area-based matching methods because LSM (least square matching) is a highly nonlinear process [40]. Therefore, in this paper, considering the illumination distortion and the complex situation in the real-site condition, we proposed a coarse-to-fine matching algorithm combining SIFT feature as an initial guess and an improved LSM algorithm to generate precise registration results. A simulation test was conducted to verify the effectiveness and precision of the proposed matching algorithm.In conclusion, to cope with the challenges, some commonly used displacement sensors may face. In this paper, we developed a wireless binocular stereovision measurement system based on an improved coarse-to-fine image matching algorithm. The system consists of two cameras, a remote-control unit and a solar panel for power supply. We proposed an improved coarse-to-fine matching algorithm to keep a balance between target installation and measurement accuracy. The proposed registration method only requires a flat structural surface with enough texture (which can be easily made using paint spray). Pregenerated look-up tables for interpolation and parallel computation are used to speed up the algorithm and improve the accuracy. Several experiments were conducted to analyze the proposed measuring system fully. A simulation test was conducted to verify the precision of the proposed matching algorithm; an indoor small shaking table test was conducted to compare the proposed binocular measurement system with other commonly used sensors, and a real experiment in the outdoor environment was performed to test the feasibility and accuracy of the whole system.


## 2. Methodology

In this paper, a wireless binocular vision measurement system was developed, which consists of two cameras, a remote control unit, and a solar panel. As shown in Figure1, the proposed system measures structural displacement by using two cameras to simultaneously capture the images of the targets mounted on the structure from different directions. The corresponding points in the image pairs of the targets are then matched through the proposed coarse-to-fine matching algorithm (SIFT-based improved LSM algorithm). Furthermore, the corresponding feature points in the left and right pictures are then triangulated to reconstruct the 3D positions of the targets in the real-world coordinate system. The target displacement can be extracted by analyzing a series of image pairs. This section introduces the wireless image acquisition instruments and the proposed image-matching algorithm.Figure 1
Schematic description of the visual displacement measurement system.


### 2.1. Wireless Binocular Vision System

The wireless binocular vision system proposed is shown in Figure1, including two industrial cameras (Hikvision MVCE200-10GM/GC) and a remote controller developed on an NVIDIA Jetson Nano B1 development board with a 4G cellular data module embedded. The technical specifications of the industrial cameras are shown in Table 1. The pictures captured by the two cameras can be transferred through the wire connection or the 4G module, and then, the data will be processed on the cloud server. The whole system can work independently in remote areas when equipped with a solar panel for power supply. Therefore, it is easy to deploy the measurement system in places where structures need to be monitored.Table 1
Camera parameters.
Parameter nameValueMaximum resolution5472× 3648Sensor modelSony IMX183Pixel size2.4μm × 2.4 μmGray scale depth8 bitResolution5472 × 3648


### 2.2. Target Selection

As mentioned previously, visual measurement methods based on artificial targets are more accurate than natural targets. However, some artificial targets with designed and refined patterns may require more work to produce or install on the structure, such as bridges and slopes in a remote area. In this paper, by the proposed coarse-to-fine matching algorithm (area-based matching algorithm), the requisite target only requires a flat surface with some textures. It can be a predestined marker with a circle-shaped pattern (widely used) or a flat surface with paint spray; as long as there are changes in pixel values, enough SIFT feature points can be extracted, as shown in Figure2. Both patterns are then used as the targets in the following sections to verify this character. For simplicity, all the diagrams in this paper use the target made by paint spray as the illustration.Figure 2
The illustration of the requirements of the target.


### 2.3. Improved Coarse-to-Fine Matching Algorithm

As shown in Figure1, the displacement calculation of the targets is performed through the 3-D coordinates generated by triangulating the corresponding points found by the matching algorithm in the image pairs. Figure 3(a) shows a more specific diagram of the displacement calculation procedure. The image sequences of the interesting points P captured by the left and right camera are denoted as ImgLt−1, ImgRt−1, ImgLt, and ImgRt, respectively, and t denotes the time series. The center point P1 in the ImgLt−1 is set at the initial point, and the goal is to find its corresponding points in the right image (ImgRt−1) and the next image pairs (ImgLt and ImgRt). The proposed matching algorithm exploits the pixel information contained within the matching window centered around the initial guess in the image to calculate three homography matrices that describe the mapping relationships between the initial point and its three corresponding points in different images, namely P2,P1′, and P2′ in image pairs ImgLt−1-ImgRt−1, ImgLt−1-ImgLt, and ImgRt−1-ImgRt. The three homography matrices are denoted as Hab, Hac, and Hbd. In each time interval, the coordinates of the P1’s corresponding points P2,P1′ and P2′ can be computed using the three homography matrices Hab, Hac, and Hbd. Then, by triangulating the corresponding pairs, P1−P2 and P1′−P2′, the coordinates Cn−1 and Cn of the interesting point P can be calculated, and the displacement dn can be then obtained.Figure 3
The illustration of the displacement calculation using binocular vision: (a) illustration of the displacement calculation through matching algorithm and (b) illustration of the LSM algorithm.
(a)(b)The proposed coarse-to-fine matching algorithm is shown in Figure4 and arranged in Algorithm 1. Firstly, the feature points in the image pairs of the targets are extracted by scale-invariant feature transform (SIFT). However, due to the limitations of the SIFT algorithm, there are some mismatched point pairs, as shown in Figure 4(b). The extracted corresponding SIFT feature points are filtered and then used to calculate an initial affine translation matrix mapping from ImgL to ImgR as shown in Figure 3(b). The obtained affine matrix is set as the initial estimation of the shape and shift parameters of the improved LSM algorithm, relying on the existing relationship between the affine warp model of the SIFT and the homography matrix of the LSM [41]. The homography matrix shares the same function as the affine translation. The corresponding relationship can be expressed as follows:(1)x′y′1=a1a2a0b1b2b0001xy1,=Affinexy1,where x,y and x′,y′ are the initial and transformed coordinates of the corresponding points, respectively, and Affine is the transforming matrix calculated through SIFT feature points. The elements of the Affine matrix are in one-to-one correspondence with the homography matrix.Figure 4
The illustration of the proposed coarse-to-fine matching algorithm: (a) illustration of the proposed coarse-to-fine matching algorithm and (b) diagram of the proposed matching algorithm.
(a)(b)Based on the initial estimation, the LSM algorithm is improved to refine the initial matching guess to generate an accurate homography matrix. As shown in Figure3(b), the LSM matching algorithm constructs two matching windows centered around the SIFT feature points, termed template and reference, respectively. Two discrete two-dimensional functions fx,y and gx,y are constructed over the template and reference region, respectively. Moreover, the algorithm is to minimize the square differences of the grey values between the template and reference region. Considering the affine translation as the geometric correction, the new location x2,y2 transformed from the original position x1,y1 can be estimated as follows:(2)x2=a0+a1x1+a2y1,y2=b0+b1x1+b2y1,where a1,a2,b1,b2 are the model shape differences and a0,b0 are the shift parameters. Considering the linear radiometric difference, the relationship between the two patches in the reference image fx,y and template image gx,y can be expressed as follows:(3)fx,y=h0+h1ga0+a1x+a2y,b0+b1x+b2y+nx,y,where h0, h1 control the contrast and brightness difference, respectively, correcting the radiation distortion. After linearizing with a first-order Taylor series, the goal function to be minimized is the L2-norm of the residuals nx,y of the least square estimation, and the error equation is expressed as follows:(4)v=c1dh0+c2dh1+c3da0+c4da1+c5da2+c6db0+c7db1+c8db2−δg,where c1,c2,c3,c4,c5,c6,c7,c8 are the equation parameters, while X=dh0,dh1,da0,da1,da2,db0,db1,db2T is the incremental parameter vector. Thus, the residual function is expressed as follows:(5)V=CX−L,where C is a coefficient matrix recording the partial derivatives of each unknown parameter. L=h0+h1ga0+a1x+a2y,b0+b1x+b2y+nx,y−fx,y is the observation term. The parameter vector X can be solved using least square adjustment iterative by updating the parameters until reaching a certain convergence criterion. The model shape differences a1,a2,b1,b2 and the shift parameters a0,b0 can then updated using the parameter vector X. In this way, the real homography matrix can then be computed. Each matching window pair can generate one precisely matched point pair. After all the initial guesses are matched, these precisely matched point pairs are gathered to calculate a precise homography matrix mapping from ImgL to ImgR using the RANSAC algorithm [42].When mapping the points from the template region to the reference region using equation (2), the mapped points may not fall exactly at the grid pixel points of the reference region. Therefore, interpolation is needed to determine the actual pixel value of the mapped position. In order to balance between computation burden and interpolation precision, the B-spline interpolation algorithm based on a look-up table is adopted. As shown in Figure 5, a subpatch of the pixel intensity is constructed containing a 2 × 2-pixel grid pijpi+1jpij+1pi+1j+1 in the B-spline interpolation algorithm. The pixel value Px,y of the subpixel location on this patch can be expressed by a polynomial form in the following equation.(6)tx,y=∑m=03∑n=03amnxmyn,where x,y denotes the local coordinates of the subpixel location according to the upper-left corner of the 4 × 4-pixel grid as shown in Figure 5, and a=aij is a matrix containing 16 interpolation coefficients generated considering the pixel value pij of the 4 × 4 grid pixels centered on the subpixel location. The global look-up table (containing the coefficient matrix a of each grid) consists of M−1×N−1 elements on a M×N-pixel speckle image is precomputed using equation (7), and the interpolation coefficients are obtained by referring to this table in each iteration [43, 44]. Thus, a considerable redundant computation burden is avoided. Furthermore, the relationship between a and A can be expressed by the following equation:(7)A=BCQC′B′,(8)aij=A3−i3−j,(9)B=16−13−313−630−30301401,C=15671−195−1−1995−2555−2595−19−15−1971,Q=pi−1j−1pi−1jpi−1j+1pi−1j+2pij−1pijpij+1pij+2pi+1j−1pi+1jpi+1j+1pi+1j+2pi+2j−1pi+2jpi+2j+1pi+2j+2.Figure 5
Illustration of the pixel grid for the calculation of 16 interpolation coefficients.The ZNSSD (zero-normalized sum of square differences) criterion, instead of ZNCC (zero-normalized sum of squared differences), is assigned to quantify the similarity between the two patches in the reference and the target images because of its better ability to resist the effects of uneven lighting [45]. The ZNSSD can be expressed by the following equation:(10)Cf.gp⟶=∑x=−MM∑y=−MMfx,y−fm∑x=−MM∑y=−MMfx,y−fm21/2−gx′,y′−gm∑x=−MM∑y=−MMgx′,y′−gm21/2,2where fm and gm are the ensemble averages of the reference and target matching window, respectively, and p⟶i=1,…,6=a0,b0,a1,a2,b1,b2 denotes a vector concerning the six desired shape mapping parameters. The smaller the ZNSSD value is, the more similar the two patches are.The convergence conditions of the proposed algorithm take into consideration of all the elements of the incremental vectorX=dh0,dh1,da0,da1,da2,db0,db1,db2T and the ZNSSD criterion. The criterion of the norm ∆X can be expressed by the following equation:(11)∆X=da02+db02+max∆xda12+max∆yda22+max∆ydb12+max∆ydb22≤0.01,where max∆X=matchingWindowSize/2. The convergence condition guarantees precise shape and shifts mapping parameters. The proposed coarse-to-fine matching algorithm is developed using C++ language and OpenMP for parallel computing to reduce the computation time. After the corresponding point pairs of each image pair are obtained, the target’s position is reconstructed using the triangulation algorithm from OpenMVG [46–48]. Because the proposed coarse-to-fine matching algorithm uses the SIFT matching results as the initial guess, we name it the SIFT-based LSM algorithm and SLSM for short. The pseudocode of the proposed SLSM algorithm is shown in Algorithm 1.Algorithm 1: Calculate LeftTiePoints, RightTiePoint.
(i)
Require.∆X≤0.01 and ZNSSD_Value≤threshold(ii)
Ensure.x2=a0+a1x+a2y and y2=b0+b1x+b2y(1)
Initial_Guess[TiePoints,AffineMatrix]⇐SIFT_Matching(2)
for TiePoints in Initial_Guess do(3)
if ZNSSD_Value of MatchingWindow [TiePoints] ≥ threshold then(4)
delete Initial_Guess[TiePoints](5)
end if(6)
TiePoints = TiePoints with MAX ZNSSD_Value(7)
end for(8)
for LeftTiePoints in Initial_Guess[TiePoints] do(9)
x1,y1 = LeftTiePoints(10)
while∆X≤0.001 and ZNSSD_Value≤thresholddo(11)
forI1,J1 in LeftMatchingWindow x1,y1do(12)
I2,J2 = a0+a1I1+a2J1,b0+b1I1+b2J1(13)
gI2,J2 = h0+h1∗bicubic_Bspline_calculationI2,J2,look_up_table(14)
RightMatchingWindowI2,J2 = gI2,J2(15)
end for(16)
ZNSSD_Value = ZNSSD(LeftMatchingWindow, Right Matching Window)(17)
CalculateX=dh0,dh1,da0,da1,da2,db0,db1,db2T(18)
Updatingp⟶i=1,…,6=a0,b0,a1,a2,b1,b2,h0,h1(19)
end while(20)
x2,y2 = a0+a1x1+a2y1,b0+b1x1+b2y1(21)
RightTiePoints.appendx2,y2(22)
end for


## 2.1. Wireless Binocular Vision System

The wireless binocular vision system proposed is shown in Figure1, including two industrial cameras (Hikvision MVCE200-10GM/GC) and a remote controller developed on an NVIDIA Jetson Nano B1 development board with a 4G cellular data module embedded. The technical specifications of the industrial cameras are shown in Table 1. The pictures captured by the two cameras can be transferred through the wire connection or the 4G module, and then, the data will be processed on the cloud server. The whole system can work independently in remote areas when equipped with a solar panel for power supply. Therefore, it is easy to deploy the measurement system in places where structures need to be monitored.Table 1
Camera parameters.
Parameter nameValueMaximum resolution5472× 3648Sensor modelSony IMX183Pixel size2.4μm × 2.4 μmGray scale depth8 bitResolution5472 × 3648


## 2.2. Target Selection

As mentioned previously, visual measurement methods based on artificial targets are more accurate than natural targets. However, some artificial targets with designed and refined patterns may require more work to produce or install on the structure, such as bridges and slopes in a remote area. In this paper, by the proposed coarse-to-fine matching algorithm (area-based matching algorithm), the requisite target only requires a flat surface with some textures. It can be a predestined marker with a circle-shaped pattern (widely used) or a flat surface with paint spray; as long as there are changes in pixel values, enough SIFT feature points can be extracted, as shown in Figure2. Both patterns are then used as the targets in the following sections to verify this character. For simplicity, all the diagrams in this paper use the target made by paint spray as the illustration.Figure 2
The illustration of the requirements of the target.


## 2.3. Improved Coarse-to-Fine Matching Algorithm

As shown in Figure1, the displacement calculation of the targets is performed through the 3-D coordinates generated by triangulating the corresponding points found by the matching algorithm in the image pairs. Figure 3(a) shows a more specific diagram of the displacement calculation procedure. The image sequences of the interesting points P captured by the left and right camera are denoted as ImgLt−1, ImgRt−1, ImgLt, and ImgRt, respectively, and t denotes the time series. The center point P1 in the ImgLt−1 is set at the initial point, and the goal is to find its corresponding points in the right image (ImgRt−1) and the next image pairs (ImgLt and ImgRt). The proposed matching algorithm exploits the pixel information contained within the matching window centered around the initial guess in the image to calculate three homography matrices that describe the mapping relationships between the initial point and its three corresponding points in different images, namely P2,P1′, and P2′ in image pairs ImgLt−1-ImgRt−1, ImgLt−1-ImgLt, and ImgRt−1-ImgRt. The three homography matrices are denoted as Hab, Hac, and Hbd. In each time interval, the coordinates of the P1’s corresponding points P2,P1′ and P2′ can be computed using the three homography matrices Hab, Hac, and Hbd. Then, by triangulating the corresponding pairs, P1−P2 and P1′−P2′, the coordinates Cn−1 and Cn of the interesting point P can be calculated, and the displacement dn can be then obtained.Figure 3
The illustration of the displacement calculation using binocular vision: (a) illustration of the displacement calculation through matching algorithm and (b) illustration of the LSM algorithm.
(a)(b)The proposed coarse-to-fine matching algorithm is shown in Figure4 and arranged in Algorithm 1. Firstly, the feature points in the image pairs of the targets are extracted by scale-invariant feature transform (SIFT). However, due to the limitations of the SIFT algorithm, there are some mismatched point pairs, as shown in Figure 4(b). The extracted corresponding SIFT feature points are filtered and then used to calculate an initial affine translation matrix mapping from ImgL to ImgR as shown in Figure 3(b). The obtained affine matrix is set as the initial estimation of the shape and shift parameters of the improved LSM algorithm, relying on the existing relationship between the affine warp model of the SIFT and the homography matrix of the LSM [41]. The homography matrix shares the same function as the affine translation. The corresponding relationship can be expressed as follows:(1)x′y′1=a1a2a0b1b2b0001xy1,=Affinexy1,where x,y and x′,y′ are the initial and transformed coordinates of the corresponding points, respectively, and Affine is the transforming matrix calculated through SIFT feature points. The elements of the Affine matrix are in one-to-one correspondence with the homography matrix.Figure 4
The illustration of the proposed coarse-to-fine matching algorithm: (a) illustration of the proposed coarse-to-fine matching algorithm and (b) diagram of the proposed matching algorithm.
(a)(b)Based on the initial estimation, the LSM algorithm is improved to refine the initial matching guess to generate an accurate homography matrix. As shown in Figure3(b), the LSM matching algorithm constructs two matching windows centered around the SIFT feature points, termed template and reference, respectively. Two discrete two-dimensional functions fx,y and gx,y are constructed over the template and reference region, respectively. Moreover, the algorithm is to minimize the square differences of the grey values between the template and reference region. Considering the affine translation as the geometric correction, the new location x2,y2 transformed from the original position x1,y1 can be estimated as follows:(2)x2=a0+a1x1+a2y1,y2=b0+b1x1+b2y1,where a1,a2,b1,b2 are the model shape differences and a0,b0 are the shift parameters. Considering the linear radiometric difference, the relationship between the two patches in the reference image fx,y and template image gx,y can be expressed as follows:(3)fx,y=h0+h1ga0+a1x+a2y,b0+b1x+b2y+nx,y,where h0, h1 control the contrast and brightness difference, respectively, correcting the radiation distortion. After linearizing with a first-order Taylor series, the goal function to be minimized is the L2-norm of the residuals nx,y of the least square estimation, and the error equation is expressed as follows:(4)v=c1dh0+c2dh1+c3da0+c4da1+c5da2+c6db0+c7db1+c8db2−δg,where c1,c2,c3,c4,c5,c6,c7,c8 are the equation parameters, while X=dh0,dh1,da0,da1,da2,db0,db1,db2T is the incremental parameter vector. Thus, the residual function is expressed as follows:(5)V=CX−L,where C is a coefficient matrix recording the partial derivatives of each unknown parameter. L=h0+h1ga0+a1x+a2y,b0+b1x+b2y+nx,y−fx,y is the observation term. The parameter vector X can be solved using least square adjustment iterative by updating the parameters until reaching a certain convergence criterion. The model shape differences a1,a2,b1,b2 and the shift parameters a0,b0 can then updated using the parameter vector X. In this way, the real homography matrix can then be computed. Each matching window pair can generate one precisely matched point pair. After all the initial guesses are matched, these precisely matched point pairs are gathered to calculate a precise homography matrix mapping from ImgL to ImgR using the RANSAC algorithm [42].When mapping the points from the template region to the reference region using equation (2), the mapped points may not fall exactly at the grid pixel points of the reference region. Therefore, interpolation is needed to determine the actual pixel value of the mapped position. In order to balance between computation burden and interpolation precision, the B-spline interpolation algorithm based on a look-up table is adopted. As shown in Figure 5, a subpatch of the pixel intensity is constructed containing a 2 × 2-pixel grid pijpi+1jpij+1pi+1j+1 in the B-spline interpolation algorithm. The pixel value Px,y of the subpixel location on this patch can be expressed by a polynomial form in the following equation.(6)tx,y=∑m=03∑n=03amnxmyn,where x,y denotes the local coordinates of the subpixel location according to the upper-left corner of the 4 × 4-pixel grid as shown in Figure 5, and a=aij is a matrix containing 16 interpolation coefficients generated considering the pixel value pij of the 4 × 4 grid pixels centered on the subpixel location. The global look-up table (containing the coefficient matrix a of each grid) consists of M−1×N−1 elements on a M×N-pixel speckle image is precomputed using equation (7), and the interpolation coefficients are obtained by referring to this table in each iteration [43, 44]. Thus, a considerable redundant computation burden is avoided. Furthermore, the relationship between a and A can be expressed by the following equation:(7)A=BCQC′B′,(8)aij=A3−i3−j,(9)B=16−13−313−630−30301401,C=15671−195−1−1995−2555−2595−19−15−1971,Q=pi−1j−1pi−1jpi−1j+1pi−1j+2pij−1pijpij+1pij+2pi+1j−1pi+1jpi+1j+1pi+1j+2pi+2j−1pi+2jpi+2j+1pi+2j+2.Figure 5
Illustration of the pixel grid for the calculation of 16 interpolation coefficients.The ZNSSD (zero-normalized sum of square differences) criterion, instead of ZNCC (zero-normalized sum of squared differences), is assigned to quantify the similarity between the two patches in the reference and the target images because of its better ability to resist the effects of uneven lighting [45]. The ZNSSD can be expressed by the following equation:(10)Cf.gp⟶=∑x=−MM∑y=−MMfx,y−fm∑x=−MM∑y=−MMfx,y−fm21/2−gx′,y′−gm∑x=−MM∑y=−MMgx′,y′−gm21/2,2where fm and gm are the ensemble averages of the reference and target matching window, respectively, and p⟶i=1,…,6=a0,b0,a1,a2,b1,b2 denotes a vector concerning the six desired shape mapping parameters. The smaller the ZNSSD value is, the more similar the two patches are.The convergence conditions of the proposed algorithm take into consideration of all the elements of the incremental vectorX=dh0,dh1,da0,da1,da2,db0,db1,db2T and the ZNSSD criterion. The criterion of the norm ∆X can be expressed by the following equation:(11)∆X=da02+db02+max∆xda12+max∆yda22+max∆ydb12+max∆ydb22≤0.01,where max∆X=matchingWindowSize/2. The convergence condition guarantees precise shape and shifts mapping parameters. The proposed coarse-to-fine matching algorithm is developed using C++ language and OpenMP for parallel computing to reduce the computation time. After the corresponding point pairs of each image pair are obtained, the target’s position is reconstructed using the triangulation algorithm from OpenMVG [46–48]. Because the proposed coarse-to-fine matching algorithm uses the SIFT matching results as the initial guess, we name it the SIFT-based LSM algorithm and SLSM for short. The pseudocode of the proposed SLSM algorithm is shown in Algorithm 1.Algorithm 1: Calculate LeftTiePoints, RightTiePoint.
(i)
Require.∆X≤0.01 and ZNSSD_Value≤threshold(ii)
Ensure.x2=a0+a1x+a2y and y2=b0+b1x+b2y(1)
Initial_Guess[TiePoints,AffineMatrix]⇐SIFT_Matching(2)
for TiePoints in Initial_Guess do(3)
if ZNSSD_Value of MatchingWindow [TiePoints] ≥ threshold then(4)
delete Initial_Guess[TiePoints](5)
end if(6)
TiePoints = TiePoints with MAX ZNSSD_Value(7)
end for(8)
for LeftTiePoints in Initial_Guess[TiePoints] do(9)
x1,y1 = LeftTiePoints(10)
while∆X≤0.001 and ZNSSD_Value≤thresholddo(11)
forI1,J1 in LeftMatchingWindow x1,y1do(12)
I2,J2 = a0+a1I1+a2J1,b0+b1I1+b2J1(13)
gI2,J2 = h0+h1∗bicubic_Bspline_calculationI2,J2,look_up_table(14)
RightMatchingWindowI2,J2 = gI2,J2(15)
end for(16)
ZNSSD_Value = ZNSSD(LeftMatchingWindow, Right Matching Window)(17)
CalculateX=dh0,dh1,da0,da1,da2,db0,db1,db2T(18)
Updatingp⟶i=1,…,6=a0,b0,a1,a2,b1,b2,h0,h1(19)
end while(20)
x2,y2 = a0+a1x1+a2y1,b0+b1x1+b2y1(21)
RightTiePoints.appendx2,y2(22)
end for


## 3. Simulation Test

A simulation test was performed by programming in C++ to examine the accuracy and feasibility of the proposed SLSM method (SIFT-based improved LSM). As shown in Figure6, the test simulated the natural conditions of the image distortion and used a painted target as the template. At the same time, the reference region was mapped from the template with a known affine translation. The elements of the known affine translation were set as the actual value and compared with the translation matrix calculated by the proposed matching algorithm (SLSM). The simulated affine translations include three types of image transforming: from the left camera to the left camera, from the left camera to the right camera, and from the right camera to the right camera. SIFT matching algorithm was used as a comparison to calculate the homography matrix. Except for accuracy, several specific factors which may influence the performance of the proposed matching algorithm (SLSM) were also analyzed through controlled variable tests, namely the size of the matching window, the convergence conditions, the matrix inversion methods in solving the residual matrix equation (equation (5)), and the iteration number limitation. As a result, five simulation tests were conducted in total.Figure 6
Illustration of the simulation test. The artificial targets mapped using different homography matrixes.To quantify the matching errors, RMSE (root mean square error) is selected as the evaluation criterion of the matching results, expressed by the following equation:(12)RMSE=19∑i=19hi−h^i,where hi is the element of the homography matrix computed by the matching algorithm and h^i is the element of the true homography matrix. The test was run on a desktop computer with CPU Intel Core i9-10900KF and 32 GB Ram.


### 3.1. Simulation Test I

Simulation experiment I examines the overall performance of the proposed matching algorithm (SLSM) in the three transformations above (L−L′,L−R, and R−R′). The comparison of the homography matrices of the simulation is shown in Table 2. The RMSE erros are shown in bold for easy comparison. The last three elements of the homography matrix are omitted because they always tend to be the same (0.0, 0.0, and 1.0, respectively), as shown in equation (1). It can be seen that the proposed SIFT-based LSM algorithm (SLSM) can generate a precise homography matrix. The results display that the RMSE of the proposed matching algorithm is much lower than the SIFT algorithm (at least 80% lower). The errors of the L−L′ and the R−R′ translation are basically the same and retain extremely low (negligible). By contrast, the errors of both the matching methods of L−R are relatively larger than the other image mappings, and the proposed matching algorithm (SLSM) displays a stable RMSE (basically 0.18 pixels).Table 2
Results of the simulation test I.
Affine typeAlgorithmHomography matrixError/pixelL−L′TRUE0.9977−0.000283.7409−0.000281.00034−43.715RMSESLSM0.99764−0.000343.75543−0.000251.00033−43.72720.0078SIFT0.986299−0.003724.76816−0.003380.989484−42.38550.6859L−L′TRUE0.99950.000229−26.47750.0002290.9969−57.7127RMSESLSM0.9996680.000386−26.50470.0001220.997347−57.73690.0149SIFT0.989675−0.003−25.9192−0.0030.974333−55.18361.0574L−RTRUE1.0937−0.422212.66410.39740.8906−6.1527RMSESLSM1.09404−0.422342.319880.3976370.890746−15.89230.1762SIFT1.07447−0.411422.833370.3815120.889261−14.90020.5161L−RTRUE1.0937−0.42222.66410.39740.8906−16.1527RMSESLSM1.09381−0.4223912.34390.3975590.890414−5.867410.1751SIFT1.08469−0.4227612.73280.3826790.894351−5.355920.3266R−R′TRUE1.0008970.000108−14.00390.0001081.0016653.8145RMSESLSM1.000880.000123−14.00579.54E − 051.0016853.82180.0031SIFT1.007130.00109−14.45580.0031151.0051853.65890.1951R−R′TRUE0.9982−0.00019−7.1839−0.000190.999637.7021RMSESLSM0.998169−0.00021−7.17462−0.000150.99956737.70770.0044SIFT0.984811−0.00574−5.82985−0.00650.98697138.62860.6699The bold values are the main quantitative indicators of the algorithm's accuracy, and the illustration was added.


### 3.2. Simulation Test II: Matching Window Size

Because the SLSM method seeks the parameter vector in (5) to minimize the difference between the matching window pairs, a larger window size contains more pixel information. It leads to higher accuracy while potentially undermining the algorithm’s robustness and speed, causing divergence. In simulation experiment II, the matching window size is analyzed, and the L−L′ and L−R image pairs are set as the test data and denoted as LL and LR, respectively. The simulation results are shown in Figure 7(a).Figure 7
Results of the simulation test on different factors. (a) Simulation test II: varying matching window size. (b) Simulation test III: matrix inversion methods. (c) Simulation test IV: iteration numbers. (d) Simulation test V: illumination variance.
(a)(b)(c)(d)It can be seen that the computing duration increases with the window size, and the RMSE of the LL decreases with the increase of the window size while the LR remains basically the same. Therefore, the matching window size is not the bigger, the better. When the size of the window is set to 31 × 31 pixels, the RMSE and computing duration reach their optimized values, respectively. Considering the size of the target images for simulation is 250 × 300 pixel, the best matching window size is approximately 13% of the short side of the target region.


### 3.3. Simulation Test III: Matrix Inversion Method

To solve the residual (5) (calculating the incremental vector), the coefficient matrix C needs to be inverted. Three commonly used methods in C++ for matrix inversion are compared, namely Cholesky, LU (LU factorization), and eigenvalue decomposition. The matching window size is set to 31 × 31 pixels, and the simulation results are shown in Figure 7(b). It can be seen that the LU factorization achieves the most satisfying result in both computing speed and precision.


### 3.4. Simulation Test IV: Iteration Number

Not all the matching windows centered at the initial estimation points can generate satisfying matching tie points. The algorithm should stop after several iterations when the initial value is too bad to converge. Meanwhile, the algorithm needs an adequate number of iterations to find a satisfying result. Therefore, a fair number of iterations will guarantee robustness and precision. The iteration numbers are evaluated in simulation test IV. The window size is set to be 31 × 31 pixels, and LU is selected as the matrix inversion method. The results of the simulation experiment IV are shown in Figure7(c).It can be noticed that the endless increase of the iteration number does not lead to lower RMSE but longer duration. When the iteration number is set to 100, both the duration and RMSE are acceptable and balanced.


### 3.5. Simulation Test V: Illumination Variance Resistance

The illumination condition in the outside field is unstable, so the brightness of the images acquired in one interval may vary. Thus, the matching algorithm should be able to resist some illumination variance. In simulation experiment V, the robustness of the uneven illumination of the proposed matching algorithm is evaluated. The image’s brightness is artificially adjusted, and the brightness contrasts between image pairs to simulate the actual condition of the local environment. The test results are shown in Figure7(d).The diagram shows that the proposed matching algorithm has better robustness than the SIFT method toward illumination variation. Although the RMSE increases while the illumination difference enlarges, the increase is relatively small and within the acceptable range, nearly 0.04 pixels.


### 3.6. Discussion of the Simulation Test

The simulation test shows that the proposed SIFT-based improved matching algorithm (SLSM) can achieve accurate matching results within an acceptable time, and the RMSE is within 0.2 pixels. Also, the SLSM performs better than the SIFT in illumination resistance and accuracy. However, SIFT provides a relatively good initial guess for the LSM matching, contributing to robustness and high precision. The real homography matrix is the basis for the subsequent 3D coordinate reconstruction and displacement measurement.


## 3.1. Simulation Test I

Simulation experiment I examines the overall performance of the proposed matching algorithm (SLSM) in the three transformations above (L−L′,L−R, and R−R′). The comparison of the homography matrices of the simulation is shown in Table 2. The RMSE erros are shown in bold for easy comparison. The last three elements of the homography matrix are omitted because they always tend to be the same (0.0, 0.0, and 1.0, respectively), as shown in equation (1). It can be seen that the proposed SIFT-based LSM algorithm (SLSM) can generate a precise homography matrix. The results display that the RMSE of the proposed matching algorithm is much lower than the SIFT algorithm (at least 80% lower). The errors of the L−L′ and the R−R′ translation are basically the same and retain extremely low (negligible). By contrast, the errors of both the matching methods of L−R are relatively larger than the other image mappings, and the proposed matching algorithm (SLSM) displays a stable RMSE (basically 0.18 pixels).Table 2
Results of the simulation test I.
Affine typeAlgorithmHomography matrixError/pixelL−L′TRUE0.9977−0.000283.7409−0.000281.00034−43.715RMSESLSM0.99764−0.000343.75543−0.000251.00033−43.72720.0078SIFT0.986299−0.003724.76816−0.003380.989484−42.38550.6859L−L′TRUE0.99950.000229−26.47750.0002290.9969−57.7127RMSESLSM0.9996680.000386−26.50470.0001220.997347−57.73690.0149SIFT0.989675−0.003−25.9192−0.0030.974333−55.18361.0574L−RTRUE1.0937−0.422212.66410.39740.8906−6.1527RMSESLSM1.09404−0.422342.319880.3976370.890746−15.89230.1762SIFT1.07447−0.411422.833370.3815120.889261−14.90020.5161L−RTRUE1.0937−0.42222.66410.39740.8906−16.1527RMSESLSM1.09381−0.4223912.34390.3975590.890414−5.867410.1751SIFT1.08469−0.4227612.73280.3826790.894351−5.355920.3266R−R′TRUE1.0008970.000108−14.00390.0001081.0016653.8145RMSESLSM1.000880.000123−14.00579.54E − 051.0016853.82180.0031SIFT1.007130.00109−14.45580.0031151.0051853.65890.1951R−R′TRUE0.9982−0.00019−7.1839−0.000190.999637.7021RMSESLSM0.998169−0.00021−7.17462−0.000150.99956737.70770.0044SIFT0.984811−0.00574−5.82985−0.00650.98697138.62860.6699The bold values are the main quantitative indicators of the algorithm's accuracy, and the illustration was added.


## 3.2. Simulation Test II: Matching Window Size

Because the SLSM method seeks the parameter vector in (5) to minimize the difference between the matching window pairs, a larger window size contains more pixel information. It leads to higher accuracy while potentially undermining the algorithm’s robustness and speed, causing divergence. In simulation experiment II, the matching window size is analyzed, and the L−L′ and L−R image pairs are set as the test data and denoted as LL and LR, respectively. The simulation results are shown in Figure 7(a).Figure 7
Results of the simulation test on different factors. (a) Simulation test II: varying matching window size. (b) Simulation test III: matrix inversion methods. (c) Simulation test IV: iteration numbers. (d) Simulation test V: illumination variance.
(a)(b)(c)(d)It can be seen that the computing duration increases with the window size, and the RMSE of the LL decreases with the increase of the window size while the LR remains basically the same. Therefore, the matching window size is not the bigger, the better. When the size of the window is set to 31 × 31 pixels, the RMSE and computing duration reach their optimized values, respectively. Considering the size of the target images for simulation is 250 × 300 pixel, the best matching window size is approximately 13% of the short side of the target region.


## 3.3. Simulation Test III: Matrix Inversion Method

To solve the residual (5) (calculating the incremental vector), the coefficient matrix C needs to be inverted. Three commonly used methods in C++ for matrix inversion are compared, namely Cholesky, LU (LU factorization), and eigenvalue decomposition. The matching window size is set to 31 × 31 pixels, and the simulation results are shown in Figure 7(b). It can be seen that the LU factorization achieves the most satisfying result in both computing speed and precision.


## 3.4. Simulation Test IV: Iteration Number

Not all the matching windows centered at the initial estimation points can generate satisfying matching tie points. The algorithm should stop after several iterations when the initial value is too bad to converge. Meanwhile, the algorithm needs an adequate number of iterations to find a satisfying result. Therefore, a fair number of iterations will guarantee robustness and precision. The iteration numbers are evaluated in simulation test IV. The window size is set to be 31 × 31 pixels, and LU is selected as the matrix inversion method. The results of the simulation experiment IV are shown in Figure7(c).It can be noticed that the endless increase of the iteration number does not lead to lower RMSE but longer duration. When the iteration number is set to 100, both the duration and RMSE are acceptable and balanced.


## 3.5. Simulation Test V: Illumination Variance Resistance

The illumination condition in the outside field is unstable, so the brightness of the images acquired in one interval may vary. Thus, the matching algorithm should be able to resist some illumination variance. In simulation experiment V, the robustness of the uneven illumination of the proposed matching algorithm is evaluated. The image’s brightness is artificially adjusted, and the brightness contrasts between image pairs to simulate the actual condition of the local environment. The test results are shown in Figure7(d).The diagram shows that the proposed matching algorithm has better robustness than the SIFT method toward illumination variation. Although the RMSE increases while the illumination difference enlarges, the increase is relatively small and within the acceptable range, nearly 0.04 pixels.


## 3.6. Discussion of the Simulation Test

The simulation test shows that the proposed SIFT-based improved matching algorithm (SLSM) can achieve accurate matching results within an acceptable time, and the RMSE is within 0.2 pixels. Also, the SLSM performs better than the SIFT in illumination resistance and accuracy. However, SIFT provides a relatively good initial guess for the LSM matching, contributing to robustness and high precision. The real homography matrix is the basis for the subsequent 3D coordinate reconstruction and displacement measurement.


## 4. Experimental Validation

The practical application of the proposed method was verified using a laboratory-scale test and an in-field test, respectively. The laboratory test is a small shaking table test. The in-field test is an actual experiment using the whole vision system to measure a target movement in the virtual environment. The purpose of the indoor experiment is to examine the accuracy of the proposed binocular vision measurement system, and the outdoor experiment is to examine whether the methodology is valid under the actual conditions of SHM.


### 4.1. Indoor Laboratory Experiment

#### 4.1.1. Experiment Setup

The indoor experiment included a laboratory-scale shaking table, and a three-story braced steel shear structure model to conduct a small shaking table test with all the commonly used displacement sensors and the proposed vision measurement system, as shown in Figure8. The small shaking table can reproduce various displacement excitation, and several methods for comparison record the displacement signals. The small electric shaking table, equipped with an RH100X30D05A read head and a raster gauge RGS40-S from RENISHAW LTD, can achieve high-precision excitation and output accurate displacement readings (with an accuracy of 15 μm/m). Therefore, the readings from the control software of the shaking table are set as the actual values. In this experimental condition, two lenses (fixed focal length 50 mm) are used, and the aperture size of the two cameras is set to f2.0 to keep a stable brightness of the pictures. The object distance is set to 4 m to keep the artificial targets occupying enough pixel information. The applied displacement sensors’ technical parameters are shown in Appendix Figure 9.Figure 8
Setup of the indoor experiment: (a) the commonly used sensors applied in the indoor experiment and (b) installation of all the sensors.
(a)(b)Figure 9
The technical specifications of all the displacement sensors used in the indoor experiment.The calibration method developed by Zhengyou [49] was used to obtain the intrinsic and extrinsic parameters of the binocular vision system. As for the excitation signals, the sine waves with a frequency of 1 Hz and an amplitude of 0.1 mm, 1 mm, and 10 mm were selected as the excitation signals of the shaking table. Three metrics are used to quantify the measurement errors [50], namely the cross-correlation coefficient (COE), the root mean square error (RMSE), and the normalized root means square error (NRMSE) as shown in the following equations, respectively.(13)ρ=∑kdk−dk¯×dk′−dk′¯∑kdk−dk¯2×∑kdk′−dk′¯2,(14)RMSEerror=∑k=1ndk−dk′2n,(15)NRMSEerror=∑k=1ndk−dk′2/ndk,max′−dk,min′,where ρ is the cross-correlation coefficient representing the correlation between two signals in the time domain, dk,dk′ are the two displacement signals, and the average is denoted as dk¯,dk′¯, respectively.Besides the commonly used displacement sensors and the proposed binocular vision measurement system, some other visual displacement measurement methods are used for comparison. The difference between these visual measurement methods is mainly the image matching algorithm, namely subpixel Harris [51], ellipse fitting [52], gray-scale centroid [53], and the proposed SLSM algorithm in this paper. All the abbreviations of the displacement measurement methods are shown in Table 3.Table 3
Abbreviations of all the displacement measurement methods.
Measurement methodsSpecificationAbbreviationCommonly used methodsLaser displacement meterLDMDisplacement transducerDTAccelerometerAMSeismometer gaugeSGVisual methodsSubpixel HarrisSHEllipse fittingEFGrayscale centroidGCThe proposed methodSLSMActual valueRaster gaugeRG


#### 4.1.2. Test Results

As shown in Figure8, the displacement of the shaking table was measured using a raster gauge inside the table and set as the actual value, so only the actual displacement response of the first floor is measured instead of all the floors. Therefore, only the displacement responses of the first floor measured by different sensors (mounted on the Shaker table) are compared in Figures 10(a)–10(c).Figure 10
Error analysis of the indoor experiment: (a) case 0.1 mm, 1 Hz, sine wave; (b) case ±1 mm, 1 Hz, sine wave; (c) case ±10 mm, 1 Hz, sine wave; (d) 3D coordinates of the top floor in case ±10 mm, 1 Hz, sine wave.
(a)(b)(c)(d)With the high-precision shaking table, we can reproduce an extreme case where the amplitude is 0.1 mm. The displacement signals of this case (0.1 mm, 1 Hz, sine wave) are shown in Figure11. Due to the minimal displacement, the performance of the displacement transducer is not good enough, with the lowest cross-correlation coefficient (0.4019). The displacement signals recorded by other visual methods are more significant than the actual value, with some deviation at the peaks and troughs, while the proposed SLSM methodology remains basically the same. As for the smoothness, the displacement transducer and the laser meter curves are relatively sharper than all the other methods. The visual method based on the gray-scale centroid records the least smooth among all the visual methods. The SLSM method achieves a more relatively stable result among all the matching algorithms. As indicated by the errors analyzed in Figure 10, the proposed methodology achieved the minimum NRMSE (0.0839 mm) and RMSE (0.0127). Except for the accelerometer, the errors of the proposed methodology are the lowest.Figure 11
Displacement signals of the indoor experiment I ±0.1 mm, 1 Hz, sine wave.Figure12 demonstrates the displacement signals of the cases (±1 mm, 1 Hz, sine wave) and (±10 mm, 1 Hz, sine wave), respectively. Unlike experiment I, the displacement signals of all the sensors are much more smooth than the case I and correspond better with the actual value (COE ρ close to 1) as the amplitude of the vibration increases. Among all the signals, the accelerometer-based method displays a smaller amplitude than the actual value. The error analysis is shown in Figure 10. In the case (±1 mm, 1 Hz, sine wave), the proposed methodology presented the minimum RMSE (0.0146 mm) and the NRMSE (0.0171). By contrast, in the case (±10 mm, 1 Hz, sine wave), the displacement transducer and the laser displacement meter show the minimum RMSE (0.0028 mm) and NRMSE (0.0568) while the errors of the proposed binocular vision method are 0.0057 mm (RMSE) and 0.0457 (NRMSE), respectively.Figure 12
Displacement signals of the indoor experiment: (a) II ±1 mm, 1 Hz, sine wave and (b) III ±10 mm, 1 Hz, sine wave.
(a)(b)The above experiment results show that the proposed binocular vision measurement system can outperform other visual methods (mainly Harris corner point, ellipse fitting, and gray-scale centroid as image matching methods) in accuracy. Also, the displacement signal generated using the proposed matching algorithm is more stable, and the peak value matches the actual values better than the other matching algorithms. In some cases, when the displacement response is relatively small, the proposed methodology can achieve more accurate measurements than the commonly used sensors. When the amplitude reaches a relatively large (10 mm), the proposed methodology performs approximately the same as the most accurate commonly used methods. Furthermore, the errors of the proposed methodology will increase as the vibration amplitude increases, but it is still within the requirement of SHM and close to the commonly used sensors.


#### 4.1.3. Discussion of the Indoor Experiment

Using the proposed binocular vision measurement system, we can acquire the 3D coordinates of the target and visually analyze the positions of the structure model in the shaking table test, as shown in Figure10(d). The target’s displayed coordinates are in the case (±10 mm, 1 Hz, sine wave). From the diagram, we can see a ±0.4 mm offset in the Z direction and ±0.25 mm offset in the Y direction. The offset indicates that the target moves in more than one direction (the X direction). When the shift in the X direction is relatively small (for example, in case I), the displacement ratio of the other two directions may be a significant proportion. However, the displacement transducer or the laser meter can only measure the displacement in one direction and may not be perfectly pointed in the vibration direction (X direction), so only the displacement in the X direction is measured and represented as the actual displacement of the target, neglecting the components in the other two directions. The misrepresentation may lead to relatively more significant errors in the case of slight amplitude vibration. When the amplitude of the vibration increases, the proportion of the shift in the other two directions decreases, and the measurement errors decrease. So the measuring errors of the displacement transducer and the laser displacement meter in case III (10 mm amplitude) are much more minor than in case II (1 mm amplitude). However, the proposed visual method can avoid this defect by directly measuring the 3D displacement.


### 4.2. Outdoor Experiment

To verify the effectiveness and the accuracy of the proposed measurement system in the actual condition of the SHM, a field test is conducted using all the image acquisition equipment mentioned previously, and the image matching method is shown in Figure13.Figure 13
Setup of the outdoor experiment: (a) installation positions of the binocular vision system and (b) arrangement of the artificial target and the encoded markers.
(a)(b)


#### 4.2.1. Experiment Setup

The outdoor experiment site is located at a parking lot near the laboratory for the indoor experiment mentioned above. The object distance is set to 24 m, which is the maximum distance we can achieve in this situation. The encoded markers shown in Figure13 are fixed to the ground around the artificial target and used to provide geometry constraints for camera calibration. An artificial target (generated using paint spray) is mounted on an adjustable platform with a micrometer to precisely control the position of the artificial target. During each experiment, the target is moved a certain distance according to the readings of the micrometer. By comparing the shift displacement measured by the visual method and the micrometer readings, the measurement system’s accuracy can be evaluated.


#### 4.2.2. Camera Calibration

A total station, encoded markers, and a calibration chessboard are used for camera calibration. The black-white chessboard is used for precise corner extraction. The intrinsic parameters of each camera are obtained using the calibration method developed by Zhengyou [49] using MATLAB, as shown in Figure 14.Figure 14
Camera calibration: (a) intrinsic parameters calculation and (b) extrinsic parameters calculation.
(a)(b)As for the extrinsic parameters of the two cameras, the encoded markers with a circle pattern shown in Figure13 are fixed to the surroundings to impose the geometry constraints for the calibration procedure. The center of the markers can be visible in the images, and the accurate coordinates of the center are obtained through the total station. The extrinsic parameters of the two cameras can be obtained by combining the coordinates of the same marker in the image plane and the world coordinate system.


#### 4.2.3. Experiment Results

The readings from the micrometer mounted on the bracket are set as the actual value and compared with the measurement results obtained by the proposed system. Experiment results are shown in Table4. The RMSE of the proposed system is 3.905 mm, and the object distance is 24 m.Table 4
Results of the outdoor experiment.
SequenceActual value (mm)Measured displacement (mm)Deviation (mm)128.531.7183.218228.531.1992.699328.532.4473.947428.533.9825.482528.532.9494.449628.531.8613.361728.532.0183.518RMSE (mm)3.905By analyzing the images from the outdoor experiment shown in Figure15, we can see a nonnegligible lighting intensity difference in the actual scene, which caused the measurement errors. When the deviation of the measuring results increases, the light intensity also increases from time sequence 3 to 6, as shown in Table 4. From the previous simulation test, the proposed image matching algorithm (SLSM) can resist illumination changes to some extent. However, as seen in Figure 15, the illumination difference is way beyond what the algorithm can do.Figure 15
The circumstance of the outdoor experiment illumination variance.


## 4.1. Indoor Laboratory Experiment

### 4.1.1. Experiment Setup

The indoor experiment included a laboratory-scale shaking table, and a three-story braced steel shear structure model to conduct a small shaking table test with all the commonly used displacement sensors and the proposed vision measurement system, as shown in Figure8. The small shaking table can reproduce various displacement excitation, and several methods for comparison record the displacement signals. The small electric shaking table, equipped with an RH100X30D05A read head and a raster gauge RGS40-S from RENISHAW LTD, can achieve high-precision excitation and output accurate displacement readings (with an accuracy of 15 μm/m). Therefore, the readings from the control software of the shaking table are set as the actual values. In this experimental condition, two lenses (fixed focal length 50 mm) are used, and the aperture size of the two cameras is set to f2.0 to keep a stable brightness of the pictures. The object distance is set to 4 m to keep the artificial targets occupying enough pixel information. The applied displacement sensors’ technical parameters are shown in Appendix Figure 9.Figure 8
Setup of the indoor experiment: (a) the commonly used sensors applied in the indoor experiment and (b) installation of all the sensors.
(a)(b)Figure 9
The technical specifications of all the displacement sensors used in the indoor experiment.The calibration method developed by Zhengyou [49] was used to obtain the intrinsic and extrinsic parameters of the binocular vision system. As for the excitation signals, the sine waves with a frequency of 1 Hz and an amplitude of 0.1 mm, 1 mm, and 10 mm were selected as the excitation signals of the shaking table. Three metrics are used to quantify the measurement errors [50], namely the cross-correlation coefficient (COE), the root mean square error (RMSE), and the normalized root means square error (NRMSE) as shown in the following equations, respectively.(13)ρ=∑kdk−dk¯×dk′−dk′¯∑kdk−dk¯2×∑kdk′−dk′¯2,(14)RMSEerror=∑k=1ndk−dk′2n,(15)NRMSEerror=∑k=1ndk−dk′2/ndk,max′−dk,min′,where ρ is the cross-correlation coefficient representing the correlation between two signals in the time domain, dk,dk′ are the two displacement signals, and the average is denoted as dk¯,dk′¯, respectively.Besides the commonly used displacement sensors and the proposed binocular vision measurement system, some other visual displacement measurement methods are used for comparison. The difference between these visual measurement methods is mainly the image matching algorithm, namely subpixel Harris [51], ellipse fitting [52], gray-scale centroid [53], and the proposed SLSM algorithm in this paper. All the abbreviations of the displacement measurement methods are shown in Table 3.Table 3
Abbreviations of all the displacement measurement methods.
Measurement methodsSpecificationAbbreviationCommonly used methodsLaser displacement meterLDMDisplacement transducerDTAccelerometerAMSeismometer gaugeSGVisual methodsSubpixel HarrisSHEllipse fittingEFGrayscale centroidGCThe proposed methodSLSMActual valueRaster gaugeRG


### 4.1.2. Test Results

As shown in Figure8, the displacement of the shaking table was measured using a raster gauge inside the table and set as the actual value, so only the actual displacement response of the first floor is measured instead of all the floors. Therefore, only the displacement responses of the first floor measured by different sensors (mounted on the Shaker table) are compared in Figures 10(a)–10(c).Figure 10
Error analysis of the indoor experiment: (a) case 0.1 mm, 1 Hz, sine wave; (b) case ±1 mm, 1 Hz, sine wave; (c) case ±10 mm, 1 Hz, sine wave; (d) 3D coordinates of the top floor in case ±10 mm, 1 Hz, sine wave.
(a)(b)(c)(d)With the high-precision shaking table, we can reproduce an extreme case where the amplitude is 0.1 mm. The displacement signals of this case (0.1 mm, 1 Hz, sine wave) are shown in Figure11. Due to the minimal displacement, the performance of the displacement transducer is not good enough, with the lowest cross-correlation coefficient (0.4019). The displacement signals recorded by other visual methods are more significant than the actual value, with some deviation at the peaks and troughs, while the proposed SLSM methodology remains basically the same. As for the smoothness, the displacement transducer and the laser meter curves are relatively sharper than all the other methods. The visual method based on the gray-scale centroid records the least smooth among all the visual methods. The SLSM method achieves a more relatively stable result among all the matching algorithms. As indicated by the errors analyzed in Figure 10, the proposed methodology achieved the minimum NRMSE (0.0839 mm) and RMSE (0.0127). Except for the accelerometer, the errors of the proposed methodology are the lowest.Figure 11
Displacement signals of the indoor experiment I ±0.1 mm, 1 Hz, sine wave.Figure12 demonstrates the displacement signals of the cases (±1 mm, 1 Hz, sine wave) and (±10 mm, 1 Hz, sine wave), respectively. Unlike experiment I, the displacement signals of all the sensors are much more smooth than the case I and correspond better with the actual value (COE ρ close to 1) as the amplitude of the vibration increases. Among all the signals, the accelerometer-based method displays a smaller amplitude than the actual value. The error analysis is shown in Figure 10. In the case (±1 mm, 1 Hz, sine wave), the proposed methodology presented the minimum RMSE (0.0146 mm) and the NRMSE (0.0171). By contrast, in the case (±10 mm, 1 Hz, sine wave), the displacement transducer and the laser displacement meter show the minimum RMSE (0.0028 mm) and NRMSE (0.0568) while the errors of the proposed binocular vision method are 0.0057 mm (RMSE) and 0.0457 (NRMSE), respectively.Figure 12
Displacement signals of the indoor experiment: (a) II ±1 mm, 1 Hz, sine wave and (b) III ±10 mm, 1 Hz, sine wave.
(a)(b)The above experiment results show that the proposed binocular vision measurement system can outperform other visual methods (mainly Harris corner point, ellipse fitting, and gray-scale centroid as image matching methods) in accuracy. Also, the displacement signal generated using the proposed matching algorithm is more stable, and the peak value matches the actual values better than the other matching algorithms. In some cases, when the displacement response is relatively small, the proposed methodology can achieve more accurate measurements than the commonly used sensors. When the amplitude reaches a relatively large (10 mm), the proposed methodology performs approximately the same as the most accurate commonly used methods. Furthermore, the errors of the proposed methodology will increase as the vibration amplitude increases, but it is still within the requirement of SHM and close to the commonly used sensors.


### 4.1.3. Discussion of the Indoor Experiment

Using the proposed binocular vision measurement system, we can acquire the 3D coordinates of the target and visually analyze the positions of the structure model in the shaking table test, as shown in Figure10(d). The target’s displayed coordinates are in the case (±10 mm, 1 Hz, sine wave). From the diagram, we can see a ±0.4 mm offset in the Z direction and ±0.25 mm offset in the Y direction. The offset indicates that the target moves in more than one direction (the X direction). When the shift in the X direction is relatively small (for example, in case I), the displacement ratio of the other two directions may be a significant proportion. However, the displacement transducer or the laser meter can only measure the displacement in one direction and may not be perfectly pointed in the vibration direction (X direction), so only the displacement in the X direction is measured and represented as the actual displacement of the target, neglecting the components in the other two directions. The misrepresentation may lead to relatively more significant errors in the case of slight amplitude vibration. When the amplitude of the vibration increases, the proportion of the shift in the other two directions decreases, and the measurement errors decrease. So the measuring errors of the displacement transducer and the laser displacement meter in case III (10 mm amplitude) are much more minor than in case II (1 mm amplitude). However, the proposed visual method can avoid this defect by directly measuring the 3D displacement.


## 4.1.1. Experiment Setup

The indoor experiment included a laboratory-scale shaking table, and a three-story braced steel shear structure model to conduct a small shaking table test with all the commonly used displacement sensors and the proposed vision measurement system, as shown in Figure8. The small shaking table can reproduce various displacement excitation, and several methods for comparison record the displacement signals. The small electric shaking table, equipped with an RH100X30D05A read head and a raster gauge RGS40-S from RENISHAW LTD, can achieve high-precision excitation and output accurate displacement readings (with an accuracy of 15 μm/m). Therefore, the readings from the control software of the shaking table are set as the actual values. In this experimental condition, two lenses (fixed focal length 50 mm) are used, and the aperture size of the two cameras is set to f2.0 to keep a stable brightness of the pictures. The object distance is set to 4 m to keep the artificial targets occupying enough pixel information. The applied displacement sensors’ technical parameters are shown in Appendix Figure 9.Figure 8
Setup of the indoor experiment: (a) the commonly used sensors applied in the indoor experiment and (b) installation of all the sensors.
(a)(b)Figure 9
The technical specifications of all the displacement sensors used in the indoor experiment.The calibration method developed by Zhengyou [49] was used to obtain the intrinsic and extrinsic parameters of the binocular vision system. As for the excitation signals, the sine waves with a frequency of 1 Hz and an amplitude of 0.1 mm, 1 mm, and 10 mm were selected as the excitation signals of the shaking table. Three metrics are used to quantify the measurement errors [50], namely the cross-correlation coefficient (COE), the root mean square error (RMSE), and the normalized root means square error (NRMSE) as shown in the following equations, respectively.(13)ρ=∑kdk−dk¯×dk′−dk′¯∑kdk−dk¯2×∑kdk′−dk′¯2,(14)RMSEerror=∑k=1ndk−dk′2n,(15)NRMSEerror=∑k=1ndk−dk′2/ndk,max′−dk,min′,where ρ is the cross-correlation coefficient representing the correlation between two signals in the time domain, dk,dk′ are the two displacement signals, and the average is denoted as dk¯,dk′¯, respectively.Besides the commonly used displacement sensors and the proposed binocular vision measurement system, some other visual displacement measurement methods are used for comparison. The difference between these visual measurement methods is mainly the image matching algorithm, namely subpixel Harris [51], ellipse fitting [52], gray-scale centroid [53], and the proposed SLSM algorithm in this paper. All the abbreviations of the displacement measurement methods are shown in Table 3.Table 3
Abbreviations of all the displacement measurement methods.
Measurement methodsSpecificationAbbreviationCommonly used methodsLaser displacement meterLDMDisplacement transducerDTAccelerometerAMSeismometer gaugeSGVisual methodsSubpixel HarrisSHEllipse fittingEFGrayscale centroidGCThe proposed methodSLSMActual valueRaster gaugeRG


## 4.1.2. Test Results

As shown in Figure8, the displacement of the shaking table was measured using a raster gauge inside the table and set as the actual value, so only the actual displacement response of the first floor is measured instead of all the floors. Therefore, only the displacement responses of the first floor measured by different sensors (mounted on the Shaker table) are compared in Figures 10(a)–10(c).Figure 10
Error analysis of the indoor experiment: (a) case 0.1 mm, 1 Hz, sine wave; (b) case ±1 mm, 1 Hz, sine wave; (c) case ±10 mm, 1 Hz, sine wave; (d) 3D coordinates of the top floor in case ±10 mm, 1 Hz, sine wave.
(a)(b)(c)(d)With the high-precision shaking table, we can reproduce an extreme case where the amplitude is 0.1 mm. The displacement signals of this case (0.1 mm, 1 Hz, sine wave) are shown in Figure11. Due to the minimal displacement, the performance of the displacement transducer is not good enough, with the lowest cross-correlation coefficient (0.4019). The displacement signals recorded by other visual methods are more significant than the actual value, with some deviation at the peaks and troughs, while the proposed SLSM methodology remains basically the same. As for the smoothness, the displacement transducer and the laser meter curves are relatively sharper than all the other methods. The visual method based on the gray-scale centroid records the least smooth among all the visual methods. The SLSM method achieves a more relatively stable result among all the matching algorithms. As indicated by the errors analyzed in Figure 10, the proposed methodology achieved the minimum NRMSE (0.0839 mm) and RMSE (0.0127). Except for the accelerometer, the errors of the proposed methodology are the lowest.Figure 11
Displacement signals of the indoor experiment I ±0.1 mm, 1 Hz, sine wave.Figure12 demonstrates the displacement signals of the cases (±1 mm, 1 Hz, sine wave) and (±10 mm, 1 Hz, sine wave), respectively. Unlike experiment I, the displacement signals of all the sensors are much more smooth than the case I and correspond better with the actual value (COE ρ close to 1) as the amplitude of the vibration increases. Among all the signals, the accelerometer-based method displays a smaller amplitude than the actual value. The error analysis is shown in Figure 10. In the case (±1 mm, 1 Hz, sine wave), the proposed methodology presented the minimum RMSE (0.0146 mm) and the NRMSE (0.0171). By contrast, in the case (±10 mm, 1 Hz, sine wave), the displacement transducer and the laser displacement meter show the minimum RMSE (0.0028 mm) and NRMSE (0.0568) while the errors of the proposed binocular vision method are 0.0057 mm (RMSE) and 0.0457 (NRMSE), respectively.Figure 12
Displacement signals of the indoor experiment: (a) II ±1 mm, 1 Hz, sine wave and (b) III ±10 mm, 1 Hz, sine wave.
(a)(b)The above experiment results show that the proposed binocular vision measurement system can outperform other visual methods (mainly Harris corner point, ellipse fitting, and gray-scale centroid as image matching methods) in accuracy. Also, the displacement signal generated using the proposed matching algorithm is more stable, and the peak value matches the actual values better than the other matching algorithms. In some cases, when the displacement response is relatively small, the proposed methodology can achieve more accurate measurements than the commonly used sensors. When the amplitude reaches a relatively large (10 mm), the proposed methodology performs approximately the same as the most accurate commonly used methods. Furthermore, the errors of the proposed methodology will increase as the vibration amplitude increases, but it is still within the requirement of SHM and close to the commonly used sensors.


## 4.1.3. Discussion of the Indoor Experiment

Using the proposed binocular vision measurement system, we can acquire the 3D coordinates of the target and visually analyze the positions of the structure model in the shaking table test, as shown in Figure10(d). The target’s displayed coordinates are in the case (±10 mm, 1 Hz, sine wave). From the diagram, we can see a ±0.4 mm offset in the Z direction and ±0.25 mm offset in the Y direction. The offset indicates that the target moves in more than one direction (the X direction). When the shift in the X direction is relatively small (for example, in case I), the displacement ratio of the other two directions may be a significant proportion. However, the displacement transducer or the laser meter can only measure the displacement in one direction and may not be perfectly pointed in the vibration direction (X direction), so only the displacement in the X direction is measured and represented as the actual displacement of the target, neglecting the components in the other two directions. The misrepresentation may lead to relatively more significant errors in the case of slight amplitude vibration. When the amplitude of the vibration increases, the proportion of the shift in the other two directions decreases, and the measurement errors decrease. So the measuring errors of the displacement transducer and the laser displacement meter in case III (10 mm amplitude) are much more minor than in case II (1 mm amplitude). However, the proposed visual method can avoid this defect by directly measuring the 3D displacement.


## 4.2. Outdoor Experiment

To verify the effectiveness and the accuracy of the proposed measurement system in the actual condition of the SHM, a field test is conducted using all the image acquisition equipment mentioned previously, and the image matching method is shown in Figure13.Figure 13
Setup of the outdoor experiment: (a) installation positions of the binocular vision system and (b) arrangement of the artificial target and the encoded markers.
(a)(b)


### 4.2.1. Experiment Setup

The outdoor experiment site is located at a parking lot near the laboratory for the indoor experiment mentioned above. The object distance is set to 24 m, which is the maximum distance we can achieve in this situation. The encoded markers shown in Figure13 are fixed to the ground around the artificial target and used to provide geometry constraints for camera calibration. An artificial target (generated using paint spray) is mounted on an adjustable platform with a micrometer to precisely control the position of the artificial target. During each experiment, the target is moved a certain distance according to the readings of the micrometer. By comparing the shift displacement measured by the visual method and the micrometer readings, the measurement system’s accuracy can be evaluated.


### 4.2.2. Camera Calibration

A total station, encoded markers, and a calibration chessboard are used for camera calibration. The black-white chessboard is used for precise corner extraction. The intrinsic parameters of each camera are obtained using the calibration method developed by Zhengyou [49] using MATLAB, as shown in Figure 14.Figure 14
Camera calibration: (a) intrinsic parameters calculation and (b) extrinsic parameters calculation.
(a)(b)As for the extrinsic parameters of the two cameras, the encoded markers with a circle pattern shown in Figure13 are fixed to the surroundings to impose the geometry constraints for the calibration procedure. The center of the markers can be visible in the images, and the accurate coordinates of the center are obtained through the total station. The extrinsic parameters of the two cameras can be obtained by combining the coordinates of the same marker in the image plane and the world coordinate system.


### 4.2.3. Experiment Results

The readings from the micrometer mounted on the bracket are set as the actual value and compared with the measurement results obtained by the proposed system. Experiment results are shown in Table4. The RMSE of the proposed system is 3.905 mm, and the object distance is 24 m.Table 4
Results of the outdoor experiment.
SequenceActual value (mm)Measured displacement (mm)Deviation (mm)128.531.7183.218228.531.1992.699328.532.4473.947428.533.9825.482528.532.9494.449628.531.8613.361728.532.0183.518RMSE (mm)3.905By analyzing the images from the outdoor experiment shown in Figure15, we can see a nonnegligible lighting intensity difference in the actual scene, which caused the measurement errors. When the deviation of the measuring results increases, the light intensity also increases from time sequence 3 to 6, as shown in Table 4. From the previous simulation test, the proposed image matching algorithm (SLSM) can resist illumination changes to some extent. However, as seen in Figure 15, the illumination difference is way beyond what the algorithm can do.Figure 15
The circumstance of the outdoor experiment illumination variance.


## 4.2.1. Experiment Setup

The outdoor experiment site is located at a parking lot near the laboratory for the indoor experiment mentioned above. The object distance is set to 24 m, which is the maximum distance we can achieve in this situation. The encoded markers shown in Figure13 are fixed to the ground around the artificial target and used to provide geometry constraints for camera calibration. An artificial target (generated using paint spray) is mounted on an adjustable platform with a micrometer to precisely control the position of the artificial target. During each experiment, the target is moved a certain distance according to the readings of the micrometer. By comparing the shift displacement measured by the visual method and the micrometer readings, the measurement system’s accuracy can be evaluated.


## 4.2.2. Camera Calibration

A total station, encoded markers, and a calibration chessboard are used for camera calibration. The black-white chessboard is used for precise corner extraction. The intrinsic parameters of each camera are obtained using the calibration method developed by Zhengyou [49] using MATLAB, as shown in Figure 14.Figure 14
Camera calibration: (a) intrinsic parameters calculation and (b) extrinsic parameters calculation.
(a)(b)As for the extrinsic parameters of the two cameras, the encoded markers with a circle pattern shown in Figure13 are fixed to the surroundings to impose the geometry constraints for the calibration procedure. The center of the markers can be visible in the images, and the accurate coordinates of the center are obtained through the total station. The extrinsic parameters of the two cameras can be obtained by combining the coordinates of the same marker in the image plane and the world coordinate system.


## 4.2.3. Experiment Results

The readings from the micrometer mounted on the bracket are set as the actual value and compared with the measurement results obtained by the proposed system. Experiment results are shown in Table4. The RMSE of the proposed system is 3.905 mm, and the object distance is 24 m.Table 4
Results of the outdoor experiment.
SequenceActual value (mm)Measured displacement (mm)Deviation (mm)128.531.7183.218228.531.1992.699328.532.4473.947428.533.9825.482528.532.9494.449628.531.8613.361728.532.0183.518RMSE (mm)3.905By analyzing the images from the outdoor experiment shown in Figure15, we can see a nonnegligible lighting intensity difference in the actual scene, which caused the measurement errors. When the deviation of the measuring results increases, the light intensity also increases from time sequence 3 to 6, as shown in Table 4. From the previous simulation test, the proposed image matching algorithm (SLSM) can resist illumination changes to some extent. However, as seen in Figure 15, the illumination difference is way beyond what the algorithm can do.Figure 15
The circumstance of the outdoor experiment illumination variance.


## 5. Conclusions

To overcome the challenges, the commonly used displacement sensors may face in the field of structural health monitoring. This paper developed a binocular vision measurement system with the SIFT-based improved LSM algorithm (SLSM). Simulation tests and indoor and outdoor experiments comprehensively analyze the visual method. The followings are the conclusions drawn from the study:(1)
The binocular vision system consists of a 4G cellular module that can upload the images to the cloud server for image process and displacement calculation. The whole system can be installed inside a waterproof box with a solar panel as the power supply, ensuring its ability to telemonitor.(2)
We proposed a SIFT-based improved LSM (SLSM) matching algorithm to balance target installation and measurement precision. The proposed matching algorithm only requires the target to have a flat surface and some textures. This reduces the target complexity, making the installation work much easier in the field of SHM, and guarantees matching accuracy. The simulation test verified the precision of the proposed algorithm and studied some factors related to the algorithm’s performance. 13% of the short side of the target image, 100 iteration number, and LU matrix inversion method are the best algorithm settings, contributing to less than 0.2 pixel RMSE. Using the look-up table for interpolation, the computing time is less than 1 second with the best settings. Plus, the time used for data transferring and the measuring time interval can be within 2 seconds. Also, compared to other image-matching methods, the SLSM method achieves visible improvement in the resistance to illumination variance, matching precision, and stability.(3)
Actual experiments were conducted using the proposed methodology, the accuracy was analyzed in the shaking table test, and the feasibility for SHM was verified in the outdoor experiment. From the displacement signals recorded in the shaking table test, the proposed methodology presents an RMSE less than 0.05 mm with object distance (4 m) in a laboratory environment, which outperforms all the other visual methods. Especially when the vibration is slight (0.1 mm), the proposed methodology performs better than the commonly used displacement meters and has nearly the same accuracy when the amplitude is large (10 mm). By analyzing the readings from the outdoor experiment, the RMSE is about 4 mm, which is larger than the indoor experiment because of the considerable object distance (24 m) and sharp illumination differences.The advantages of the proposed measurement system are apparent; it can measure multipoint high-precision 3D displacement using flat regions with some textures as the targets. The installation is much easier than the commonly used sensors. Except for the displacement, the structure’s status is visible in the images acquired by the binocular vision system. However, there are also some disadvantages shown by the experiments. Although the visual method can achieve accurate results in the laboratory environment, the accuracy will be negatively impacted when the illumination varies widely. This paper has exerted some effort to compensate for the impact of the varying lighting by combining SIFT and improved LSM matching. However, the effect remains, and future work is still needed to solve this problem.



---

*Source: 1132569-2023-02-09.xml*