# A Real-Time Deformation Monitoring Method for Large-Scale Structure Based on Relay Camera

Authors: Chuan He; Lianxiong Liu; Changhua Hu
Journal: Journal of Sensors
 (2021)

---

## Abstract

In the process of the deformation monitoring for large-scale structure, the mobile vision method is often used. However, most of the existent researches rarely consider the real-time property and the variation of the intrinsic parameters. This paper proposes a real-time deformation monitoring method for the large-scale structure based on a relay camera. First, we achieve the real-time pose-position relationship by using the relay camera and the coded mark points whose coordinates are known. The real-time extrinsic parameters of the measuring camera are then solved according to the constraint relationship between the relay camera and the measuring camera. Second, the real-time intrinsic parameters of the measuring camera are calculated based on the real-time constraint relationship among the extrinsic parameters, the intrinsic parameters, and the fundamental matrix. Finally, the coordinates of the noncoded measured mark points, which are affixed to the surface of the structure, are achieved. Experimental results show that the accuracy of the proposed method is higher than 1.8 mm. Besides, the proposed method also possesses the real-time and automation property.


---

## Body

## 1. Introduction

Deformation monitoring is vital for the safety of large-scale structure [1], and parameter calibration of instruments is essential for high-precision measurement [2]. In most mobile videometric methods, in order to achieve the global mobile measurement, the intrinsic camera parameters are calibrated in advance of the extrinsic parameters. Once the intrinsic parameters are achieved, they are assumed to be fixed. Then, the real-time varying extrinsic parameters, which reflect the pose-position relationship between the camera and the world coordinate system, are calibrated by using different methods. For instance, if the measuring camera is carried by a high-precision controllable platform, then its extrinsic parameters can be gained directly [3], or else, the extrinsic parameters should be calculated by acquiring the pose-position relationship of the camera at different positions. On the basis of the intrinsic and extrinsic parameters, the relevant measurement work is completed according to the principle of videometrics. On the one hand, although these methods can obtain extrinsic camera parameters with high precision, some of them request costly controllable platforms with high precision, and others request frequent operations with low automation. On the other hand, the environmental vibrations and other factors may influence the solution accuracy of the extrinsic parameters and change the intrinsic parameters to some extent [4]. Therefore, in actual mobile measurement works with high precision requirements, the solution error of the extrinsic parameters and the ignorance of the intrinsic parameter variation will cause large measurement error.Many works try to overcome the influence of disturbance factors such as environmental vibration and improve the accuracy of the intrinsic and extrinsic parameters. Some works, such as Yu et al. [5], Wu et al. [6], and Wu et al. [7], compensate the camera shake referring to the known control points in the scene. Their methods effectively overcome the camera shake caused by the insufficient platform accuracy and thus ensure the accuracy of the extrinsic parameters. However, in order to ensure the stability of the error-correction result, the above methods need to perform multidirectional measurement, and it is difficult to ensure the real-time performance of the measurement. Moreover, the camera shake in the mobile measurement process will bring in significant change of intrinsic parameters. Thus, there is still a certain error existing in the measurement results. In addition, some works use auxiliary equipment, such as laser range finder [8, 9] or inclinometer [10], to overcome the interference of environmental factors. These methods achieve the real-time measurement of the extrinsic parameters. In [8, 9], laser range finder with high stability and high precision is adopted to capture the pose-position parameters of the camera. Two inclinometer sensors are used in [10] to obtain the pose change of the camera in measurement and to complete the calibration of the system parameters. However, the potential change of the intrinsic parameters of the camera is not considered in [8–10].The self-calibration is the most commonly used method [11, 12] to overcome the variation of the intrinsic parameters. Pollefeys et al. [11] adopt the absolute quadratic surface to obtain the optimization constraint and to identify the intrinsic parameters, under the assumption that the tilt factor is zero. Wu and Wang [12] first achieve the simplified Kruppa equation by decomposing the fundamental matrix and then gain the intrinsic parameters by the constraints of two sets of equations. In contrast of the above two methods, some works [13] utilize the constraint relationship of the known coded mark points in the scene to complete the optimization of the intrinsic parameters. In [13], Batteoui et al. use parallelograms in at least five scenes to identify the constraint equations of the intrinsic camera parameters. Then, the real-time varying intrinsic parameters are obtained through optimization. In fact, this method is still a self-calibration method in essence. The methods mentioned above complete the solution of the intrinsic parameters through various constraints. However, for the mobile videometric tasks, these methods are too complicated to guarantee the real-time request.In order to overcome shortcomings of the above works for solving real-time intrinsic and extrinsic parameters, this paper proposes a real-time calibration method with high precision based on a parameter-known relay camera. First, the real-time extrinsic parameters of the measuring camera in the mobile measurement are identified, based on the real-time pose-position relationship between the relay camera and the known control points. Second, the fundamental matrix is gained by using the two adjacent images. Referring to the constraint between the camera parameters and the fundamental matrix, the pose-position parameters of the measuring camera at the adjacent moments are captured. Then, the varying intrinsic camera parameters are real-timely calculated. At last, with the real-time intrinsic and extrinsic parameters of the measuring camera, the real-time monitoring of structural deformation is achieved.The remainder of the paper is organized as follows. The problem formulation is given in Section2. Section 3 presents the high-precision mobile measurement method based on relay camera. In Section 4, two experiments are shown to demonstrate the efficiency and effectiveness of the proposed method. Finally, Section 5 concisely concludes this paper.


## 2. Problem Formulation

In mobile videometric measurements, the measuring camera is often installed on a relatively fixed measuring platform. The camera-relevant coordinate system is built based on the relative pose-position relationship between the measuring platform and the world coordinate system. Due to limitations in production processes, experimental equipment, and existing research strengths, the accuracy of measurement results is often affected by factors such as environmental vibration and platform sway. These factors often cause solution error of the intrinsic and extrinsic camera parameters. In the following, we will carry out the error analysis from these two aspects.


### 2.1. Extrinsic Parameter Acquisition Error

The capture of extrinsic camera parameters in mobile videometric is a difficult problem. The extrinsic camera parameters mainly comprise the extrinsic parametersRWC,TWC between the camera coordinate system and the world coordinate system, and the extrinsic parameters RC,ij,TC,ij of the pose transformation between the cameras at the adjacent time. Suffering from the disturbance factors such as environmental vibration and platform sway, different degrees of interference will directly lead to different degrees of solving errors of extrinsic parameter between the camera coordinate system and the world coordinate system. The error will affect the final measurement. For example, Yu et al. [5] point out that, when the measured object distance is 20 m, a change of 1 centimeter of the camera tilt angle will cause a displacement error of 5.8 mm in the measurement object.As shown in Figure1, the environment vibration will cause an error δθ,t between the actual camera coordinate system C0,δ−XCYCZC and the original camera coordinate system C0−XCYCZC. Under the assumption that the intrinsic camera parameters are unvarying, this error will directly lead to the error of the extrinsic parameters between the camera coordinate system and the world coordinate system:
(1)XC,0=RWCXW,0+TWC⇒environmentvibration,etc.XC,δ=RWCθXW,δ+TWCt.Figure 1
Measurement error caused by the environment vibration.As described in the Introduction, to solve the varying extrinsic camera parameters between adjacent moments, most of the existing methods refer to a controllable measuring platform or a control camera which performs measurable pose transformation. However, the pose-position between the cameras is neither easily changed according to the controllable means set by people nor accurately measured by means of manual measurement. Therefore, the acquisition error of relevant parameters is inevitable in these methods.As shown in Figure2, we assume that the camera moves from the original position C1 to the preset ideal position C2′. However, due to the defect of the platform control, the inaccuracy of the direct measurement method, or the environment vibration, etc., the camera will be in the actual position C2 but not the preset ideal position C2′. We assume that the corresponding extrinsic parameters of the actual position C2 are RC,ij and TC,ij, and the corresponding extrinsic parameters of the preset ideal position C2′ are R~C,ij and T~C,ij.Figure 2
Schematic diagram of extrinsic parameter acquisition error: (a) front view; (b) top view.
(a)(b)


### 2.2. Intrinsic Parameter Acquisition Error

In the long-term mobile measurement process, due to the environmental vibration, rotation measurement, and other interference factors, the intrinsic parameters will change frame by frame [4]. Following the pose-position change of the measuring instrument, the image principal point will change [14], which ultimately affects the accuracy of the measurement results.As shown in equation (2), the equivalent focal length fx,fy, the scale factor, and the image principal point u0,v0 in the intrinsic parameter matrix K are defaultly assumed unchanged in most cases. However, during the mobile measurement, the intrinsic parameter matrix K will gradually change to Km, with the influence of environmental vibration, camera rotation, and other factors. Therefore, the invariability assumption of the intrinsic parameters during the mobile measurement process will cause obvious measurement error.
(2)K=fxsuo0fyv0001⇒environmentvibration,etc.Km=fxmsmuom0fymv0m001.Based on the above analysis, in order to further overcome the error effects in the mobile measurement process and improve the measurement accuracy and automation of the measurement method, this paper put efforts on solving the following problems:(1)
Considering the influence of factors such as measuring platform shaking, the extrinsic parameters of the moving camera are real-timely calculated with high-precision based on the relay camera. The relay camera is used to obtain the known coded mark point information in the mobile measurement process. It is assumed that the pose-position relationship between the relay camera and the measuring camera is known and unvarying, due to their relative fixed installation position in one structure. However, in practical application, much more measures such as vibration isolation are needed to guarantee this fixed pose-position relationship(2)
Based on the constraint between the intrinsic and extrinsic camera parameters and the fundamental matrix, the intrinsic camera parameters are real-timely calculated after obtaining the extrinsic camera parameters with high-reliability and high-precision.


## 2.1. Extrinsic Parameter Acquisition Error

The capture of extrinsic camera parameters in mobile videometric is a difficult problem. The extrinsic camera parameters mainly comprise the extrinsic parametersRWC,TWC between the camera coordinate system and the world coordinate system, and the extrinsic parameters RC,ij,TC,ij of the pose transformation between the cameras at the adjacent time. Suffering from the disturbance factors such as environmental vibration and platform sway, different degrees of interference will directly lead to different degrees of solving errors of extrinsic parameter between the camera coordinate system and the world coordinate system. The error will affect the final measurement. For example, Yu et al. [5] point out that, when the measured object distance is 20 m, a change of 1 centimeter of the camera tilt angle will cause a displacement error of 5.8 mm in the measurement object.As shown in Figure1, the environment vibration will cause an error δθ,t between the actual camera coordinate system C0,δ−XCYCZC and the original camera coordinate system C0−XCYCZC. Under the assumption that the intrinsic camera parameters are unvarying, this error will directly lead to the error of the extrinsic parameters between the camera coordinate system and the world coordinate system:
(1)XC,0=RWCXW,0+TWC⇒environmentvibration,etc.XC,δ=RWCθXW,δ+TWCt.Figure 1
Measurement error caused by the environment vibration.As described in the Introduction, to solve the varying extrinsic camera parameters between adjacent moments, most of the existing methods refer to a controllable measuring platform or a control camera which performs measurable pose transformation. However, the pose-position between the cameras is neither easily changed according to the controllable means set by people nor accurately measured by means of manual measurement. Therefore, the acquisition error of relevant parameters is inevitable in these methods.As shown in Figure2, we assume that the camera moves from the original position C1 to the preset ideal position C2′. However, due to the defect of the platform control, the inaccuracy of the direct measurement method, or the environment vibration, etc., the camera will be in the actual position C2 but not the preset ideal position C2′. We assume that the corresponding extrinsic parameters of the actual position C2 are RC,ij and TC,ij, and the corresponding extrinsic parameters of the preset ideal position C2′ are R~C,ij and T~C,ij.Figure 2
Schematic diagram of extrinsic parameter acquisition error: (a) front view; (b) top view.
(a)(b)


## 2.2. Intrinsic Parameter Acquisition Error

In the long-term mobile measurement process, due to the environmental vibration, rotation measurement, and other interference factors, the intrinsic parameters will change frame by frame [4]. Following the pose-position change of the measuring instrument, the image principal point will change [14], which ultimately affects the accuracy of the measurement results.As shown in equation (2), the equivalent focal length fx,fy, the scale factor, and the image principal point u0,v0 in the intrinsic parameter matrix K are defaultly assumed unchanged in most cases. However, during the mobile measurement, the intrinsic parameter matrix K will gradually change to Km, with the influence of environmental vibration, camera rotation, and other factors. Therefore, the invariability assumption of the intrinsic parameters during the mobile measurement process will cause obvious measurement error.
(2)K=fxsuo0fyv0001⇒environmentvibration,etc.Km=fxmsmuom0fymv0m001.Based on the above analysis, in order to further overcome the error effects in the mobile measurement process and improve the measurement accuracy and automation of the measurement method, this paper put efforts on solving the following problems:(1)
Considering the influence of factors such as measuring platform shaking, the extrinsic parameters of the moving camera are real-timely calculated with high-precision based on the relay camera. The relay camera is used to obtain the known coded mark point information in the mobile measurement process. It is assumed that the pose-position relationship between the relay camera and the measuring camera is known and unvarying, due to their relative fixed installation position in one structure. However, in practical application, much more measures such as vibration isolation are needed to guarantee this fixed pose-position relationship(2)
Based on the constraint between the intrinsic and extrinsic camera parameters and the fundamental matrix, the intrinsic camera parameters are real-timely calculated after obtaining the extrinsic camera parameters with high-reliability and high-precision.


## 3. High-Precision Mobile Measurement Method Based on Relay Camera

The principle of the mobile measurement method based on relay camera is shown in Figure3. First, three coordinate systems are established: the world coordinate system W−XWYWZW, the measuring camera coordinate system C−XCYCZC, and the relay camera coordinate system CT−XTYTZT. Then, the motion platform is driven to measure around the outer surface of the structure, and simultaneously, the relay camera real-timely takes picture of the coded mark points on the ground whose positions are unvarying in the world coordinate system. The extrinsic parameters of the camera motion are solved firstly. At the same time, the measuring camera, which is fixed to the relay camera, can take picture of the measured mark points attached to the outer surface of the structure. After obtaining the real-time extrinsic parameters solved based on the relay camera, the real-time intrinsic parameters can be further solved by using the relevant constraints. According to the principle of the videometrics, the real-time measurement of the spatial position of the surface of the structure is completed.Figure 3
Schematic diagram of mobile measurement based on the relay camera: (1) large-scale structure; (2) coded mark point; (3) measured mark point; (4) motion measuring platform; (5) relay camera; (6) measuring camera.


### 3.1. Real-Time Extrinsic Camera Parameter Solution Based on Relay Camera

#### 3.1.1. Real-Time Pose-Position Relationship between the Relay Camera and the World Coordinate System

As shown in Figure3, the position of the target is unvarying in the world coordinate system, and the coordinate of the coded mark point P on the ground is known. The intrinsic parameters of the relay camera can be precalibrated by the videometrics method [15]. Therefore, for the relay camera, the relationship between the feature image point pu,v and the corresponding coordinate PX,Y,Z in the world coordinate system can be expressed by the following homogeneous coordinate equation:
(3)Zcuv1T=KTRWTTWTXYZT.In equation (3), Zc is the projection of the distance from the space point P to the optical center in the direction of the optical axis; RWT and TWT are the pose rotation matrix and the translation matrix between the relay camera coordinate system and the world coordinate system, respectively. RWT and TWT are unknown and can be solved by the pose estimation algorithm [16–18].


#### 3.1.2. Real-Time Pose-Position Relationship between the Measuring Camera and the World Coordinate System

The relay camera and the measuring camera are both fixed on the measuring platform, and their pose-position relationship is stable. When there is interference such as shaking in the measuring platform, the real-time pose-position relationship between the relay camera and the world coordinate system can be solved first. Then, according to the solid-state relationship between the measuring camera coordinate system and the relay camera coordinate system, the real-time extrinsic parameters of the measuring camera can be obtained, and the influence of interference is suppressed.LetXi and Xj be the positional coordinates of a spatial point in the coordinate systems i and j, respectively. Then, the pose-position transformation relationship between the two coordinates can be expressed as
(4)Xi=RjiXj+Tji,where Rji and Tji are the rotation matrix and the translation matrix between the two coordinate systems, respectively.According to equation (4), the relationship between the measuring camera coordinate system and the world coordinate system can be expressed as
(5)XC=RWCXW+TWC.The relationship between the relay camera coordinate system and the world coordinate system and the measuring camera coordinate system can be, respectively, expressed as(6)XT=RWTXW+TWT,XT=RCTXC+TCT.In equation (6), RCT and TCT can be calibrated based on the fixed relationship between the measuring camera and the relay camera. RWT and TWT can be solved by the pose-position estimation algorithm described in Section 3.1.1. Therefore, according to equation (6), the relationship between measuring camera coordinate system and the world coordinate can be expressed by
(7)XC=RCT−1RWTXW+RCT−1TWT−TCT.The extrinsic parameters between the measuring camera coordinate system and the world coordinate system can be gained by comparing equations (5) and (7):
(8)RWC=RCT−1RWT,TWC=RCT−1TWT−TCT.In equation (8), RCT−1 and TCT are determined by the initial pose-position relationship of the measuring platform and can be seen as fixed constants after initial calibration. In contrast, RWT and TWT are varying as long as the movement of the measuring platform. At any time i, they are specifically expressed as Ri,WT and Ti,WT and can be obtained by the pose-position estimation algorithm described in Section 3.1. Therefore, at an arbitrary time i, the real-time pose-position relationship between the camera coordinate system and the world coordinate system can be expressed as
(9)XC=RCT−1Ri,WTXW+RCT−1Ti,WT−TCT.


### 3.2. Intrinsic Camera Parameter Solution Method Based on Fundamental Matrix

When the measuring camera moves, it can measure the same area on the structure at different positions within a certain measuring range. Profiting from this, we firstly solve the fundamental matrix based on the polarity constraint relationship of the same feature points between the two images. Then, we use the constraint relationship between the extrinsic camera parameters and the fundamental matrix, which are solved in real time in Section3.1, to complete the intrinsic camera parameters.


#### 3.2.1. Solution of the Fundamental Matrix

Referring to the opposite geometry theory, the fundamental matrixFij can be gained by using two different images Frame i and Frame j of the same scene (captured from different positions). The fundamental matrix can be described by the following equation [19, 20]:
(10)pjnFijpin=0,where pin=ui,vi,1T and pjn=uj,vj,1T are the homogeneous coordinates of the point PnXpn,Ypn,Zpn in the image i and image j, respectively. The fundamental matrix has a dimension of 3×3, a rank of 2, and a freedom degree of 7.For the solution of the fundamental matrix, Hartley [21] proposed the classical eight-point algorithm based on the practical application. The eight-point algorithm is simple and easy to implement. However, experiments indicate that when there is noise in the image; this method is not easy to get the accurate solution [22]. Besides, the distortion in the image may cause solution error of the fundamental matrix and affect the solution of the subsequent camera parameters.To effectively improve the recognizing and matching accuracy of the corresponding points, we first calibrate the distortion based on the linear characteristics existing in the measurement scene by using the method in [23, 24]. Then, the robust Random Sample Consensus (RANSAC) algorithm [25] is utilized to perform the feature point matching between the two images. This strategy eliminates the mismatching in the presence of noise, thereby ensuring the solution accuracy of the fundamental matrix.


#### 3.2.2. Solution of the Intrinsic Camera Parameters

Assuming that the corresponding homogeneous coordinates of the points on the target areXT,i and XT,j at the two adjacent moments ti and tj, respectively, then the pose-position relationship between the relay camera coordinate system and the world coordinate system can be expressed as
(11)XT,i=RWT,iXW+TWT,i,XT,j=RWT,jXW+TWT,j.According to equation (12), the pose-position transformation between the camera coordinate systems at the two adjacent moments ti and tj is
(12)XT,j=RWT,jRWT,i−1XT,i−RWT,jRWT,i−1TWT,i+TWT,j.Combining the pose-position relationship between the relay camera coordinate system and the measuring camera coordinate systemRTC,TTC, the pose-position transformation of the measuring camera at the two adjacent moments ti and tj is
(13)XC,j=RC,ijXC,i+TC,ij,which satisfies
(14)RC,ij=RTC−1RWT,jRWT,i−1RTC,TC,ij=RTC−1RWT,jRWT,i−1TTC‐TWT,i+RTC−1TWT,j‐TTC.In the self-calibration of the camera, the following equation is commonly used to solve the fundamental matrix [26, 27]:
(15)Fij=λKj−TTC,ij×RC,ijKi−1,where
(16)TC,ij×=0−t3t2t30−t1−t2t10,is the antisymmetric matrix of the translation matrix t⇀j=t1,t2,t3T.It can be seen from equations (10) and (14) that the fundamental matrix Fij in equation (15) and the pose-position relationship of the measuring camera at the two adjacent moments ti and tjRC,ij,TC,ij can be figured out. Thus, only the intrinsic camera parameters Ki,Kj and the constant factor λ are unknown. Equation (15) can be transformed to
(17)KjTFijKi=λTC,ij×RC,ij.If we setB=TC,ij×RC,ij in equation (17), referring to equation (14), B can be real-timely solved after acquiring RC,ij,TC,ij. Therefore, we have
(18)fxj00sjfyj0CxjCyj1f11f12f13f21f22f23f31f32f33fxisiCxi0fyiCyi001=λb11b12b13b21b22b23b31b32b33.The initial calibration parameter can be considered as an initial value for iterative solution. Therefore, the intrinsic parameterKi can be seen as known in equation (17) for an iterative method. With the fast development of the camera hardware technology, the intrinsic parameter s is very close to the ideal value [22, 28] 0 in most digital camera models. Therefore, we can set s=0. Then, we have
(19)f1p=fxifxjf11−λb11=0;f2p=fxjfyif12−λb12=0;f3p=fxjf11Cxi+fxjf12Cyi+fxjf13−λb13=0;f4p=fyjf21fxi−λb21=0;f5p=fyjf22fyi−λb22=0;f6p=fyjf21Cx+fyjf22Cy+fyjf23−λb23=0;f7p=f11Cxj+f21Cyj+f31fxi−λb31=0;f8p=f12Cxj+f22Cyj+f32fyi−λb32=0;f9p=f11Cxj+f21Cyj+f31Cxi+f12Cxj+f22Cyj+f32Cyi+f13Cxj+f23Cyj+f33−λb33=0.In equation (19), p=fxj,fyj,Cx,Cy,λT is the parameter vector to be solved. In the process of real-time monitoring using mobile cameras, it is often necessary to perform high-precision initial calibration of the camera parameters to ensure the reliability of the whole monitoring process. With equations (17) and (19), we can gain the initial approximate solution of the intrinsic parameter Kj0 and the constant factor λ0. Our goal is to find fxj,fyj,Cx,Cy,λ.In order to ensure the accuracy and convergence of the solution process, nonlinear optimization method can be used for iterative solution (the LM optimization algorithm [29, 30] is used in this paper). Supposing the solution at iterative step k is pk, we have pk+1=pk+δp and
(20)δp=JTJ+μE−1JTε.In equation (20), E is identity matrix, and J is the Jacobi matrix at pk as follows
(21)J=∂f1∂fxi∂f1∂fxj∂f1Cxi∂f1Cyi∂f1λ⋮⋮⋮⋮⋮∂f9∂fxi∂f9∂fxj∂f9Cxi∂f9Cyi∂f9λ.ε=−f1pk⋯−f9pkT is the residual error vector at pk, and μ is damping ratio. For the sake of simplicity, we suggest referring to [29, 30] for finding the update strategy of μ. After achieving δp, we can use pk+1=pk+δp to update p until δp is less than a predetermined value such as 10-4.


### 3.3. The Real-Time Mobile Measurement Process of the Measuring Camera

By the solution of the extrinsic and intrinsic camera parameters in Section3.1 and Section 3.2, we can real-timely get the camera parameters and the pose-position information of the measured structure. By detecting the same area at different time, the variation of the pose-position information of the area can be obtained, and then, the deformation monitoring of the measured structure part is completed. The specific measurement process is as follows:Step 1.
Before the measurement, the initial value of the intrinsic measuring camera parameterK0, the intrinsic relay camera parameter KT, and the constant pose-position conversion relationship RCT and TCT between the measuring camera and the relay camera are calibrated first.Step 2.
The coded mark points with known spatial information are measured with high-precision by the relay camera, and the real-time pose-position parametersRWT and TWT of the moving relay camera are obtained.Step 3.
With the parametersRCT, TCT, RWT, and TWT, the real-time pose-position of the measuring camera related to the world coordinate system can be calculated by Ri,WC=RCT−1Ri,WT and Ti,WC=RCT−1Ti,WT−TCT.Step 4.
After obtaining two different images Framei and Frame j of the same scene at the adjacent moments ti and tj, the fundamental matrix Fij can be figured out by the method in Section 3.2.1. The pose-position transfer relationship RC,ij and TC,ij can be gained by the method in Section 3.2.2. With the intrinsic camera parameter Ki at the moment i, the intrinsic camera parameter Kj can be iteratively calculated, and then, the precision of the following measurement is guaranteed.Step 5.
After obtaining the intrinsic and extrinsic parameters of the measuring camera, with the bundle adjustment method in [4], the coordinate PXW,YW,ZW of the measured mark point in the world coordinate system can be gained with high-precision by the common coded mark points between different images acquired at the adjacent moments.Step 6.
In the measurement process, Step 3, Step 4, and Step 5 are repeated, and the coordinates of the measured mark points in the world coordinate system are gained at different moments. Then, the real-time deformation measurement of the structure is completed.


## 3.1. Real-Time Extrinsic Camera Parameter Solution Based on Relay Camera

### 3.1.1. Real-Time Pose-Position Relationship between the Relay Camera and the World Coordinate System

As shown in Figure3, the position of the target is unvarying in the world coordinate system, and the coordinate of the coded mark point P on the ground is known. The intrinsic parameters of the relay camera can be precalibrated by the videometrics method [15]. Therefore, for the relay camera, the relationship between the feature image point pu,v and the corresponding coordinate PX,Y,Z in the world coordinate system can be expressed by the following homogeneous coordinate equation:
(3)Zcuv1T=KTRWTTWTXYZT.In equation (3), Zc is the projection of the distance from the space point P to the optical center in the direction of the optical axis; RWT and TWT are the pose rotation matrix and the translation matrix between the relay camera coordinate system and the world coordinate system, respectively. RWT and TWT are unknown and can be solved by the pose estimation algorithm [16–18].


### 3.1.2. Real-Time Pose-Position Relationship between the Measuring Camera and the World Coordinate System

The relay camera and the measuring camera are both fixed on the measuring platform, and their pose-position relationship is stable. When there is interference such as shaking in the measuring platform, the real-time pose-position relationship between the relay camera and the world coordinate system can be solved first. Then, according to the solid-state relationship between the measuring camera coordinate system and the relay camera coordinate system, the real-time extrinsic parameters of the measuring camera can be obtained, and the influence of interference is suppressed.LetXi and Xj be the positional coordinates of a spatial point in the coordinate systems i and j, respectively. Then, the pose-position transformation relationship between the two coordinates can be expressed as
(4)Xi=RjiXj+Tji,where Rji and Tji are the rotation matrix and the translation matrix between the two coordinate systems, respectively.According to equation (4), the relationship between the measuring camera coordinate system and the world coordinate system can be expressed as
(5)XC=RWCXW+TWC.The relationship between the relay camera coordinate system and the world coordinate system and the measuring camera coordinate system can be, respectively, expressed as(6)XT=RWTXW+TWT,XT=RCTXC+TCT.In equation (6), RCT and TCT can be calibrated based on the fixed relationship between the measuring camera and the relay camera. RWT and TWT can be solved by the pose-position estimation algorithm described in Section 3.1.1. Therefore, according to equation (6), the relationship between measuring camera coordinate system and the world coordinate can be expressed by
(7)XC=RCT−1RWTXW+RCT−1TWT−TCT.The extrinsic parameters between the measuring camera coordinate system and the world coordinate system can be gained by comparing equations (5) and (7):
(8)RWC=RCT−1RWT,TWC=RCT−1TWT−TCT.In equation (8), RCT−1 and TCT are determined by the initial pose-position relationship of the measuring platform and can be seen as fixed constants after initial calibration. In contrast, RWT and TWT are varying as long as the movement of the measuring platform. At any time i, they are specifically expressed as Ri,WT and Ti,WT and can be obtained by the pose-position estimation algorithm described in Section 3.1. Therefore, at an arbitrary time i, the real-time pose-position relationship between the camera coordinate system and the world coordinate system can be expressed as
(9)XC=RCT−1Ri,WTXW+RCT−1Ti,WT−TCT.


## 3.1.1. Real-Time Pose-Position Relationship between the Relay Camera and the World Coordinate System

As shown in Figure3, the position of the target is unvarying in the world coordinate system, and the coordinate of the coded mark point P on the ground is known. The intrinsic parameters of the relay camera can be precalibrated by the videometrics method [15]. Therefore, for the relay camera, the relationship between the feature image point pu,v and the corresponding coordinate PX,Y,Z in the world coordinate system can be expressed by the following homogeneous coordinate equation:
(3)Zcuv1T=KTRWTTWTXYZT.In equation (3), Zc is the projection of the distance from the space point P to the optical center in the direction of the optical axis; RWT and TWT are the pose rotation matrix and the translation matrix between the relay camera coordinate system and the world coordinate system, respectively. RWT and TWT are unknown and can be solved by the pose estimation algorithm [16–18].


## 3.1.2. Real-Time Pose-Position Relationship between the Measuring Camera and the World Coordinate System

The relay camera and the measuring camera are both fixed on the measuring platform, and their pose-position relationship is stable. When there is interference such as shaking in the measuring platform, the real-time pose-position relationship between the relay camera and the world coordinate system can be solved first. Then, according to the solid-state relationship between the measuring camera coordinate system and the relay camera coordinate system, the real-time extrinsic parameters of the measuring camera can be obtained, and the influence of interference is suppressed.LetXi and Xj be the positional coordinates of a spatial point in the coordinate systems i and j, respectively. Then, the pose-position transformation relationship between the two coordinates can be expressed as
(4)Xi=RjiXj+Tji,where Rji and Tji are the rotation matrix and the translation matrix between the two coordinate systems, respectively.According to equation (4), the relationship between the measuring camera coordinate system and the world coordinate system can be expressed as
(5)XC=RWCXW+TWC.The relationship between the relay camera coordinate system and the world coordinate system and the measuring camera coordinate system can be, respectively, expressed as(6)XT=RWTXW+TWT,XT=RCTXC+TCT.In equation (6), RCT and TCT can be calibrated based on the fixed relationship between the measuring camera and the relay camera. RWT and TWT can be solved by the pose-position estimation algorithm described in Section 3.1.1. Therefore, according to equation (6), the relationship between measuring camera coordinate system and the world coordinate can be expressed by
(7)XC=RCT−1RWTXW+RCT−1TWT−TCT.The extrinsic parameters between the measuring camera coordinate system and the world coordinate system can be gained by comparing equations (5) and (7):
(8)RWC=RCT−1RWT,TWC=RCT−1TWT−TCT.In equation (8), RCT−1 and TCT are determined by the initial pose-position relationship of the measuring platform and can be seen as fixed constants after initial calibration. In contrast, RWT and TWT are varying as long as the movement of the measuring platform. At any time i, they are specifically expressed as Ri,WT and Ti,WT and can be obtained by the pose-position estimation algorithm described in Section 3.1. Therefore, at an arbitrary time i, the real-time pose-position relationship between the camera coordinate system and the world coordinate system can be expressed as
(9)XC=RCT−1Ri,WTXW+RCT−1Ti,WT−TCT.


## 3.2. Intrinsic Camera Parameter Solution Method Based on Fundamental Matrix

When the measuring camera moves, it can measure the same area on the structure at different positions within a certain measuring range. Profiting from this, we firstly solve the fundamental matrix based on the polarity constraint relationship of the same feature points between the two images. Then, we use the constraint relationship between the extrinsic camera parameters and the fundamental matrix, which are solved in real time in Section3.1, to complete the intrinsic camera parameters.


### 3.2.1. Solution of the Fundamental Matrix

Referring to the opposite geometry theory, the fundamental matrixFij can be gained by using two different images Frame i and Frame j of the same scene (captured from different positions). The fundamental matrix can be described by the following equation [19, 20]:
(10)pjnFijpin=0,where pin=ui,vi,1T and pjn=uj,vj,1T are the homogeneous coordinates of the point PnXpn,Ypn,Zpn in the image i and image j, respectively. The fundamental matrix has a dimension of 3×3, a rank of 2, and a freedom degree of 7.For the solution of the fundamental matrix, Hartley [21] proposed the classical eight-point algorithm based on the practical application. The eight-point algorithm is simple and easy to implement. However, experiments indicate that when there is noise in the image; this method is not easy to get the accurate solution [22]. Besides, the distortion in the image may cause solution error of the fundamental matrix and affect the solution of the subsequent camera parameters.To effectively improve the recognizing and matching accuracy of the corresponding points, we first calibrate the distortion based on the linear characteristics existing in the measurement scene by using the method in [23, 24]. Then, the robust Random Sample Consensus (RANSAC) algorithm [25] is utilized to perform the feature point matching between the two images. This strategy eliminates the mismatching in the presence of noise, thereby ensuring the solution accuracy of the fundamental matrix.


### 3.2.2. Solution of the Intrinsic Camera Parameters

Assuming that the corresponding homogeneous coordinates of the points on the target areXT,i and XT,j at the two adjacent moments ti and tj, respectively, then the pose-position relationship between the relay camera coordinate system and the world coordinate system can be expressed as
(11)XT,i=RWT,iXW+TWT,i,XT,j=RWT,jXW+TWT,j.According to equation (12), the pose-position transformation between the camera coordinate systems at the two adjacent moments ti and tj is
(12)XT,j=RWT,jRWT,i−1XT,i−RWT,jRWT,i−1TWT,i+TWT,j.Combining the pose-position relationship between the relay camera coordinate system and the measuring camera coordinate systemRTC,TTC, the pose-position transformation of the measuring camera at the two adjacent moments ti and tj is
(13)XC,j=RC,ijXC,i+TC,ij,which satisfies
(14)RC,ij=RTC−1RWT,jRWT,i−1RTC,TC,ij=RTC−1RWT,jRWT,i−1TTC‐TWT,i+RTC−1TWT,j‐TTC.In the self-calibration of the camera, the following equation is commonly used to solve the fundamental matrix [26, 27]:
(15)Fij=λKj−TTC,ij×RC,ijKi−1,where
(16)TC,ij×=0−t3t2t30−t1−t2t10,is the antisymmetric matrix of the translation matrix t⇀j=t1,t2,t3T.It can be seen from equations (10) and (14) that the fundamental matrix Fij in equation (15) and the pose-position relationship of the measuring camera at the two adjacent moments ti and tjRC,ij,TC,ij can be figured out. Thus, only the intrinsic camera parameters Ki,Kj and the constant factor λ are unknown. Equation (15) can be transformed to
(17)KjTFijKi=λTC,ij×RC,ij.If we setB=TC,ij×RC,ij in equation (17), referring to equation (14), B can be real-timely solved after acquiring RC,ij,TC,ij. Therefore, we have
(18)fxj00sjfyj0CxjCyj1f11f12f13f21f22f23f31f32f33fxisiCxi0fyiCyi001=λb11b12b13b21b22b23b31b32b33.The initial calibration parameter can be considered as an initial value for iterative solution. Therefore, the intrinsic parameterKi can be seen as known in equation (17) for an iterative method. With the fast development of the camera hardware technology, the intrinsic parameter s is very close to the ideal value [22, 28] 0 in most digital camera models. Therefore, we can set s=0. Then, we have
(19)f1p=fxifxjf11−λb11=0;f2p=fxjfyif12−λb12=0;f3p=fxjf11Cxi+fxjf12Cyi+fxjf13−λb13=0;f4p=fyjf21fxi−λb21=0;f5p=fyjf22fyi−λb22=0;f6p=fyjf21Cx+fyjf22Cy+fyjf23−λb23=0;f7p=f11Cxj+f21Cyj+f31fxi−λb31=0;f8p=f12Cxj+f22Cyj+f32fyi−λb32=0;f9p=f11Cxj+f21Cyj+f31Cxi+f12Cxj+f22Cyj+f32Cyi+f13Cxj+f23Cyj+f33−λb33=0.In equation (19), p=fxj,fyj,Cx,Cy,λT is the parameter vector to be solved. In the process of real-time monitoring using mobile cameras, it is often necessary to perform high-precision initial calibration of the camera parameters to ensure the reliability of the whole monitoring process. With equations (17) and (19), we can gain the initial approximate solution of the intrinsic parameter Kj0 and the constant factor λ0. Our goal is to find fxj,fyj,Cx,Cy,λ.In order to ensure the accuracy and convergence of the solution process, nonlinear optimization method can be used for iterative solution (the LM optimization algorithm [29, 30] is used in this paper). Supposing the solution at iterative step k is pk, we have pk+1=pk+δp and
(20)δp=JTJ+μE−1JTε.In equation (20), E is identity matrix, and J is the Jacobi matrix at pk as follows
(21)J=∂f1∂fxi∂f1∂fxj∂f1Cxi∂f1Cyi∂f1λ⋮⋮⋮⋮⋮∂f9∂fxi∂f9∂fxj∂f9Cxi∂f9Cyi∂f9λ.ε=−f1pk⋯−f9pkT is the residual error vector at pk, and μ is damping ratio. For the sake of simplicity, we suggest referring to [29, 30] for finding the update strategy of μ. After achieving δp, we can use pk+1=pk+δp to update p until δp is less than a predetermined value such as 10-4.


## 3.2.1. Solution of the Fundamental Matrix

Referring to the opposite geometry theory, the fundamental matrixFij can be gained by using two different images Frame i and Frame j of the same scene (captured from different positions). The fundamental matrix can be described by the following equation [19, 20]:
(10)pjnFijpin=0,where pin=ui,vi,1T and pjn=uj,vj,1T are the homogeneous coordinates of the point PnXpn,Ypn,Zpn in the image i and image j, respectively. The fundamental matrix has a dimension of 3×3, a rank of 2, and a freedom degree of 7.For the solution of the fundamental matrix, Hartley [21] proposed the classical eight-point algorithm based on the practical application. The eight-point algorithm is simple and easy to implement. However, experiments indicate that when there is noise in the image; this method is not easy to get the accurate solution [22]. Besides, the distortion in the image may cause solution error of the fundamental matrix and affect the solution of the subsequent camera parameters.To effectively improve the recognizing and matching accuracy of the corresponding points, we first calibrate the distortion based on the linear characteristics existing in the measurement scene by using the method in [23, 24]. Then, the robust Random Sample Consensus (RANSAC) algorithm [25] is utilized to perform the feature point matching between the two images. This strategy eliminates the mismatching in the presence of noise, thereby ensuring the solution accuracy of the fundamental matrix.


## 3.2.2. Solution of the Intrinsic Camera Parameters

Assuming that the corresponding homogeneous coordinates of the points on the target areXT,i and XT,j at the two adjacent moments ti and tj, respectively, then the pose-position relationship between the relay camera coordinate system and the world coordinate system can be expressed as
(11)XT,i=RWT,iXW+TWT,i,XT,j=RWT,jXW+TWT,j.According to equation (12), the pose-position transformation between the camera coordinate systems at the two adjacent moments ti and tj is
(12)XT,j=RWT,jRWT,i−1XT,i−RWT,jRWT,i−1TWT,i+TWT,j.Combining the pose-position relationship between the relay camera coordinate system and the measuring camera coordinate systemRTC,TTC, the pose-position transformation of the measuring camera at the two adjacent moments ti and tj is
(13)XC,j=RC,ijXC,i+TC,ij,which satisfies
(14)RC,ij=RTC−1RWT,jRWT,i−1RTC,TC,ij=RTC−1RWT,jRWT,i−1TTC‐TWT,i+RTC−1TWT,j‐TTC.In the self-calibration of the camera, the following equation is commonly used to solve the fundamental matrix [26, 27]:
(15)Fij=λKj−TTC,ij×RC,ijKi−1,where
(16)TC,ij×=0−t3t2t30−t1−t2t10,is the antisymmetric matrix of the translation matrix t⇀j=t1,t2,t3T.It can be seen from equations (10) and (14) that the fundamental matrix Fij in equation (15) and the pose-position relationship of the measuring camera at the two adjacent moments ti and tjRC,ij,TC,ij can be figured out. Thus, only the intrinsic camera parameters Ki,Kj and the constant factor λ are unknown. Equation (15) can be transformed to
(17)KjTFijKi=λTC,ij×RC,ij.If we setB=TC,ij×RC,ij in equation (17), referring to equation (14), B can be real-timely solved after acquiring RC,ij,TC,ij. Therefore, we have
(18)fxj00sjfyj0CxjCyj1f11f12f13f21f22f23f31f32f33fxisiCxi0fyiCyi001=λb11b12b13b21b22b23b31b32b33.The initial calibration parameter can be considered as an initial value for iterative solution. Therefore, the intrinsic parameterKi can be seen as known in equation (17) for an iterative method. With the fast development of the camera hardware technology, the intrinsic parameter s is very close to the ideal value [22, 28] 0 in most digital camera models. Therefore, we can set s=0. Then, we have
(19)f1p=fxifxjf11−λb11=0;f2p=fxjfyif12−λb12=0;f3p=fxjf11Cxi+fxjf12Cyi+fxjf13−λb13=0;f4p=fyjf21fxi−λb21=0;f5p=fyjf22fyi−λb22=0;f6p=fyjf21Cx+fyjf22Cy+fyjf23−λb23=0;f7p=f11Cxj+f21Cyj+f31fxi−λb31=0;f8p=f12Cxj+f22Cyj+f32fyi−λb32=0;f9p=f11Cxj+f21Cyj+f31Cxi+f12Cxj+f22Cyj+f32Cyi+f13Cxj+f23Cyj+f33−λb33=0.In equation (19), p=fxj,fyj,Cx,Cy,λT is the parameter vector to be solved. In the process of real-time monitoring using mobile cameras, it is often necessary to perform high-precision initial calibration of the camera parameters to ensure the reliability of the whole monitoring process. With equations (17) and (19), we can gain the initial approximate solution of the intrinsic parameter Kj0 and the constant factor λ0. Our goal is to find fxj,fyj,Cx,Cy,λ.In order to ensure the accuracy and convergence of the solution process, nonlinear optimization method can be used for iterative solution (the LM optimization algorithm [29, 30] is used in this paper). Supposing the solution at iterative step k is pk, we have pk+1=pk+δp and
(20)δp=JTJ+μE−1JTε.In equation (20), E is identity matrix, and J is the Jacobi matrix at pk as follows
(21)J=∂f1∂fxi∂f1∂fxj∂f1Cxi∂f1Cyi∂f1λ⋮⋮⋮⋮⋮∂f9∂fxi∂f9∂fxj∂f9Cxi∂f9Cyi∂f9λ.ε=−f1pk⋯−f9pkT is the residual error vector at pk, and μ is damping ratio. For the sake of simplicity, we suggest referring to [29, 30] for finding the update strategy of μ. After achieving δp, we can use pk+1=pk+δp to update p until δp is less than a predetermined value such as 10-4.


## 3.3. The Real-Time Mobile Measurement Process of the Measuring Camera

By the solution of the extrinsic and intrinsic camera parameters in Section3.1 and Section 3.2, we can real-timely get the camera parameters and the pose-position information of the measured structure. By detecting the same area at different time, the variation of the pose-position information of the area can be obtained, and then, the deformation monitoring of the measured structure part is completed. The specific measurement process is as follows:Step 1.
Before the measurement, the initial value of the intrinsic measuring camera parameterK0, the intrinsic relay camera parameter KT, and the constant pose-position conversion relationship RCT and TCT between the measuring camera and the relay camera are calibrated first.Step 2.
The coded mark points with known spatial information are measured with high-precision by the relay camera, and the real-time pose-position parametersRWT and TWT of the moving relay camera are obtained.Step 3.
With the parametersRCT, TCT, RWT, and TWT, the real-time pose-position of the measuring camera related to the world coordinate system can be calculated by Ri,WC=RCT−1Ri,WT and Ti,WC=RCT−1Ti,WT−TCT.Step 4.
After obtaining two different images Framei and Frame j of the same scene at the adjacent moments ti and tj, the fundamental matrix Fij can be figured out by the method in Section 3.2.1. The pose-position transfer relationship RC,ij and TC,ij can be gained by the method in Section 3.2.2. With the intrinsic camera parameter Ki at the moment i, the intrinsic camera parameter Kj can be iteratively calculated, and then, the precision of the following measurement is guaranteed.Step 5.
After obtaining the intrinsic and extrinsic parameters of the measuring camera, with the bundle adjustment method in [4], the coordinate PXW,YW,ZW of the measured mark point in the world coordinate system can be gained with high-precision by the common coded mark points between different images acquired at the adjacent moments.Step 6.
In the measurement process, Step 3, Step 4, and Step 5 are repeated, and the coordinates of the measured mark points in the world coordinate system are gained at different moments. Then, the real-time deformation measurement of the structure is completed.


## 4. Experimental Results

### 4.1. Verification Experiment

Based on the above theoretical analysis, in order to complete the real-time deformation monitoring of large cylindrical structure, we need to set circular measure orbit around the structure. However, considering the limit of the experimental conditions, we replace the circular orbit with the parallel orbit, and this setting is adequate for the experimental validation. The images shown in Figure4 are the measuring platform in the actual experiment and some of the pictures captured during the measuring process. The relay camera and measuring camera are fixed on the motion measuring platform. The movement of the platform is controlled by the output of the torque motor. The cameras used in the experiment are microview MVC14KSAC-GE6 with a resolution of 4384pixels×3288pixels, and the lenses are the C3516-M produced by Pentax. The intrinsic parameters of the cameras and the relative pose-position relationship between the cameras are calibrated with high-precision aforehand.Figure 4
(a) Experimental measuring device. (b) Pictures captured by relay camera. (c) Pictures captured by measuring camera.
(a)(b)(c)


#### 4.1.1. Extrinsic Parameter Acquisition Accuracy Experiment

In order to verify that the proposed method can effectively overcome the effects of environmental vibration and can real-timely solve the intrinsic and extrinsic parameters, the variable-control method is utilized in the comparing measurement experiments of the intrinsic and extrinsic parameters. In order to effectively verify the acquisition accuracy of the extrinsic parameters, we first use the precalibrated intrinsic parameters and ignore the influence of the intrinsic parameter variation in the comparing experiment. We, respectively, use the solution parameters of the relay camera and the rotation parameters of the torque motor to solve the extrinsic parameters, including three pose anglesAx, Ay, and Az and three translation quantities Tx, Ty, and Tz. The experiment results are shown in Table 1. After obtaining the intrinsic and extrinsic parameters, we figure out the coordinates of the 20 measured mark points on the surface of the structure and then compare them with the actual coordinate information to obtain the distance deviation between the measured coordinate and the actual standard coordinate (as shown in Figure 5(a)).Table 1
Measured data of the extrinsic parameters.
External parametersFrame 1Frame 2Frame 3By relay cameraBy torque motorBy relay cameraBy torque motorBy relay cameraBy torque motorAx,Ay,Az/″(7.2, 3.5, 28.8)(14.0, 7.2, 32.4)(10.8, 3.6, 52.6)Tx,Ty,Tz/mm(0.07, 49.97, 0)(0, 50.0, 0)(0.04, 99.94, 0.01)(0, 100.0, 0)(0.03, 149.95, 0.01)(0, 150.0, 0)Figure 5
Comparison of the extrinsic and intrinsic parameter solution error: (a) extrinsic parameter solution error; (b) intrinsic parameter solution error.
(a)(b)We can learn from the experiment result that, when the extrinsic parameters are figured out by using the relay camera, the coordinate error of the measured mark points is not more than 2.8 mm. The minimum value is 1.5 mm, and the average error is 2.029 mm. In contrast, when the extrinsic parameters are figured out by using torque motor rotation parameters, the maximum coordinate error of the measured mark points is 3.5 mm, the minimum value is 2.2 mm, and the average error is 2.586 mm. From the experimental result, we can find that, when the intrinsic camera parameters are unified to the initial calibration value, the proposed extrinsic parameter solution method using the relay camera can effectively improve the measuring accuracy of the measured mark points.


#### 4.1.2. Intrinsic Parameter Acquisition Accuracy Experiment

In order to verify the acquisition accuracy of the intrinsic parameters, we set the extrinsic camera parameters to the values obtained by using the relay camera in the comparing experiment. Then, we use the initial calibrated intrinsic parameters and the real-time intrinsic parameters achieved by the method in 2.2 for comparison. The comparing results are shown in Table2. After obtaining the intrinsic parameters, we calculate the coordinates of the 20 measured mark points on the structure surface and scale the measuring accuracy by the coordinate error of the measured mark points as demonstrated in Figure 5(b).Table 2
Measured data of the intrinsic parameters.
Initial valueFrame 1Frame 2Frame 3fx,fy(22201.81, 22395.01)(22947.64, 22647.23)(22753.21, 22486.35)(23053.61, 22931.02)Cx,Cy(1957.36, 1405.73)(1982.66, 1485.21)(2031.43, 1501.82)(1998.36, 1465.73)From the experimental results we learn that, when the extrinsic parameters are consistent through the experiment, the coordinate error of the measured mark points of the proposed method is not more than 2.31 mm. The minimum value is 1.38 mm, and the average error is 1.78 mm. In contrast, when only the initial calibrated intrinsic parameters are used, the maximum coordinate error of the measured mark points is 2.74 mm, the minimum value is 1.58 mm, and the average error is 2.087 mm. It is learned from the experimental result that the measuring accuracy of the measured mark points by using the real-time intrinsic parameters is higher than that by using the initial calibrated intrinsic parameters.


### 4.2. Accuracy Analysis

From the verification experiment in Section4.1, we learn that the proposed method utilizing the relay camera can not only effectively improve the accuracy of the extrinsic parameters but also achieve the real-time intrinsic parameters. To further verify the proposed method meets the request of the real-time on-line deformation monitoring for large-scale structure, we design the following experiment. We adopt the proposed method in the on-line monitoring for a metal structure surface, which contains 60 measured mark points. The measuring results of the proposed method are compared to the results of the measuring with electronic total station. The detailed comparison is shown in Figure 6.Figure 6
(a) Measuring result of measured mark point coordinates. (b) Measuring error of measured mark point coordinates.
(a)(b)From the coordinate error of the measured mark points shown in Figure6(b) we can learn that the maximum, minimum, mean, and standard errors of the measured mark point coordinate are 2.38 mm, 1.35 mm, 1.797 mm, and 0.2156 mm, respectively. The mean of the coordinate measuring error is less than 1.8 mm, and the standard error is less than 0.25 mm. The coordinate accuracy of the measured mark points obtained by on-line measuring is similar to the accuracy of the verification experiment in Section 4.1. This indicates the efficiency and stability of the proposed method. The proposed method using relay camera can achieve the intrinsic and extrinsic parameters of the measuring camera with high precision and can meet the demand of the on-line deformation monitoring of the structure.


## 4.1. Verification Experiment

Based on the above theoretical analysis, in order to complete the real-time deformation monitoring of large cylindrical structure, we need to set circular measure orbit around the structure. However, considering the limit of the experimental conditions, we replace the circular orbit with the parallel orbit, and this setting is adequate for the experimental validation. The images shown in Figure4 are the measuring platform in the actual experiment and some of the pictures captured during the measuring process. The relay camera and measuring camera are fixed on the motion measuring platform. The movement of the platform is controlled by the output of the torque motor. The cameras used in the experiment are microview MVC14KSAC-GE6 with a resolution of 4384pixels×3288pixels, and the lenses are the C3516-M produced by Pentax. The intrinsic parameters of the cameras and the relative pose-position relationship between the cameras are calibrated with high-precision aforehand.Figure 4
(a) Experimental measuring device. (b) Pictures captured by relay camera. (c) Pictures captured by measuring camera.
(a)(b)(c)


### 4.1.1. Extrinsic Parameter Acquisition Accuracy Experiment

In order to verify that the proposed method can effectively overcome the effects of environmental vibration and can real-timely solve the intrinsic and extrinsic parameters, the variable-control method is utilized in the comparing measurement experiments of the intrinsic and extrinsic parameters. In order to effectively verify the acquisition accuracy of the extrinsic parameters, we first use the precalibrated intrinsic parameters and ignore the influence of the intrinsic parameter variation in the comparing experiment. We, respectively, use the solution parameters of the relay camera and the rotation parameters of the torque motor to solve the extrinsic parameters, including three pose anglesAx, Ay, and Az and three translation quantities Tx, Ty, and Tz. The experiment results are shown in Table 1. After obtaining the intrinsic and extrinsic parameters, we figure out the coordinates of the 20 measured mark points on the surface of the structure and then compare them with the actual coordinate information to obtain the distance deviation between the measured coordinate and the actual standard coordinate (as shown in Figure 5(a)).Table 1
Measured data of the extrinsic parameters.
External parametersFrame 1Frame 2Frame 3By relay cameraBy torque motorBy relay cameraBy torque motorBy relay cameraBy torque motorAx,Ay,Az/″(7.2, 3.5, 28.8)(14.0, 7.2, 32.4)(10.8, 3.6, 52.6)Tx,Ty,Tz/mm(0.07, 49.97, 0)(0, 50.0, 0)(0.04, 99.94, 0.01)(0, 100.0, 0)(0.03, 149.95, 0.01)(0, 150.0, 0)Figure 5
Comparison of the extrinsic and intrinsic parameter solution error: (a) extrinsic parameter solution error; (b) intrinsic parameter solution error.
(a)(b)We can learn from the experiment result that, when the extrinsic parameters are figured out by using the relay camera, the coordinate error of the measured mark points is not more than 2.8 mm. The minimum value is 1.5 mm, and the average error is 2.029 mm. In contrast, when the extrinsic parameters are figured out by using torque motor rotation parameters, the maximum coordinate error of the measured mark points is 3.5 mm, the minimum value is 2.2 mm, and the average error is 2.586 mm. From the experimental result, we can find that, when the intrinsic camera parameters are unified to the initial calibration value, the proposed extrinsic parameter solution method using the relay camera can effectively improve the measuring accuracy of the measured mark points.


### 4.1.2. Intrinsic Parameter Acquisition Accuracy Experiment

In order to verify the acquisition accuracy of the intrinsic parameters, we set the extrinsic camera parameters to the values obtained by using the relay camera in the comparing experiment. Then, we use the initial calibrated intrinsic parameters and the real-time intrinsic parameters achieved by the method in 2.2 for comparison. The comparing results are shown in Table2. After obtaining the intrinsic parameters, we calculate the coordinates of the 20 measured mark points on the structure surface and scale the measuring accuracy by the coordinate error of the measured mark points as demonstrated in Figure 5(b).Table 2
Measured data of the intrinsic parameters.
Initial valueFrame 1Frame 2Frame 3fx,fy(22201.81, 22395.01)(22947.64, 22647.23)(22753.21, 22486.35)(23053.61, 22931.02)Cx,Cy(1957.36, 1405.73)(1982.66, 1485.21)(2031.43, 1501.82)(1998.36, 1465.73)From the experimental results we learn that, when the extrinsic parameters are consistent through the experiment, the coordinate error of the measured mark points of the proposed method is not more than 2.31 mm. The minimum value is 1.38 mm, and the average error is 1.78 mm. In contrast, when only the initial calibrated intrinsic parameters are used, the maximum coordinate error of the measured mark points is 2.74 mm, the minimum value is 1.58 mm, and the average error is 2.087 mm. It is learned from the experimental result that the measuring accuracy of the measured mark points by using the real-time intrinsic parameters is higher than that by using the initial calibrated intrinsic parameters.


## 4.1.1. Extrinsic Parameter Acquisition Accuracy Experiment

In order to verify that the proposed method can effectively overcome the effects of environmental vibration and can real-timely solve the intrinsic and extrinsic parameters, the variable-control method is utilized in the comparing measurement experiments of the intrinsic and extrinsic parameters. In order to effectively verify the acquisition accuracy of the extrinsic parameters, we first use the precalibrated intrinsic parameters and ignore the influence of the intrinsic parameter variation in the comparing experiment. We, respectively, use the solution parameters of the relay camera and the rotation parameters of the torque motor to solve the extrinsic parameters, including three pose anglesAx, Ay, and Az and three translation quantities Tx, Ty, and Tz. The experiment results are shown in Table 1. After obtaining the intrinsic and extrinsic parameters, we figure out the coordinates of the 20 measured mark points on the surface of the structure and then compare them with the actual coordinate information to obtain the distance deviation between the measured coordinate and the actual standard coordinate (as shown in Figure 5(a)).Table 1
Measured data of the extrinsic parameters.
External parametersFrame 1Frame 2Frame 3By relay cameraBy torque motorBy relay cameraBy torque motorBy relay cameraBy torque motorAx,Ay,Az/″(7.2, 3.5, 28.8)(14.0, 7.2, 32.4)(10.8, 3.6, 52.6)Tx,Ty,Tz/mm(0.07, 49.97, 0)(0, 50.0, 0)(0.04, 99.94, 0.01)(0, 100.0, 0)(0.03, 149.95, 0.01)(0, 150.0, 0)Figure 5
Comparison of the extrinsic and intrinsic parameter solution error: (a) extrinsic parameter solution error; (b) intrinsic parameter solution error.
(a)(b)We can learn from the experiment result that, when the extrinsic parameters are figured out by using the relay camera, the coordinate error of the measured mark points is not more than 2.8 mm. The minimum value is 1.5 mm, and the average error is 2.029 mm. In contrast, when the extrinsic parameters are figured out by using torque motor rotation parameters, the maximum coordinate error of the measured mark points is 3.5 mm, the minimum value is 2.2 mm, and the average error is 2.586 mm. From the experimental result, we can find that, when the intrinsic camera parameters are unified to the initial calibration value, the proposed extrinsic parameter solution method using the relay camera can effectively improve the measuring accuracy of the measured mark points.


## 4.1.2. Intrinsic Parameter Acquisition Accuracy Experiment

In order to verify the acquisition accuracy of the intrinsic parameters, we set the extrinsic camera parameters to the values obtained by using the relay camera in the comparing experiment. Then, we use the initial calibrated intrinsic parameters and the real-time intrinsic parameters achieved by the method in 2.2 for comparison. The comparing results are shown in Table2. After obtaining the intrinsic parameters, we calculate the coordinates of the 20 measured mark points on the structure surface and scale the measuring accuracy by the coordinate error of the measured mark points as demonstrated in Figure 5(b).Table 2
Measured data of the intrinsic parameters.
Initial valueFrame 1Frame 2Frame 3fx,fy(22201.81, 22395.01)(22947.64, 22647.23)(22753.21, 22486.35)(23053.61, 22931.02)Cx,Cy(1957.36, 1405.73)(1982.66, 1485.21)(2031.43, 1501.82)(1998.36, 1465.73)From the experimental results we learn that, when the extrinsic parameters are consistent through the experiment, the coordinate error of the measured mark points of the proposed method is not more than 2.31 mm. The minimum value is 1.38 mm, and the average error is 1.78 mm. In contrast, when only the initial calibrated intrinsic parameters are used, the maximum coordinate error of the measured mark points is 2.74 mm, the minimum value is 1.58 mm, and the average error is 2.087 mm. It is learned from the experimental result that the measuring accuracy of the measured mark points by using the real-time intrinsic parameters is higher than that by using the initial calibrated intrinsic parameters.


## 4.2. Accuracy Analysis

From the verification experiment in Section4.1, we learn that the proposed method utilizing the relay camera can not only effectively improve the accuracy of the extrinsic parameters but also achieve the real-time intrinsic parameters. To further verify the proposed method meets the request of the real-time on-line deformation monitoring for large-scale structure, we design the following experiment. We adopt the proposed method in the on-line monitoring for a metal structure surface, which contains 60 measured mark points. The measuring results of the proposed method are compared to the results of the measuring with electronic total station. The detailed comparison is shown in Figure 6.Figure 6
(a) Measuring result of measured mark point coordinates. (b) Measuring error of measured mark point coordinates.
(a)(b)From the coordinate error of the measured mark points shown in Figure6(b) we can learn that the maximum, minimum, mean, and standard errors of the measured mark point coordinate are 2.38 mm, 1.35 mm, 1.797 mm, and 0.2156 mm, respectively. The mean of the coordinate measuring error is less than 1.8 mm, and the standard error is less than 0.25 mm. The coordinate accuracy of the measured mark points obtained by on-line measuring is similar to the accuracy of the verification experiment in Section 4.1. This indicates the efficiency and stability of the proposed method. The proposed method using relay camera can achieve the intrinsic and extrinsic parameters of the measuring camera with high precision and can meet the demand of the on-line deformation monitoring of the structure.


## 5. Conclusions

This paper focuses on the solution of the difficulties existing in the deformation monitoring for large-scale structure using mobile vision, including real-time acquirement of the extrinsic parameters, low precision of the measuring, and low level of the automation. We first achieve the real-time extrinsic parameters of the measuring camera by using a relay camera that is fixed to the measuring camera. Then, the real-time intrinsic parameters of the measuring camera are calculated with the constraint between the fundamental matrix and the intrinsic and extrinsic parameters. At last, the deformation monitoring for the large-scale structure is fulfilled with the obtained intrinsic and extrinsic parameters. The experimental results verify the validity and stability of the proposed method for solving the coordinates of the measured mark points on the structure surface. The mean of the coordinate measuring error is less than 1.8 mm, and the standard error is less than 0.25 mm, and this accuracy generally meets the needs of the deformation monitoring of large-scale structure. At the same time, the proposed method improves the automation level and the solution accuracy of the camera parameters.



---

*Source: 8481230-2021-07-29.xml*