# High-Precision 3D Reconstruction of Cooperative Markers under Motion Blur

Authors: Yun Shi; Cong Tao; Xiaoping Wang; Liyan Zhang
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

The application of artificial intelligence and deep learning in the fields of wireless communication, image and speech recognition, and 3D reconstruction has successfully solved some difficult modeling problems. This paper focuses on the high-precision 3D reconstruction of the motion-blurred cooperative markers, including the Chinese character coded targets (CCTs) and the noncoded circular markers. A simulation-based motion-blurred image generation model is constructed to provide sufficient samples for training the convolutional neural network to identify and match the motion-blurred CCTs on the moving object. The blurred noncoded marker matching is performed through homography. The 3D reconstruction of the markers is realized via the optimization of the spatial moving path within the exposure period. The midpoint of the moving path of the markers is taken as the final reconstruction result. The experimental results show that the 3D reconstruction accuracy of the markers with a certain motion blur effect is about 0.08 mm.


---

## Body

## 1. Introduction

Machine vision has been widely used in object positioning, pose detection, motion tracking, and 3D shape reconstruction [1–5]. However, the imaging blur effect of moving targets makes these tasks difficult.There are two potential ways to the 3D reconstruction of motion-blurred targets: one is the reconstruction after deblurring, and the other is the reconstruction directly from the blurred images. Image deblurring is a typical inverse problem. The classic nonblind deconvolution algorithms alleviate the motion blur effect with given blur kernels such as point spread function [6] and Richardson and Lucy [7, 8]. For blind deconvolution with unknown blur kernel, Ref. [9] first used Radon transformation to estimate the blur parameters based on the spectrum information. The nonblind deconvolution is then used to recover the clear image. In [10], the gradient information of the motion-blurred image was used to determine the length and angle of the blur path. In [11], the unsaturated region was selected to estimate the blur kernel by using prior knowledge. The regularization equation was established, and the variation Bayesian was used to solve the optimal problem. In [12], the sharp boundary template was extracted from the downsampled image of the blurred image, and then, the blurred and the predicted value images were used to calculate the blur kernel. Although the above-mentioned methods can restore the motion-blurred images to some extent, the algorithms still need to make assumptions about the motion of the target or the camera. These assumptions implied the uniformity of the blur, which is usually not the case in practice.For reducing spatially varying motion blur, Tai et al. [13] used a hybrid camera system that simultaneously captures high-resolution video at a low-frame rate together with low-resolution video at a high-frame rate. Extra information available in the hybrid camera is utilized to reformulate the correction process to achieve better deblurring effect. The optical flow method proposed in [14] calculated the spatial change. The pose data output by the gyroscope attached to the camera was used by [15, 16] to determine the path of camera shaking and calculate the blur kernel caused by the shake. Tai et al. [17] used the coded exposure method to collect motion-blurred images, where the user estimated the homography matrix of the intermittent motion through multiple interventions.The different depths of the image points lead to the different motion blur kernel functions, which reveal that the blurring effect is inherently related to the 3D geometry of the scene. In [18], Xu and Jia used the depth information generated by the binocular stereo image to deblur the image in any form of motion. However, this was essentially a quasiuniform blur model with obvious ring artifacts. In [19], Lee and Lee developed a depth reconstruction method based on blur perception. Although the deblurring effect was good, the premise of the reconstruction was to know the motion path of the camera, which made it unsuitable for the 3D reconstruction of an object with unknown motion path. Hong et al. [20] achieved better motion blur removal effect in multiview images. However, they did not provide 3D reconstruction result.Several researchers have tried to extract useful information directly from the blurred images without deblurring. In [21], a segmentation-based symmetrical stereo vision matching method was proposed to address the high matching error rates of the images with motion blur. Although this method effectively reduced the false matching rates, it was only suitable for images with slight local blurring. In [22], a motion model based on the affine transformation principle was established. This method estimated the motion blur parameters of each subregion, but it was only suitable for problems tolerating relative low accuracy. In [23], the circular coded target under the effect of motion blur was 3D reconstructed. However, the circular coded targets have a simple structure and relatively low distinguishability; therefore, they are difficult to be correctly recognized under motion blur effect. Ref. [24] proposed a set of cooperative markers which used Chinese characters as the feature target and achieved better recognition performance under motion blur effect.For the 3D reconstruction of a high-speed moving target, the movement of the target within the exposure time cannot be ignored. The trajectory of the target during the exposure period will result in superimposed imaging, forming a certain degree of motion blur effect. Taking multiview images with motion blur effect as input, we propose an approach to 3D reconstruction of corporative markers, including both coded and noncoded ones. The remaining sections are organized as follows. Section2 introduces the structure and the segmentation algorithm of the marker points and establishes a simulation model for the motion-blurred marker targets. Section 3 uses convolutional neural network (CNN) to match the Chinese character coded targets (CCTs) with motion blur effect. Section 4 establishes the objective function of the marker targets and provides the initial values of the parameters. Section 5 designs an experimental procedure to reconstruct the marked targets in motion. Finally, Section 6 summarizes the article.


## 2. The Marker Targets

The cooperative markers in this paper include the CCTs [24] and noncoded circular markers. Each of the CCTs has a unique identity and is relatively easy for establishing correspondences among the multiview images. However, the CCTs are too large to be employed for all interest points on the object. Therefore, the relative small noncoded circular markers are also utilized. The correspondences between the noncoded markers are established based on the correspondences of the CCTs.


### 2.1. Structure of the Markers

The structure of each Chinese character is unique. Even if a motion blur occurs, they still possess certain characteristic information. The CCTs proposed in [24] are shown in Figure 1(a), which are composed of three concentric black/white/black circles (the center of the circles is also the center of the CCTs) and a combination of different Chinese characters. The diameter of the three circles and the size of the Chinese characters are in the ratio of 1 : 2 : 3 : 6. We choose 100 Chinese characters as the target to be coded and use numbers 0-99 as the encoding value of the corresponding CCTs. The Chinese characters in the coded targets are different. Thus, they can be used as the target point of the unique identity characterization. The noncoded target, shown in Figure 1(b), is a square, where the positioning area is composed of a white auxiliary circle concentric with a black positioning circle. The diameter of the two circles and the square is in the ratio of 1 : 2 : 3. In different applications, the size of the markers, including the CCTs and the noncoded markers, can be scaled to fit in the range of the scene. In our experiment, the size of the marker point is set to x=4 mm.Figure 1
Marker structure diagram.
(a)(b)


### 2.2. Simulation of the Motion-Blurred CCTs

We generate a large number of CCT images via soft program to provide sufficient samples for training the recognition network.


#### 2.2.1. The Virtual Camera Method

Figure2 illustrates a schematic diagram that defines the blur degree (intensity) σ, which can be expressed as(1)σ=Oobje−Oobjs/D2′−D1′,where Oobjs and Oobje are the starting and ending points of the marker’s center on the imaging plane, respectively, and D1′ and D2′ are the two intersection points.Figure 2
The schematic diagram defining the degree of blur.Figure3 shows the simulation steps of motion blur imaging. uvo2 andxyo1, respectively, represent the pixel coordinate system and the image coordinate system. The camera coordinate system OcXcYcZc and the marker coordinate system OobjXobjYobjZobj are described by the rotation matrix Roc1 and the translation vector toc1. The point Oobj is translated to the point O′obj, where the spatial displacement is Δtoc. The new camera coordinate system O′objX′objY′objZ′obj and the translated coordinate system of the marker targets are described by the rotation matrix Roce and the translation vector toce, which satisfy(2)Roce=Roc1,toce=toc1+Δtoc.Figure 3
Virtual camera method to simulate motion blur imaging.The displacement of the marker targets is evenly discretized intoN points. The rotation matrix Roc and the translation vector toc between the camera and the marker targets’ coordinate systems with the ith point used as the origin can be expressed as(3)Roc=Roc1,toc=toc1+i−1N−1Δtoc.The process of generating the dynamic simulation segmentation imageIb from the original image Iobj of the marker point can be expressed as(4)IbDd′=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,where IbDd′ is the gray value at Dd′ in Ib, and  refers to the smallest bounding box of the simulated images.The rotation matrixRocis not intuitive enough. In this paper, the rotation vector roc=rx,ry,rzT is used to represent the spatial orientation of the markers, where rx, ry, and rz represent the angles of the marker coordinate system rotating around the axes OobjXobj, OobjYobj, and OobjZobj in turn. In the actual measurements, the range of roc is set as(5)roc=rx,ry,rzT∣rx,ry∈−π3,π3,rz∈−π,π.


#### 2.2.2. The Perturbation Model

The real images are affected by noise, light distribution, and light intensity. The perturbation model adds noiseη, gray contrast φ, and gray increment β to simulate these imaging factors. Hence, we assume the relationship between IobjDobj and the gray value ΙimgDd′ at the corresponding point Dd′ on the simulated image Ιimg satisfying(6)IimgDd′=φIobjDobj+β∣φ∈0.3,1,β∈0,25.By combining with the perturbation model (6), the virtual camera method (4) is modified to generate the CCT simulation segmentation image as(7)IbD′d=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β+η,where η refers to the noise. Then, the parameters of the simulation model are determined to generate various motion-blurred CCTs. Figure 4 shows the simulated motion-blurred image examples with various spatial positions, directions, motion paths, blur levels, and noise levels.Figure 4
The motion-blurred CCTs “典” generated by simulations.


## 2.1. Structure of the Markers

The structure of each Chinese character is unique. Even if a motion blur occurs, they still possess certain characteristic information. The CCTs proposed in [24] are shown in Figure 1(a), which are composed of three concentric black/white/black circles (the center of the circles is also the center of the CCTs) and a combination of different Chinese characters. The diameter of the three circles and the size of the Chinese characters are in the ratio of 1 : 2 : 3 : 6. We choose 100 Chinese characters as the target to be coded and use numbers 0-99 as the encoding value of the corresponding CCTs. The Chinese characters in the coded targets are different. Thus, they can be used as the target point of the unique identity characterization. The noncoded target, shown in Figure 1(b), is a square, where the positioning area is composed of a white auxiliary circle concentric with a black positioning circle. The diameter of the two circles and the square is in the ratio of 1 : 2 : 3. In different applications, the size of the markers, including the CCTs and the noncoded markers, can be scaled to fit in the range of the scene. In our experiment, the size of the marker point is set to x=4 mm.Figure 1
Marker structure diagram.
(a)(b)


## 2.2. Simulation of the Motion-Blurred CCTs

We generate a large number of CCT images via soft program to provide sufficient samples for training the recognition network.


### 2.2.1. The Virtual Camera Method

Figure2 illustrates a schematic diagram that defines the blur degree (intensity) σ, which can be expressed as(1)σ=Oobje−Oobjs/D2′−D1′,where Oobjs and Oobje are the starting and ending points of the marker’s center on the imaging plane, respectively, and D1′ and D2′ are the two intersection points.Figure 2
The schematic diagram defining the degree of blur.Figure3 shows the simulation steps of motion blur imaging. uvo2 andxyo1, respectively, represent the pixel coordinate system and the image coordinate system. The camera coordinate system OcXcYcZc and the marker coordinate system OobjXobjYobjZobj are described by the rotation matrix Roc1 and the translation vector toc1. The point Oobj is translated to the point O′obj, where the spatial displacement is Δtoc. The new camera coordinate system O′objX′objY′objZ′obj and the translated coordinate system of the marker targets are described by the rotation matrix Roce and the translation vector toce, which satisfy(2)Roce=Roc1,toce=toc1+Δtoc.Figure 3
Virtual camera method to simulate motion blur imaging.The displacement of the marker targets is evenly discretized intoN points. The rotation matrix Roc and the translation vector toc between the camera and the marker targets’ coordinate systems with the ith point used as the origin can be expressed as(3)Roc=Roc1,toc=toc1+i−1N−1Δtoc.The process of generating the dynamic simulation segmentation imageIb from the original image Iobj of the marker point can be expressed as(4)IbDd′=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,where IbDd′ is the gray value at Dd′ in Ib, and  refers to the smallest bounding box of the simulated images.The rotation matrixRocis not intuitive enough. In this paper, the rotation vector roc=rx,ry,rzT is used to represent the spatial orientation of the markers, where rx, ry, and rz represent the angles of the marker coordinate system rotating around the axes OobjXobj, OobjYobj, and OobjZobj in turn. In the actual measurements, the range of roc is set as(5)roc=rx,ry,rzT∣rx,ry∈−π3,π3,rz∈−π,π.


### 2.2.2. The Perturbation Model

The real images are affected by noise, light distribution, and light intensity. The perturbation model adds noiseη, gray contrast φ, and gray increment β to simulate these imaging factors. Hence, we assume the relationship between IobjDobj and the gray value ΙimgDd′ at the corresponding point Dd′ on the simulated image Ιimg satisfying(6)IimgDd′=φIobjDobj+β∣φ∈0.3,1,β∈0,25.By combining with the perturbation model (6), the virtual camera method (4) is modified to generate the CCT simulation segmentation image as(7)IbD′d=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β+η,where η refers to the noise. Then, the parameters of the simulation model are determined to generate various motion-blurred CCTs. Figure 4 shows the simulated motion-blurred image examples with various spatial positions, directions, motion paths, blur levels, and noise levels.Figure 4
The motion-blurred CCTs “典” generated by simulations.


## 2.2.1. The Virtual Camera Method

Figure2 illustrates a schematic diagram that defines the blur degree (intensity) σ, which can be expressed as(1)σ=Oobje−Oobjs/D2′−D1′,where Oobjs and Oobje are the starting and ending points of the marker’s center on the imaging plane, respectively, and D1′ and D2′ are the two intersection points.Figure 2
The schematic diagram defining the degree of blur.Figure3 shows the simulation steps of motion blur imaging. uvo2 andxyo1, respectively, represent the pixel coordinate system and the image coordinate system. The camera coordinate system OcXcYcZc and the marker coordinate system OobjXobjYobjZobj are described by the rotation matrix Roc1 and the translation vector toc1. The point Oobj is translated to the point O′obj, where the spatial displacement is Δtoc. The new camera coordinate system O′objX′objY′objZ′obj and the translated coordinate system of the marker targets are described by the rotation matrix Roce and the translation vector toce, which satisfy(2)Roce=Roc1,toce=toc1+Δtoc.Figure 3
Virtual camera method to simulate motion blur imaging.The displacement of the marker targets is evenly discretized intoN points. The rotation matrix Roc and the translation vector toc between the camera and the marker targets’ coordinate systems with the ith point used as the origin can be expressed as(3)Roc=Roc1,toc=toc1+i−1N−1Δtoc.The process of generating the dynamic simulation segmentation imageIb from the original image Iobj of the marker point can be expressed as(4)IbDd′=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,where IbDd′ is the gray value at Dd′ in Ib, and  refers to the smallest bounding box of the simulated images.The rotation matrixRocis not intuitive enough. In this paper, the rotation vector roc=rx,ry,rzT is used to represent the spatial orientation of the markers, where rx, ry, and rz represent the angles of the marker coordinate system rotating around the axes OobjXobj, OobjYobj, and OobjZobj in turn. In the actual measurements, the range of roc is set as(5)roc=rx,ry,rzT∣rx,ry∈−π3,π3,rz∈−π,π.


## 2.2.2. The Perturbation Model

The real images are affected by noise, light distribution, and light intensity. The perturbation model adds noiseη, gray contrast φ, and gray increment β to simulate these imaging factors. Hence, we assume the relationship between IobjDobj and the gray value ΙimgDd′ at the corresponding point Dd′ on the simulated image Ιimg satisfying(6)IimgDd′=φIobjDobj+β∣φ∈0.3,1,β∈0,25.By combining with the perturbation model (6), the virtual camera method (4) is modified to generate the CCT simulation segmentation image as(7)IbD′d=1N∑i=1NfAC,kC1,kC2,pC1,pC2,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β+η,where η refers to the noise. Then, the parameters of the simulation model are determined to generate various motion-blurred CCTs. Figure 4 shows the simulated motion-blurred image examples with various spatial positions, directions, motion paths, blur levels, and noise levels.Figure 4
The motion-blurred CCTs “典” generated by simulations.


## 3. Recognition of the Motion-Blurred CCTs

The CNN utilized for motion-blurred CCT recognition is composed of five layers: the input layer, convolutional layer (C), pooling layer (P), fully connected layer (F), and the output layer.Figure5 shows the CNN structure diagrams of the Chinese character coded target “典.” The CNN adopts a convolutional layer and a pooling layer to alternately set four convolutional layers and three pooling layers. To be specific, 12×5×51@64×64 means that the layer has 12 convolution kernels with the window size of 5×5, the sliding step length of which on its input feature surface is 1, and the output feature surface size is 64×64. The output layer uses SoftMax [25] as the regression model, where the category with the highest probability is used as the output category. Finally, the crossentropy loss function is used as the objective function of the optimization problem.Figure 5
The CNN’s structure diagrams of the CCT “典.”


## 4. The 3D Reconstruction Based on Image Difference Minimization

In [23], Zhou et al. proposed a method to reconstruct the spatial motion path of the circular coded targets within the camera exposure time. In this paper, we investigate the 3D reconstruction of both motion-blurred CCTs and noncoded targets, including the preliminary reconstruction process and the fine-tune optimization.


### 4.1. The Preliminary 3D Reconstruction of Motion-Blurred Markers

The preliminary 3D reconstruction of the motion-blurred markers refers to the 3D reconstruction of the midpoint of the path within the exposure period to obtain the initial 3D coordinates of the markers.


#### 4.1.1. The Preliminary 3D Reconstruction of the CCTs

Figure6 shows the recognition result of the CCTs, where the rectangular boxes enclose the CCTs, and the numbers on top of them represent the code values of the CCTs. The CCTs with the same code values are regarded as the images of the same coded target in the left and right cameras. The pixel coordinates, namely, Dcinitl′ and Dcinitr′, of the center of each bounding rectangle are used to obtain the initial 3D coordinates Dcinitr′ of the underlying CCT.Figure 6
Recognition result of the CCTs.


#### 4.1.2. The Preliminary 3D Reconstruction of the Noncoded Targets

The preliminary 3D reconstruction of the noncoded targets is performed after the 3D reconstruction and optimization of the CCTs. Since we mark the optimized result ofDcinit as Dcopt, the pixel coordinates Dcoptl′ and Dcoptr′ of Dcopt on the left and right images can be expressed as(8)Dcoptl′=projectACl,kC1l,kC2l,pC1l,pC2l,Rl,tl,Dcopt,Dcoptr′=projectACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Rl,Rrl−1tl−trl,Dcopt.The homography matrixH is obtained by the correspondence between the imaging plane of the left and the right camera, respectively. Divide an area on the scene shot by the binocular stereo vision system. The divided area requires a small curvature and contains at least 5 CCTs. Thus, the homography relationship can be established as(9)Dr′=HrHl−1Dl′=HlrDl′,where Dl′ and are the pixel coordinates of the imaging point in the left and right corresponding images, Hl and Hr are the homography relationship with Dl′ and Dr′, and Hlr is the homography between Dl′ and Dr′. The corresponding relationship between Dcoptl′ and Dcoptr′ can be obtained from (8) and (9) as(10)Dcoptr′=HlrDcoptl′.Use at least five sets of corresponding pixel coordinatesDcoptl′ and Dcoptr′ in the left and right corresponding images. According to formula (10), the homography relationship Hlr between the left and right corresponding images is optimized. After the solution of Hlr is completed, the center coordinates of the bounding box of the noncoded targets are obtained. The center coordinates of the noncoded markers are shown in the left imageDuinitl′and the corresponding coordinateDlHr′=HlrDuinitl′in the right image. We then calculate the pixel distance between DlHr′ and the center coordinates of the bounding box of all noncoded targets in the right picture. Finally, we take Duinitr′, which is nearest to DlHr′, as the matching point of Duinitl′.Figure7 shows the matching results of the noncoded targets, in which the box is the division rectangle, and the same number represents the corresponding noncoded targets after matching. When the noncoded target matching is completed, D′uinitl and D′uinitr are taken as the initial values of the pixel coordinate of the noncoded targets’ positioning center, and hence, the initial 3D coordinates of the positioning center Duinit is obtained.Figure 7
The matching results of the noncoded targets.


### 4.2. Establishment of the Objective Function

In the motion-blurred images, the markers are imaged along a 3D moving path instead of at a certain spatial point during the exposure period. Therefore, the optimization parameters in our method concern the spatial moving path of the markers.


#### 4.2.1. Binocular System Virtual Camera Method

The virtual camera method of the binocular system is used to simulate the motion-blurred images of the CCTs on the camera-imaging plane under high-speed motion. The process of generating the dynamic simulation segmentation imagesIbl and Ibr of the original image Iobj of the CCT in the left and right cameras can be expressed as(11)IblD′d=1N∑i=1NfACl,kC1l,kC2l,pC1l,pC2l,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,(12)IbrD′d=1N∑i=1NfACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Roc1,Rrl−1toc1−trl+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,where the noise is removed as it is not a suitable optimization parameter.


#### 4.2.2. Optimization Goal Based on Differential Images

As in [23], we also minimize the difference between the dynamic analog segmented images and the real-shot segmented images of the markers in the left and right cameras, namely, IΔl and IΔr, to fine tune the 3D reconstruction. The process is as follows:(13)IΔl=Irl−Ibl,IΔr=Irr−Ibr,where • ensures that the gray values of all pixels in differential images are nonnegative numbers. Here, firstly, the sizes of Ibl, Ibr, Irl, and Irr have been unified to U×V. Next, the optimized function FΔ that minimizes the average gray value of the difference image is set as the optimization goal, which can be expressed as(14)FΔ=∑u=0U−1∑v=0V−1IΔlu,v+∑u=0U−1∑v=0V−1IΔru,v2UV,where IΔlu,v and IΔru,v represent the gray value at u,v of the differential image. The optimization function FΔ can be expressed in terms of Roc1, toc1, Δtoc, φ, and β as(15)FΔ=optRoc1,toc1,Δtoc,φ,β.


### 4.3. Estimation of the Initial Values of the Parameters

This subsection describes how the initial values of the optimized parameters are obtained for both CCTs and the noncoded targets. The initial values of the CCTs are obtained by improving the method in [23]. An effective method for determining the initial value of the noncoded targets is independently proposed.


#### 4.3.1. Initial Values of the CCTs

(1) Determination of the Initial Valuestoc1 and Δtoc of the Space Position. Figure 8 shows the motion relationship diagram of the CCTs, while the relationship between the camera exposure time Δt and ΔT is shown in Figure 9. Three sets of images with motion blur effect are shot under high-speed motion at equal time intervals of ΔT, where -1, 0, and 1 represent the previous frame, current frame, and the next frame in the shooting process, respectively.Figure 8
The diagram showing the motion relationships of the CCTs.Figure 9
The relationship betweenΔt and ΔT.For a CCT, the initial values of the 3D coordinates of the midpoint of the path within the previous, current, and the next frames areDcinit−1, Dcinit0, and Dcinit1, which are shown by the points Dcinit−1, Dcinit0, and Dcinit1 in Figure 8. Take points Dcinits and Dcinite and the midpoint Dcinit0 on the fitted curve. If the arc length DcinitsDcinite is used to represent the intraframe path of the current frame, the relationship can be given as(16)lsLS=Δt2ΔT,where LS represents the arc length of the curve and ls represents the arc length of DcinitsDcinite. After calculating ls, the spatial coordinates Dcinits and Dciniteof the points Dcinits and Dcinite can be solved. The relationships between Dcinits and Dcinite and toc1 and Δtoc are as follows:(17)toc1=Dcinits,Δtoc=Dcinite−Dcinits.(2) Determination of the Initial Gray Image Valuesφ and β. To improve the initial value setting of β provided in [23], we distinguish the black background of the dynamic real-shot segmented image and take its gray average value as β to generate the dynamic analog-segmented image. Hence, the initial value of φ can be calculated as(18)φ=∑u=0U−1∑v=0V−1Iru,v−β∑u=0U−1∑v=0V−1Iobju,v,where Ir represents the dynamic real shot segmented image and Iobju,v and Iru,v represent the gray value of Iobj and Ir at the pixel coordinate u,v.(3) Determination of the Initial Space AttitudeRoc1. We first divided rx and ry in roc into 10 equal parts and rz into 50 equal parts; hence, a total of 10×10×50=5000 combinations were obtained. We then calculated the initial value of roc using rx, ry, and rz when FΔ is the smallest and hence determined the initial value of Roc1.


#### 4.3.2. Initial Values of the Noncoded Targets

(1) Determination of the Initial Values oftoc1 and Δtoc. For a rigid body moving in the 3D space, its relative pose between the beginning and the end of the motion can be described by a rotation and a translation transformation. If the rotation matrix Robj represents its rotation transformation and the translation vector tobj represents its translation transformation, the coordinate transformation of any point on the rigid body can be expressed as(19)P′obj=RobjPobj+tobj,where Pobj and P′obj are the 3D coordinates of the point on the rigid body before and after the movement. If the midpoint and the endpoint of the intraframe path after the optimization of the CCTs are marked as Dcoptm and Dcopte, their relation can be given as(20)Dcopte=RobjDcoptm+tobj,where Robj and tobj, respectively, represent the rotation matrix and the translation vector of the movement of the marker targets from the midpoint of the camera exposure time to the end of the camera exposure. In this paper, Robj and tobj are obtained by parameter fitting through the intraframe path relations of more than five CCTs. Regarding the center of the reconstructed noncoded targets, the midpoint and endpoint of the nonoptimized intraframe path are denoted as Duinitm and Duinite, respectively. Their relation can be expressed as(21)Duinite=RobjDuinitm+tobj.The starting pointDuinits of the noncoded intraframe path without optimization is Duinits=2Duinitm−Duinite. Here, toc1 and Δtoc between the marker point coordinate system of the noncoded targets and the left camera coordinate system can be expressed as(22)toc1=Duinits,Δtoc=Duinite−Duinits.(2) Determination of the Initial Values ofφ and β. The method for determining the initial values of φ and β of the noncoded targets is the same as that of the CCTs.(3) Determination of the Initial ValueRoc1. When determining Roc1 of the noncoded targets, rx and ry of roc need to be sampled at equal intervals within their value range. The parameter rz is set to zero for the circular symmetry property of the noncoded targets. Hence, roc of the noncoded targets has only 10×10=100 combinations.


### 4.4. Optimization Method and Results

We used Powell optimization to minimizeFΔ. Figure 10 shows the difference image during the optimization process for various iterations of the CCT “典.” The first row and the fourth row in Figure 10, respectively, represent the enlarged results of the red rectangular boxes in the second and the third rows. When the optimization is over, the difference image is nearly black, which indicates that the simulated segmented image is very close to the real segmented image. In other words, the optimization achieves good convergence.Figure 10
The difference image at different stages of the optimization process.In the final error calculation, the space coordinates of the static marker targets reconstructed by a commercial structured-light device ATOS® are assumed as the true values. Accordingly, the midpoints of the 3D moving paths of the markers are taken as the final 3D reconstruction result for the error evaluation. So far, the whole 3D reconstruction procedure of the two types of the cooperative markers with motion blur effect is completed.


## 4.1. The Preliminary 3D Reconstruction of Motion-Blurred Markers

The preliminary 3D reconstruction of the motion-blurred markers refers to the 3D reconstruction of the midpoint of the path within the exposure period to obtain the initial 3D coordinates of the markers.


### 4.1.1. The Preliminary 3D Reconstruction of the CCTs

Figure6 shows the recognition result of the CCTs, where the rectangular boxes enclose the CCTs, and the numbers on top of them represent the code values of the CCTs. The CCTs with the same code values are regarded as the images of the same coded target in the left and right cameras. The pixel coordinates, namely, Dcinitl′ and Dcinitr′, of the center of each bounding rectangle are used to obtain the initial 3D coordinates Dcinitr′ of the underlying CCT.Figure 6
Recognition result of the CCTs.


### 4.1.2. The Preliminary 3D Reconstruction of the Noncoded Targets

The preliminary 3D reconstruction of the noncoded targets is performed after the 3D reconstruction and optimization of the CCTs. Since we mark the optimized result ofDcinit as Dcopt, the pixel coordinates Dcoptl′ and Dcoptr′ of Dcopt on the left and right images can be expressed as(8)Dcoptl′=projectACl,kC1l,kC2l,pC1l,pC2l,Rl,tl,Dcopt,Dcoptr′=projectACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Rl,Rrl−1tl−trl,Dcopt.The homography matrixH is obtained by the correspondence between the imaging plane of the left and the right camera, respectively. Divide an area on the scene shot by the binocular stereo vision system. The divided area requires a small curvature and contains at least 5 CCTs. Thus, the homography relationship can be established as(9)Dr′=HrHl−1Dl′=HlrDl′,where Dl′ and are the pixel coordinates of the imaging point in the left and right corresponding images, Hl and Hr are the homography relationship with Dl′ and Dr′, and Hlr is the homography between Dl′ and Dr′. The corresponding relationship between Dcoptl′ and Dcoptr′ can be obtained from (8) and (9) as(10)Dcoptr′=HlrDcoptl′.Use at least five sets of corresponding pixel coordinatesDcoptl′ and Dcoptr′ in the left and right corresponding images. According to formula (10), the homography relationship Hlr between the left and right corresponding images is optimized. After the solution of Hlr is completed, the center coordinates of the bounding box of the noncoded targets are obtained. The center coordinates of the noncoded markers are shown in the left imageDuinitl′and the corresponding coordinateDlHr′=HlrDuinitl′in the right image. We then calculate the pixel distance between DlHr′ and the center coordinates of the bounding box of all noncoded targets in the right picture. Finally, we take Duinitr′, which is nearest to DlHr′, as the matching point of Duinitl′.Figure7 shows the matching results of the noncoded targets, in which the box is the division rectangle, and the same number represents the corresponding noncoded targets after matching. When the noncoded target matching is completed, D′uinitl and D′uinitr are taken as the initial values of the pixel coordinate of the noncoded targets’ positioning center, and hence, the initial 3D coordinates of the positioning center Duinit is obtained.Figure 7
The matching results of the noncoded targets.


## 4.1.1. The Preliminary 3D Reconstruction of the CCTs

Figure6 shows the recognition result of the CCTs, where the rectangular boxes enclose the CCTs, and the numbers on top of them represent the code values of the CCTs. The CCTs with the same code values are regarded as the images of the same coded target in the left and right cameras. The pixel coordinates, namely, Dcinitl′ and Dcinitr′, of the center of each bounding rectangle are used to obtain the initial 3D coordinates Dcinitr′ of the underlying CCT.Figure 6
Recognition result of the CCTs.


## 4.1.2. The Preliminary 3D Reconstruction of the Noncoded Targets

The preliminary 3D reconstruction of the noncoded targets is performed after the 3D reconstruction and optimization of the CCTs. Since we mark the optimized result ofDcinit as Dcopt, the pixel coordinates Dcoptl′ and Dcoptr′ of Dcopt on the left and right images can be expressed as(8)Dcoptl′=projectACl,kC1l,kC2l,pC1l,pC2l,Rl,tl,Dcopt,Dcoptr′=projectACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Rl,Rrl−1tl−trl,Dcopt.The homography matrixH is obtained by the correspondence between the imaging plane of the left and the right camera, respectively. Divide an area on the scene shot by the binocular stereo vision system. The divided area requires a small curvature and contains at least 5 CCTs. Thus, the homography relationship can be established as(9)Dr′=HrHl−1Dl′=HlrDl′,where Dl′ and are the pixel coordinates of the imaging point in the left and right corresponding images, Hl and Hr are the homography relationship with Dl′ and Dr′, and Hlr is the homography between Dl′ and Dr′. The corresponding relationship between Dcoptl′ and Dcoptr′ can be obtained from (8) and (9) as(10)Dcoptr′=HlrDcoptl′.Use at least five sets of corresponding pixel coordinatesDcoptl′ and Dcoptr′ in the left and right corresponding images. According to formula (10), the homography relationship Hlr between the left and right corresponding images is optimized. After the solution of Hlr is completed, the center coordinates of the bounding box of the noncoded targets are obtained. The center coordinates of the noncoded markers are shown in the left imageDuinitl′and the corresponding coordinateDlHr′=HlrDuinitl′in the right image. We then calculate the pixel distance between DlHr′ and the center coordinates of the bounding box of all noncoded targets in the right picture. Finally, we take Duinitr′, which is nearest to DlHr′, as the matching point of Duinitl′.Figure7 shows the matching results of the noncoded targets, in which the box is the division rectangle, and the same number represents the corresponding noncoded targets after matching. When the noncoded target matching is completed, D′uinitl and D′uinitr are taken as the initial values of the pixel coordinate of the noncoded targets’ positioning center, and hence, the initial 3D coordinates of the positioning center Duinit is obtained.Figure 7
The matching results of the noncoded targets.


## 4.2. Establishment of the Objective Function

In the motion-blurred images, the markers are imaged along a 3D moving path instead of at a certain spatial point during the exposure period. Therefore, the optimization parameters in our method concern the spatial moving path of the markers.


### 4.2.1. Binocular System Virtual Camera Method

The virtual camera method of the binocular system is used to simulate the motion-blurred images of the CCTs on the camera-imaging plane under high-speed motion. The process of generating the dynamic simulation segmentation imagesIbl and Ibr of the original image Iobj of the CCT in the left and right cameras can be expressed as(11)IblD′d=1N∑i=1NfACl,kC1l,kC2l,pC1l,pC2l,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,(12)IbrD′d=1N∑i=1NfACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Roc1,Rrl−1toc1−trl+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,where the noise is removed as it is not a suitable optimization parameter.


### 4.2.2. Optimization Goal Based on Differential Images

As in [23], we also minimize the difference between the dynamic analog segmented images and the real-shot segmented images of the markers in the left and right cameras, namely, IΔl and IΔr, to fine tune the 3D reconstruction. The process is as follows:(13)IΔl=Irl−Ibl,IΔr=Irr−Ibr,where • ensures that the gray values of all pixels in differential images are nonnegative numbers. Here, firstly, the sizes of Ibl, Ibr, Irl, and Irr have been unified to U×V. Next, the optimized function FΔ that minimizes the average gray value of the difference image is set as the optimization goal, which can be expressed as(14)FΔ=∑u=0U−1∑v=0V−1IΔlu,v+∑u=0U−1∑v=0V−1IΔru,v2UV,where IΔlu,v and IΔru,v represent the gray value at u,v of the differential image. The optimization function FΔ can be expressed in terms of Roc1, toc1, Δtoc, φ, and β as(15)FΔ=optRoc1,toc1,Δtoc,φ,β.


## 4.2.1. Binocular System Virtual Camera Method

The virtual camera method of the binocular system is used to simulate the motion-blurred images of the CCTs on the camera-imaging plane under high-speed motion. The process of generating the dynamic simulation segmentation imagesIbl and Ibr of the original image Iobj of the CCT in the left and right cameras can be expressed as(11)IblD′d=1N∑i=1NfACl,kC1l,kC2l,pC1l,pC2l,Roc1,toc1+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,(12)IbrD′d=1N∑i=1NfACr,kC1r,kC2r,pC1r,pC2r,Rrl−1Roc1,Rrl−1toc1−trl+i−1N−1Δtoc,Dobj,IobjDobj,φ,β,where the noise is removed as it is not a suitable optimization parameter.


## 4.2.2. Optimization Goal Based on Differential Images

As in [23], we also minimize the difference between the dynamic analog segmented images and the real-shot segmented images of the markers in the left and right cameras, namely, IΔl and IΔr, to fine tune the 3D reconstruction. The process is as follows:(13)IΔl=Irl−Ibl,IΔr=Irr−Ibr,where • ensures that the gray values of all pixels in differential images are nonnegative numbers. Here, firstly, the sizes of Ibl, Ibr, Irl, and Irr have been unified to U×V. Next, the optimized function FΔ that minimizes the average gray value of the difference image is set as the optimization goal, which can be expressed as(14)FΔ=∑u=0U−1∑v=0V−1IΔlu,v+∑u=0U−1∑v=0V−1IΔru,v2UV,where IΔlu,v and IΔru,v represent the gray value at u,v of the differential image. The optimization function FΔ can be expressed in terms of Roc1, toc1, Δtoc, φ, and β as(15)FΔ=optRoc1,toc1,Δtoc,φ,β.


## 4.3. Estimation of the Initial Values of the Parameters

This subsection describes how the initial values of the optimized parameters are obtained for both CCTs and the noncoded targets. The initial values of the CCTs are obtained by improving the method in [23]. An effective method for determining the initial value of the noncoded targets is independently proposed.


### 4.3.1. Initial Values of the CCTs

(1) Determination of the Initial Valuestoc1 and Δtoc of the Space Position. Figure 8 shows the motion relationship diagram of the CCTs, while the relationship between the camera exposure time Δt and ΔT is shown in Figure 9. Three sets of images with motion blur effect are shot under high-speed motion at equal time intervals of ΔT, where -1, 0, and 1 represent the previous frame, current frame, and the next frame in the shooting process, respectively.Figure 8
The diagram showing the motion relationships of the CCTs.Figure 9
The relationship betweenΔt and ΔT.For a CCT, the initial values of the 3D coordinates of the midpoint of the path within the previous, current, and the next frames areDcinit−1, Dcinit0, and Dcinit1, which are shown by the points Dcinit−1, Dcinit0, and Dcinit1 in Figure 8. Take points Dcinits and Dcinite and the midpoint Dcinit0 on the fitted curve. If the arc length DcinitsDcinite is used to represent the intraframe path of the current frame, the relationship can be given as(16)lsLS=Δt2ΔT,where LS represents the arc length of the curve and ls represents the arc length of DcinitsDcinite. After calculating ls, the spatial coordinates Dcinits and Dciniteof the points Dcinits and Dcinite can be solved. The relationships between Dcinits and Dcinite and toc1 and Δtoc are as follows:(17)toc1=Dcinits,Δtoc=Dcinite−Dcinits.(2) Determination of the Initial Gray Image Valuesφ and β. To improve the initial value setting of β provided in [23], we distinguish the black background of the dynamic real-shot segmented image and take its gray average value as β to generate the dynamic analog-segmented image. Hence, the initial value of φ can be calculated as(18)φ=∑u=0U−1∑v=0V−1Iru,v−β∑u=0U−1∑v=0V−1Iobju,v,where Ir represents the dynamic real shot segmented image and Iobju,v and Iru,v represent the gray value of Iobj and Ir at the pixel coordinate u,v.(3) Determination of the Initial Space AttitudeRoc1. We first divided rx and ry in roc into 10 equal parts and rz into 50 equal parts; hence, a total of 10×10×50=5000 combinations were obtained. We then calculated the initial value of roc using rx, ry, and rz when FΔ is the smallest and hence determined the initial value of Roc1.


### 4.3.2. Initial Values of the Noncoded Targets

(1) Determination of the Initial Values oftoc1 and Δtoc. For a rigid body moving in the 3D space, its relative pose between the beginning and the end of the motion can be described by a rotation and a translation transformation. If the rotation matrix Robj represents its rotation transformation and the translation vector tobj represents its translation transformation, the coordinate transformation of any point on the rigid body can be expressed as(19)P′obj=RobjPobj+tobj,where Pobj and P′obj are the 3D coordinates of the point on the rigid body before and after the movement. If the midpoint and the endpoint of the intraframe path after the optimization of the CCTs are marked as Dcoptm and Dcopte, their relation can be given as(20)Dcopte=RobjDcoptm+tobj,where Robj and tobj, respectively, represent the rotation matrix and the translation vector of the movement of the marker targets from the midpoint of the camera exposure time to the end of the camera exposure. In this paper, Robj and tobj are obtained by parameter fitting through the intraframe path relations of more than five CCTs. Regarding the center of the reconstructed noncoded targets, the midpoint and endpoint of the nonoptimized intraframe path are denoted as Duinitm and Duinite, respectively. Their relation can be expressed as(21)Duinite=RobjDuinitm+tobj.The starting pointDuinits of the noncoded intraframe path without optimization is Duinits=2Duinitm−Duinite. Here, toc1 and Δtoc between the marker point coordinate system of the noncoded targets and the left camera coordinate system can be expressed as(22)toc1=Duinits,Δtoc=Duinite−Duinits.(2) Determination of the Initial Values ofφ and β. The method for determining the initial values of φ and β of the noncoded targets is the same as that of the CCTs.(3) Determination of the Initial ValueRoc1. When determining Roc1 of the noncoded targets, rx and ry of roc need to be sampled at equal intervals within their value range. The parameter rz is set to zero for the circular symmetry property of the noncoded targets. Hence, roc of the noncoded targets has only 10×10=100 combinations.


## 4.3.1. Initial Values of the CCTs

(1) Determination of the Initial Valuestoc1 and Δtoc of the Space Position. Figure 8 shows the motion relationship diagram of the CCTs, while the relationship between the camera exposure time Δt and ΔT is shown in Figure 9. Three sets of images with motion blur effect are shot under high-speed motion at equal time intervals of ΔT, where -1, 0, and 1 represent the previous frame, current frame, and the next frame in the shooting process, respectively.Figure 8
The diagram showing the motion relationships of the CCTs.Figure 9
The relationship betweenΔt and ΔT.For a CCT, the initial values of the 3D coordinates of the midpoint of the path within the previous, current, and the next frames areDcinit−1, Dcinit0, and Dcinit1, which are shown by the points Dcinit−1, Dcinit0, and Dcinit1 in Figure 8. Take points Dcinits and Dcinite and the midpoint Dcinit0 on the fitted curve. If the arc length DcinitsDcinite is used to represent the intraframe path of the current frame, the relationship can be given as(16)lsLS=Δt2ΔT,where LS represents the arc length of the curve and ls represents the arc length of DcinitsDcinite. After calculating ls, the spatial coordinates Dcinits and Dciniteof the points Dcinits and Dcinite can be solved. The relationships between Dcinits and Dcinite and toc1 and Δtoc are as follows:(17)toc1=Dcinits,Δtoc=Dcinite−Dcinits.(2) Determination of the Initial Gray Image Valuesφ and β. To improve the initial value setting of β provided in [23], we distinguish the black background of the dynamic real-shot segmented image and take its gray average value as β to generate the dynamic analog-segmented image. Hence, the initial value of φ can be calculated as(18)φ=∑u=0U−1∑v=0V−1Iru,v−β∑u=0U−1∑v=0V−1Iobju,v,where Ir represents the dynamic real shot segmented image and Iobju,v and Iru,v represent the gray value of Iobj and Ir at the pixel coordinate u,v.(3) Determination of the Initial Space AttitudeRoc1. We first divided rx and ry in roc into 10 equal parts and rz into 50 equal parts; hence, a total of 10×10×50=5000 combinations were obtained. We then calculated the initial value of roc using rx, ry, and rz when FΔ is the smallest and hence determined the initial value of Roc1.


## 4.3.2. Initial Values of the Noncoded Targets

(1) Determination of the Initial Values oftoc1 and Δtoc. For a rigid body moving in the 3D space, its relative pose between the beginning and the end of the motion can be described by a rotation and a translation transformation. If the rotation matrix Robj represents its rotation transformation and the translation vector tobj represents its translation transformation, the coordinate transformation of any point on the rigid body can be expressed as(19)P′obj=RobjPobj+tobj,where Pobj and P′obj are the 3D coordinates of the point on the rigid body before and after the movement. If the midpoint and the endpoint of the intraframe path after the optimization of the CCTs are marked as Dcoptm and Dcopte, their relation can be given as(20)Dcopte=RobjDcoptm+tobj,where Robj and tobj, respectively, represent the rotation matrix and the translation vector of the movement of the marker targets from the midpoint of the camera exposure time to the end of the camera exposure. In this paper, Robj and tobj are obtained by parameter fitting through the intraframe path relations of more than five CCTs. Regarding the center of the reconstructed noncoded targets, the midpoint and endpoint of the nonoptimized intraframe path are denoted as Duinitm and Duinite, respectively. Their relation can be expressed as(21)Duinite=RobjDuinitm+tobj.The starting pointDuinits of the noncoded intraframe path without optimization is Duinits=2Duinitm−Duinite. Here, toc1 and Δtoc between the marker point coordinate system of the noncoded targets and the left camera coordinate system can be expressed as(22)toc1=Duinits,Δtoc=Duinite−Duinits.(2) Determination of the Initial Values ofφ and β. The method for determining the initial values of φ and β of the noncoded targets is the same as that of the CCTs.(3) Determination of the Initial ValueRoc1. When determining Roc1 of the noncoded targets, rx and ry of roc need to be sampled at equal intervals within their value range. The parameter rz is set to zero for the circular symmetry property of the noncoded targets. Hence, roc of the noncoded targets has only 10×10=100 combinations.


## 4.4. Optimization Method and Results

We used Powell optimization to minimizeFΔ. Figure 10 shows the difference image during the optimization process for various iterations of the CCT “典.” The first row and the fourth row in Figure 10, respectively, represent the enlarged results of the red rectangular boxes in the second and the third rows. When the optimization is over, the difference image is nearly black, which indicates that the simulated segmented image is very close to the real segmented image. In other words, the optimization achieves good convergence.Figure 10
The difference image at different stages of the optimization process.In the final error calculation, the space coordinates of the static marker targets reconstructed by a commercial structured-light device ATOS® are assumed as the true values. Accordingly, the midpoints of the 3D moving paths of the markers are taken as the final 3D reconstruction result for the error evaluation. So far, the whole 3D reconstruction procedure of the two types of the cooperative markers with motion blur effect is completed.


## 5. Experiments

To validate the effectiveness of the 3D reconstruction algorithm, the multiview images of the motion target under different exposure times are captured, and the intraframe motion paths of the motion blur markers are reconstructed.


### 5.1. Experimental Setup

We select the rotating ceiling fan blades as the experimental object. A synchronous controller is used to control the left and right cameras to obtain the multiview images of the moving blade at 19 Hz frequency. The image resolution is1392×1040, obtained by a Basler A102f camera.When the fan blades are in a static state, the camera exposure time is set to 1 ms, and the binocular stereo vision system is used to obtain clear images. When the fan blade rotates at a fixed speed, the camera exposure time is set to 0.6 ms, 0.8 ms, 1.0 ms, 1.2 ms, 1.4 ms, 1.6 ms, and 1.8 ms, respectively. Then, we shoot three sets of images with motion blur effect under high-speed motion at each exposure time.


### 5.2. Reconstruction Results

We perform 3D reconstruction and optimization on the markers in all the images and acquired the 3D coordinates of the marker points before and after the optimization. The time-consuming steps in the optimization process mainly include the initial value search of the marker space attitude and the minimization. The average time-consuming statistics are provided in Table1.Table 1
The mean time-consuming of different types of markers.
Time-consuming stepsThe initial search of the spatial attitudeMinimizationCCTs647.4 s523.9 sNoncoded targets12.6 s494.5 sIt can be seen from Table1 that the average times of the two types of markers are very close for the minimization optimization. In the process of searching for the initial value of the spatial pose, the time of noncoded targets to be searched is about 1/50 of the time of the CCTs to be searched. Therefore, using the noncoded targets as the 3D reconstruction target can greatly improve the optimization efficiency.We take the 3D coordinates of the markers reconstructed by the commercial ATOS system as the true values to measure the reconstruction accuracy. Due to the inconsistent coordinate systems between the two results, it is necessary to align the 3D points in a unique coordinate system through the best fitting function. Then, the errors of the reconstructed 3D coordinates of the markers are calculated. To validate the optimization algorithm based on the image difference minimization, the errors before and after the optimization are calculated. The results are shown in Table2.Table 2
The 3D coordinate reconstruction error of the markers.
Fan blade stateStaticRotate at a fixed speedExposure time (ms)1.00.60.81.01.21.41.61.8Before optimization (mm)0.1950.9141.2251.2511.4511.4921.5981.655After optimization (mm)0.0360.0750.0880.0860.0880.0910.0950.136


## 5.1. Experimental Setup

We select the rotating ceiling fan blades as the experimental object. A synchronous controller is used to control the left and right cameras to obtain the multiview images of the moving blade at 19 Hz frequency. The image resolution is1392×1040, obtained by a Basler A102f camera.When the fan blades are in a static state, the camera exposure time is set to 1 ms, and the binocular stereo vision system is used to obtain clear images. When the fan blade rotates at a fixed speed, the camera exposure time is set to 0.6 ms, 0.8 ms, 1.0 ms, 1.2 ms, 1.4 ms, 1.6 ms, and 1.8 ms, respectively. Then, we shoot three sets of images with motion blur effect under high-speed motion at each exposure time.


## 5.2. Reconstruction Results

We perform 3D reconstruction and optimization on the markers in all the images and acquired the 3D coordinates of the marker points before and after the optimization. The time-consuming steps in the optimization process mainly include the initial value search of the marker space attitude and the minimization. The average time-consuming statistics are provided in Table1.Table 1
The mean time-consuming of different types of markers.
Time-consuming stepsThe initial search of the spatial attitudeMinimizationCCTs647.4 s523.9 sNoncoded targets12.6 s494.5 sIt can be seen from Table1 that the average times of the two types of markers are very close for the minimization optimization. In the process of searching for the initial value of the spatial pose, the time of noncoded targets to be searched is about 1/50 of the time of the CCTs to be searched. Therefore, using the noncoded targets as the 3D reconstruction target can greatly improve the optimization efficiency.We take the 3D coordinates of the markers reconstructed by the commercial ATOS system as the true values to measure the reconstruction accuracy. Due to the inconsistent coordinate systems between the two results, it is necessary to align the 3D points in a unique coordinate system through the best fitting function. Then, the errors of the reconstructed 3D coordinates of the markers are calculated. To validate the optimization algorithm based on the image difference minimization, the errors before and after the optimization are calculated. The results are shown in Table2.Table 2
The 3D coordinate reconstruction error of the markers.
Fan blade stateStaticRotate at a fixed speedExposure time (ms)1.00.60.81.01.21.41.61.8Before optimization (mm)0.1950.9141.2251.2511.4511.4921.5981.655After optimization (mm)0.0360.0750.0880.0860.0880.0910.0950.136


## 6. Conclusions

The 3D reconstruction error after the optimization is at least one order of magnitude lower than that of before optimization. This indicates that the optimization algorithm based on the differential image has an obvious effect on improving the accuracy of the reconstructed 3D spatial coordinates of the cooperative markers.In the experiments, the rotating fan blade is not a rigid body due to the air perturbation, resulting in small changes in the relative position between the markers. Therefore, the actual errors of the reconstructed 3D coordinates of the markers might be smaller than the results provided in Table2.



---

*Source: 6467337-2021-09-06.xml*