# Vision-Based Object Recognition and Precise Localization for Space Body Control

Authors: Zeyu Shangguan; Lingyu Wang; Jianquan Zhang; Wenbo Dong
Journal: International Journal of Aerospace Engineering
 (2019)

---

## Abstract

The space motion control is an important issue on space robot, rendezvous and docking, small satellite formation, and some on-orbit services. The motion control needs robust object detection and high-precision object localization. Among many sensing systems such as laser radar, inertia sensors, and GPS navigation, vision-based navigation is more adaptive to noncontact applications in the close distance and in high-dynamic environment. In this work, a vision-based system serving for a free-floating robot inside the spacecraft is introduced, and the method to measure space body 6-DOF position-attitude is presented. At first, the deep-learning method is applied for robust object detection in the complex background, and after the object is navigated at the close distance, the reference marker is used for more precise matching and edge detection. After the accurate coordinates are gotten in the image sequence, the object space position and attitude are calculated by the geometry method and used for fine control. The experimental results show that the recognition method based on deep-learning at a distance and marker matching in close range effectively eliminates the false target recognition and improves the precision of positioning at the same time. The testing result shows the recognition accuracy rate is 99.8% and the localization precision is far less than 1% in 1.5 meters. The high-speed camera and embedded electronic platform driven by GPU are applied for accelerating the image processing speed so that the system works at best by 70 frames per second. The contribution of this work is to introduce the deep-learning method for precision motion control and in the meanwhile ensure both the robustness and real time of the system. It aims at making such vision-based system more practicable in the real-space applications.


---

## Body

## 1. Introduction

Space programs on space robot, debris removing, rendezvous and docking, satellite formation, and other on-orbit service applications all involves the technology of moving body control [1–4]. The precondition for moving body control is to at first be acquainted with the body movement information, such as inertia, position, attitude, and velocity. Figure 1 shows several examples of on-orbit service application with vision system.Figure 1
Examples of on-orbit service application with vision system. (a) On-orbit servicing, (b) SHPERE, (c) minCAM, and (d) our system on the ground.
(a)
(b)
(c)
(d)Generally, there are many techniques for measuring the relative position and attitude between two objects. Sensors such as GPS, gyroscopes, accelerometers, and star sensors are commonly used for self-navigation, and their position information is sent to each other by wireless communication. Optical-electronic sensors such as laser radar and vision-based system may be more suitable to measure relative position and attitude when the two objects are at the close range, especially in autonomous vehicles or aircrafts [5–7]. In addition, the vision-based system based on computer vision is also widely used for object localization in industry manufacturing lines, medical instruments, and some intelligent applications. Since the vision-based measurement is of low cost and flexible to setup, vision system is increasingly applied in space body control.Vision system also has many schemes such as monocular vision [8], stereo vision [9], and active vision with structure light [10]. Besides, the active cameras such as Flash-LIDAR’s can be used to detect the unknown object [11]. For the noncooperative system localization, stereo vision and monocular vision [12] can both recognize the unknown objects by edge detection and feature matching. Active vision with structure light obtains the object 3D information when structure light scans the object surface, and it can not only help to recognize the object but also reconstruct the 3D information of unknown object. While monocular vision is more difficult to handle the unknown object, its precision and speed is not worse than stereo vision or active vision in applications of known target and environment.This work is involving in an on-orbit service application. A free-floating robot can move inside the spacecraft, and the function of the robot includes routing inspection, astronaut assistant, and autonomous docking and charging. Many similar programs have been carried out in the satellite or Space Station such as the SHPHEREs [13], SCAMP [14], mini AERCam [15], and Astrobee [16], which are shown in Figure 2. This kind of robot does not require space orbit control but only serves for relative movement inside the spacecraft. However, once the environment suitability permitted, it can also work out of the spacecraft. In this work [17], a vision navigation camera is configured on a robot, and another camera is fixed at the docking place to recognize this robot. These two kinds of vision system above have the same function to measure the relative position and attitude.Figure 2
System working mode: long-distance navigation and close dock in.The main problem of the positioning system is that the complex background and light environment may influence the image recognition. Another problem is the requirement of real-time processing speed and high precision in the control system. To resolve these problems, firstly, the deep-learning method is introduced robust for object detection and localization in the image sequence. Secondly, after the object is detected, the geometry of object position and attitude calculation is solved by P4P (perspective in 4 points) method, and the explicit calculation is gotten. The embedded electronic platform driven by GPU is applied for accelerating the image processing speed.The ground test platform is established, and its testing result indicates that our measures greatly improved the recognition accept rate so that the precision of object localization is up to 1% and the embedded platform can process the image sequence at best by 70 frames per second. The works aimed at making such vision-based system more practicable in the real dynamic environment.


## 2. System Design and Working Mode

The vision-based positioning system works as shown in Figure2. The system consists of a light source, camera and lens, and an embedded computer base on GPU and ARM. The system is tested in a scene with both simple and complex background that simulate the real-space environment. The reference marker with robust patterns of hamming code is set up both on the fixed place and on the robot, which are shown in Figure 2.The system works in two modes as follows:(1)
Long-Distance Navigation and Control. When the object is at a distance, the system recognizes the target from the image sequence and gets its coarse distance and position. This can be realized by identifying the position of the object located in the image and the image size d, then the coarse distance of the object can be known according to the real size of it. The robot is controlled close to the object under the help of a navigation system. In this working mode, we only estimate the position and distance of the object, without calculating its accurate attitudes
(2)
Close Docking and Precise Control. When the two objects are less than 1.5 meters apart, the camera seeks the known marker on the target and gets its accurate information to implement the precise control


## 3. Object Detection

According to the system working modes, the object detection process includes two parts: one is coarse recognition to find the object at a distance in the complex scenes, and the other is fine localization for calculating the accurate spatial position at close range.


### 3.1. Object Detection by Deep Learning

The traditional method to localize the target from the image is to directly adopt the precise image matching such as by SIFT features and ORB features [18, 19], but this method does not work well to distinguish the target in a complex background and may lead to mismatch and decrease in positioning accuracy. With the development of machine learning, the deep learning shows higher robustness and effectiveness to solve the problem of target detection in complex scenes [20, 21]. In this work, we use a method of target detection based on convolution neural network.The target detection process is as follows: firstly, prepare the training set, label the positive and negative samples, and train and get the model with this dataset. Then, the algorithm is embedded in the computer, and the software will automatically determine whether our target is in the scene.


### 3.2. Dataset Organization and Image Preprocessing

For offline training, we select about 4000 positive samples and 9700 negative samples. It is common practice to use about 2/3 to 4/5 samples for training. The training set and test set used in this paper is 7 : 3. The positive samples are captured in different angles and different light intensity, and some are half-sheltered as shown in Figure3. Negative samples without the target are also captured in the same environment, and some figures are downloaded from the internet as shown in Figure 4. Before data is imported into the network, the data need to be preprocessed. Firstly, all images are normalized to the dimension of 256×256 pixels before importing the network. Secondly, image type is converted to a certain format, so that training software can use it directly. Finally, the mean of the training set is generated to test the training accuracy.Figure 3
Positive sample set in different environment.Figure 4
Negative sample set in different environment.


### 3.3. Training Configuration

In this paper, the neural network model is divided into eight layers, five convolution layers, and three fully connected layers. In each convolution layer, the excitation function ReLU (rectified linear unit) and local response normalized LRN (Local Response Normalization) are included and then downsampling for pooling. The batch_size is set to 50, and the training sample has 9590 images, so it takes 192 iterations to complete all the samples once. The test sample has 4110 images, batch_size is 50, and it takes 83 times to complete the test at a time. We set the snapshot to 1000, and the loss curve and accuracy curve are shown in Figure5.Figure 5
Loss curve and accuracy curve during training.When the iterations are 768, the accuracy rate of training is 99.976% and the error rate of testing is 0.157%. The system basically reaches the steady state. And when the iterations are 30000, the model can basically meet the needs of this experiment. The training is done offline, and the time is about 60 hours in CPU or 24 hours in GPU.


### 3.4. Recognition Algorithm Test

To verify the reliability of the method, we use the database of 4110 samples to match, as shown in Table1. With the alternation of ratio values, 10 groups of FPR (true positive rate) and TPR (false positive rate) are obtained. The recognition algorithm is compared with the traditional image matching method of ORB features. The former results are FPR1 and TPR1 while the latter is FPR2 and TPR2. It is often recommended that the threshold should be taken as 0.8 for image matching. In our application, we set the threshold value between 0.4 and 0.6, where the matching result was the best. If the threshold value is greater, the generalization performance will decrease.Table 1
Image matching accuracy with different threshold.
Threshold
0.45
0.50
0.60
0.63
0.65
0.67
0.70
0.73
0.75
0.80
TPR1
23.36%
37.68%
72.28%
80.75%
85.73%
89.73%
94.19%
97.36%
98.91%
99.87%
FPR1
0
0
0
0
0
0.02%
0.06%
4.67%
18.21%
79.94%
TPR2
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
FPR2
0
0
0
0
0
0
0
0
0
0Among the 4110 tested samples, the method with CNN has no mistake to identified negative samples as positive samples, and only 21 positive samples are not identified. The accuracy rate reached 99.847%, which is far better than the traditional object detection method. It benefits from the CNN and deep-learning technology.


### 3.5. Marker Recognition and Fine Localization

Marker recognition is performed when the object is less than the range of 1.5 meter and after the object has been recognized to a small range. Marker detection includes the following steps: firstly, the image is normalized and converted to binary image with a certain threshold. Then, after image segmentation and contour extraction, the polygon is approximated and four vertices of polygons are found. According to the polygon position, the polygon image can be segmented and represented by 1 and 0. This binary vector is matched with hamming codes, and the number of the April Tag and its orientation is recognized in the image, as shown in Figure6. In addition, the marker is not the only target we can use during this stage. In fact, we tried other targets that we can apply, as shown in Figure 7. However, considering the self-correcting property of the marker, the stability of it is better.Figure 6
Reference marker recognition and decoding process.Figure 7
Various targets we can choose.After the four vertices of the marker are confirmed, they are used for further calculation. Considering that if one marker covers 50 pixels in the640×480 image, then the precision of one pixel leads to 1% error of localization, that is, 10 mm within the range of 1 m. For more precise localization, the coordinates of four control points are corrected to subpixel, and the error could decrease to 0.1%. There are many methods for subpixel edge detection, and we directly use the classic function of OpenCV for subpixel processing in this work. The subpixel edge detection is as shown in Figure 8. Since the subpixel detection is time-consuming, it is only used in fine operation control.Figure 8
Schematic picture of subpixel edge detection.


## 3.1. Object Detection by Deep Learning

The traditional method to localize the target from the image is to directly adopt the precise image matching such as by SIFT features and ORB features [18, 19], but this method does not work well to distinguish the target in a complex background and may lead to mismatch and decrease in positioning accuracy. With the development of machine learning, the deep learning shows higher robustness and effectiveness to solve the problem of target detection in complex scenes [20, 21]. In this work, we use a method of target detection based on convolution neural network.The target detection process is as follows: firstly, prepare the training set, label the positive and negative samples, and train and get the model with this dataset. Then, the algorithm is embedded in the computer, and the software will automatically determine whether our target is in the scene.


## 3.2. Dataset Organization and Image Preprocessing

For offline training, we select about 4000 positive samples and 9700 negative samples. It is common practice to use about 2/3 to 4/5 samples for training. The training set and test set used in this paper is 7 : 3. The positive samples are captured in different angles and different light intensity, and some are half-sheltered as shown in Figure3. Negative samples without the target are also captured in the same environment, and some figures are downloaded from the internet as shown in Figure 4. Before data is imported into the network, the data need to be preprocessed. Firstly, all images are normalized to the dimension of 256×256 pixels before importing the network. Secondly, image type is converted to a certain format, so that training software can use it directly. Finally, the mean of the training set is generated to test the training accuracy.Figure 3
Positive sample set in different environment.Figure 4
Negative sample set in different environment.


## 3.3. Training Configuration

In this paper, the neural network model is divided into eight layers, five convolution layers, and three fully connected layers. In each convolution layer, the excitation function ReLU (rectified linear unit) and local response normalized LRN (Local Response Normalization) are included and then downsampling for pooling. The batch_size is set to 50, and the training sample has 9590 images, so it takes 192 iterations to complete all the samples once. The test sample has 4110 images, batch_size is 50, and it takes 83 times to complete the test at a time. We set the snapshot to 1000, and the loss curve and accuracy curve are shown in Figure5.Figure 5
Loss curve and accuracy curve during training.When the iterations are 768, the accuracy rate of training is 99.976% and the error rate of testing is 0.157%. The system basically reaches the steady state. And when the iterations are 30000, the model can basically meet the needs of this experiment. The training is done offline, and the time is about 60 hours in CPU or 24 hours in GPU.


## 3.4. Recognition Algorithm Test

To verify the reliability of the method, we use the database of 4110 samples to match, as shown in Table1. With the alternation of ratio values, 10 groups of FPR (true positive rate) and TPR (false positive rate) are obtained. The recognition algorithm is compared with the traditional image matching method of ORB features. The former results are FPR1 and TPR1 while the latter is FPR2 and TPR2. It is often recommended that the threshold should be taken as 0.8 for image matching. In our application, we set the threshold value between 0.4 and 0.6, where the matching result was the best. If the threshold value is greater, the generalization performance will decrease.Table 1
Image matching accuracy with different threshold.
Threshold
0.45
0.50
0.60
0.63
0.65
0.67
0.70
0.73
0.75
0.80
TPR1
23.36%
37.68%
72.28%
80.75%
85.73%
89.73%
94.19%
97.36%
98.91%
99.87%
FPR1
0
0
0
0
0
0.02%
0.06%
4.67%
18.21%
79.94%
TPR2
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
99.85%
FPR2
0
0
0
0
0
0
0
0
0
0Among the 4110 tested samples, the method with CNN has no mistake to identified negative samples as positive samples, and only 21 positive samples are not identified. The accuracy rate reached 99.847%, which is far better than the traditional object detection method. It benefits from the CNN and deep-learning technology.


## 3.5. Marker Recognition and Fine Localization

Marker recognition is performed when the object is less than the range of 1.5 meter and after the object has been recognized to a small range. Marker detection includes the following steps: firstly, the image is normalized and converted to binary image with a certain threshold. Then, after image segmentation and contour extraction, the polygon is approximated and four vertices of polygons are found. According to the polygon position, the polygon image can be segmented and represented by 1 and 0. This binary vector is matched with hamming codes, and the number of the April Tag and its orientation is recognized in the image, as shown in Figure6. In addition, the marker is not the only target we can use during this stage. In fact, we tried other targets that we can apply, as shown in Figure 7. However, considering the self-correcting property of the marker, the stability of it is better.Figure 6
Reference marker recognition and decoding process.Figure 7
Various targets we can choose.After the four vertices of the marker are confirmed, they are used for further calculation. Considering that if one marker covers 50 pixels in the640×480 image, then the precision of one pixel leads to 1% error of localization, that is, 10 mm within the range of 1 m. For more precise localization, the coordinates of four control points are corrected to subpixel, and the error could decrease to 0.1%. There are many methods for subpixel edge detection, and we directly use the classic function of OpenCV for subpixel processing in this work. The subpixel edge detection is as shown in Figure 8. Since the subpixel detection is time-consuming, it is only used in fine operation control.Figure 8
Schematic picture of subpixel edge detection.


## 4. Position-Attitude Measurement

### 4.1. The Localization Algorithm

In the camera projective model, define the center of the marker as (XW, YW, ZW) in the marker coordinate system and (XC, YC, ZC) in the camera coordinate,Here, the translation vector isT=TX,TY,TZT and the rotation matrix is
(1)R=r11r12r13r21r22r23r31r32r33.The translation vectorT can represent the three-dimensional position of the target in the camera coordinate system, and the rotation matrix R can characterize the attitude of the target in the camera coordinate system. Support P0 is the original position of the moving object. The position measurement question can be shown as the below images: the question to calculate the robot’s position and attitude relative to the camera. Here is the equivalent to resolve the matrix rotation matrix R and translation vector T in
(2)XCYCZC=RXWYWZW+T.LetP0, P1, P2, P3 as four coordinates of the marker vertices in the space and q0, q1, q2, q3 as their projection coordinates in the camera plane, which is shown in Figure 9. Suppose that c, P0P1//P2P3, P0P1=d1, P0P3=d2, and the camera intrinsic parameters are known. The question to resolve the R and T is a classical P4P problem [22].Figure 9
Three-dimensional pose solution.Define the control points in the camera coordinate asq0x0,y0,f, q1x1,y1,f, and q2x2,y2,f, where f is the camera focus, and define the optical center as OC, the plane π1 is formed by OC and q0, q1. The normal vector of π1 is N1=nx,ny,nzT, which can be calculated by linear equation of q0, q1 and camera intrinsic parameter.Letk0=OCP0/OCq0, k1=OCP1/OCq1.Since vectorP0P1 is perpendicular to vector N1, there is
(3)k1x1−k0x0⋅nx+k1y1−k0y0⋅ny+k1f−k0f⋅nz=0,(4)k1x1−k0x02+k1y1−k0y02+k1f−k0f2=d1.From equations (3)~(4), k0 and k1 can be solved, and correspondingly P0 and P1 is solved as xP0,yP0,zP0=k0x0,k0y0,k0f, xP1,yP1,zP1=k1x1,k1y1,k1f. If we take P0 as the origin point, then the displacement is T=xP0,yP0,zP0T.TheXW axis in the camera coordinate is xP1−xP0,yP1−yP0,zP1−zP0 and is normalized as r11,r21,r31, which is the first column of rotation matrix R.TheYW axis in the camera coordinate is xP3−xP0,yP3−yP0,zP3−zP0 and is normalized as r12,r22,r32, which is the second column of rotation matrix R.TheZW axis in the camera coordinate r13,r23,r33 can be calculated by the cross multiplication as
(5)r13,r23,r33=r11,r21,r31×r12,r22,r32.It is the last column ofR, and by now the rotation matrix R has been solved.


## 4.1. The Localization Algorithm

In the camera projective model, define the center of the marker as (XW, YW, ZW) in the marker coordinate system and (XC, YC, ZC) in the camera coordinate,Here, the translation vector isT=TX,TY,TZT and the rotation matrix is
(1)R=r11r12r13r21r22r23r31r32r33.The translation vectorT can represent the three-dimensional position of the target in the camera coordinate system, and the rotation matrix R can characterize the attitude of the target in the camera coordinate system. Support P0 is the original position of the moving object. The position measurement question can be shown as the below images: the question to calculate the robot’s position and attitude relative to the camera. Here is the equivalent to resolve the matrix rotation matrix R and translation vector T in
(2)XCYCZC=RXWYWZW+T.LetP0, P1, P2, P3 as four coordinates of the marker vertices in the space and q0, q1, q2, q3 as their projection coordinates in the camera plane, which is shown in Figure 9. Suppose that c, P0P1//P2P3, P0P1=d1, P0P3=d2, and the camera intrinsic parameters are known. The question to resolve the R and T is a classical P4P problem [22].Figure 9
Three-dimensional pose solution.Define the control points in the camera coordinate asq0x0,y0,f, q1x1,y1,f, and q2x2,y2,f, where f is the camera focus, and define the optical center as OC, the plane π1 is formed by OC and q0, q1. The normal vector of π1 is N1=nx,ny,nzT, which can be calculated by linear equation of q0, q1 and camera intrinsic parameter.Letk0=OCP0/OCq0, k1=OCP1/OCq1.Since vectorP0P1 is perpendicular to vector N1, there is
(3)k1x1−k0x0⋅nx+k1y1−k0y0⋅ny+k1f−k0f⋅nz=0,(4)k1x1−k0x02+k1y1−k0y02+k1f−k0f2=d1.From equations (3)~(4), k0 and k1 can be solved, and correspondingly P0 and P1 is solved as xP0,yP0,zP0=k0x0,k0y0,k0f, xP1,yP1,zP1=k1x1,k1y1,k1f. If we take P0 as the origin point, then the displacement is T=xP0,yP0,zP0T.TheXW axis in the camera coordinate is xP1−xP0,yP1−yP0,zP1−zP0 and is normalized as r11,r21,r31, which is the first column of rotation matrix R.TheYW axis in the camera coordinate is xP3−xP0,yP3−yP0,zP3−zP0 and is normalized as r12,r22,r32, which is the second column of rotation matrix R.TheZW axis in the camera coordinate r13,r23,r33 can be calculated by the cross multiplication as
(5)r13,r23,r33=r11,r21,r31×r12,r22,r32.It is the last column ofR, and by now the rotation matrix R has been solved.


## 5. Precision and Speed Testing

### 5.1. Experiment Platform

#### 5.1.1. Experiment Design

A platform is set up to validate the precision of the system as shown in Figure10. The reference marker is fixed on a 6-D precision displacement table, and the camera is installed on the other mechanic table. We record the displacement of the 6-DOF table as the real value, and the calculated displacement by vision system as the test value. The difference between the real value and the test value represents the precision of the system.Figure 10
Precision experiment table.When the mechanical platform is moving, the computer software samples the image sequence, calculates the accurate position and attitude information of each picture, and draws the moving trajectory in the real time, as shown in Figure11. The red square indicates the marker in camera coordinate. To reflect the direction of the marker, the upper left corner of it is defined as the first corner when it is standing and the other three corners in the counterclockwise order.Figure 11
Real-time trajectory of the marker.


#### 5.1.2. Camera Calibration

Before the setup of the camera, the camera should be calibrated offline at first. The camera intrinsic parameter commonly is described as(6)A=αxγu00αyv0001,where αx=f/dX is the normalized focus of u axis, αy=f/dY is of v axis, γ is the distortion factor, and u0, v0 is the image coordinate of optical center. These factors only depend on the camera itself and can be calibrated in advance.The method of camera calibration is a commonly used operation. According the theory of Zhang [23], we only need one 2D checkerboard marker and a picture of it in various angle and position, then the inner parameters can be obtained by the least-square method.In this work, we use a 5 mm lens and a camera with 640∗480 gray in 1/3 inch; the camera parameters are calibrated as Table 2.Table 2
Intrinsic parameter result of camera calibration.
αx/pixels
αy/pixels
γ
u0/pixels
v0/pixels
k1
k2
1338.84
1345.02
0
618.15
475.73
-0.1116
0.1242According to Table2, the normalized focus of u and v axes is αx,αy=1338.84,1345.02, the image coordinate of the optical center is u0,v0=618.15,475.73, the distortion factor of u and v axes is γ=0, and the radial distortion parameter is k=−0.1116,0.1242.


### 5.2. Precision Test Results

#### 5.2.1. Single-Dimension Precision Test

Move the mechanical table, respectively, inX\Y\Z and Rx\Ry\Rz axes, and test whether the calculated value is consistent with the real value. Figure 12 is the test results in Z and RyRz, and the error is less than 1%, except for some points in the side face of Ry.Figure 12
The calculated value when the object moves in different dimension. (a) Move inZ axis, (b) move in SZ axis, and (c) Move in SY axis.
(a)
(b)
(c)


#### 5.2.2. Multidimension Decoupling Test

Since the measurement of multiple degrees of freedom may have the coupling and its error model is difficult to analysis, the decoupling influence must be given by an actual test.The corresponding experimental schematic diagram of the multidimension decoupling test is as shown in Figure13, and Figure 14 is the curve when the object moves in a rectangle of 40×60 cm at a surface. At the same time, we record the value change of Z axis and RZ; the results show the decoupling with Z-XY and RZ-XY is, respectively, less than 1 mm and 2 degrees. This error may be caused by the camera calibration or the position calculation or other system error.Figure 13
The corresponding experimental schematic diagram of the multidimension decoupling test. 1: the moving trajectory, 2: marker, and 3: camera.Figure 14
The coupling value when the object moves in a X-Y plane. (a) The XY curve, (b) Z-XY curve, and (c) Rz-XY curve.
(a)
(b)
(c)


### 5.3. Real-Time Speed Test Results

For real-time operation, we choose high-speed camera with low exposure time and use a GPU embedded platform to implement and speed up the image processing algorithm. Here, the camera is from Basler gc300 (640×480 and 300 fps), and GPU is the Tegra TX1 of NVIDIA company. These industry components are for ground test and may be enhanced for space environment.GPU performs well on big-size images and can be accelerated for image processing and CNN applications [24]. FPGA performs well on small-size images and parallel processing, and the heterogeneous computing with GPU and FPGA will be more powerful for real-time applications. The information below is only for the GPU as the processor individually.The vision-based system works with the following steps:(1)
Image sequence sampling
(2)
Image normalization or image preprocessing
(3)
Image recognition and matching
(4)
Position and attitude calculationAmong them, the image sequence sampling is mainly related with the camera performance, and this step costs the most time. For image processing, the processing time depends on the working modes. The image recognition by CNN is a little time-consuming, since it will seek and match the feature in the whole image. When the objects are less than 1.5 meters, the recognition algorithm is much faster, since the reference marker feature is matched in a small range and the image has been converted to binary image. The time is shown in the Table3.Table 3
Time consuming for object detection and localization.
Time consuming (ms)
GPU
CPU
Image sampling and buffering
3 ms for 1 frame
3 ms for 1 frame
Image preprocessing
0.5 ms
2 ms
Object detection (navigation mode)
~30 ms
~100 ms
Object tracking (docking mode)
~10 ms
~14 ms
Coordinates calculating
<0.5 ms
<0.5 ms
Summary
Navigation mode 34 ms (~30 Hz)
Navigation mode 105 ms (~10 Hz)
Docking mode 14 ms (~70 Hz)
Docking mode 19.5 ms (~20 Hz)As a conclusion in this chapter, ground experiment shows that the system sampling and the fasted processing speed can get up to 70 Hz in docking mode and 30 Hz in navigation mode, which has been able to satisfy most demand of vision-based control system. Actually, there are more strategies to improve the system speed, such as using the image tracking method to reduce the seeking size of image, or optimizing GPU program codes in CUDA accelerator, that will be discussed in the future.


## 5.1. Experiment Platform

### 5.1.1. Experiment Design

A platform is set up to validate the precision of the system as shown in Figure10. The reference marker is fixed on a 6-D precision displacement table, and the camera is installed on the other mechanic table. We record the displacement of the 6-DOF table as the real value, and the calculated displacement by vision system as the test value. The difference between the real value and the test value represents the precision of the system.Figure 10
Precision experiment table.When the mechanical platform is moving, the computer software samples the image sequence, calculates the accurate position and attitude information of each picture, and draws the moving trajectory in the real time, as shown in Figure11. The red square indicates the marker in camera coordinate. To reflect the direction of the marker, the upper left corner of it is defined as the first corner when it is standing and the other three corners in the counterclockwise order.Figure 11
Real-time trajectory of the marker.


### 5.1.2. Camera Calibration

Before the setup of the camera, the camera should be calibrated offline at first. The camera intrinsic parameter commonly is described as(6)A=αxγu00αyv0001,where αx=f/dX is the normalized focus of u axis, αy=f/dY is of v axis, γ is the distortion factor, and u0, v0 is the image coordinate of optical center. These factors only depend on the camera itself and can be calibrated in advance.The method of camera calibration is a commonly used operation. According the theory of Zhang [23], we only need one 2D checkerboard marker and a picture of it in various angle and position, then the inner parameters can be obtained by the least-square method.In this work, we use a 5 mm lens and a camera with 640∗480 gray in 1/3 inch; the camera parameters are calibrated as Table 2.Table 2
Intrinsic parameter result of camera calibration.
αx/pixels
αy/pixels
γ
u0/pixels
v0/pixels
k1
k2
1338.84
1345.02
0
618.15
475.73
-0.1116
0.1242According to Table2, the normalized focus of u and v axes is αx,αy=1338.84,1345.02, the image coordinate of the optical center is u0,v0=618.15,475.73, the distortion factor of u and v axes is γ=0, and the radial distortion parameter is k=−0.1116,0.1242.


## 5.1.1. Experiment Design

A platform is set up to validate the precision of the system as shown in Figure10. The reference marker is fixed on a 6-D precision displacement table, and the camera is installed on the other mechanic table. We record the displacement of the 6-DOF table as the real value, and the calculated displacement by vision system as the test value. The difference between the real value and the test value represents the precision of the system.Figure 10
Precision experiment table.When the mechanical platform is moving, the computer software samples the image sequence, calculates the accurate position and attitude information of each picture, and draws the moving trajectory in the real time, as shown in Figure11. The red square indicates the marker in camera coordinate. To reflect the direction of the marker, the upper left corner of it is defined as the first corner when it is standing and the other three corners in the counterclockwise order.Figure 11
Real-time trajectory of the marker.


## 5.1.2. Camera Calibration

Before the setup of the camera, the camera should be calibrated offline at first. The camera intrinsic parameter commonly is described as(6)A=αxγu00αyv0001,where αx=f/dX is the normalized focus of u axis, αy=f/dY is of v axis, γ is the distortion factor, and u0, v0 is the image coordinate of optical center. These factors only depend on the camera itself and can be calibrated in advance.The method of camera calibration is a commonly used operation. According the theory of Zhang [23], we only need one 2D checkerboard marker and a picture of it in various angle and position, then the inner parameters can be obtained by the least-square method.In this work, we use a 5 mm lens and a camera with 640∗480 gray in 1/3 inch; the camera parameters are calibrated as Table 2.Table 2
Intrinsic parameter result of camera calibration.
αx/pixels
αy/pixels
γ
u0/pixels
v0/pixels
k1
k2
1338.84
1345.02
0
618.15
475.73
-0.1116
0.1242According to Table2, the normalized focus of u and v axes is αx,αy=1338.84,1345.02, the image coordinate of the optical center is u0,v0=618.15,475.73, the distortion factor of u and v axes is γ=0, and the radial distortion parameter is k=−0.1116,0.1242.


## 5.2. Precision Test Results

### 5.2.1. Single-Dimension Precision Test

Move the mechanical table, respectively, inX\Y\Z and Rx\Ry\Rz axes, and test whether the calculated value is consistent with the real value. Figure 12 is the test results in Z and RyRz, and the error is less than 1%, except for some points in the side face of Ry.Figure 12
The calculated value when the object moves in different dimension. (a) Move inZ axis, (b) move in SZ axis, and (c) Move in SY axis.
(a)
(b)
(c)


### 5.2.2. Multidimension Decoupling Test

Since the measurement of multiple degrees of freedom may have the coupling and its error model is difficult to analysis, the decoupling influence must be given by an actual test.The corresponding experimental schematic diagram of the multidimension decoupling test is as shown in Figure13, and Figure 14 is the curve when the object moves in a rectangle of 40×60 cm at a surface. At the same time, we record the value change of Z axis and RZ; the results show the decoupling with Z-XY and RZ-XY is, respectively, less than 1 mm and 2 degrees. This error may be caused by the camera calibration or the position calculation or other system error.Figure 13
The corresponding experimental schematic diagram of the multidimension decoupling test. 1: the moving trajectory, 2: marker, and 3: camera.Figure 14
The coupling value when the object moves in a X-Y plane. (a) The XY curve, (b) Z-XY curve, and (c) Rz-XY curve.
(a)
(b)
(c)


## 5.2.1. Single-Dimension Precision Test

Move the mechanical table, respectively, inX\Y\Z and Rx\Ry\Rz axes, and test whether the calculated value is consistent with the real value. Figure 12 is the test results in Z and RyRz, and the error is less than 1%, except for some points in the side face of Ry.Figure 12
The calculated value when the object moves in different dimension. (a) Move inZ axis, (b) move in SZ axis, and (c) Move in SY axis.
(a)
(b)
(c)


## 5.2.2. Multidimension Decoupling Test

Since the measurement of multiple degrees of freedom may have the coupling and its error model is difficult to analysis, the decoupling influence must be given by an actual test.The corresponding experimental schematic diagram of the multidimension decoupling test is as shown in Figure13, and Figure 14 is the curve when the object moves in a rectangle of 40×60 cm at a surface. At the same time, we record the value change of Z axis and RZ; the results show the decoupling with Z-XY and RZ-XY is, respectively, less than 1 mm and 2 degrees. This error may be caused by the camera calibration or the position calculation or other system error.Figure 13
The corresponding experimental schematic diagram of the multidimension decoupling test. 1: the moving trajectory, 2: marker, and 3: camera.Figure 14
The coupling value when the object moves in a X-Y plane. (a) The XY curve, (b) Z-XY curve, and (c) Rz-XY curve.
(a)
(b)
(c)


## 5.3. Real-Time Speed Test Results

For real-time operation, we choose high-speed camera with low exposure time and use a GPU embedded platform to implement and speed up the image processing algorithm. Here, the camera is from Basler gc300 (640×480 and 300 fps), and GPU is the Tegra TX1 of NVIDIA company. These industry components are for ground test and may be enhanced for space environment.GPU performs well on big-size images and can be accelerated for image processing and CNN applications [24]. FPGA performs well on small-size images and parallel processing, and the heterogeneous computing with GPU and FPGA will be more powerful for real-time applications. The information below is only for the GPU as the processor individually.The vision-based system works with the following steps:(1)
Image sequence sampling
(2)
Image normalization or image preprocessing
(3)
Image recognition and matching
(4)
Position and attitude calculationAmong them, the image sequence sampling is mainly related with the camera performance, and this step costs the most time. For image processing, the processing time depends on the working modes. The image recognition by CNN is a little time-consuming, since it will seek and match the feature in the whole image. When the objects are less than 1.5 meters, the recognition algorithm is much faster, since the reference marker feature is matched in a small range and the image has been converted to binary image. The time is shown in the Table3.Table 3
Time consuming for object detection and localization.
Time consuming (ms)
GPU
CPU
Image sampling and buffering
3 ms for 1 frame
3 ms for 1 frame
Image preprocessing
0.5 ms
2 ms
Object detection (navigation mode)
~30 ms
~100 ms
Object tracking (docking mode)
~10 ms
~14 ms
Coordinates calculating
<0.5 ms
<0.5 ms
Summary
Navigation mode 34 ms (~30 Hz)
Navigation mode 105 ms (~10 Hz)
Docking mode 14 ms (~70 Hz)
Docking mode 19.5 ms (~20 Hz)As a conclusion in this chapter, ground experiment shows that the system sampling and the fasted processing speed can get up to 70 Hz in docking mode and 30 Hz in navigation mode, which has been able to satisfy most demand of vision-based control system. Actually, there are more strategies to improve the system speed, such as using the image tracking method to reduce the seeking size of image, or optimizing GPU program codes in CUDA accelerator, that will be discussed in the future.


## 6. Conclusions

A vision-based system that is mainly for space body 6-DOF position-attitude measurement and control is introduced in this paper. The configuration of this system, the image processing and object detection algorithms, and the position-attitude measurement formula are given. Our work mainly focuses on the practical problems for the system. Firstly, we bring forward CNN with deep-learning method for object detection and get 99.8% accuracy rate. Secondly, we resolve the P4P questions for object position and attitude problem and test it on the ground. The results show the precision is far less than 1% of range. The third is we use high-speed camera and GPU as the processor and accelerate the system to nearly 70 frames per second. The computer technology’s fast development such as GPU and deep-learning brings us great benefits for object detection applications. In the future, we will continue to optimize algorithms and reduce the time consuming and ensure this kind of vision-based system more robust and faster and to be used in more space applications.



---

*Source: 7050915-2019-03-25.xml*