# A Novel Method for Automatic Extrinsic Parameter Calibration of RGB-D Cameras

Authors: Qin Shi; Huansheng Song; Shijie Sun
Journal: Discrete Dynamics in Nature and Society
 (2021)

---

## Abstract

Calibration of extrinsic parameters of the RGB-D camera can be applied in many fields, such as 3D scene reconstruction, robotics, and target detection. Many calibration methods employ a specific calibration object (i.e., a chessboard, cuboid, etc.) to calibrate the extrinsic parameters of the RGB-D color camera without using the depth map. As a result, it is difficult to simplify the calibration process, and the color sensor gets calibrated instead of the depth sensor. To this end, we propose a method that employs the depth map to perform extrinsic calibration automatically. In detail, the depth map is first transformed to a 3D point cloud in the camera coordinate system, and then the planes in the 3D point cloud are automatically detected using the Maximum Likelihood Estimation Sample Consensus (MLESAC) method. After that, according to the constraint relationship between the ground plane and the world coordinate system, all planes are traversed and screened until the ground plane is obtained. Finally, the extrinsic parameters are calculated using the spatial relationship between the ground plane and the camera coordinate system. The results show that the mean roll angle error of extrinsic parameter calibration was −1.14°. The mean pitch angle error was 4.57°, and the mean camera height error was 3.96 cm. The proposed method can accurately and automatically estimate the extrinsic parameters of a camera. Furthermore, after parallel optimization, it can achieve real-time performance for automatically estimating a robot’s attitude.


---

## Body

## 1. Introduction

RGB-D cameras, such as Kinect [1–6], PrimeSense, and Asus Xtion Pro, are traditional RGB cameras with added infrared cameras. Figure 1 shows the structure of a Kinect camera. It can be seen that it includes a color camera, infrared camera, and an infrared illuminator. The color camera outputs a color image, the infrared camera outputs a depth image, and the infrared illuminator emits infrared light for the calculation of the depth image. With the emergence of low-cost RGB-D cameras such as Kinect, the camera and illuminator are increasingly used for tasks such as 3D scene reconstruction and navigation [8, 9], target recognition and tracking [10, 11], 3D measurement [12–15], and even social networks [16], for which the extrinsic parameters of the camera often must be calibrated.Figure 1
Sensor structure of the Kinect V2 [7].For example, in target detection and recognition [11], the extrinsic parameters of the camera should be calibrated first, and then pedestrians in the scene are tracked. If the extrinsic parameters of the camera can be obtained, the negative effects of perspective can be eliminated. In addition, the detection algorithm can be unified, and the recognition process can be simplified, thus accelerating recognition speed.However, studies of the calibration of RGB-D cameras mainly focus on the extrinsic parameters of the color camera relative to the infrared camera [17, 18]. Calibrating the extrinsic parameters of an RGB-D camera is often similar to the calibration of the color camera, and the method of chessboard calibration is generally used to calibrate the color camera [19]. Munaro et al. [10] used a chessboard to calibrate the extrinsic parameters of multiple cameras and carried out pedestrian detection based on this. This calibration method did not make full use of the depth information provided by the RGB-D camera, and the calibration results were essentially the extrinsic parameters of the color camera. If it were directly used for 3D reconstruction of the depth map, then there would be a large error. Shibo and Qing [20] designed a calibration board for RGB-D infrared camera recognition, which had holes with regular intervals and calibrated the infrared camera by automatically identifying holes. The need to design special calibration objects increased the difficulty of calibration. Liao et al. [17] divided the calibration into three categories: calibration that required a calibrator, calibration that required human intervention, and fully automatic calibration. The proposed method belonged to the third category.Liao et al. [17] divided the extrinsic parameter calibration methods for RGB-D cameras into three categories: (1) the first was to calibrate the extrinsic parameters of the color camera and then to use the extrinsic parameter TC′D of the color camera and images collected by the infrared camera to obtain the extrinsic parameter TD of the infrared camera through transformation TD=TC′DTC′. This method can directly use the color image calibration method, but it needs the extrinsic parameters of the color camera relative to the infrared camera and does not use the information provided by the depth map, so the process is complicated. (2) The second was to detect the features on the depth map provided by the infrared camera by designing a specific calibration object to obtain the extrinsic parameters of the camera based on the features (such as chessboard corners). (3) The third method used the depth map to calibrate the extrinsic parameters of the camera. The calibration was carried out by detecting the target on the depth map and using the relationship between the target and the world coordinate system. This method directly used the depth information, which greatly simplified the process and improved the efficiency of calibration. Our proposed method, which we will call the ground plane calibration method, is based on the automatic calibration of the extrinsic parameters of a ground plane detection camera and belongs to the third category. This method can directly calibrate the infrared camera of an RGB-D camera using its depth information.


## 2. Extrinsic Parameter Calibration

RGB-D cameras can be divided into color cameras and infrared cameras. For this study, the extrinsic parameters of an infrared camera were calibrated. Thus, extrinsic parameter calibration refers to that of infrared cameras. The extrinsic parameter of the camera is(1)TCW=RCW0tCW1,where TCW∈ℝ4×4 and RCW∈ℝ3×3 are the rotation matrices of the camera, tCW∈ℝ1×3 is the translation matrix of the camera, and ⋅CW represents the transformation from the camera coordinate system to the world coordinate system.Figure2 shows the flowchart of extrinsic parameter calibration. After the establishment of a world coordinate system, the depth image is first obtained from the RGB-D camera. Then, the depth image is transformed to a 3D point cloud under the camera coordinate system using the internal parameters of the infrared camera. After that, the ground plane in the point cloud is solved by iteration. Subsequently, the extrinsic parameters of the camera are obtained by calculation.Figure 2
Process of extrinsic parameter calibration of the camera.


### 2.1. Establishment of the Camera Coordinate System and World Coordinate System

The camera is in a 3D coordinate system, where the infrared camera is the origin, as shown in Figure3. The infrared camera is the origin OC of the camera coordinate system. The Xc axis is along the transverse direction of Kinect. The ZC is perpendicular to Kinect and points in the shooting direction.Figure 3
Camera coordinate.The world coordinates can generally be established at will. For facilitating the calculation of extrinsic parameters of the camera, as shown in Figure4, the world coordinates should meet the following requirements.(1)
The origin of the world coordinate system is the projection point of the origin of the camera coordinate system on the ground plane(2)
TheYW axis is the projection of the ZC axis of the camera coordinate system on the ground(3)
TheZW axis is downwardly perpendicular to the ground plane(4)
The coordinate system is a right-hand systemFigure 4
The position of world coordinate.In this way, it is convenient to find the point group corresponding to camera coordinates and world coordinates and simplify the calibration process.


### 2.2. Calculation of Extrinsic Parameters of the Camera

The matrix of transformation from the camera coordinate system to the world coordinate system, that is, the extrinsic parameterTCW of the camera, is a 4×4 matrix with 12 unknown parameters, as shown in (1). First, the transformation matrix TWC from the world coordinate system to the camera coordinate system is calculated to obtain the extrinsic parameters of the camera:(2)TCW=TWC−1.A point in the camera coordinate system isPCi, and the corresponding point in the world coordinate system is PWi, as shown in Figure 5. Four special points are selected in the world coordinate system:Figure 5
Selected points in world coordinate.(3)PW0=000,PW1=100,PW2=010,PW3=001.By calculating the corresponding points in the camera coordinate system, that is,PC0,PC1,PC2,andPC3, the value of TWC can be obtained as(4)TWC=PC1−PC0TPC1−PC0T2PC2−PC0TPC2−PC0T2PC3−PC0TPC3−PC0T20PC0PC021.It is known that the plane in the camera coordinate system isax+by+cz+d=0. Its normal vector perpendicular to the ground is n⟶C=abc and n⟶C2=1. As the origin of the world coordinate system is the projection of the origin of the camera coordinate system on the plane, the coordinates corresponding to the origin PW0 of the world coordinate in the camera coordinate system are(5)PC0=da2+b2+c2n⟶C.If pointPW3=001 is on the Z axis of the world coordinate system and OC,PC0,andPC3 are collinear, then(6)PC3=PC0+n⟶C.Because theYW axis is the projection of the ZC axis on the plane,(7)PC2=PC0+n⟶y,where n⟶y is the unit vector of PC0PC2⟶,(8)n⟶y=PZc−PC0⟶PZc−PC0⟶2.PZc is the projection of point 001 in the camera coordinate system to the plane. By solving (9), the projection point can be obtained as(9)ax+by+cz+d=0,x−x0a=y−y0b=z−z0c,where a,b,c,andd are the parameters of the plane equation and x0,y0,z0 is a random point on the plane.AfterPC0,PC2,andPC3 are obtained, PC1 can be obtained by vector cross-multiplication as(10)PC1=PC2−PC0⟶×PC3−PC0⟶PC2−PC0⟶×PC3−PC0⟶2.By solving (5)–(7) and (10), the coordinates of four points selected from the world coordinate system in the camera coordinate system are obtained. Then, the transformation matrix from the world coordinate point to the camera coordinate point is obtained by solving (4). Finally, the extrinsic parameter matrix of the infrared camera is obtained by solving (2).


## 2.1. Establishment of the Camera Coordinate System and World Coordinate System

The camera is in a 3D coordinate system, where the infrared camera is the origin, as shown in Figure3. The infrared camera is the origin OC of the camera coordinate system. The Xc axis is along the transverse direction of Kinect. The ZC is perpendicular to Kinect and points in the shooting direction.Figure 3
Camera coordinate.The world coordinates can generally be established at will. For facilitating the calculation of extrinsic parameters of the camera, as shown in Figure4, the world coordinates should meet the following requirements.(1)
The origin of the world coordinate system is the projection point of the origin of the camera coordinate system on the ground plane(2)
TheYW axis is the projection of the ZC axis of the camera coordinate system on the ground(3)
TheZW axis is downwardly perpendicular to the ground plane(4)
The coordinate system is a right-hand systemFigure 4
The position of world coordinate.In this way, it is convenient to find the point group corresponding to camera coordinates and world coordinates and simplify the calibration process.


## 2.2. Calculation of Extrinsic Parameters of the Camera

The matrix of transformation from the camera coordinate system to the world coordinate system, that is, the extrinsic parameterTCW of the camera, is a 4×4 matrix with 12 unknown parameters, as shown in (1). First, the transformation matrix TWC from the world coordinate system to the camera coordinate system is calculated to obtain the extrinsic parameters of the camera:(2)TCW=TWC−1.A point in the camera coordinate system isPCi, and the corresponding point in the world coordinate system is PWi, as shown in Figure 5. Four special points are selected in the world coordinate system:Figure 5
Selected points in world coordinate.(3)PW0=000,PW1=100,PW2=010,PW3=001.By calculating the corresponding points in the camera coordinate system, that is,PC0,PC1,PC2,andPC3, the value of TWC can be obtained as(4)TWC=PC1−PC0TPC1−PC0T2PC2−PC0TPC2−PC0T2PC3−PC0TPC3−PC0T20PC0PC021.It is known that the plane in the camera coordinate system isax+by+cz+d=0. Its normal vector perpendicular to the ground is n⟶C=abc and n⟶C2=1. As the origin of the world coordinate system is the projection of the origin of the camera coordinate system on the plane, the coordinates corresponding to the origin PW0 of the world coordinate in the camera coordinate system are(5)PC0=da2+b2+c2n⟶C.If pointPW3=001 is on the Z axis of the world coordinate system and OC,PC0,andPC3 are collinear, then(6)PC3=PC0+n⟶C.Because theYW axis is the projection of the ZC axis on the plane,(7)PC2=PC0+n⟶y,where n⟶y is the unit vector of PC0PC2⟶,(8)n⟶y=PZc−PC0⟶PZc−PC0⟶2.PZc is the projection of point 001 in the camera coordinate system to the plane. By solving (9), the projection point can be obtained as(9)ax+by+cz+d=0,x−x0a=y−y0b=z−z0c,where a,b,c,andd are the parameters of the plane equation and x0,y0,z0 is a random point on the plane.AfterPC0,PC2,andPC3 are obtained, PC1 can be obtained by vector cross-multiplication as(10)PC1=PC2−PC0⟶×PC3−PC0⟶PC2−PC0⟶×PC3−PC0⟶2.By solving (5)–(7) and (10), the coordinates of four points selected from the world coordinate system in the camera coordinate system are obtained. Then, the transformation matrix from the world coordinate point to the camera coordinate point is obtained by solving (4). Finally, the extrinsic parameter matrix of the infrared camera is obtained by solving (2).


## 3. Ground Plane Estimation

Toward facilitating plane detection, the depth map is first transformed to a 3D point cloud in the camera coordinate system using internal parameters. Then, the maximum likelihood estimation sample consensus (MLESAC) method [21] is used to extract the plane. The ergodic plane is iterated to obtain the extrinsic parameters. Next, the extrinsic parameters and point cloud are used to determine whether the plane is a ground plane, and we finally obtain the extrinsic parameters of the camera, as shown in Figure 6.Figure 6
The process of ground plane estimation.


### 3.1. Transformation of Depth Map to 3D Point Cloud in the Camera Coordinate System

Consider the following:(11)fx=fdx,fy=fdy,ZC=dp∗s,XC=uI−cx⋅ZCfx,YC=vI−cy⋅ZCfy.Figure7 shows the imaging model of the camera. The coordinate axes of the image plane are XI and YI, respectively. The imaging point of point PC=XC,YC,ZC in the camera coordinate system on the image plane is pI=uI,vI. The focal length of the camera is f. The coordinates of the intersection of the optical axis and the image plane are cx and cy. The length and width of the pixels are dx and dy, respectively. The value of the pixel is dp. The proportion of Z coordinate values corresponding to the value of pixels in the camera coordinate system is s.Figure 7
The model of the camera.Then,u,v,dp can be transformed to XC,YC,ZC, as shown in (11).


### 3.2. Ground Plane Estimation

Ground plane estimation is the basis of subsequent extrinsic parameter calculation of the camera. Its accuracy determines the accuracy of extrinsic camera parameters. In a scene, multiple planes will be fitted out. Then, whether the plane is a ground plane is determined according to the following conditions:(12)θ<π4,medianPWIZ>minPCOZ+ε,where θ is the angle between the Xc axis of the camera and the plane; median is the operation of taking the median value; PWIZ is the set of interior point Z-values of the fitted plane model; PCOZ is the set of exterior point Z-values of the fitted plane model; and ε is the set tolerance value. Equation (12) represents two conditions: (1) the inclination angle of the camera relative to the plane does not exceed 45° and (2) after the exterior parameter is calculated according to the plane, the point cloud is transformed to the world coordinate system. The point set on the plane has the largest Z-value.According to this definition, the planes were screened to calculate the qualified ground plane. The flowchart is shown in Figure6.First, the 3D point cloud in the camera coordinate system was input, and the plane was calculated using the MLESAC method. The set of interior points satisfying the plane was recorded. By using this plane and the method in Section2.2, the extrinsic parameters of the camera were calculated. Combined with the internal parameters of the camera, the 3D point cloud in the camera coordinate system was transformed to the 3D point cloud in the world coordinate system, and it was judged whether the conditional equation (12) was met. If not, then the recorded set of internal points was removed from the point cloud, and fitting of the plane continued. Otherwise, the plane was the ground plane. No further operation was conducted, and the extrinsic parameters of the camera were output.


## 3.1. Transformation of Depth Map to 3D Point Cloud in the Camera Coordinate System

Consider the following:(11)fx=fdx,fy=fdy,ZC=dp∗s,XC=uI−cx⋅ZCfx,YC=vI−cy⋅ZCfy.Figure7 shows the imaging model of the camera. The coordinate axes of the image plane are XI and YI, respectively. The imaging point of point PC=XC,YC,ZC in the camera coordinate system on the image plane is pI=uI,vI. The focal length of the camera is f. The coordinates of the intersection of the optical axis and the image plane are cx and cy. The length and width of the pixels are dx and dy, respectively. The value of the pixel is dp. The proportion of Z coordinate values corresponding to the value of pixels in the camera coordinate system is s.Figure 7
The model of the camera.Then,u,v,dp can be transformed to XC,YC,ZC, as shown in (11).


## 3.2. Ground Plane Estimation

Ground plane estimation is the basis of subsequent extrinsic parameter calculation of the camera. Its accuracy determines the accuracy of extrinsic camera parameters. In a scene, multiple planes will be fitted out. Then, whether the plane is a ground plane is determined according to the following conditions:(12)θ<π4,medianPWIZ>minPCOZ+ε,where θ is the angle between the Xc axis of the camera and the plane; median is the operation of taking the median value; PWIZ is the set of interior point Z-values of the fitted plane model; PCOZ is the set of exterior point Z-values of the fitted plane model; and ε is the set tolerance value. Equation (12) represents two conditions: (1) the inclination angle of the camera relative to the plane does not exceed 45° and (2) after the exterior parameter is calculated according to the plane, the point cloud is transformed to the world coordinate system. The point set on the plane has the largest Z-value.According to this definition, the planes were screened to calculate the qualified ground plane. The flowchart is shown in Figure6.First, the 3D point cloud in the camera coordinate system was input, and the plane was calculated using the MLESAC method. The set of interior points satisfying the plane was recorded. By using this plane and the method in Section2.2, the extrinsic parameters of the camera were calculated. Combined with the internal parameters of the camera, the 3D point cloud in the camera coordinate system was transformed to the 3D point cloud in the world coordinate system, and it was judged whether the conditional equation (12) was met. If not, then the recorded set of internal points was removed from the point cloud, and fitting of the plane continued. Otherwise, the plane was the ground plane. No further operation was conducted, and the extrinsic parameters of the camera were output.


## 4. Experiment

### 4.1. Experiment Process

In this experiment, a PrimeSense camera was used to collect video data, and MATLAB was used for simulation to validate the proposed algorithm. To facilitate the accuracy comparison, the chessboard calibration results of the color camera were used as the reference data. Because the matrix was not suitable for comparison, the extrinsic parameters were transformed to camera height (H), roll angle (θ), and pitch angle (ϕ). The calibration accuracy was measured by comparing the camera height, roll angle, and pitch angle. Following the world coordinates established in Section 2, the values can be obtained as(13)H=Zc,ϕ=arccosPc2⋅Pw2,θ=arccosPc0⋅Pw2,where Pc2 is the camera Z-axis in the world coordinate, Pc0 is the camera X-axis in the world coordinate, and Pw2 is the world Z-axis.The experiment was carried out according to the following steps.(1)
A PrimeSense camera was used to collect video data (each frame of the video had a clear chessboard), including color video and depth video.(2)
NN>20 video frames were selected randomly from the color video as the input to the Zhang camera calibration method [19], and the internal camera parameters were estimated.(3)
Each frame of the color video was traversed. The camera’s extrinsic parameters were estimated using the internal parameters and the chessboard corner detected by the current frame, and the camera attitudes (Hchessboard, ϕchessboard, and θchessboard) of each frame of color video were obtained.(4)
Each frame of depth video was traversed. The ground plane was detected using the proposed method, and the extrinsic parameters of the camera were estimated to obtain the camera attitudes (Hplane, ϕplane, and θplane) of each frame of depth video.The camera height difference (ΔH), roll angle difference (Δθ), and pitch angle difference (Δϕ) of each frame was calculated, using the following equations:(14)ΔH=Hchessboard−Hplane,Δϕ=ϕchessboard−ϕplane,Δθ=θchessboard−θplane.The video formats collected in Step 1 are shown in Table1.Table 1
Parameters of video.
Color videoDepth videoVideo size (pixel)320 × 240320 × 240Frame rate (frame/s)3030Duration (s)89.489.4Pixel byte (byte)2416Video formatRGB24Mono16Each frame of the video contained a chessboard, as shown in Figure8. In the experiment, the chessboard was used to estimate the camera’s extrinsic parameters, which were taken as the reference data. Because different chessboards corresponded to different internal parameters when the camera’s internal parameters were calculated using the Zhang calibration method [19], Steps 1 to 3 were executed 86 times. Each video frame could estimate 86 extrinsic parameters, with their medians as the reference data.Figure 8
Color video frame comprising chessboard.The size of a chessboard was7×9, and both the length and width of each chessboard were 40 mm. The camera’s extrinsic parameters, as obtained through chessboard calibration, are shown in Figure 8.In Figure9, θchessboard, ϕchessboard, and Hchessboard are the medians of the camera pitch angle, roll angle, and height, respectively, as measured using the chessboard method and θplane, ϕplane, and Hplane are the medians of the camera pitch angle, roll angle, and height, respectively, as measured using the proposed method.Figure 9
The result of the checkboard calibration method. The units ofϕchessboard, θchessboard, ϕplane, and θplane are degree, and the units of Hchessboard and Hplane are centimeter.Figure10 shows the experimental errors, and Table 2 shows the final measured results.Figure 10
Test result. The camera height difference (ΔH), roll angle difference (Δθ), and yaw angle difference (Δϕ) of each frame were calculated by (13). The units of Δθ and Δϕ are degree, and the unit of ΔH is centimeter.Table 2
Table of extrinsic parameters error of camera.
MaximumMinimumMeanθ7.49−0.754.47ϕ4.87−9.62−3.87H15.21−1.153.96


### 4.2. Error Analysis

Because there was no high-precision instrument to measure the extrinsic parameters of the camera in the experiment, these were measured using the chessboard calibration method and taken as the reference data to measure the accuracy of the proposed method. The factors affecting the accuracy of this experiment are as follows.(1)
First is quantization error of corner detection. Because the video frame used was240×320 pixels, the quantization error of corner detection was quite large when calculating the camera's internal and extrinsic parameters.(2)
The chessboard method and the proposed method were, respectively, used to measure the extrinsic parameters of the RGB-D camera's color sensor and infrared camera.(3)
The noise of the RGB-D camera scanning scene data would affect the detection of the ground plane, thus affecting the accuracy of the camera's extrinsic parameters.(4)
The parameters to stop MLESAC iteration in ground plane detection would also affect the results of this experiment.


### 4.3. Influence of Scene Noise on Ground Plane Detection

RGB-D cameras have many noise sources, such as temperature, incident angle and intensity of ambient light, and texture [22]. The MLESAC used in this study could deal with small amounts of noise, but not with scenes with too much noise or data loss.(1)
Strong sunlight would cause too many ground plane noise points, resulting in inaccurate estimation of plane parameters and a slight influence on the camera height and roll angle.(2)
If the reflectivity of the ground was too high (for example, a mirror was placed on the ground), the data of this area would be lost. If too much data were missing, then the ground plane could not be detected, and the extrinsic parameters of the camera could not be estimated.To verify the influence of noise on camera attitude, Figure11 shows the error change of camera attitude with the increase of Gaussian noise variance in the 929th frame. The mean value of Gaussian noise was 0, and the variance was δg. It can be seen that with the increase of noise variance, the height error increases, and the stability of pitch angle and roll angle decreases. In addition, when the variance is greater than 0.25, the plane cannot be correctly estimated.Figure 11
The effect of noise on the result. The error units of roll angle difference (Δθ) and yaw angle difference (Δϕ) are degrees, and the error unit of camera height difference (ΔH) is centimeter.


## 4.1. Experiment Process

In this experiment, a PrimeSense camera was used to collect video data, and MATLAB was used for simulation to validate the proposed algorithm. To facilitate the accuracy comparison, the chessboard calibration results of the color camera were used as the reference data. Because the matrix was not suitable for comparison, the extrinsic parameters were transformed to camera height (H), roll angle (θ), and pitch angle (ϕ). The calibration accuracy was measured by comparing the camera height, roll angle, and pitch angle. Following the world coordinates established in Section 2, the values can be obtained as(13)H=Zc,ϕ=arccosPc2⋅Pw2,θ=arccosPc0⋅Pw2,where Pc2 is the camera Z-axis in the world coordinate, Pc0 is the camera X-axis in the world coordinate, and Pw2 is the world Z-axis.The experiment was carried out according to the following steps.(1)
A PrimeSense camera was used to collect video data (each frame of the video had a clear chessboard), including color video and depth video.(2)
NN>20 video frames were selected randomly from the color video as the input to the Zhang camera calibration method [19], and the internal camera parameters were estimated.(3)
Each frame of the color video was traversed. The camera’s extrinsic parameters were estimated using the internal parameters and the chessboard corner detected by the current frame, and the camera attitudes (Hchessboard, ϕchessboard, and θchessboard) of each frame of color video were obtained.(4)
Each frame of depth video was traversed. The ground plane was detected using the proposed method, and the extrinsic parameters of the camera were estimated to obtain the camera attitudes (Hplane, ϕplane, and θplane) of each frame of depth video.The camera height difference (ΔH), roll angle difference (Δθ), and pitch angle difference (Δϕ) of each frame was calculated, using the following equations:(14)ΔH=Hchessboard−Hplane,Δϕ=ϕchessboard−ϕplane,Δθ=θchessboard−θplane.The video formats collected in Step 1 are shown in Table1.Table 1
Parameters of video.
Color videoDepth videoVideo size (pixel)320 × 240320 × 240Frame rate (frame/s)3030Duration (s)89.489.4Pixel byte (byte)2416Video formatRGB24Mono16Each frame of the video contained a chessboard, as shown in Figure8. In the experiment, the chessboard was used to estimate the camera’s extrinsic parameters, which were taken as the reference data. Because different chessboards corresponded to different internal parameters when the camera’s internal parameters were calculated using the Zhang calibration method [19], Steps 1 to 3 were executed 86 times. Each video frame could estimate 86 extrinsic parameters, with their medians as the reference data.Figure 8
Color video frame comprising chessboard.The size of a chessboard was7×9, and both the length and width of each chessboard were 40 mm. The camera’s extrinsic parameters, as obtained through chessboard calibration, are shown in Figure 8.In Figure9, θchessboard, ϕchessboard, and Hchessboard are the medians of the camera pitch angle, roll angle, and height, respectively, as measured using the chessboard method and θplane, ϕplane, and Hplane are the medians of the camera pitch angle, roll angle, and height, respectively, as measured using the proposed method.Figure 9
The result of the checkboard calibration method. The units ofϕchessboard, θchessboard, ϕplane, and θplane are degree, and the units of Hchessboard and Hplane are centimeter.Figure10 shows the experimental errors, and Table 2 shows the final measured results.Figure 10
Test result. The camera height difference (ΔH), roll angle difference (Δθ), and yaw angle difference (Δϕ) of each frame were calculated by (13). The units of Δθ and Δϕ are degree, and the unit of ΔH is centimeter.Table 2
Table of extrinsic parameters error of camera.
MaximumMinimumMeanθ7.49−0.754.47ϕ4.87−9.62−3.87H15.21−1.153.96


## 4.2. Error Analysis

Because there was no high-precision instrument to measure the extrinsic parameters of the camera in the experiment, these were measured using the chessboard calibration method and taken as the reference data to measure the accuracy of the proposed method. The factors affecting the accuracy of this experiment are as follows.(1)
First is quantization error of corner detection. Because the video frame used was240×320 pixels, the quantization error of corner detection was quite large when calculating the camera's internal and extrinsic parameters.(2)
The chessboard method and the proposed method were, respectively, used to measure the extrinsic parameters of the RGB-D camera's color sensor and infrared camera.(3)
The noise of the RGB-D camera scanning scene data would affect the detection of the ground plane, thus affecting the accuracy of the camera's extrinsic parameters.(4)
The parameters to stop MLESAC iteration in ground plane detection would also affect the results of this experiment.


## 4.3. Influence of Scene Noise on Ground Plane Detection

RGB-D cameras have many noise sources, such as temperature, incident angle and intensity of ambient light, and texture [22]. The MLESAC used in this study could deal with small amounts of noise, but not with scenes with too much noise or data loss.(1)
Strong sunlight would cause too many ground plane noise points, resulting in inaccurate estimation of plane parameters and a slight influence on the camera height and roll angle.(2)
If the reflectivity of the ground was too high (for example, a mirror was placed on the ground), the data of this area would be lost. If too much data were missing, then the ground plane could not be detected, and the extrinsic parameters of the camera could not be estimated.To verify the influence of noise on camera attitude, Figure11 shows the error change of camera attitude with the increase of Gaussian noise variance in the 929th frame. The mean value of Gaussian noise was 0, and the variance was δg. It can be seen that with the increase of noise variance, the height error increases, and the stability of pitch angle and roll angle decreases. In addition, when the variance is greater than 0.25, the plane cannot be correctly estimated.Figure 11
The effect of noise on the result. The error units of roll angle difference (Δθ) and yaw angle difference (Δϕ) are degrees, and the error unit of camera height difference (ΔH) is centimeter.


## 5. Conclusion

During pedestrian detection based on RGB-D cameras, due to the impact of environmental vibration, an RGB-D camera will shift its original position, and the extrinsic parameters will change greatly, which will directly affect the detection accuracy. By using the automatic extrinsic parameter calibration method proposed in this study, the extrinsic parameters can be automatically corrected when there is no pedestrian, so as to solve the above problem.The proposed method can be applied to the automatic adjustment of extrinsic parameters of 3D cameras (speckle, TOF, and binocular camera) and has high parallelism. After parallel implementation, it is instantaneous and can be used for the automatic calibration of extrinsic parameters of 3D cameras on mobile robots.In this method, we extracted the plane from the 3D point cloud and used the position relationship between the plane and world coordinates to obtain a camera’s extrinsic parameters, which were used to determine whether the current plane was a ground plane. If not, then the next plane would be used to calculate the extrinsic parameters until the ground plane was found, and the extrinsic parameters of the infrared camera were obtained. In this study, the conditions of the ground plane were given, which could guarantee the correctness of the established world coordinate system and creatively combine the plane detection with the extrinsic parameter estimation, so as to achieve the objective of automatic extrinsic parameter calibration. This method does not require an additional calibration object, and it is aimed at the calibration of infrared cameras. The results are reliable.The currently proposed method has the limitation that there needs to be a ground plane in the scene. If the ground plane cannot get detected, then the calibration cannot be carried out regularly. Two problems still must be solved: (1) to improve the accuracy of calibration and use targets such as pedestrians in scenes to carry out fine calibration and (2) to calibrate the extrinsic parameters of multiple cameras automatically according to the common area taken by multiple cameras. If no common area is taken by the two cameras, then a simple calibration object should be designed for automatic calibration to be carried out according to the geometric size of the calibration object.The MATLAB resource code and the video data used to support the findings of this study are available from the corresponding author upon request.



---

*Source: 5251898-2021-07-15.xml*