# Calibration of Camera and Flash LiDAR System with a Triangular Pyramid Target
## 3 Method
### Methodology
#### 3.2.2 Extrinsic Parameters Calibration between Camera and Flash LiDAR  
Our method applies a geometric constraint of the triangular pyramid target to calibrate the extrinsic parameters. The method estimates the rotation matrix and translation vector from \(O_{W}-X_{W}Y_{W}Z_{W}\) to \(O_{\mathbb{C}}-X_{\mathbb{C}}Y_{\mathbb{C}}Z_{\mathbb{C}}\) (and \(O_{L}-X_{\mathbb{L}}Y_{\mathbb{L}}Z_{L}\)) separately. Finally, we can obtain the extrinsic parameters from \(O_{\mathbb{C}}-X_{\mathbb{C}}Y_{\mathbb{C}}Z_{\mathbb{C}}\) to \(O_{L}-X_{\mathbb{L}}Y_{\mathbb{L}}Z_{L}\). It is essential to convert the pixel values of LiDAR to 3D point clouds. As mentioned in Section 3.1.1, the output of flash LiDAR is the distance between the object and LiDAR on each pixel. The relation between the coordinate of the arbitrary point \(P(x,y,z)\) in \(O_{L}-X_{\mathbb{L}}Y_{\mathbb{L}}Z_{L}\) and distance is as follows:  
\[d=\sqrt{x^{2}+y^{2}+z^{2}}, \tag{7}\]  
where \(d\) represents the distance between origin \(O_{L}\) and point \(P\), which can be acquired by each pixel value of LiDAR. Utilizing the intrinsic parameters of flash LiDAR offered by manufacturers, we can obtain the relationship between the pixel coordinate system and LiDAR coordinate system according to Equation (1):  
\[\left\{ \begin{array}{c}x=\dfrac{\left(u-u_{0}\right)d}{f_{\mathbb{x}}\sqrt{ \left(u-u_{0}\right)/f_{\mathbb{x}}}^{2}+\left(\left(v-v_{0}\right)/f_{ \mathbb{y}}\right)^{2}+1}\\ y=\dfrac{\left(v-v_{0}\right)d}{f_{\mathbb{y}}\sqrt{\left(u-u_{0}\right)/f_{ \mathbb{x}}}^{2}+\left(\left(v-v_{0}\right)/f_{\mathbb{y}}\right)^{2}+1}\\ z=\dfrac{d}{\sqrt{\left(u-u_{0}\right)/f_{\mathbb{x}}}^{2}+\left(\left(v-v_{0} \right)/f_{\mathbb{y}}\right)^{2}+1}\end{array} \right., \tag{8}\]  
To calibrate the extrinsic parameter between \(O_{L}-X_{\mathbb{L}}Y_{\mathbb{L}}Z_{L}\) and \(O_{W}-X_{W}Y_{W}Z_{W}\), we adopt a 3D plane fitting method based on the RANSAC algorithm to acquire the equation of three planes of the triangular pyramid, and the steps can be described as follows:* Apply Kmeans cluster algorithm [30] to acquire point cloud of triangular pyramid target apart from background points. * Select three points from point cloud P of triangular pyramid, and calculate the equation of the plane formed by these three points. * Classify the other points to either inlier point or outlier point by comparing the distance between these points and the plane to a threshold value, and tally the amount of inlier points. * Repeat steps (2) and (3) a certain number of times to find the first best plane, or until the amount of inlier points reaches a threshold value. * Remove the inlier points and repeat steps (2), (3), and (4) twice to find the second and third best planes. Then, we can obtain the equations of plane \(i\) in \(O_{L}-X_{L}Y_{L}Z_{L}\):  
\[N_{i}^{T}P^{L}-d_{i}=0, \tag{9}\]  
where \(P^{L}\) is arbitrary point of plane \(i\) (\(i=1,2,3\)), vector \(N_{i}\) represents the unit normal vector of plane \(i\), and \(d_{i}\) denotes the distance of origin \(O_{W}\) and plane \(i\). The coordinate of origin \(O_{W}=[x_{W},y_{W},z_{W}]^{T}\) can be calculated by Equation (10), and normal vectors of three axes can be determined by cross product:  
\[[N_{1},N_{2},N_{3}]^{T}O_{W}-[d_{1},d_{2},d_{3}]^{T}=0, \tag{10}\]  
\[\left\{ \begin{array}{l}N_{X}=\dfrac{N_{1}\times N_{3}}{\|N_{1}\times N_{3}\| }\\ N_{Y}=\dfrac{N_{X}\times N_{1}}{\|N_{X}\times N_{1}\|}\\ N_{Z}=N_{X}\times N_{Y}\end{array} \right., \tag{11}\]  
Then, we can obtain the rotation matrix \(\mathbf{R}_{WL}\) and translation vector \(\mathbf{T}_{WL}\) from the world coordinate system to LiDAR coordinate system:  
\[\left\{ \begin{array}{l}\mathbf{R}_{WL}=[N_{X},N_{Y},N_{Z}]\\ \mathbf{T}_{WL}=[x_{W},y_{W},z_{W}]^{T}\end{array} \right., \tag{12}\]  
The next step is to estimate the rotation matrix and translation vector from the world coordinate system to camera coordinate system. This time, we need to acquire the expressions of plane \(i\) in \(O_{\mathbb{C}}-X_{\mathbb{C}}Y_{\mathbb{C}}Z_{\mathbb{C}}\).