# Calibration of Camera and Flash LiDAR System with a Triangular Pyramid Target
## 3 Method
### Methodology
#### 3.2.3 Calibration Problem Restated  
The proposed method can estimate the intrinsic parameters and extrinsic parameters through a single capture of the camera and LiDAR in Sections 3.2.1 and 3.2.2. We can further increase the accuracy by acquiring more data. Keeping the camera and LiDAR static, we randomly move the triangular pyramid \(K\) times. Each time, the camera and LiDAR frame are denoted by \(\{C_{k}\}\) and \(\{L_{k}\}\), where \(k=1,\ldots,K\). According to Equation (5), the intrinsic parameter calibration problem of the camera can be formulated according to the equations below:  
\[\mathbf{H}_{i,k}=\mathbf{M}\big{[}R_{i,k},R^{\prime}{}_{i,k},T_{i,k}\big{]}, \tag{20}\]  
where \(i=1,2,3\). Hence, we have \(3K\) equations of Equation (20), and it can be estimated by the Levenberg Marquardt algorithm [31]. The extrinsic calibration problem can be transformed to the optimization of the rotation matrix and translation vector separately. After rigid transformation, the unit normal vectors of \(\{C_{k}\}\) and \(\{L_{k}\}\) are aligned to each other.  
\[\text{argmin}\sum_{i=1}^{3}\sum_{k=1}^{K}\Big{|}\big{(}\mathbf{R}N_{i,k}^{L} \big{)}^{T}N_{i,k}^{C}-1\Big{|}, \tag{21}\]  
where \(N_{i,K}^{L}\) and \(N_{i,K}^{C}\) represent the unit normal vectors of plane \(i\) in \(\{Lk\}\) and \(\{Ck\}\), available by the Singular Value Decomposition (SVD) method. After rigid transformation, the points of triangular pyramid are on the chessboard plane, formulated by the following:  
\[\text{argmin}\sum_{k=1}^{K}\sum_{i=1}^{3}\sum_{j=1}^{M(i,k)}\|N_{i,k}^{T}( \mathbf{R}P_{i,j,k}^{L}+T)-d_{i,k}\|, \tag{22}\]  
where \(M(i,k)\) is the amount of points of plane \(i\) in frame \(k\), \(N_{i,k}\) and \(d_{i,k}\) are the expression of plane \(i\) in frame \(k\) according to Equation (9), and \(P_{i,j,k}^{L}\) denotes the \(j\)th point of plane \(i\) in frame \(k\). The translation vector \(T\) can also be estimated by the Levenberg Marquardt algorithm [31].  
Figure 3 is a flow chart of our method to calibrate the camera and flash LiDAR system. Firstly, we turn on the camera and LiDAR to acquire image and point cloud of the triangular pyramid target simultaneously. The image of the target is divided into three parts and each part displays the image of one chessboard. Secondly, we apply the feature points correspondence method based on [13] to obtain homography matrix \(\mathbf{H}\) and intrinsic matrix \(\mathbf{M}\) of the camera. Thirdly, we calculate rigid transformation from the chessboard to camera by combining \(\mathbf{H}\) and \(\mathbf{M}\) to obtain the extrinsic parameters from the world coordinate system to camera coordinate system. At the same time, we fit planes of three chessboards in the point cloud and obtain extrinsic parameters from the world coordinate system to LiDAR coordinate system. Finally, we calculate the initial extrinsic parameters from the LiDAR coordinate system to camera coordinate system, and optimize the result by minimizing the distance from the points to the chessboard. Furthermore, if we take more frames of the camera and LiDAR, the result can be more accurate by applying the Levenberg Marquardt algorithm [31].