# Ore Rock Fragmentation Calculation Based on Multi-Modal Fusion of Point Clouds and Images
## 3 Ore Rock Fragmentation Calculation Method
### Ore Rock Fragmentation Calculation Based on Multi-Modal Calibration Mapping
#### 3.6.1 Calibration Mapping  
The calibration mapping process between the point cloud and image is depicted in Figure 7.  
Figure 6: Ore rock fragmentation calculation based on multi-modal calibration mapping.  
Figure 7: Three-dimensional color laser ranging system and the calibration board.  
The camera is internally calibrated with the use of a chessboard calibration board to obtain the camera's intrinsic calibration matrix, denoted as \(A\). Furthermore, a joint calibration is performed between the camera and the LiDAR to acquire the extrinsic calibration matrix, denoted as \([R|\mathbf{t}]\). Assuming \(\mathbf{p}=[x,y,z]^{T}\) represents an arbitrary point in the LiDAR coordinate system, \(\mathbf{q}=[u,v]^{T}\) represents the corresponding mapped pixel in the image coordinate system, and the homogeneous coordinates of point \(\mathbf{p}\) and the mapped pixel \(\mathbf{q}\) are expressed as \(\widetilde{\mathbf{p}}=[x,y,z,1]^{T}\) and \(\widetilde{\mathbf{q}}=[u,v,1]^{T}\), respectively. Based on the pinhole camera model, the mapping relationship between \(\widetilde{\mathbf{p}}\) and \(\widetilde{\mathbf{q}}\) can be computed as follows:  
\[s\widetilde{\mathbf{q}}=A[\,R|\mathbf{t}]\widetilde{\mathbf{p}}, \tag{17}\]  
where \(\mathbf{s}\) is an arbitrary scaling factor.  
The main steps involved in the joint calibration between the 3D LiDAR and the camera are outlined as follows:  
1. Point cloud and image data acquisition. Multiple sets of 3D LiDAR point clouds and 2D images, featuring a black and white chessboard calibration board, are captured in various poses using the 3D LiDAR and camera from a multimodal 3D environmental perception system. Figure 8 showcases the simultaneous collection results of each set of point clouds and images;
2. Camera intrinsic calibration. We utilize multiple sets of images, featuring a calibration board with chessboard patterns captured in different poses, to compute the camera's intrinsic matrix using [PERSON]'s calibration method [24]. Simultaneously, through intrinsic calibration, we determine the positional relationship between the calibration board coordinate system and the camera coordinate system for various poses;
3. The extrinsic calibration of the 3D LiDAR and camera involves determining the geometric mapping relationship between the 3D point cloud and image pixels. Our research group has developed a method for calibrating the extrinsic parameters [25]. This method incorporates both line-plane and plane-plane geometric constraints to create a set of linear equations. The initial extrinsic matrix is obtained using the least squares method. Subsequently, a global error is formed by combining the pixel projection error from the camera's intrinsic calibration and the vertical distance error of laser points resulting from the initial extrinsic calibration. To refine the calibration, this global error undergoes nonlinear global optimization based on the Levenberg-Marquardt method. Through this optimization process, the final extrinsic calibration is achieved, establishing the desired geometric mapping relationship between the 3D point cloud and image pixels.  
Figure 8: Synchronization data of point clouds and images of the calibration board.