# Ore Rock Fragmentation Calculation Based on Multi-Modal Fusion of Point Clouds and Images
###### Abstract  
The accurate calculation of ore rock fragmentation is important for achieving the autonomous mining operation of mine excavators. However, a single mode cannot accurately calculate the ore rock fragmentation due to the low resolution of the point cloud mode and the lack of spatial position information of the image mode. To solve this problem, we propose an ore rock fragmentation calculation method (ORFCM) based on the multi-modal fusion of point clouds and images. The ORFCM makes full use of the advantages of multi-modal data, including the fine-grained object segmentation of images and spatial location information of point clouds. To solve the problem of image under-segmentation, we propose a multiscale adaptive edge-detection method based on an innovative standard deviation map to enhance the weak edges. Furthermore, an improved marked watershed segmentation algorithm is proposed to solve the problem of low segmentation accuracy caused by excessive noise of the gradient map and weak edges submerged. Experiments demonstrate that ORFCM can accurately calculate ore rock fragmentation in the local excavation area without relying on external markers for pixel calibration. The average error of the equivalent diameter of ore rock blocks is 0.66 cm, the average error of the elliptical long diameter is 1.42 cm, and the average error of the elliptical short diameter is 1.06 cm, which can effectively meet practical engineering needs.  
oe rock fragmentation; multi-modal fusion; gradient map; weak edges; segmentation map
recent years, the widespread adoption of LiDAR technology has prompted researchers to use three-dimensional point cloud data for calculating the sizes of ore blocks [2]. In the point cloud modality, the spatial and geometric position information of the ore rock point cloud can be directly utilized for size calculations. However, 3D point clouds obtained from vehicle-mounted 3D LiDAR systems typically have lower resolutions compared with images. Moreover, point cloud data often lack fine-grained information, such as color and texture. This limitation can impact the effectiveness of mine rock segmentation, thereby reducing the accuracy of rock block size calculations.  
To address the aforementioned challenges, this research paper proposes a novel approach that integrates point cloud and image multimodalities for calculating the size of rock fragments. This method capitalizes on the high resolution and comprehensive color-texture information available in the image modality to generate segmented images of ore and rock blocks. Firstly, the mapping relationship between the image and point cloud mode is established based on the joint calibration of the camera and LiDAR. Then, the pixels in the segmented map are dimensionally calibrated using the spatial position information of the point cloud modality. As a result, accurate computations of rock block sizes are achieved. This approach maximizes the fusion potential of point cloud and image multimodalities and effectively utilizes the spatial position information from the point cloud modality, as well as the high-resolution capabilities and texture information from the image modality. By synergistically integrating these modal data, the method overcomes the limitations of relying on a single modality for calculating rock fragment sizes. This integration significantly enhances the accuracy and practicality of the algorithm, leading to improved overall computation accuracy.