# MPVF: Multi-Modal 3D Object Detection Algorithm with Pointwise and Voxelwise Fusion
## 3 Methodology
### Detection Network  
In our MPVF architecture, the detection network integrates the IRFP and PVWF modules, embedding the 2D detection network RetinaNet [36] and the 3D detection network VoxelNet [33], and optimizing the 3D candidate boxes through projection consistency correction, ultimately accomplishing the 3D object detection task.  
2D Detection Network. We utilize an improved version of RetinaNet, incorporating multi-scale features from the IRFP module to enhance detection performance. The specific optimizations are as follows: (1) Base Network: We use the IR module pre-trained on ImageNet [40] as the base network, fine-tuning it with RGB images and corresponding bounding box annotations from the KITTI dataset. Once training is completed, the high-level features from the IR module (i.e., output from the C5 layer) will be utilized for the 3D object detection task; (2) Utilization of High-Level Semantic Features: The output of the C5 layer from the IR module provides high-level semantic features, serving as prior information to better infer the presence of targets; (3) Multi-Scale Feature Integration: The multi-scale features extracted from the P3 to P7 layers of the FP module are integrated into the classification and regression subnetworks of RetinaNet, allowing the model to capture fine-grained features of targets at different resolutions, thereby enhancing detection accuracy and robustness; (4) Classification and Regression Subnetworks: These subnetworks process the pyramid feature maps (P3 to P7) to predict the classes of anchor boxes and their boundary box coordinates, enabling target classification and localization. The classification subnetwork applies 4 3 \(\times\) 3 convolutional layers (with 256 filters and ReLU activation) to each feature map, followed by a 3 \(\times\) 3 convolutional layer with KA filters, where each spatial location produces KA binary classification predictions through a sigmoid function, with A representing the number of anchor boxes and K the number of classes. The boundary regression subnetwork has the same structure as the classification subnetwork but outputs 4A linear values, predicting the offsets for each anchor box relative to the ground truth boxes.  
3D Detection Network. We adopt an improved VoxelNet [33] structure, combined with pointwise and voxelwise fused features extracted by the PVWF module for 3D object detection. Specifically, the point cloud data are first voxelized, dividing the 3D space into voxels of size \(v_{D}=0.4\), \(v_{H}=0.2\), and \(v_{W}=0.2\). The detection range for the point cloud is set to Z direction [\(-3\), 1], Y direction [\(-40\), 40], and X direction [\(0\), 70.4], resulting in a 3D voxel grid size of \(D^{\prime}=10\), \(H^{\prime}=400\), and \(W^{\prime}=352\). Thus, the features extracted by the PVWF module form a 4D tensor with a shape of 256 \(\times\) 10 \(\times\) 400 \(\times\) 352. Next,this tensor is input into a 3D convolutional layer for feature aggregation, where 3 3D convolutions aggregate the pointwise and voxelwise fused features, transforming the shape into 256 \(\times\) 400 \(\times\) 352 3D features. These processed features are then fed into the 3D Region Proposal Network (RPN) to generate a high-resolution feature map of 1536 \(\times\) 200 \(\times\) 176. Subsequently, 2D convolutions are used to predict the probability score map and regression map, resulting in preliminary 3D candidate boxes. Finally, the 3D candidate boxes are projected onto the 2D detection image using calibration data for consistency correction, optimizing the localization of the 3D candidate boxes and further improving the accuracy of 3D object detection.  
By integrating the improved RetinaNet and VoxelNet detection frameworks and applying projection correction to the 3D candidate boxes, the MPVF architecture achieves high-precision 3D object detection. This design effectively utilizes multi-modal information, enhancing the robustness and accuracy of the system.