# Multi-Layer Fusion 3D Object Detection via Lidar Point Cloud and Camera Image
## 2 Related Works
### Fusion-Based 3D Object Detection  
The method based on visual images can provide texture information but lacks depth information. The method based on point cloud provides spatial geometry information but lacks texture information. Texture information is helpful for object detection and classification, while depth information is helpful for object spatial location estimation. At present, it is a research direction of 3D object detection methods to improve the overall performance by using image and point cloud data at the same time. The methods based on fusion can be classified into three types: early fusion network, medium fusion network and late fusion network. Early fusion refers to the fusion of information at the original pixel level. Pointpainting [14] and PI-RCNN [15] will color the point cloud by splicing the color information of each pixel of the image with the corresponding point cloud features and then use the existing detection network (such as PointRCNN [16], PointPillars [17], etc.) for the object detection of the colored point cloud. MVX-Net [18] uses two simple and effective early fusion methods, PointFusion and VoxelFusion, to integrate visual texture information and point cloud spatial geometry information to achieve high-precision object detection. This kind of network improves the effect well, but increases the amount of computation, and there is a problem that causes difficulty to align pixels. F-PointNet [19], proposed by [PERSON] et al., Faraway-frustum [20], proposed by [PERSON] et al. and F-ConvNet [21], proposed by [PERSON] et al., etc., generate high-quality two-dimensional candidate boxes using image data; they then map them to the three-dimensional space of the original point cloud. Three-dimensional candidate boxes were generated by extracting regional point cloud features. Unfortunately, such practices are largely limited by 2D detection results. Medium fusion refers to the fusion of information at the feature map level or RoI (Regions of Interest) level. The MV3D [22] method proposed by [PERSON] et al. uses the point cloud to generate the corresponding front view (FV) and bird's eye view (BEV), which, together with RGB images, serve as the input of the three feature extraction networks, respectively, and realizes the task of 3D object classifications and boundary box regression through deep fusion. Different from the MV3D, AVOD [23], proposed by [PERSON] et al., only uses the BEV generated by the image data and point cloud data coding as the network input and realizes the 3D object detection and classification and boundary box regression tasks through early fusion. SCANet [24], proposed by [PERSON] and [PERSON], aims to effectively integrate multi-scale and global context information and, at the same time, generate attention from space and channels to select discriminant features. ContFuse [25] projects the image in a BEV to supplement the sparse BEV information. Crossfusion [26] realizes the cross-projection and fusion of an image and BEV on the basis of ContFuse. The cross-modality 3D object detection model [27] proposed by [PERSON] et al. not only realizes interactive fusion at the feature level but also combines 2D and 3D candidate boxes to optimize the results. RoIFusion [28], proposed by [PERSON] et al., saves a lot of computation by integrating 3D RoI with 2D RoI. Medium fusion will not be very computation-intensive. However, in the process of fusion, the problem of information degradation still exists because of insufficient integration. Late fusion refers to the fusion optimization of the detection results. A typical example is the CLOCs [29] proposed by [PERSON] et al. The network first obtains 2D detection results and 3D detection results from the image and point cloud, respectively, and then filters the final 3D frame and adjusts the scale according to the geometric and semantic consistency of the 2D and 3D frames. This kind of network is difficult to apply because of the difficulty of training and real-time problems, so there are few related follow-up studies.