# Multi-Layer Fusion 3D Object Detection via Lidar Point Cloud and Camera Image
## 3 The Proposed Approach
### Inputs  
The multi-layer fusion 3D object detection network has two types of input data: the RGB image and the Lidar point cloud. The camera is a typical representative of passive sensors. Images have rich color and texture information, which can help us intuitively understand the traffic scene and identify the object. Moreover, the image occupies very little memory. Therefore, we use the image as an input. However, this method lacks depth information, which is essential for accurate position estimation in the real 3D world. Using images as a standalone visual system is far from sufficient, as the brightness of the camera can easily degrade its accuracy at night or in rainy weather conditions. Lidar is a representative of active sensors. Lidar can not only acquire depth information but is less affected by external lighting conditions (i.e., at night) because it emits its own light pulses. Therefore, the Lidar system has higher accuracy and reliability than the camera system. We use the Lidar point cloud and RGB image input at the same time, which can complement their advantages and greatly improve the applicability of detection.  
The point cloud data of the KITTI [34] dataset were collected using the Velodyne HDL-64E Lidar. Each collected point cloud file contains hundreds of thousands of points. Each point represents three-dimensional position and intensity information in a three-dimensional space and is distributed irregularly. Point cloud data are usually stored in the form of x, y, z and intensity, with the four values representing the 3D coordinates and reflectance of a point. Due to the uneven distribution of point cloud data and the large number of points, direct processing of point cloud data will take up a large amount of computing. Therefore, we use the BEV representation to represent the point cloud while preserving the information of the point cloud. Since all objects are not covered in the vertical direction of the road, the size and shape information of the object will be retained. The BEV includes the height map and density map of the point cloud obtained by encoding the height information and density information of the point cloud. The height map discretized the point cloud according to a certain resolution, projected the voxel onto the ground plane to generate a BEV, and took the highest height of the point in each voxel as the height feature. The slices are evenly divided over a certain height range so that the BEV contains more height features. The height feature map was calculated within each slice. The density map represents the number of points in each voxel. Considering the camera's viewing range, we selected the point cloud range of [-40,40] \(\times\) [0,70] meters.  
We discretized the projected point cloud into a two-dimensional grid with a resolution of 0.1 m. For each grid, the height feature is calculated as the maximum height of a point in the cell. To encode more detailed height information, the point cloud is evenly divided into five slices. By calculating a height plot for each slice, we obtain five distinct height  
Figure 1: Overall frame of multi-layer fusion 3D object detection structure.  
features. Point cloud density represents the number of points in each cell. To standardize the feature, it is calculated as \(min(1.0,\ log(N+1)log(64))\), where \(N\) is the number of points in the cell. Note that the density feature is calculated for the entire point cloud, while the height feature is calculated for five slices, so the BEV is coded as a six-channel feature.