# A Novel Interactive Fusion Method with Images and Point Clouds for 3D Object Detection
## 2 Related Work and Proposed Method  
There are a few works that exploit multi-sensor data including the combination of RGB images and depth images and the fusion of RGB images and point clouds [1]. Ref. [7] utilizes RGB images and depth images with early fusion strategy and trains pose-based classifiers for 2D detection [14]. Refs. [4; 5] applying early fusion strategy by projecting the point cloud to the plane and augmenting the image channels after upsampling. Ref. [10] fuses images and point clouds by late fusion strategy for urban segmentation and computes size, shape, position, color features, a high-dimension Bag-of-words (BoW) descriptor in images and point clouds. Ref. [11] fuses lidar bird view, lidar front view and image for 3D object detection with deep fusion. Besides, it projects point cloud to the bird's eye view instead of the image plane and produces good results in KITTI [15]; however, it performs poorly in the test of average heading similarity (AHS).  
We evaluated our fusion algorithm in KITTI by 3D object detection with images and point clouds based on AVOD [16], a two-stage 3D object detector for autonomous driving scenarios on 3D localization, orientation estimation, and category classification tasks with a low memory overhead and a linear fusion algorithm. Different from existing fusion algorithms, our fusion algorithms combine a linear model and nonlinear model and enhance interaction and independence. We can find a detailed visual structure for one of our proposal fusion algorithms in Figure 1. It could be observed that two types or one group input feature maps to fusion output feature maps, our proposed fusion algorithms contain a linear model element-wise mean, and concatenation and nonlinear model convolution with a Relu [17] activation function. Besides, it reinforces interaction and independence by adding one type feature maps into the other type and considering the mixing ratio, then the two types of fused feature maps are separately through convolution layers. Finally, the two branches fuse again by an element-wise mean. We name this framework as the single level feature maps fusion algorithm. In addition, the multilevel feature maps fusion algorithm is illustrated in Figure 2, where four type or two group input feature maps are adopted for fusion of output feature maps. The proposed multilevel fusion algorithm contains la inear model element-wise mean, as well as concatenation and a nonlinear model convolution with a Relu [17] activation function. Moreover, it reinforces interaction and independence by adding one type feature maps into the other type and considering the mixing ratio, then the four type fused feature maps are inputted through convolution layers separately, finally, the four branches fuse again by the element-wise mean.  
We examined our proposed fusion algorithm by replacing the two fusion parts of AVOD [16] with our fusion algorithm. To be specific, the single level feature maps fusion algorithm uses the Figure 1 structure to displace the second fusion part of AVOD [16], and the multilevel feature maps fusion algorithm uses the Figure 2 structure to displace the first fusion part of AVOD.