# PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving
## 1 Introduction  
Object detection plays an important role in an Advanced Driver Assistance System (ADAS). Since the accuracy of the object detection in an ADAS is essential for the safety of drivers and pedestrians, it is considered one of the most challenging computer vision problems. The traditional object detection based on images is to transform the detection task into a classification task by using a sliding window. These classifiers usually use manual features and classifiers like Support Vector Machines (SVMs) [1, 2]. The main weakness of this method is that the amount of computation is very large because the classifier must scan each position in the space. Light Detection and Ranging (LiDAR) sensors are important for ADAS to perceive the environment and obstacles. Therefore, it is crucial to effectively process irregular and uneven LiDAR point clouds. The classical method [3] to process LiDAR data in 3D detection task achieved good performance. For example, [PERSON] et al. [3] proposed a dynamic clustering algorithm that applied an elliptical function to adjust each position of key points. Convolution Neural Networks (CNNs) have improved the performance of object detection because CNNs can extract useful featuresfrom 2D images, learn features, and classify objects. Many researchers have proposed various neural network architectures for 2D object detection, and these architectures can be extended for 3D object detection tasks. In [4], the authors proposed to generate specific object proposals instead of classification by using deep learning. With the emergence of more and more 3D object detection datasets and sensors, developing new 3D object detection algorithms attracts more interest. Recently the majority of state-of-the-art and popular 3D object detection methods utilize deep learning technologies. Most existing 3D detection methods based on deep learning can be classified into three categories in terms of representations, i.e., the point cloud-based methods [5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19], camera image-based methods [20; 21; 22; 23], and the point cloud and image fusion-based methods [24; 25; 26; 27; 28; 29; 30; 31; 32]. In general, it is not easy to achieve good performance in 3D object detection by using only camera sensors because there is no depth information in RGB images. Thus, LiDAR sensors and RGB-D camera sensors are extensively used to support the depth information for 3D object detection. In the last decade, significant progresses have been reported in 3D object detection using RGB images and LiDAR point clouds. Rich semantic information in color and texture can be provided by camera RGB images, whereas depth information and geometric structure features can be provided by LiDAR point clouds. Therefore, effective fusion of camera images and LiDAR point clouds appears to be extremely beneficial for understanding 3D objects and 3D scene construction. LiDAR sensors are commonly used in various self-driving vehicles and robot applications to capture the surrounding 3D scene information. Although point clouds provide depth information for environmental perception and 3D structure understanding, LiDAR points are usually sparse and unordered. We designed an effective fusion module to combine data from different sensors targeting a more accurate 3D object detection. In this paper, we propose a new 3D object detection framework, Point-Image Fusion network (PIFNet), which fuses camera image features and LiDAR point cloud features. In fact, the problem of fusing camera and LiDAR sensors is challenging as the features obtained from camera images and LiDAR point clouds are represented from different points of views, i.e., camera-view and 3D world view. The information fusion is achieved using two object detection streams. In one stream, we use PointNet++ [11] to generate the point-wise features. In the other stream, we aim to extract the image features and generate the strong joint Camera-LiDAR features. The main contributions of our work can be summarized as follows.