Query: sensor fusion point cloud autonomous driving

title: Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
abstract: Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep learning based camera LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep learning based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in depth reviews of camera LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over looked challenges between current academic researches and real world applications. Based on these observations, we provide our insights and point out promising research directions.

title: Object Recognition Based Interpolation With 3D LIDAR and Vision for Autonomous Driving of an Intelligent Vehicle
abstract: An algorithm has been developed for fusing 3D LIDAR (Light Detection and Ranging) systems that receive objects detected in deep learning based image sensors and object data in the form of 3D point clouds. 3D LIDAR represents 3D point data in a planar rectangular coordinate system with a 360deg representation of the detected object surface, including the front face. However, only the direction and distance data of the object can be obtained, and point cloud data cannot be used to create a specific definition of the object. Therefore, only the movement of the point cloud data can be tracked using probability and classification algorithms based on image processing. To overcome this limitation, the study matches 3D LIDAR data with 2D image data through the fusion of hybrid level multi sensors. First, because 3D LIDAR data represents all objects in the sensor's detection range as dots, all unnecessary data, including ground data, is filtered out. The 3D Random Sample Consensus (RANSAC) algorithm enables the extraction of ground data perpendicular to the reference estimation 3D plane and data at both ends through ground estimation. Classified environmental data facilitates the labeling of all objects within the viewing angle of 3D LIDAR based on the presence or absence of movement. The path of motion of the platform can be established by detecting whether objects within the region of interest are movable or static. Because LIDAR is based on 8 and 16 channel rotation mechanisms, real time data cannot be used to define objects. Instead, point clouds can be used to detect obstacles in the image through deep learning in the preliminary processing phase of the classification algorithm. By matching the labeling information of defined objects with the classified object cloud data obtained using 3D LIDAR, the exact dynamic trajectory and position of the defined objects can be calculated. Consequently, to process the acquired object data efficiently, we devised an active region of interest technique to ensure a fast processing speed while maintaining a high detection rate.

title: Point cloud based 3D object detection and classification methods for self driving applications: A survey and taxonomy
abstract: Autonomous vehicles are becoming central for the future of mobility, supported by advances in deep learning techniques. The performance of a self driving system is highly dependent on the quality of the perception task. Developments in sensor technologies have led to an increased availability of 3D scanners such as LiDAR, allowing for a more accurate representation of the vehicle's surroundings, leading to safer systems. The rapid development and consequent rise of research studies around self driving systems since early 2010, resulted in a tremendous increase in the number and novelty of object detection methods. After the first wave of works that essentially tried to expand known techniques from object detection in images, more recently there has been a notable development in newer and more adapted to LiDAR data works. This paper addresses the existing literature on object detection using LiDAR data within the scope of self driving and brings a systematic way for analysing it. Unlike general object detection surveys, we will focus on point cloud data, which presents specific challenges, notably its high dimensional and sparse nature. This work introduces a common object detection pipeline and taxonomy to facilitate a thorough comparison between different techniques and, departing from it, this work will critically examine the representation of data (critical for complexity reduction) feature extraction and finally the object detection models. A comparison between performance results of the different models is included, alongside with some future research challenges.

title: 2D 3D scene understanding for autonomous driving
abstract: In this thesis, we address the challenges of label scarcity and fusion of heterogeneous 3D point clouds and 2D images. We adopt the strategy of end to end race driving where a neural network is trained to directly map sensor input (camera image) to control output, which makes this strategy independent from annotations in the visual domain. We employ deep reinforcement learning where the algorithm learns from reward by interaction with a realistic simulator. We propose new training strategies and reward functions for better driving and faster convergence. However, training time is still very long which is why we focus on perception to study point cloud and image fusion in the remainder of this thesis. We propose two different methods for 2D 3D fusion. First, we project 3D LiDAR point clouds into 2D image space, resulting in sparse depth maps. We propose a novel encoder decoder architecture to fuse dense RGB and sparse depth for the task of depth completion that enhances point cloud resolution to image level. Second, we fuse directly in 3D space to prevent information loss through projection. Therefore, we compute image features with a 2D CNN of multiple views and then lift them all to a global 3D point cloud for fusion, followed by a point based network to predict 3D semantic labels. Building on this work, we introduce the more difficult novel task of cross modal unsupervised domain adaptation, where one is provided with multi modal data in a labeled source and an unlabeled target dataset. We propose to perform 2D 3D cross modal learning via mutual mimicking between image and point cloud networks to address the source target domain shift. We further showcase that our method is complementary to the existing uni modal technique of pseudo labeling.

title: Vision Only 3 D Shape Estimation for Autonomous Driving
abstract: We present a probabilistic framework for detailed 3 D shape estimation and tracking using only vision measurements. Vision detections are processed via a bird's eye view representation, creating accurate detections at far ranges. A probabilistic model of the vision based point cloud measurements is learned and used in the framework. A 3 D shape model is developed by fusing a set of point cloud detections via a recursive Best Linear Unbiased Estimator (BLUE) The point cloud fusion accounts for noisy and inaccurate measurements, as well as minimizing growth of points in the 3 D shape. The use of a tracking algorithm and sensor pose enables 3 D shape estimation of dynamic objects from a moving car. Results are analyzed on experimental data, demonstrating the ability of our approach to produce more accurate and cleaner shape estimates.

title: CurbScan: Curb Detection and Tracking Using Multi Sensor Fusion
abstract: Reliable curb detection is critical for safe autonomous driving in urban contexts. Curb detection and tracking are also useful in vehicle localization and path planning. Past work utilized a 3D LiDAR sensor to determine accurate distance information and the geometric attributes of curbs. However, such an approach requires dense point cloud data and is also vulnerable to false positives from obstacles present on both road and off road areas. In this paper, we propose an approach to detect and track curbs by fusing together data from multiple sensors: sparse LiDAR data, a mono camera and low cost ultrasonic sensors. The detection algorithm is based on a single 3D LiDAR and a mono camera sensor used to detect candidate curb features and it effectively removes false positives arising from surrounding static and moving obstacles. The detection accuracy of the tracking algorithm is boosted by using Kalman filter based prediction and fusion with lateral distance information from low cost ultrasonic sensors. We next propose a line fitting algorithm that yields robust results for curb locations. Finally, we demonstrate the practical feasibility of our solution by testing in different road environments and evaluating our implementation in a real vehicle1. Our algorithm maintains over 90% accuracy within 4.5 22 meters and 0 14 meters for the KITTI dataset and our dataset respectively, and its average processing time per frame is approximately 10 ms on Intel i7 x86 and 100ms on NVIDIA Xavier board.

title: A spaceborne camera pose estimate method based on high precision point cloud model
abstract: Obtaining the position and orientation information of camera or sensor is a key task in many fields such as robot navigation, autonomous driving, DSM (digital surface model) reconstruction, etc. The pose can be recovered by matching a 2D image and a corresponding digital surface model/point cloud model of the scene. A 3D point cloud model of very high spatial accuracy can be created with a combination of stereophotogrammetry and big data processing. So far, the most accurate 3D point cloud model created with satellites imagery can reach the accuracy of 3m@SE90 (3 meters error with SE90, which is the abbreviation for Spherical Error 90% In this paper, a novel method of estimating the pose of spaceborne cameras based on the fusion of high resolution point cloud models and remote sensing images. The core of our method is to project a high precision 3D point cloud model into the image space of a virtual camera, then the 3D 2D pose estimation method is transformed into a 2D 2D registration method. The registration results between two images are used to estimate the camera pose parameters. Simulation experiments were carried out to evaluate the performance of our method. The results showed that acceptable accuracy of camera pose can be achieved by using the proposed approach.

title: Real Time clustering and LiDAR camera fusion on embedded platforms for self driving cars
abstract: 3D object detection and classification are crucial tasks for perception in Autonomous Driving (AD) To promptly and correctly react to environment changes and avoid hazards, it is of paramount importance to perform those operations with high accuracy and in real time. One of the most widely adopted strategies to improve the detection precision is to fuse information from different sensors, like e.g. cameras and LiDAR. However, sensor fusion is a computationally intensive task, that may be difficult to execute in real time on an embedded platforms. In this paper, we present a new approach for LiDAR and camera fusion, that can be suitable to execute within the tight timing requirements of an autonomous driving system. The proposed method is based on a new clustering algorithm developed for the LiDAR point cloud, a new technique for the alignment of the sensors, and an optimization of the Yolo v3 neural network. The efficiency of the proposed method is validated comparing it against state of the art solutions on commercial embedded platforms.

title: Raw fusion of camera and sparse LiDAR for detecting distant objects
abstract: Environment perception plays a significant role in autonomous driving since all traffic participants in the vehicle's surroundings must be reliably recognized and localized in order to take any subsequent action. The main goal of this paper is to present a neural network approach for fusing camera images and LiDAR point clouds in order to detect traffic participants in the vehicle's surroundings more reliably. Our approach primarily addresses the problem of sparse LiDAR data (point clouds of distant objects) where due to sparsity the point cloud based detection might become ambiguous. In the proposed model each 3D point in the LiDAR point cloud is augmented by semantically strong image features allowing us to inject additional information for the network to learn from. Experimental results show that our method increases the number of correctly detected 3D bounding boxes in sparse point clouds by at least 13 21 and thus raw sensor fusion is validated as a viable approach for enhancing autonomous driving safety in difficult sensory conditions.

title: 3D LiDAR and Color Camera Data Fusion
abstract: 3D LiDAR sensor and 2D color camera provide complementary information of a scene. It is essential to fuse both types of data to obtain a holistic representation of the real world scene. In this paper, we present a method to fuse the data captured by rigidly mounted LiDAR and camera. We first cocalibrate the LiDAR and camera with a calibration target. We then establish 3D 2D correspondences and fuse the LiDAR and camera data through geometric transformations. Experimental results demonstrate that our method is able to fuse the 3D LiDAR point cloud with the 2D color image efficiently and accurately. Our method is also fast and can be used in autonomous driving applications.