Query: semantic segmentation survey point cloud autonomous driving

title: Lightningnet: Fast and Accurate Semantic Segmentation for Autonomous Driving Based on 3D LIDAR Point Cloud
abstract: Semantic segmentation is A series of earlier works have demonstrated that 3D LiDAR point cloud segmentation is a promising approach. However, these methods usually rely on pretrained models. And their requiring improvements in either speed or accuracy prevent them to be applied to mobile platforms. To address these problems, a smaller Convolutional Neural Network (CNN) architecture, namely, LightningNet is presented. Furthermore, we propose a lightweight pipeline with Feature Refinement Modules to boost the performance of real time 3D LiDAR point cloud segmentation. Our model is trained on spherical images projected from LiDAR point clouds on KITTI  dataset. Experiments show accuracy improvements of 4.1 8.2% in small objects and 4% in terms of mIoU over the state of the art of spherical image based method with 111 fps on a single 1080Ti GPU and 14fps on a Jetson TX2, which enables deployment for real time semantic segmentation in autonomous driving. We achieve superior performance on another large dataset called SemanticKITTI (illustrated in Figure 1) both in speed and accuracy also.

title: SalsaNext: Fast Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving
abstract: In this paper, we introduce SalsaNext for the semantic segmentation of a full 3D LiDAR point cloud in real time. SalsaNext is the next version of SalsaNet  which has an encoder decoder architecture where the encoder unit has a set of ResNet blocks and the decoder part combines upsampled features from the residual blocks. In contrast to SalsaNet, we have an additional layer in the encoder and decoder, introduce the context module, switch from stride convolution to average pooling and also apply central dropout treatment. To directly optimize the Jaccard index, we further combine the weighted cross entropy loss with Lovasz Softmax loss  We provide a thorough quantitative evaluation on the Semantic KITTI dataset  which demonstrates that the proposed SalsaNext outperforms other state of the art semantic segmentation networks in terms of accuracy and computation time. We also release our source code https:/github.com/TiagoCortinhal/SalsaNext.

title: Ground Aware Point Cloud Semantic Segmentation for Autonomous Driving
abstract: Semantic understanding of 3D scenes is essential for autonomous driving. Although a number of efforts have been devoted to semantic segmentation of dense point clouds, the great sparsity of 3D LiDAR data poses significant challenges in autonomous driving. In this paper, we work on the semantic segmentation problem of extremely sparse LiDAR point clouds with specific consideration of the ground as reference. In particular, we propose a ground aware framework that well solves the ambiguity caused by data sparsity. We employ a multi section plane fitting approach to roughly extract ground points to assist segmentation of objects on the ground. Based on the roughly extracted ground points, our approach implicitly integrates the ground information in a weakly supervised manner and utilizes ground aware features with a new ground aware attention module. The proposed ground aware attention module captures long range dependence between ground and objects, which significantly facilitates the segmentation of small objects that only consist of a few points in extremely sparse point clouds. Extensive experiments on two large scale LiDAR point cloud datasets for autonomous driving demonstrate that the proposed method achieves state of the art performance both quantitatively and qualitatively.

title: Ground distance segmentation of 3D LiDAR point cloud toward autonomous driving
abstract: In this paper, we study the semantic segmentation of 3D LiDAR point cloud data in urban environments for autonomous driving, and a method utilizing the surface information of the ground plane was proposed. In practice, the resolution of a LiDAR sensor installed in a self driving vehicle is relatively low and thus the acquired point cloud is indeed quite sparse. While recent work on dense point cloud segmentation has achieved promising results, the performance is relatively low when directly applied to sparse point clouds. This paper is focusing on semantic segmentation of the sparse point clouds obtained from 32 channel LiDAR sensor with deep neural networks. The main contribution is the integration of the ground information which is used to group ground points far away from each other. Qualitative and quantitative experiments on two large scale point cloud datasets show that the proposed method outperforms the current state of the art.

title: Semantic Point Cloud Mapping of LiDAR Based on Probabilistic Uncertainty Modeling for Autonomous Driving
abstract: LiDAR based Simultaneous Localization And Mapping (SLAM) which provides environmental information for autonomous vehicles by map building, is a major challenge for autonomous driving. In addition, the semantic information has been used for the LiDAR based SLAM with the advent of deep neural network based semantic segmentation algorithms. The semantic segmented point clouds provide a much greater range of functionality for autonomous vehicles than geometry alone, which can play an important role in the mapping step. However, due to the uncertainty of the semantic segmentation algorithms, the semantic segmented point clouds have limitations in being directly used for SLAM. In order to solve the limitations, this paper proposes a semantic segmentation based LiDAR SLAM system considering the uncertainty of the semantic segmentation algorithms. The uncertainty is explicitly modeled by proposed probability models which are come from the data driven approaches. Based on the probability models, this paper proposes semantic registration which calculates the transformation relationship of consecutive point clouds using semantic information with proposed probability models. Furthermore, the proposed probability models are used to determine the semantic class of the points when the multiple scans indicate different classes due to the uncertainty. The proposed framework is verified and evaluated by the KITTI dataset and outdoor environments. The experiment results show that the proposed semantic mapping framework reduces the errors of the mapping poses and eliminates the ambiguity of the semantic information of the generated semantic map.

title: Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
abstract: Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep learning based camera LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep learning based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in depth reviews of camera LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over looked challenges between current academic researches and real world applications. Based on these observations, we provide our insights and point out promising research directions.

title: SalsaNet: Fast Road and Vehicle Segmentation in LiDAR Point Clouds for Autonomous Driving
abstract: In this paper, we introduce a deep encoder decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free space, and vehicles in the scene by employing the Bird Eye View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an auto labeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of image like projection of LiDAR data in semantic segmentation by comparing BEV with spherical front view projection and show that SalsaNet is projection agnostic. We perform quantitative and qualitative evaluations on the KITTI dataset, which demonstrate that the proposed SalsaNet outperforms other state of the art semantic segmentation networks in terms of accuracy and computation time. Our code and data are publicly available at https:/gitlab.com/aksoyeren/salsanet.git.

title: Improving Point Cloud Semantic Segmentation by Learning 3D Object Proposal Generation
abstract: Point cloud semantic segmentation plays an essential role in autonomous driving, providing vital information about drivable surfaces and nearby objects that can aid higher level tasks such as path planning and collision avoidance. While current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes, they show a significant drop in performance for underrepresented classes that share similar geometric features. We propose a novel Detection Aware 3D Semantic Segmentation (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task. By utilizing multitask training, the shared feature representation of the network is guided to be aware of per class detection features that aid tackling the differentiation of geometrically similar classes. We additionally provide a pipeline that uses DASS to generate high recall proposals for existing 2 stage detectors and demonstrate that the added supervisory signal can be used to improve 3D orientation estimation capabilities. Extensive experiments on both the SemanticKITTI and KITTI object datasets show that DASS can improve 3D semantic segmentation results of geometrically similar classes up to 37.8% IoU in image FOV while maintaining high precision BEV detection results.

title: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation
abstract: 3D point cloud semantic segmentation has attracted wide attention with its extensive applications in autonomous driving, AR/VR, and robot sensing fields. However, in existing methods, each point in the segmentation results is predicted independently from each other. This property causes the non contiguity of label sets in three dimensional space and produces many noisy label points, which hinders the improvement of segmentation accuracy. To address this problem, we first extend adversarial learning to this task and propose a novel framework Attention Adversarial Networks (AttAN) With high order correlations in label sets learned from the adversarial learning, segmentation network can predict labels closer to the real ones and correct noisy results. Moreover, we design an additive attention block for the segmentation network, which is used to automatically focus on regions critical to the segmentation task by learning the correlation between multi scale features. Adversarial learning, which explores the underlying relationship between labels in high dimensional space, opens up a new way in 3D point cloud semantic segmentation. Experimental results on ScanNet and S3DIS datasets show that this framework effectively improves the segmentation quality and outperforms other state of the art methods.

title: Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection
abstract: Point cloud semantic segmentation plays an essential role in autonomous driving, providing vital information about drivable surfaces and nearby objects that can aid higher level tasks such as path planning and collision avoidance. While current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes, they show a significant drop in performance for underrepresented classes that share similar geometric features. We propose a novel Detection Aware 3D Semantic Segmentation (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task. By utilizing multitask training, the shared feature representation of the network is guided to be aware of per class detection features that aid tackling the differentiation of geometrically similar classes. We additionally provide a pipeline that uses DASS to generate high recall proposals for existing 2 stage detectors and demonstrate that the added supervisory signal can be used to improve 3D orientation estimation capabilities. Extensive experiments on both the SemanticKITTI and KITTI object datasets show that DASS can improve 3D semantic segmentation results of geometrically similar classes up to 37.8% IoU in image FOV while maintaining high precision bird's eye view (BEV) detection results.