Multimodal Virtual Point 3D Detection

1 Introduction . 3D perception is a core component in safe autonomous driving [ 1 , 55 ] . A 3D Lidar sensor provides accurate depth measurements of the surrounding environment [ 23 , 49 , 75 ] , but is costly and has low resolution at long range . A top-of-the-line 64-lane Lidar sensor can easily cost more than a small car with an input resolution that is at least two orders of magnitude lower than a $ 50 RGB sensor . This Lidar sensor receives one or two measurements for small or far away objects , whereas a corresponding RGB sensor sees hundreds of pixels . However , the RGB sensor does not perceive the depth and can not directly place its measurements into a scene . In this paper , we present a simple and effective framework to fuse 3D Lidar and high-resolution color measurements . We lift RGB measurements into 3D virtual points by mapping them into the scene using close-by depth measurements of a Lidar sensor ( See Figure 1 for an example ) . Our Multi-modal Virtual Point detector , MVP , generates high-resolution 3D point-cloud near target objects . A center-based 3D detector [ 66 ] then identifies all objects in the scene . Specifically , MVP uses 2D object detections to crop the original point cloud into instance frustums . MVP then generates dense 3D virtual points near these foreground points by lifting 2D pixels into 3D space . We use depth completion in image space to infer the depth of each virtual point . Finally , MVP combines virtual points with the original Lidar measurements as input to a standard center-based 3D detector [ 66 ] . Our multi-modal virtual point method has several key advantages : First , 2D object detections are well optimized [ 17 , 74 ] and highly accurate even for small objects . See Figure 2 for a comparison of two state-of-the-art 2D and 3D detectors on the same scene . The 2D detector has a significantly higher 2D detection accuracy but lacks the necessary 3D information used in the downstream driving task . Secondly , virtual points reduce the density imbalance between close and faraway objects . MVP augments objects at different distances with the same number of virtual points , making the point cloud measurement of these objects more consistent . Finally , our framework is a plug-and- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . play module to any existing or new 2D or 3D detectors . We test our model on the large-scale nuScenes dataset [ 2 ] . Adding multi-modal virtual points brings 6.6 mAP improvements over a strong CenterPoint baseline [ 66 ] . Without any ensembles or test-time augmentation , our best model achieves 66.4 mAP and 70.5 NDS on nuScenes , outperforming all competing non-ensembled methods on the nuScenes leaderboard at the time of submission . 2 Related work . 2D Object Detection has great progress in recent years . Standard approaches include the RCNN family [ 13 , 17 , 43 ] which first predict class-agnostic bounding boxes based on predefined anchor boxes and then classify and refine them in a two-stage fashion with deep neural networks . YOLO [ 42 ] , SSD [ 33 ] , and RetinaNet [ 30 ] predicts the class specific bounded boxes in one shot . Recent anchorfree detectors like CornerNet [ 24 ] and CenterNet [ 74 ] directly localize objects through keypoints without the need of predefined anchors . In our approach , we use CenterNet [ 74 ] as our 2D detector for its simplicity and superior performance for detecting small objects . See Figure 2 for an example of a 2D detectors output . Lidar-based 3D Object Detection estimates rotated 3D bounding boxes from 3D point clouds [ 7 , 12 , 23 , 37 , 60–63 , 65 , 76 ] . 3D detectors share a common output representation and network structure with 2D detectors but encode the input differently . VoxelNet [ 75 ] uses a PointNe-based feature extractor to generate a voxel-wise feature representation from which a backbone consisted of sparse 3D convolutions and bird-eye view 2D convolution produces detection outputs . SECOND [ 60 ] introduces more efficient sparse convolution operations . PIXOR [ 61 ] and PointPillars [ 23 ] directly process point clouds in bird-eye view , further improving efficiency . Two-stage 3D detectors [ 8 , 45– 47 , 63 ] use a PointNet-based set abstraction layer [ 39 ] to aggregate RoI-specifc features inside first stage proposals to refine outputs . Anchor-free approaches [ 5 , 36 , 57 , 59 , 61 , 66 ] remove the need for axis-aligned bird-eye view anchor boxes . VoteNet [ 36 ] detects 3D objects through Hough voting and clustering . CenterPoint [ 66 ] proposes a center-based representation for 3D object detection and tracking and achieved state-of-the-art performance on nuScenes and Waymo benchmarks . However , as Figure 2 shows a Lidar-only detector still misses small or far-away objects due to the sparsity of depth measurements . In this work , we build upon the CenterPoint detector and demonstrate significant 6.6 mAP improvements by adding our multi-modal virtual point approach . Camera-based 3D Object Detection Camera-based 3D object detection predicts 3D bounding boxes from camera images . Mono3D [ 6 ] uses the ground-plane assumption to generate 3D candidate boxes and scores the proposals using 2D semantic cues . CenterNet [ 74 ] first detects 2D objects in images and predicts the corresponding 3D depth and bounding box attributes using center features . Despite rapid progress , monocular 3D object detectors still perform far behind the Lidar-based methods . On state-of-the-art 3D detection benchmarks [ 2 , 12 ] , state-of-the-art monocular methods [ 26 , 41 ] achieve about half the mAP detection accuracy , of standard Lidar based baselines [ 60 ] . PseudoLidar [ 56 ] based methods produce a virtual point cloud from RGB images , similar to our approach . However , they rely on noisy stereo depth estimates [ 25 , 40 , 56 ] while we use more accurate Lidar measurements . Again , the performance of purely color-based approaches lags slightly behind Lidar or fusion-based methods [ 2 , 12 ] . Multi-modal 3D Object Detection fuses information of Lidar and color cameras [ 20 , 28 , 28 , 29 , 35 , 37 , 53 , 67 , 71 , 72 ] . Frustum PointNet [ 38 ] and Frustum ConvNet [ 58 ] first detect objects in image space to identify regions of interest in the point cloud for further processing . It improves the efficiency and precision of 3D detection but is fundamentally limited by the quality of 2D detections . In contrast , we adopt a standard 3D backbone [ 75 ] to process the augmented Lidar point cloud , combining the benefit of both sensor modalities . MV3D [ 7 ] and AVOD [ 22 ] performs object-centric fusion in a two-stage framework . Objects are first detected in each sensor and fused at the proposal stage using RoIPooling [ 43 ] . Continuous fusion [ 20 , 29 ] shares image and Lidar features between their backbones . Closest to our approach are MVX-Net [ 48 ] , PointAugmenting [ 54 ] , and PointPainting [ 52 ] , which utilize point-wise correspondence to annotate each lidar point with image-based segmentation or CNN features . We instead augment the 3D lidar point cloud with additional points surrounding 3D measurements . These additional points make full use of the higher dimensional RGB measurements . Point Cloud Augmentation generates denser point clouds from sparse Lidar measurements . Lidarbased methods like PUNet [ 69 ] , PUGAN [ 27 ] , and Wang et al . [ 64 ] learn high level point-wise features from raw Lidar scans . They then reconstruct multiple upsampled point clouds from each high dimensional feature vector . Image-based methods [ 18 , 21 , 51 ] perform depth completion from sparse measurements . We build upon these depth completion methods and demonstrate state-of-the-art 3D detection results through point upsampling . 3 Preliminary . Our framework relies on both 2D detection , existing 3D detectors , and a mapping between 2D and 3D . We introduce the necessary concepts and notations below . 2D Detection . Let I be a camera image . A 2D object detector aims to localize and classify all objects in I . A bounding-box bi ∈ R4 describes the objects location . A class score si ( c ) predicts the likelihood of detection bi to be of class c. An optional instance mask mi ∈ [ 0 , 1 ] W×H predicts a pixel-level segmentation of each object . In this paper , we use the popular CenterNet [ 74 ] detector . CenterNet detects objects through keypoint estimation . It takes the input image I and predicts a heatmap for each class c. Peaks ( local maxima ) of the heatmap corresponds to an object . The model regresses to other bounding box attributes using peak features with an L1 [ 74 ] or box IoU [ 44 ] objective . For instance segmentation , we use CenterNet2 [ 73 ] which adds a cascade RoI heads [ 3 ] on top of the first stage proposal network . The overall network runs at 40 FPS and achieves 43.3 instance segmentation mAP on the nuScenes image dataset [ 2 ] . 3D Detection . Let P = { ( x , y , z , r ) i } be a point cloud with 3D location ( x , y , z ) and reflectance r. The goal of a 3D detector is to predict a set of 3D bounding boxes { bi } from the point cloud P . The bounding box b = ( u , v , o , w , l , h , θ ) includes the 3D center location ( u , v , o ) , object size ( w , l , h ) and the yaw rotation along z axis θ . In this paper , we build upon the state-of-the-art CenterPoint [ 66 ] detector . We experiment with two popular 3D backbones : VoxelNet [ 75 ] and PointPillars [ 23 ] . VoxelNet quantizes the irregular point clouds into regular bins followed by a simple average pooling to extract features from all points inside a bin [ 60 ] . After that , a backbone consisted of sparse 3D convolutions [ 14 ] processes the quantized 3D feature volumes and the output is a map view feature map M ∈ RW×H×F . PointPillar directly processes point clouds as bird-eye view pillar , a single elongated voxel per map location , and extracts features with fast 2D convolution to get the map view feature map M . With the map view features , a detection head inspired by CenterNet [ 74 ] localizes objects in bird-eye view and regress to other box parameters using center features . 2D-3D Correspondence . Multi-modal fusion approaches [ 48 , 52 , 53 , 72 ] often rely on a pointwise correspondence between 3D point clouds and 2D pixels . In absence of calibration noise , the projection from the 3D Lidar coordinate into a 2D image coordinate involves an SE ( 3 ) transformation from the Lidar measurement to the camera frame and a perspective projection from the camera frame into image coordinates . All transformations may be described with homogeneous , time-dependent transformations . Let t1 and t2 be the capture time of the Lidar measurement and RGB image respectively . Let T ( car←lidar ) be the transformation from the Lidar sensor to the reference frame of the car . Let T ( t1←t2 ) be the transformation of the car between t2 and t1 . Let T ( rgb←car ) be the transformation from the cars reference frame to the RGB sensor . Finally , let Prgb be the projection matrix of the RGB camera defined by the camera intrinsic . The transformation from the Lidar to RGB sensor is then defined by T t1←t2rgb←lidar = T ( rgb←car ) T ( t1←t2 ) T ( car←lidar ) , ( 1 ) followed by a perspective projection with camear matrix Prgb and a perspective division . The perspective division makes the mapping from Lidar to RGB surjective and non-invertible . In the next section , we show how to recover an inverse mapping by using depth measurements of Lidar when mapping RGB to Lidar .

Summary: This paper proposes a new method (MVP) to use multimodal (RGB+LiDAR) sensor data in 3D perception. MVP is simple in principle: compute the 2D segmentation mask, project LiDAR points on the mask, and generate the virtual points by sampling the points in the mask and attaching them with the closest LiDAR depth. Though simple, the authors show empirically the proposed method can bring performance gain and achieve SOTA results on the nuScenes 3D detection task.