Object DGCNN: 3D Object Detection using Dynamic Graphs

Methods for 3D object detection have progressed rapidly , yielding deployable autonomous driving perception systems . Following common practice in 2D vision , 3D object detection often employs complex training and testing pipelines including many post-processing operations to achieve superior performance . These operations are typically non-parallelizable and inefficient even with modern deep learning frameworks , implying a steep trade-off between between efficiency and effectiveness . Modern methods usually employ two stages [ 1 , 2 ] , including a region proposal network [ 3 ] that can introduce significant training overhead . Subsequent efforts simplify this pipeline for 3D object detection . PointPillars [ 4 ] introduces a one-stage anchor-based design , simplfying training . PillarOD [ 5 ] and CenterPoint [ 6 ] improve the one-stage model by making per-pillar predictions , that is , one prediction per point on the ground plane . They assign ground-truth bounding boxes to multiple outputs while training to ease optimization . However , they predict redundant boxes , which can overlap in the same positions ; extra boxes are eliminated a posteriori using non-maximum suppression ( NMS ) . It remains elusive to remove hand-designed components like NMS in training and testing . We introduce Object DGCNN , a streamlined architecture for 3D object detection from point clouds . Like DETR for 2D object detection [ 7 ] , we predict a set of bounding boxes from the raw data , enabling an NMS-free pipeline that achieves real-time performance . A critical new component is to treat each object query as a point in a set whose embedding is learned using DGCNN [ 8 ] . Compared to the self-attention module [ 9 ] in DETR , DGCNN leverages a sparse set of object relations , which reflects the real object distribution in the scene . In contrast to PointPillars [ 4 ] , PillarOD [ 5 ] , and CenterPoint [ 6 ] , our method does not require post-processing . We also provide a knowledge distillation approach customized to 3D object detection . Existing methods typically distill dense feature maps from a teacher model to a student model , whose training objective does not necessarily capture 3D object detection performance [ 10 ] . In contrast , we propose set-to-set distillation training that aligns the outputs of the teacher and the student in a permutationinvariant fashion . This process is enabled by the unified Object DGCNN architecture . In addition to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . obtaining better performance , through this process our model can benefit from privileged information ( e.g. , dense point clouds ) only available at training time . Contributions . We summarize our key contributions as follows : • We propose a post-processing-free 3D object detection model achieving state-of-the-art performance . To our knowledge , this is the first NMS-free 3D object detector . • We generalize DGCNN to model objects as a point set . The DGCNN module outperforms its self-attention counterpart thanks to its sparse structure . • We propose a set-to-set distillation method for 3D object detection . In our construction , knowledge distillation on object detection simply penalizes differences between the outputs of the teacher model and the student model . • We show our model can use privileged information ( such as dense point clouds ) that is naturally available at training time to improve the model performance at inference time . • We release our code to promote reproducibility and future research . 1 1 Related Work . 2D object detection . Object recognition research has been transitioning from models with handcrafted components to models with limited post-processing . One-stage detectors [ 11–13 ] remove the complicated region proposal networks in two-stage objectors [ 3 , 14 ] , yielding more efficient training and testing . Anchor-free methods [ 15 , 16 ] further simplify the one-stage pipeline by shifting from per-anchor prediction to per-pixel prediction . However , these methods still make dense predictions and rely on NMS to reduce redundancy . To alleviate this issue , DETR [ 7 ] formulates object detection as a set-to-set prediction problem . It introduces a set-to-set loss that implicitly penalizes redundant boxes , removing the necessity of post-processing . To accelerate convergence , Deformable DETR [ 17 ] proposes deformable self-attention and streamlines the optimization process . Our method also formulates 3D object detection as set prediction , but with a customized design for 3D . 3D object detection . VoxelNet [ 18 ] generalizes one-stage object detection to 3D . It uses 3D dense convolutions to learn representations on voxelized point clouds , which is too inefficient to capture fine-grained features . To address that , PIXOR [ 19 ] and PointPillars [ 4 ] project points to a birds-eye view ( BEV ) and operate on 2D feature maps ; PointNet [ 20 ] aggregates features within each BEV pixel . We use a variant of PointPillars [ 4 ] for 3D detection ( §3 ) . These methods are efficient but drop information along the vertical axis . To accompany the BEV projection , MVF [ 21 ] adds a spherical projection . PillarOd [ 5 ] and CenterPoint [ 6 ] use pillar-centric object detection , making predictions per BEV pixel ( pillar ) rather than per anchor . These anchor-free methods simplify 3D object detection while maintaining efficiency . Beyond SSD-style [ 11 ] one-stage models , Complex-YOLO [ 22 ] extends YOLO to 3D for real-time perception . PointRCNN [ 23 ] employs a two-stage architecture for high-quality detection . To improve representations of two-stage models , PVRCNN [ 2 ] proposes a point-voxel feature set abstraction layer to leverage the flexible receptive fields of PointNet-based networks . Unlike works on point clouds , LaserNet [ 24 ] operates on raw range scans with comparable performance . [ 25–27 ] combine point clouds with camera images . Frustum-PointNet [ 28 ] leverages 2D object detectors to form a frustum crop of points and then uses PointNet to aggregate features . [ 29 ] describes an end-to-end learnable architecture that exploits continuous convolutions to fuse feature maps . VoteNet [ 30 , 31 ] generalizes Hough voting [ 32 ] for 3D object detection in point clouds . DOPS [ 33 ] extends VoteNet and predicts 3D object shapes . In addition to visual input , [ 34 ] shows that high-definition ( HD ) maps can boost performance of 3D object detectors . [ 35 ] argues that multi-tasking can learn better representations than single-tasking . Beyond supervised learning , [ 36 ] learns a perception model for unknown classes . DGCNN . DGCNN [ 8 ] pioneered learning point cloud representations via dynamic graphs . It models point clouds as connected graphs , which are dynamically built using k-nearest neighbors in the latent space . DGCNN learns per-point features through message passing . However , it operates on point clouds for single object recognition and semantic segmentation . One of our key contributions is to generalize DGCNN to model scene-level object relations for 3D detection . Knowledge distillation ( KD ) . KD compresses knowledge from an ensemble of models into a single smaller model [ 37 ] . [ 38 ] generalizes this idea and combines it with deep learning . KD transfers 1https : //github.com/WangYueFt/detr3d knowledge from a teacher model to a student model by minimizing a loss , in which the target is the distribution of class probabilities induced by the teacher . [ 39–50 ] improve knowledge distillation for classification . Beyond image classification , KD has been extended to improve object detection . [ 51 ] leverages FitNets for object detection , addressing obstacles such as class imbalance , loss instability , and feature distribution mismatch . [ 52 ] distills between region proposals , accelerating training with added instability . To address this issue , [ 53 ] uses fine-grained representation imitation using object masks . [ 54 ] uses KD to tackle a continual learning problem . Privileged information . [ 55 ] introduces the framework of learning with privileged information in the context of support vector machines ( SVMs ) , wherein additional information is accessible during training but not testing . [ 56 ] unifies KD and learning using privileged information theoretically . [ 57 ] identifies practical applications , e.g. , transferring knowledge from localized data to non-localized data , from high resolution to low resolution , from color images to edge images , and from regular images to distorted images . To mediate uncertainty and improve training efficiency , [ 58 ] makes the variance of Dropout [ 59 ] a function of privileged information . We extend these methods to 3D data , in which privileged information consists of dense point clouds aggregated from LiDAR sequences . 2 Overview . Our target application of object detection differs from the recognition and segmentation tasks considered for DGCNN . Our point clouds typically contain too many points to apply DGCNN and its peers directly to the entire scene . Moreover , the size of our output set , a small set of bounding boxes , differs from the size of our input set , a huge set of points in R3 . Following state-of-the-art in large-scale object detection , our pipeline learns a grid-based intermediate representation to capture local features ( §3 ) . We test two standard learning-based methods for collecting local point cloud features on a birds-eye view ( BEV ) grid . While in principle it might be possible to avoid grids entirely in our pipeline , this BEV representation is far more efficient and—as observed in previous work—is sufficient to find objects reliably in autonomous driving , where there is likely only one object above any given grid cell on the ground plane . Our main architecture contribution is the Object DGCNN pipeline ( §4 ) , which transitions from this BEV grid of features to a set of object bounding boxes . Object DGCNN draws inspiration from the DGCNN architecture ; its layers alternate between local feature transformations and knearest neighbor aggregation to capture relationships between objects . Unlike conventional DGCNN , however , Object DGCNN incorporates features from the BEV grid in each of its layers ; each layer incorporates several queries into the BEV to refine object position estimates . The output of Object DGCNN is a set of objects in the scene . We use a permutation-invariant loss ( 10 ) to measure divergence from the ground truth set of objects . The pipeline above does not require hand-designed post-processing like NMS ; our output boxes are usable directly for object detection . Beyond simplifying the object detection pipeline , this allows us to propose object detection-specific distillation procedures ( §6.3 ) that further improve performance . These use one network to train another , e.g. , to train a network operating on sparse point clouds to output features that imitate those learned by a network trained on denser , more detailed point clouds . 3 Local Features . We begin with a point cloud X = { x1 , . . . , xi , . . . , xN } ⊂ R3 with per-point features F = { f1 , . . . , f i , . . . , fN } ⊂ RK , ground-truth bounding boxes B = { b1 , . . . , bj , . . . , bM } ⊂ R9 , and categorical labels C = { cj , . . . , cj , . . . , cM } ⊂ Z . Each bj contains position , size , heading angle , and velocity in the birds-eye view ( BEV ) ; our architecture aims to predict these boxes and their labels from the point cloud and its features . As an initial step , modern 3D object detection models scatter points into either BEV pillars or 3D voxels and then use convolutional neural networks to extract features on a grid . This strategy accelerates object detection for large point clouds . We test two neural network architectures for BEV feature extraction , detailed below . PointPillars [ 4 ] maps sparse point clouds onto a dense BEV pillar map on which 2D convolutions can be applied . Suppose FP ( i ) returns the points in pillar i , that is , the set of points in a vertical column above point i on the ground . When collecting features from points to pillars , multiple points can fall into the same pillar . In this case , PointNet [ 20 ] ( PN ) is used to obtain pillar-wise features : fpillari = PN ( { f j |xj ∈ FP ( pi ) } ) , ( 1 ) where fpillari is the feature of pillar pi . We set the features for empty pillars to 0 . This results in a dense 2D grid Fpillar ⊂ RHp×Wp×Cp , where Hp , W p and Cp are the height , width , number of channels of this 2D pillar map , respectively . Multiple stacked convolutional layers further embed the pillar features to the final feature map Fd ⊂ RHd×Wd×Cd . An alternative BEV embedding is SparseConv [ 60 ] . If FV ( i ) returns the set of points in voxel i , SparseConv collects point-wise features into voxel-wise features by fvoxeli = PN ( { f j |xj ∈ FV ( i ) } ) , ( 2 ) where fvoxeli contains the features of voxel i . In contrast to PointPillars , SparseConv conducts 3D sparse convolutions to refine the voxel-wise features . Finally , we compress these sparse voxels to a BEV 2D grid by filling empty voxels with zeros and averaging along the z-axis . For ease of notation , we also denote the resulting 2D grid Fd ⊂ RHd×Wd×Cd .

Summary: This paper focuses on the task of 3D object detection from point cloud. Inspired by recent non-maximum suppression-free 2D object detection modes, this paper proposes a new 3D object detection architecture. Concretely, it models 3D object detection as message passing on a dynamic graph, generalizing the framework to predict a set of objects. Furthermore, it removes the necessity of post-processing via object confidence aggregation or non-maximum suppression. To facilitate object detection from sparse point clouds, the authors also propose a set-to-set distillation approach customized to 3D detection. Experiments are conducted on nuScenes dataset to demonstrate the effectiveness of the proposed method.