Object DGCNN: 3D Object Detection using Dynamic Graphs

Methods for 3D object detection have progressed rapidly , yielding deployable autonomous driving perception systems . Following common practice in 2D vision , 3D object detection often employs complex training and testing pipelines including many post-processing operations to achieve superior performance . These operations are typically non-parallelizable and inefficient even with modern deep learning frameworks , implying a steep trade-off between between efficiency and effectiveness . Modern methods usually employ two stages [ 1 , 2 ] , including a region proposal network [ 3 ] that can introduce significant training overhead . Subsequent efforts simplify this pipeline for 3D object detection . PointPillars [ 4 ] introduces a one-stage anchor-based design , simplfying training . PillarOD [ 5 ] and CenterPoint [ 6 ] improve the one-stage model by making per-pillar predictions , that is , one prediction per point on the ground plane . They assign ground-truth bounding boxes to multiple outputs while training to ease optimization . However , they predict redundant boxes , which can overlap in the same positions ; extra boxes are eliminated a posteriori using non-maximum suppression ( NMS ) . It remains elusive to remove hand-designed components like NMS in training and testing . We introduce Object DGCNN , a streamlined architecture for 3D object detection from point clouds . Like DETR for 2D object detection [ 7 ] , we predict a set of bounding boxes from the raw data , enabling an NMS-free pipeline that achieves real-time performance . A critical new component is to treat each object query as a point in a set whose embedding is learned using DGCNN [ 8 ] . Compared to the self-attention module [ 9 ] in DETR , DGCNN leverages a sparse set of object relations , which reflects the real object distribution in the scene . In contrast to PointPillars [ 4 ] , PillarOD [ 5 ] , and CenterPoint [ 6 ] , our method does not require post-processing . We also provide a knowledge distillation approach customized to 3D object detection . Existing methods typically distill dense feature maps from a teacher model to a student model , whose training objective does not necessarily capture 3D object detection performance [ 10 ] . In contrast , we propose set-to-set distillation training that aligns the outputs of the teacher and the student in a permutationinvariant fashion . This process is enabled by the unified Object DGCNN architecture . In addition to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . obtaining better performance , through this process our model can benefit from privileged information ( e.g. , dense point clouds ) only available at training time . Contributions . We summarize our key contributions as follows : • We propose a post-processing-free 3D object detection model achieving state-of-the-art performance . To our knowledge , this is the first NMS-free 3D object detector . • We generalize DGCNN to model objects as a point set . The DGCNN module outperforms its self-attention counterpart thanks to its sparse structure . • We propose a set-to-set distillation method for 3D object detection . In our construction , knowledge distillation on object detection simply penalizes differences between the outputs of the teacher model and the student model . • We show our model can use privileged information ( such as dense point clouds ) that is naturally available at training time to improve the model performance at inference time . • We release our code to promote reproducibility and future research . 1 1 Related Work . 2D object detection . Object recognition research has been transitioning from models with handcrafted components to models with limited post-processing . One-stage detectors [ 11–13 ] remove the complicated region proposal networks in two-stage objectors [ 3 , 14 ] , yielding more efficient training and testing . Anchor-free methods [ 15 , 16 ] further simplify the one-stage pipeline by shifting from per-anchor prediction to per-pixel prediction . However , these methods still make dense predictions and rely on NMS to reduce redundancy . To alleviate this issue , DETR [ 7 ] formulates object detection as a set-to-set prediction problem . It introduces a set-to-set loss that implicitly penalizes redundant boxes , removing the necessity of post-processing . To accelerate convergence , Deformable DETR [ 17 ] proposes deformable self-attention and streamlines the optimization process . Our method also formulates 3D object detection as set prediction , but with a customized design for 3D . 3D object detection . VoxelNet [ 18 ] generalizes one-stage object detection to 3D . It uses 3D dense convolutions to learn representations on voxelized point clouds , which is too inefficient to capture fine-grained features . To address that , PIXOR [ 19 ] and PointPillars [ 4 ] project points to a birds-eye view ( BEV ) and operate on 2D feature maps ; PointNet [ 20 ] aggregates features within each BEV pixel . We use a variant of PointPillars [ 4 ] for 3D detection ( §3 ) . These methods are efficient but drop information along the vertical axis . To accompany the BEV projection , MVF [ 21 ] adds a spherical projection . PillarOd [ 5 ] and CenterPoint [ 6 ] use pillar-centric object detection , making predictions per BEV pixel ( pillar ) rather than per anchor . These anchor-free methods simplify 3D object detection while maintaining efficiency . Beyond SSD-style [ 11 ] one-stage models , Complex-YOLO [ 22 ] extends YOLO to 3D for real-time perception . PointRCNN [ 23 ] employs a two-stage architecture for high-quality detection . To improve representations of two-stage models , PVRCNN [ 2 ] proposes a point-voxel feature set abstraction layer to leverage the flexible receptive fields of PointNet-based networks . Unlike works on point clouds , LaserNet [ 24 ] operates on raw range scans with comparable performance . [ 25–27 ] combine point clouds with camera images . Frustum-PointNet [ 28 ] leverages 2D object detectors to form a frustum crop of points and then uses PointNet to aggregate features . [ 29 ] describes an end-to-end learnable architecture that exploits continuous convolutions to fuse feature maps . VoteNet [ 30 , 31 ] generalizes Hough voting [ 32 ] for 3D object detection in point clouds . DOPS [ 33 ] extends VoteNet and predicts 3D object shapes . In addition to visual input , [ 34 ] shows that high-definition ( HD ) maps can boost performance of 3D object detectors . [ 35 ] argues that multi-tasking can learn better representations than single-tasking . Beyond supervised learning , [ 36 ] learns a perception model for unknown classes . DGCNN . DGCNN [ 8 ] pioneered learning point cloud representations via dynamic graphs . It models point clouds as connected graphs , which are dynamically built using k-nearest neighbors in the latent space . DGCNN learns per-point features through message passing . However , it operates on point clouds for single object recognition and semantic segmentation . One of our key contributions is to generalize DGCNN to model scene-level object relations for 3D detection . Knowledge distillation ( KD ) . KD compresses knowledge from an ensemble of models into a single smaller model [ 37 ] . [ 38 ] generalizes this idea and combines it with deep learning . KD transfers 1https : //github.com/WangYueFt/detr3d knowledge from a teacher model to a student model by minimizing a loss , in which the target is the distribution of class probabilities induced by the teacher . [ 39–50 ] improve knowledge distillation for classification . Beyond image classification , KD has been extended to improve object detection . [ 51 ] leverages FitNets for object detection , addressing obstacles such as class imbalance , loss instability , and feature distribution mismatch . [ 52 ] distills between region proposals , accelerating training with added instability . To address this issue , [ 53 ] uses fine-grained representation imitation using object masks . [ 54 ] uses KD to tackle a continual learning problem . Privileged information . [ 55 ] introduces the framework of learning with privileged information in the context of support vector machines ( SVMs ) , wherein additional information is accessible during training but not testing . [ 56 ] unifies KD and learning using privileged information theoretically . [ 57 ] identifies practical applications , e.g. , transferring knowledge from localized data to non-localized data , from high resolution to low resolution , from color images to edge images , and from regular images to distorted images . To mediate uncertainty and improve training efficiency , [ 58 ] makes the variance of Dropout [ 59 ] a function of privileged information . We extend these methods to 3D data , in which privileged information consists of dense point clouds aggregated from LiDAR sequences . 2 Overview . Our target application of object detection differs from the recognition and segmentation tasks considered for DGCNN . Our point clouds typically contain too many points to apply DGCNN and its peers directly to the entire scene . Moreover , the size of our output set , a small set of bounding boxes , differs from the size of our input set , a huge set of points in R3 . Following state-of-the-art in large-scale object detection , our pipeline learns a grid-based intermediate representation to capture local features ( §3 ) . We test two standard learning-based methods for collecting local point cloud features on a birds-eye view ( BEV ) grid . While in principle it might be possible to avoid grids entirely in our pipeline , this BEV representation is far more efficient and—as observed in previous work—is sufficient to find objects reliably in autonomous driving , where there is likely only one object above any given grid cell on the ground plane . Our main architecture contribution is the Object DGCNN pipeline ( §4 ) , which transitions from this BEV grid of features to a set of object bounding boxes . Object DGCNN draws inspiration from the DGCNN architecture ; its layers alternate between local feature transformations and knearest neighbor aggregation to capture relationships between objects . Unlike conventional DGCNN , however , Object DGCNN incorporates features from the BEV grid in each of its layers ; each layer incorporates several queries into the BEV to refine object position estimates . The output of Object DGCNN is a set of objects in the scene . We use a permutation-invariant loss ( 10 ) to measure divergence from the ground truth set of objects . The pipeline above does not require hand-designed post-processing like NMS ; our output boxes are usable directly for object detection . Beyond simplifying the object detection pipeline , this allows us to propose object detection-specific distillation procedures ( §6.3 ) that further improve performance . These use one network to train another , e.g. , to train a network operating on sparse point clouds to output features that imitate those learned by a network trained on denser , more detailed point clouds . 3 Local Features . We begin with a point cloud X = { x1 , . . . , xi , . . . , xN } ⊂ R3 with per-point features F = { f1 , . . . , f i , . . . , fN } ⊂ RK , ground-truth bounding boxes B = { b1 , . . . , bj , . . . , bM } ⊂ R9 , and categorical labels C = { cj , . . . , cj , . . . , cM } ⊂ Z . Each bj contains position , size , heading angle , and velocity in the birds-eye view ( BEV ) ; our architecture aims to predict these boxes and their labels from the point cloud and its features . As an initial step , modern 3D object detection models scatter points into either BEV pillars or 3D voxels and then use convolutional neural networks to extract features on a grid . This strategy accelerates object detection for large point clouds . We test two neural network architectures for BEV feature extraction , detailed below . PointPillars [ 4 ] maps sparse point clouds onto a dense BEV pillar map on which 2D convolutions can be applied . Suppose FP ( i ) returns the points in pillar i , that is , the set of points in a vertical column above point i on the ground . When collecting features from points to pillars , multiple points can fall into the same pillar . In this case , PointNet [ 20 ] ( PN ) is used to obtain pillar-wise features : fpillari = PN ( { f j |xj ∈ FP ( pi ) } ) , ( 1 ) where fpillari is the feature of pillar pi . We set the features for empty pillars to 0 . This results in a dense 2D grid Fpillar ⊂ RHp×Wp×Cp , where Hp , W p and Cp are the height , width , number of channels of this 2D pillar map , respectively . Multiple stacked convolutional layers further embed the pillar features to the final feature map Fd ⊂ RHd×Wd×Cd . An alternative BEV embedding is SparseConv [ 60 ] . If FV ( i ) returns the set of points in voxel i , SparseConv collects point-wise features into voxel-wise features by fvoxeli = PN ( { f j |xj ∈ FV ( i ) } ) , ( 2 ) where fvoxeli contains the features of voxel i . In contrast to PointPillars , SparseConv conducts 3D sparse convolutions to refine the voxel-wise features . Finally , we compress these sparse voxels to a BEV 2D grid by filling empty voxels with zeros and averaging along the z-axis . For ease of notation , we also denote the resulting 2D grid Fd ⊂ RHd×Wd×Cd .

Summary: The paper introduces an NMS-free framework for 3D object detection in self-driving scenarios, following the 2D representative work DETR. Instead of using Transformers, it uses a dynamic graph CNN (DGCNN) to generate the set of detected objects. Besides, it proposes a set-to-set distillation loss to distill the knowledge of the teacher model, e.g., trained on multiple frames, for the student model, which will be tested on single frames. The experiments are only conducted on nuScenes.