Object DGCNN: 3D Object Detection using Dynamic Graphs
Authors: Yue Wang
Justin Solomon
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D object detection
point clouds
dynamic graph CNN

Abstract: 3D object detection often involves complicated training and testing pipelines, which require substantial domain knowledge about individual datasets. Inspired by recent non-maximum suppression-free 2D object detection models, we propose a 3D object detection architecture on point clouds. Our method models 3D object detection as message passing on a dynamic graph, generalizing the DGCNN framework to predict a set of objects. In our construction, we remove the necessity of post-processing via object confidence aggregation or non-maximum suppression. To facilitate object detection from sparse point clouds, we also propose a set-to-set distillation approach customized to 3D detection. This approach aligns the outputs of the teacher model and the student model in a permutation-invariant fashion, significantly simplifying knowledge distillation for the 3D detection task. Our method achieves state-of-the-art performance on autonomous driving benchmarks. We also provide abundant analysis of the detection model and distillation framework. 

Object DGCNN:
3D Object Detection using Dynamic Graphs
Yue Wang
Justin Solomon
Massachusetts Institute of Technology yuewang@csail .ww.i.m..............m...... .mit.eu
Massachusetts Institute of Technology jsolomon@mit.. edu
Abstract
3D object detection often involves complicated training and testing pipelines, which require substantial domain knowledge about individual datasets. Inspired by re- cent non-maximum suppression-free 2D object detection models, we propose a 3D object detection architecture point clouds. Our method models 3D object detection as message passing on a dynamic graph, generalizing the DGCNN frame- work to predict set of objects. In our construction, we remove the necessity of post post-processing via object confidence aggregation or non- mon-maximum suppression. To facilitate object detection from sparse point clouds, also propose set-to-set distillation approach customized to 3D) detection. This approach aligns the outputs of the teacher model and the student model in permutation-invante nificantly simplifying knowledge distillation for the 3D detection task. Our method achieves state-of-the- performance autonomous driving benchmarks. We also provide abundant analysis of the detection model and distillation framework.
fashion, sig-
Methods for 3D object detection have progressed rapidly, yielding deployable autonomous driving perception systems. Following common practice in 2D vision, 3D object detection often employs complex training and testing pipelines including many post-processing operations to achieve superior performance. These operations are typically paraaallllllizbbl and inefficient even with modern deep learning frameworks, implying a steep trade-off between between efficiency and effectiveness.
Modern methods usually employ two stages I 2], including a region proposal network 3 that can introduce significant training overhead. Subsequent efforts simplify this pipeline for 3D object detec- tion. PointPillars
4 introduces a one-stage anchor- design, simplfying training. PillarOD (5) (6) improve the one-stage model by making per-pillar predictions, that is, one pre- diction per point on the ground plane. They assign ground-truth bounding boxes to multiple outputs while training to ease optimization. However, they predict redundant boxes, which can overlap in the same positions; extra boxes are eliminated a posteriori using non- mon-maximum suppression (NMS). It remains elusive remove hand-designed components like NMS training and testing.
and CenterPoint
We introduce Object DGCNN, a streamlined architecture for 3D object detection from point clouds. Like DETR for 2D object detection 77, we predict set of bounding boxes from the raw data, enabling an NMS-free pipeline that achieves real-time performance. A critical new component is to treat each object query point in whose embedding is learned using DGCNN g8. Compared to the self-attention module [9] in DETR, DGCNN leverages sparse set of object relations, which reflects the real object distribution in the scene. In contrast to PointPillars CenterPoint (6), our method does not require post-processing.
PillarOD [5), and
We also provide knowledge distillation approach customized to 3D object detection. Existing methods typically distill dense feature maps from a teacher model to a student model, whose training objective does not necessarily capture 3D object detection performance [10]. In contrast, propose set-to-set distillation training that aligns the outputs of the teacher and the student in permutation- invariant fashion. This process is enabled by the unified Object DGCNN architecture. In addition to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

obtaining better performance, through this process our model can benefit from privileged information (e.g,, dense point clouds) only available training time.
Contributions. We summarize our key contributions as follows:
We propose post- petcprrsssin---ree 3D object detection model achieving state-of-the-an perfor- mance. To our knowledge, this is the first NMS-free 3D object detector.
We generalize DGCNN to model objects as a point set. The DGCNN module outperforms its self-attention counterpart thanks to its sparse structure.
We propose set-to-set distillation method for 3D object detection. In our construction, knowledge distillation on object detection simply penalizes differences between the outputs of the teacher model and the student model.
We show our model can use privileged information (such as dense point clouds) that is naturally available at training time to improve the model performance at inference time.
We release our code to promote reproducibility and future research.
1 Related Work
2D object detection. Object recognition research has been transitioning from models with hand- crafted components to models with limited post-processing. One-stage detectors [1|]3]] remove the complicated region proposal networks in two-stage objectors K 14], yielding more efficient training and testing. Anchor-free methods [6] further simplify the one-stage pipeline by shifting from per-anchor prediction per-pixel prediction. However, these methods still make dense predictions and rely on NMS to reduce redundancy. To alleviate this issue, DETR 7 formulates object detection set-to-set prediction problem. It introduces set-to-set loss that implicitly penalizes redundant boxes, removing the necessity of post-pro post-processing.. To accelerate convergence, Deformable DETR [17] proposes deformable self-attention and streamlines the optimization process. Our method also formulates 3D object detection as set prediction, but with a customized design for 3D).
3D object detection. VoxelNet [8 generalizes one- object detection to 3D. It uses 3D dense convolutions to learn representations on voxelized point clouds, which too inefficient to capture fine-grained features. To address that, PIXOR [1] and PointPillars [4] project points to a birds-eye view (BEV) and operate on 2D feature maps; PointNet (0 aggregates features within each BEV pixel. We use variant of PointPillars 4 for 3D detection (3.) These methods are efficient but drop information along the vertical axis. To accompany the BEV projection, MVF 21 adds a spherical projection. PillarOd [5] and CenterPoint [6] use pillar-centric object detection, making predictions per BEV pixel (pillar) rather than per anchor. These anchor-free methods simplify 3D object detection while maintaining efficiency. Beyond SSD-style [T] one-stage models, Complex- YOLO [22] extends YOLO to 3D for real-time perception. PointRCNN (3 employs a two-stage architecture for high-quality detection. To improve representations of two-stage models, PVRCNN 2 proposes point-voxel feature set abstraction layer to leverage the flexible receptive fields of PointNet-basee based networks. Unlike works on point clouds, LaserNet [24] operates on raw range scans with comparable performance. 27 combine point clouds with camera images. Frustum-Point (28 leverages 2D object detectors to form a frustum crop of points and then uses PointNet to aggregate features. 29 describes an end-to-end learnable architecture that exploits continuous convolutions to fuse feature maps. VoteNet [30 3T generalizes Hough voting [32] for 3D object detection in point clouds.
DOPS [33] extends VoteNet and predicts 3D object shapes. In addition to visual input, [34] shows that high-defiittion (HD) maps can boost performance of 3D object detectors.
(35] argues that
multi-tasking can learn better representations than single-tasking. Beyond supervised learning, (3 learns a perception model for unknown classes.
DGCNN. DGCNN g pioneered learning point cloud representations via dynamic graphs. It models point clouds as connected graphs, which are dynamically built using k-nearest neighbors in the latent space. DGCNN learns per-point features through message passing. However, it operates on point clouds for single object recognition and semantic segmentation. One of our key contributions is to generalize DGCNN to model scene-level object relations for 3D detection.
Knowledge distillation (KD). KD compresses knowledge from ensemble of models into single smaller model 337
38
generalizes this idea and combines with deep learning. KD transfers

---

knowledge from a teacher model to student model by minimizing loss, in which the target is the distribution of class probabilities induced by the teacher. [3] improve knowledge distillation for classification. Beyond image classification, KD has been extended to improve object detection. [51] leverages FitNets for object detection, addressing obstacles such as class imbalance, loss instability, and feature distribution mismatch, [52] distills between region proposals, accelerating training with added instability. To address this issue, [53] uses fine-grained representation imitation using object masks. [54] uses KD to tackle a continual learning problem.
(50
Privileged information. [55] introduces the framework of learning with privileged information in the context of support vector machines (SVMs), wherein additional information is accessible during training but not testing. [56] unifies KD and learning using privileged information theoretically. [57] identifies practical applications, c.g., transferring knowledge from localized data to non-localized data, from high resolution to low resolution, from color images to edge images, and from regular images to distorted images. To mediate uncertainty and improve training efficiency, (58) makes the variance of Dropout [59] a function of privileged information. We extend these methods to 3D data, in which privileged information consists of dense point clouds aggregated from LiDAR sequences.
2 Overview
Our target application of object detection differs from the recognition and segmentation tasks considered for DGCNN. Our point clouds typically contain too many points apply DGCNN and its peers directly to the entire scene. Moreover, the size of our output set, a small set of bounding boxes, differs from the size of our input set, a huge of points in R
Following state-of of-the--rt in large-scale object detection, our pipeline learns a grid- intermediate representation to capture local features ($3). We test two standard learning- based methods for collecting local point cloud features on a birds-cye view (BEV) grid. While in principle it might be possible to avoid grids entirely in our pipeline, this BEV representation is far more efficient and-an as observed in previous work-is sufficient to find objects reliably in autonomous driving, where there is likely only one object above any given grid cell on the ground plane.
Our main architecture contribution is the Object DGCNN pipeline ($4), which transitions from this BEV grid of features to a set of object bounding boxes. Object DGCNN draws inspiration from the DGCNN architecture; its layers alternate between local feature transformations and nearest neighbor aggregation to capture relationships between objects. Unlike conventional DGCNN, however, Object DGCNN incorporates features from the BEV grid in each of its layers; each layer incorporates several queries into the BEV to refine object position estimates. The output of Object DGCNN is a set of objects in the scene. We use a cermutation-invariaa loss (TO) to measure divergence from the ground truth set of objects.
The pipeline above does not require hand-designed post-processing like NMS; our output boxes are usable directly for object detection. Beyond simplifying the object detection pipeline, this allows us to propose object detection-sseciffi distillation procedures ($6.3) that further improve performance. These use one network to train another, e.g., to train a network operating sparse point clouds to output features that imitate those learned by a network trained on denser, more detailed point clouds.
3 Local Features
We begin with point cloud X' {ar1,.... ..., Di, .i,...,NNN NN R3 with per-point features F f (fi....,f: f fN C KR ground-truth bounding boxes B = (br....,by. bj....,bay C TR9, and categorical labels C {c....., .....,, cM} Z. Each by contains position, size, heading angle, and velocity in the birds-eye view (BEV); our architecture aims to predict these boxes and their labels from the point cloud and its features.
As an initial step. modern 3D object detection models seatter points into either BEV pillars or 3D voxels and then use convolutional neural networks to extract features on grid. This strategy accelerates object detection for large point clouds. We test two neural network architectures for BEV feature extraction, detailed below.
PointPillars 4 maps sparse point clouds onto a dense BEV pillar map on which 2D convolutions can be applied. Suppose Fp (i) returns the points in pillar i, that is, the set of points in a vertical column
3

---

Spanstorr
Sqaata
Figure 1: Overview. Point cloud features are learned in BEV, followed by L DGCNNs to model object relations. We predict set of bounding boxes and compute loss in one-to-one manner.
above point on the ground. When collecting features from points to pillars, multiple points can fall into the same pillar. In this case, PointNet (20) (PN) is used to obtain pillar-wie wise features:
ppillar PN((f my E Fr(p:))),
(1)
where fhillar the feature of pillar P.. We set the features for empty pillars to 0. This results in a dense 2D grid F C H" xW" where 11P, WP and CP are the height, width, number of channels of this 2D pillar map, respectively. Multiple stacked convolutional layers further embed the pillar features the final feature map F C R xC"
An alternative BEV embedding is SparseConv [60]. If (t) returns the set of points in voxel i, SparseConv collects point-wise features into voxel-wise features by
f PN((f Fv)))))
where f! contains the features of voxel i. contrast to PointPillars, SparseConv conducts 3D sparse convolutions to refine the voxel-wise features. Finally, compress these sparse voxels to BEV 2D grid by filling empty voxels with zeros and averaging along the z-axis. For case notation, we also denote the resulting 2D grid F C HB
4 Object DGCNN
After obtaining the BEV features Fdl using one of the architectures above, we predict a set of bounding boxes well as a label for each box. The key difference between our architecture and most recent 3D object detection methods is that ours produces ser of bounding boxes rather than a box per grid cell followed by NMS, as in 5 6). Hence, need to transition from a grid of per- per-pillar features to an unordered set of objects; we detail our approach below. We address two key issues: prediction of the bounding boxes and evaluation of the loss.
Desiderata. Object DGCNN uses DGCNN -nnspirdd architecture but incorporates grid-based BEV features, built on the philosophy that local features ($3) are reasonable to store on a dense grid, but object predictions are better modeled using sets. Hence, we require new architecture and set-to-set loss that encourage bounding box diversity.
Object DGCNN uses L layers that follow series of set-based computations produce bounding box predictions from the BEV feature maps. Each layer employs the following steps (Figure []:
1. predict a of query points and attention weights: collect BEV features from keypoints determined by the queries; and
3. model object-object interactions via DGCNN.
Each layer results a more refined set of bounding box predictions, one per query. At the end of these layers, match the prediction set with the ground-truth set in one-to-one fashion and evaluate set-to-set object detection loss.
Single layer. Inspired by DETR 7 each layer & E (O...., L - 1} of Object DGCNN operates set of object queries Q = (gr1: .ar......--- } c R producing new set Qr-1- Although queries are fully learnable, our intuition is that they represent progressively refined object positions.

---

The initial set of object queries Qo is learned jointly with the neural network weights, yielding a dataset-specific prior. Beyond this fixed initial set, below we detail how incorporate scene information to obtain 2.+1 from Oe using an approach inspired by DGCNN g and deformable self-attention [7]. For notational convenience, we drop the subscript.
Starting from each query q, (or, without the index dropped, q::). we decode reference point P; E IR?, R:
offsets {0:0..... .......,,, 6.k] R2, and set of attention weights {wio..... ...., w.k)
= 1el (q.), {wf j.....
"......,,....................... o = & neighbor (q.),
................................ w w K = atten (q,),
where Pref. L neeghhoor- and @ atten shared neural networks among the queries. We think of p, hypothesis for the center of the i-th object; the 8's represent the positions K informative points relative to the position of the object that determine its geometry.
Next, we collect a BEV feature is associated to each neighbor point Pis: = P; + 5
f ck fbilinear (F PP; + 8ik),
(4)
where Sbilinear bilinearly interpolates the BEV feature map F Note this step is the interaction between our set-based architecture manipulating query points q, and the grid-based feature map F We then aggregate single object query feature fü from the &S:
f: Σ Exewa fik.
(5)
This generates scene-aware features; each object query "attends" to certain area in the scene.
In the current layer é, the queries have not yet interacted with each other. incorporate neighborhood information in object detection estimates, we use DGCNN-style operations to model a sparse set of relations. We construct graph between the queries using a nearest neighbor search in feature space. In particular, connect each query feature f" to its 16 nearest neighbors as ablated in Table 7 Identically to DGCNN, we learn feature per edge egg and then aggregate back to the vertices to produce the new set of object queries. In detail, write:
max Pedge (f:.
q(e+1)
edges esj
where max denotes a channel-wise maximum and Dedge is neural network for computing edge features. This completes our layer for computing 2c+1 from 2r. Optionally, we repeat this last step multiple times, in effect applying DGCNN to the features fo to get the point set Qe+1-
Set-to-set loss. After L Object DGCNN layers as described above, we are left with set of M queries Qr. used to predict our bounding boxes. For each query 4 we use a classification network to predict a categorical label é; and regression network to predict bounding box parameters 6,. Our final task is to assign the predictions to the ground boxes and compute a set-to-set loss.
Most object detection models minimize a loss Lod given by
M
Lod
log (og + (eac Frox boo)).
where M = Hdd Wd. o(+) returns the corresponding index of the ground-truth bounding box. (C;) is the probability of class Ca()) for the prediction with index a (j), Ø denotes an invalid box, and Lbox is typically the L, distance. Different matchings ô yield different optimization landscapes and hence different prediction models. Pillar-OI) [5] and CenterPoint (6] employ a simple á to determine the ground-truth box used evaluate the box predicted at BEV pixel j:
if by overlaps with BEV pixel j; doverlap (j) = j' Ø otherwise.
This strategy can assign a box to multiple nearby BEV pixels. This one-to-many assignment provides dense supervision for the object detector and cases optimization. Since the training objective
5

---

Box
Prod
Prod
Prodi
Prodi
Pred
Predi
Box
Prodf
Pred
Pred
Object DGONN
Box
Object DGCNN
(Student)
(Teacher)
Prod
None
Bipartite matching
Bipertite matching
Student network
(b) Teacher network (fixed)
Figure 2: The set-to-set distillation pipeline. The student network is trained with the ground-truth supervision as well as with the supervision from a fixed teacher network.
encourages each BEV pixel to predict the same surrounding box, however, redundant boxes are inevitable. So, NMS is usually required to remove redundant boxes at inference time.
Rather than performing dense predictions in the BEV, we make per-query predictions. Typically, M is much larger than the number of ground-truth boxes M. To account for this difference, we pad the set of ground-truth boxes with Øs (no object) up M Following (7., we use an objective built on an optimal matching between these two sets. We define the optimal bipartite matching as
M
arg min OEP
- ((HH)))0 + Yc,= bo());
where denotes the set of permutations, (c)) is the probability class ej for the prediction with index a(j), and Lbox is the L1 loss for bounding box parameters. We use the Hungarian algorithm [6] to solve this assignment problem, as in 62 7. Our final set-to-set loss adapts
Caup
Loggr'))o + Looz b))
(10)
5 Distillation
Object DGCNN enables new set-to-set knowledge distillation (KD) pipeline. KD usually involves a teacher model T and a student model S. The common practice is to align the outputs of the student with those of the teacher using L2 distance KL-divergence. In past 3D object detection methods, since final performance heavily relies on NMS and the predictions are post-processe to be smaller set, distilling the teacher to the student is neither efficient effective. Since set-based detection model is NMS-free, we can easily distill the information between models with homogeneous detection heads (per-query object detection head in case). First, train teacher T using the method above with the loss in (1O). Then, we train student S with supervision given by T and the ground-truth. The class label and box parameters predicted by the teacher for each object query are c, and b/. respectively. The corresponding student outputs cf and b,. We find an optimal matching between the output set of the teacher and that of the student:
T arg min ddEP
log Poal))())) Loox (67 bs bear))
(11)
Then, the optimal matching's KD loss is given by
N
Laistill Σ
Togioz0) + Loox (bJ. S bo
(12)
So the overall loss during KD aLCuu + &Listilll, where a balance the supervised loss and distillation loss. In practice, use = I.

---

Table I: Comparisons to recent works. Our method is robust to whether to use NMS. *: imple- mentations with the same PointPillars backbone. $ implementations with the same SparseCony backbone.
Method
NDS
MAP1 1
mATE
MASE
MAOE
mAVE
mAAE
NMS
Point Pillass
53.3
40.0
SSN
63
54.83
41.56
FreeAnchor 64
55.3
43.7
RegNetX R0gNNetX-400MF-N0 -ECPP-
66
55.2
41.2
Pillar-OD S
56.84
44.41
CenterPoint (pillar)
59.56
47.48
31.27
25.81
33.78
32.25
20.20
CenterPoint (pillar)
55.08
40.27
35.14
26.44
36.75
32,66
19.55
CenterPoint (voxel) 6
64.19
54.99
29.83
25.71
32.56
26.08
18.89
CenterPoint (voxel) 6
57.00
45.32
31.66
27.14
40.47
37.23
20.14
Ours (pillar) *
62.97
53.31
34.62
26.56
31.61
26.02
18.71
(pillar) *
62.80
53.20
34.62
26.56
31.62
26.07
19.10
Ours (voxcl) [
66.10
38.73
33.31
26.32
28.80
25.11
19.08
Ours (voxel)
66.04
58.62
33.33
26.34
28.80
25.11
19.06
Table Comparisons of different distillation approaches.
Method
NDS
mAP
mATE +
mASE
mAOE
mAVE +
mAAE
Baseline (without distillation)
62.80
53.20
34.62
26.56
31.62
26.02
19.10
Feature distill (voxel- spillar)
62.84
53.29
34.55
26.78
31.61
26.11
19.02
Pscudo labels (voxel-tpillar)
63.18
53.31
34.58
26.55
31.29
25.99
18.87
Set-lo-set distill (volel-<pillar)
63.37
53.89
34.34
26.25
31.01
25.57
18.77
6 Experiments
We present our experiments in four parts. We introduce the dataset, metrics, implementation, and optimization details $6.1 Then, we demonstrate performance on the nuScenes dataset [66] 66.
We present knowledge distillation results in $6.3 Finally, we provide ablation studies in $6.4
6.1 Training & testing procedures
Dataset. We experiment on the nuScenes dataset 666 nuScenes provides rich annotations and diverse scenes. It has IK short sequences captured in Boston and Singapore with 700, 150, 150 sequences for training, validation, and testing, respectively. Each sequence is ~ 20s and contains 400 frames. This dataset provides annotation every 0.5s, leading to 28K, 6K, 6K annotated frames for training, validation, and testing. nuScenes uses 32-beam LiDAR, producing 30K points per frame. Following common practice, use calibrated vehicle pose information aggregate every 9 non-key frames to key frames, so each annotated frame has ~ 300K points. The annotations include 23 classes with long-tail distribution, of which 10 classes are included in the benchmark.
Metrics. The major metrics are mean average precision (mAP) and the nuScenes detection score (NDS). In addition, we use set of true positive metrics (TP metrics), which include average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE). These metrics are computed in the physical unit.
Model architecture. Our model consists of three parts: point-based feature extractor, DGCNN to encode object queries and to connect the point cloud features to object queries, and a detection head output the categorical label and bounding box parameters. We experiment with PointPillars and SparseConv [60] as feature extractors. The three blocks of the PointPillars backbone have [3, 5, 5] convolutional layers, with dimensions [64, 128, 256] and strides [2. 2, 2]: the input features are downsampled to 1/2, 1/4, 1/8 of the original feature map. For SparseConv, we use four blocks of [3, 3, 3, 3D sparse convolutional layers, with dimensions [16, 32, 64, 128) and strides [2, 2, 1]; the input features are downsampled to 1/2. 1/4, 1/8, 1/8 of the original feature map. For SparseConv, we transform
features into BEV by collapsing the z-axis. Both backbones use two deformable self-attention [17] layers with dimensions [256, 256) to transform the BEV features. Then, we use two DGCNN' to encode the object queries. Each DGCNN g contains two EdgeConv layers with dimensions [56,, 256], both with 16 nearest neighbors. For each object query, predict four points

---

Table 3: Comparisons of self-distillation versus baselines.
Method
NDS
MAPt
mATE +
mASE +
mAOE 1
mAVE
mAAE
Baseline (pillar, without distillation)
62.80
53.20
34.62
26.36
31.62
26.02
19.10
Self-distillaiin (pillar >pillar)
63.41
53.89
34.21
26.19
31.11
25.67
18.54
Baseline (voxel, without distillation)
66.04
58.62
33.33
26.34
28.80
25.11
19.06
Sell-distillation (voxel voxel)
66.45
59.25
31.17
25.77
30.73
25.72
18.77
Table 4: Self- distillation with privileged information.
Method
NDS f
mAP1
mATE
mASE
mAOE
mAVE 144.01
mAAE 4
Sparse >sparse (pillar)
42.12
38.89
44.01
27.78
64.01
39.21
Dense->sparee (pillar)
42.79
39.10
43.89
27.77
64.01
143.97
39.11
Sparse (voxel)
59.55
49.84
31.17
25.77
33.73
32.22
20.22
Dense->sparse (voxel)
59,89
50.12
31.11
25.76
33.70
32.19
20.11
in the BEV to obtain and aggregate the BEV features. The final feature for this object query is the weighted sum of features of these four BEV points. The final detection head takes the features of each object query and predicts class label and bounding box parameters w.r.t. the reference point.
Training& & inference. We use AdamW W 67 to train the model. The weight decay for Adam is 10-2 Following a cyclic schedule [68] the learning rate is initially 10 and gradually increased to 10-3, which is finally decreased 10 The model is initialized with a pre-trained PointPillars network on the same dataset. We train for 20 epochs on 8 RTX 3090 GPUs. During inference, we take the top 100 objects with highest classification scores as the final predictions. We do not use any post-processing such as NMS. For evaluation, we use the toolkit provided with the nuScenes dataset.
6.2 Object DGCNN
We compare to top-performing methods the nuScenes dataset in Table PointPillars 4 is anchor-based method with reasonable trade-off between performance and efficiency. Frehor [64] extends PointPillars by learning how to assign anchors the ground-truth. RegNetX-400MFF SECFPN (65] uses neural architecture search (NAS) to learn a flexible neural network for 3D detection; it is essentially variant of PointPillars with an enhanced backbone network. Different from anchor-
methods, Pillar-OD [5] makes predictions per pillar, alleviating the class imbalance issue caused by anchors. CenterPoint [6] exploits similar detection heads, with better performance using better training scheduling and data augmentation. For these methods, we use re- re-implementations in MMDetection3D [69, which match the performances in the original papers.
We mainly compare to CenterPoint with both PointPillars and SparseConv backbones, denoted as "voxel" and "pillar" respectively. Our method outperforms other methods significantly including CenterPoint with NMS. Without NMS, the performance of CenterPoint drops considerably while our method is unaffected by NMS. This finding verifies the DGCNN implicitly models object relations and removes redundant boxes.
6.3 Set-to-set distillation
In this section, we present experiments involving our set-to-set distillation pipeline. We conduct three types of distillation. First, we distill a teacher model with a SparseConv backbone to a student model with PointPillars backbone (denoted as "voxel-xpillar". This aligns with the common knowledge distillation setup for classification. We compare to feature-based distillation and pseudo label based methods. The objective of feature-based distillation is to align the middle-level features of the teacher model and the student model while the pseudo label based methods generate pseudo training examples with the pre-trained teacher networks. As Table 2 shows, our set-to-set distillation achieves better performance, confirming that distilling the last stage of the object detection model is more efffcctive than distilling feature maps.
Second, we perform self-distillation [49] (denoted as "voxel- and "pillar-xpillar" where the teacher and the student are identical and take the same point clouds input. As Table shows,, even when the teacher network and the student network have the same capacity, self-distillation still introduces performance boost. This finding is consistent with the results in (49
Finally, we try distillation with privileged information [56], where the teacher gets access to privileged information but the student does not. Following [10], the teacher takes dense point clouds, and the
8

---

student takes sparse point clouds (denoted as "dense->spane train each model over shorter period of time. The goal for the student model learn the same representations as the teacher model without knowing the dense inputs. In Table 4 we compare this setup with self-distillation, where the difference is the teacher model and the student model take the same sparse point clouds in self-distillatio.. The student achieves better performance when the teacher takes dense point clouds. The result suggests that set-to-set knowledge distillation is an effective approach to transfer insight from privileged information.
To limit computation time, we
6.4 Ablation
We provide ablation studies on different components of our model to verify assorted design choices. First, we study the improvements of DGCNN over its counterpart, multi-head self-attention [9]. The multi-head self-attention has 8 heads with embedding dimension 256 and LayerNorm [70], following common usage. The DGCNN has two EdgeConv layers with dimensions [256. 256]. The number of neighbors K in EdgeConv is 16. In principle, DGCNN is a sparse version of multi-head self-attention; the sparse structure reduces overhead in back-propagation and leads sharper "attention maps" as well as faster convergence.
5 shows the comparisons: DGCNN con- sistently outperforms multi-head self-attention. This aligns with our hypothesis: objects are dis- tributed sparsely in the scene, so dense inter- actions among objects neither efficient nor effective. Furthermore, we study the effect of number of neighbors in DGCNN. When it is 1, The model reduces to an architecture without object interaction. As increase the number, it approaches multi-head self-attention. As shown in Table 7 the sweet spot is 16, which appears to balance object interactions and sparsity.
Table DGCNN versus multi-hear self-attention
Method
Multi-head self-attention
DGCNN
Metric
NDS
39.89
41.32
MAP
36.35
37.81
Table 6: Models with different # DGCNNs.
Metric
# layers
3
NDS
35.91
39.75
41.15
41.26
41.07
41.32
mAP
32.32
36.54
37.25
37,75
37.78
37.81
Table 7: The number of neighbors in DGCNN.
Metrie
# neighhors
16
32
64
NDS
40.21
40.45
40.51
41.32
40.17
39.80
We also investigate improvements introduced when more DGCNNs are stacked in Table 6
36.81
37.15
37.81
37.12
36.70
Table 8: The distribution of the output scores with respect overlapping boxes.
This result suggests it is beneficial to incorporate multiple DGCNNs to model the dynamic object relations.
Method
filtrred boxes
remaining
Moreover, we hypothesize our method produces different distribution of the output scores with respect to overlapping boxes. To verify this hy- pothesis (Table 8 we compute average scores for three types of boxes: filtered boxes after NMS, remaining boxes after NMS, and all boxes. Below we show the results. We also compute the percentage of filtered boxes by NMS in our method and Centerpoint. In our method, 21.9% boxes are removed while in Centerpoint 85.16% boxes are filtered. Hence, conclude that method indeed exhibits a different distribution pattern from Centerpoint.
CenterPoint
0.0764
0.1859
0.0829
Ours 0.1222
0.1711
0.1604
Table 9: Complexity comparsion between DGCNN and Multi-head self-attention.
Module
Complexity d(
parameters 262144
DGCNN
Multi-head self-attention
O(
263168
Finally, we include complexity comparison between DGCNN and Multi-head self-attention. Table shows the results; DGCNN layer on a par with Multi-head self-attention.
7 Conclusion
Object DGCNN is highly-efficient 3D object detector for point clouds. It is able to learn object interactions via dynamic graphs and is optimized through a set-to-set loss, leading to NMS-free detection. The success of Object DGCNN indicates that many post-processing operations in 3D object detection are likely unnecessary and can be replaced with suitable neural network modules. Moreover, we introduce a set-to-set knowledge distillation pipeline enabled by the Object DGCNN. This new pipeline significantly simplifies knowledge distillation for 3D object detection and may

---

be applicable to other tasks like 3D model compression. Beyond the direct usage of our model, our experiments suggest several future directions to address current limitations. For example, our method is initialized with pre-trained backbone network. Training the model from scratch remains elusive due to the sparse set-to-set supervision; solving this issue may yield improved generalization as in [71] Furthermore, studying 3D-specific feature extractors will improve the speed and generalizability of 3D object detection. Finally, the large amount of unlabeled data available at training time can serve as another type of privileged information to apply self-supervised learning to 3D domains through set-to-set distillation.
Potential impact. Our method aims to improve the object detection pipeline, which is crucial for the safety of autonomous driving systems. One potential negative impact of our work is that still lacks theoretical guarantees, similar to many deep learning methods. Future work to improve applicability in this domain might consider challenges of explainability and transparency.
8 Acknowledgement
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants @911NF2010168 and W91 INF21 10293, of Air Force Office of Scientific Research award FA9550-19-11 19-1-31,, of National Science Foundation grants IIS-1838077 and CHS-1955697 from the CSAIL Systems that Learn program, from the MIT-IBM Watson AI Laboratory, from the -oyota-CSAIL Joint Research Center, from a gift from Adobe Systems, from an MIT.nano Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoltech-MIT Next Generation Program.
References
[1] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR). June 2019.
[2] Shaoshuai Shi, Chaoxu Guo, Li Jiang, 7he Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10538, 2020.
[3] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Neural Information Processing Systems (NeurIPS), 2015.
[4] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[5] Yue Wang, Alireza Fathi, Abhijit Kundu, David Ross, Caroline Pantofaru, Thomas Funkhouser, and Justin Solomon. Pillar-based object detection for autonomous driving. In The European Conference on Computer Vision, 2020.
[6] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. CVPR, 2021.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] Yue Wang, Yongbin Sun, Sanjay E. Sarma Ziwei Liu, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics (TOG), 38:146, 2019.
19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[10] Yue Wang, Alireza Fathi. Jiajun Wu, Thomas Funkhouser, and Justin Solomon. Multi-frame to single- frame: Knowledge distillation for 3d object detection. In The Workshop on Perception for Autonomous Driving at the European Conference on Computer Vision, 2020.
[11] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yen Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In The European Conference on Computer Vision (ECCV), 2016.
10

---

[12] Joseph Redmon and Ali Farhadi. YOL09000: better, faster, stronger. In The IEEE Conference on Computer Vision and Patiern Recognition (CVPR), 2017.
[13] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. In The International Conference on Computer Vision (ICCV), 2017.
[14] Kaiming He, Georgia Gkioxari. Piotr Dollár, and Ross Girshick. Conference on Computer Vision (ICCV), 2017.
Mask R-CNN. In The International
[15] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. arXiv:/9044 07850, 2019.
Objects as points. In arXiv preprint
[16] Hei Law and Jia Deng. ComerNet: Detecting objects as paired keypoints. In European Conference on Computer Vision (ECCV), Septemher 2018.
[17] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable [dete]: Deformable transformers for end-to-ena object detection. In International Conference on Learning Representations, 2021.
[18] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[19] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time object detection from point clouds. The Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[20] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[21] Yin Zhou, Pei Sun, Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In The Conference on Robot Learning (CoRL). 2019.
[22] Martin Simon, Stefan Milz, Karl Amende. and Horst-Michael GroB. detection point clouds. In The Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Complex-yolo: Real-time 3d object
[23] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: object proposal generation and detection from point cloud. In The Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[24] Gregory P. Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzalez, and Carl K. Wellington. Lasernet: An efficient probabilistic 3d object detector for autonomous driving. In The Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
[25] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. The International Conference on Intelligent Robots and Systems (IROS), 2018.
[26] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In The Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[27] Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In The Conference on Computer Vision and Pattern Recognition (CVPR). 2018
[28] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In The Conference on Computer Vision and Puttern Recognition (CVPR). 2018.
[29] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi multi-sensor 3d object detection. In The European Conference Computer Vision (ECCV), 2018.
[30] Charles Qi. Or Litany. Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In The International Conference on Computer Vision, 2019.
[31] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Invotenet: Boosting 3d object detection in point clouds with image votes. arXiv preprint arXiv:2001.10692, 2020.
[32] Paul V.C. Hough. Machine analysis of bubble chamber pictures. The International Conference in High Energy Accelerators and Instrumentation, 1959.

---

[33] Mahyar Najibi, Guangda Lai, Abhijit Kundu, Zhichao Lu, Vivek Rathod, Tom Funkhouser, Caroline Pantofaru, David Ross, Larry S. Davis, and Alireza Fathi. Dops: Learning detect 3d objects and predict their 3d shapes. ArXiv, abs/2004.00 17700 2020.
[34] Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting maps for 3d object detection. In The Conference on Robot Learning (CORL), 2018.
[35] Ming Liang, Bin Yang. Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In The Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[36] Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, and Raquel Urtasun. Identifying unknown instances for autonomous driving. In The Conference on Robot Learning (CORI.), 2019.
[37] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD, 2006.
[38] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015.
[39] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In The International Conference on Learning Representations (ICLR), 2015.
[40] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In The International Conference on Learning Representations (ICLR), 2017.
[41] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. Conference on Computer Vision (ICCV), pages 1365-1374, 2019.
In The International
[42] Wonpyo Park, Dongju Kim, Yan La, and Minsu Cho. Relational knowledge distillation. In The Conference on Computer Vision and Pattern Recognition (CVPR), pages 3962-3971. 1971, 2019.
[43] Junho Yim, Donggyu Joo, Jihoon Bae. and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[44] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In The Inierna- tional Conference on Learning Representations (ICLR), 2020.
[45] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, ArXiv, abs/1887.00744 2018
[46] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.0572,, 2019.
[47] Ting Chen, Simon Korblith,, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
[48] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 2019.
[49] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Bor-aaaan neural networks. In ICML, 2018
[50] Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoe V. Le. Bam! born-again multi-task networks for natural language understanding. In ACI, 2019.
[51] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Leaming efficient object detection models with knowledge distillation. In Neural Information Processing Syxtems (NeurIPS). 2017.
[52] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[53] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[54] Xialei Liu, Hao Yang, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto, Continual universal object detection. ArXiv, abs/2002.05347, 2020.
12

---

[55] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 2009.
[56] David Lopez-Paze Léon Bottou, Berbaard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. In International Conference Learning Representations (ICLR), 2016.
[57] Long-Chyi Su and Subhransu Maji. Adapting models to signal degradation using distillation. In British Machine Vision Conference (BMVC), 2017.
[58] John Lambert. Ozan Sener, and Silvio Savarese. Deep learning under privileged information using heterosecdastic dropout. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[59] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting Journal of Machine Learning Research, 1929)9999991198. 2014.
[60] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. 2018
[61] H. W. Kuhn and Bryn Yaw. The hungarian method for the assignment problem. Naval Res. Logist. Quart, pages 83-97, 1955.
[62] Russell Stewart, Mykhaylo Andriluka, and Andrew Y. Ng. End-to-end people detection in crowded scenes. In 2016 IEEE Conference on Computer Vision and Puttern Recognition CVPR, pages 2325-2333 IEEE Computer Society, 2016.
[63] Xinge Zhu, Yuexin Ma, Tai Wang, Yan Xu, Jianping Shi, Dahua Lin. Ssn: Shape signature networks for multi-class object detection from point clouds. In Proceedings of the European Conference on Computer Vision, 2020.
[64] Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye. FreeAnchor: Learning to match anchors for visual object detection. In Neural Information Processing Systems, 2019.
[65] Iliaa Radosavovic, Raj Pratcek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. 2020.
[66] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint Xivi::
eix::vv90000 1903. /027, 2019.
[67] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[68] Leslie N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464 472, 2017.
[69] MMDetectionSD Contributors. MMDetection Open next-generation platform for general 3D object detection. https::/gi com/ open wat.aa//mmnetettiiniii 2020.
[70] Lei Jimmy Ba. Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 2016.
[71] Kaiming He, Ross Girshick, Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv.1811.08882 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
13

---

(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you re
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "kc5PDSqhkYF", "reviewer": "Reviewer_UvFF", "summary": "The paper introduces an NMS-free framework for 3D object detection in self-driving scenarios, following the 2D representative work DETR. Instead of using Transformers, it uses a dynamic graph CNN (DGCNN) to generate the set of detected objects. Besides, it proposes a set-to-set distillation loss to distill the knowledge of the teacher model, e.g., trained on multiple frames, for the student model, which will be tested on single frames. The experiments are only conducted on nuScenes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strongness:\n- The writing is clear.\n\n---\n\nWeakness:\n1. The major contributions are ambiguous. The connection between the NMS-free detector and the distillation is not tight. It confuses me when I read the introduction for the first time, as the knowledge distillation seems to suddenly appear. The authors seem to plan to equip the paper with two contributions, but it turns out that they are not connected well. Motivations, especially the connection of the two contributions, can be discussed more in the introduction.\n2. The experiments are only done on one dataset (nuScenes). Most 3D object detection papers recently conduct experiments on at least two datasets (KITTI, Waymo apart from nuScenes), or conduct more comprehensive experiments on one dataset, in order to make the results more convincing. Can the authors show the results on any other dataset?\n3. For the distillation experiments, the models are trained only for 1 epoch (mentioned in Sec 6.3). Can the authors report the results in Sec 6.3 and 6.4 with longer training time (like 20 epochs), so that they can be directly compared with the numbers in Table 1? \n4. As the authors have mentioned in Sec 6.4, dense graph networks, like Transformers, might be slower to converge compared to sparse graph networks, like DGCNN. Thus, can the authors also report the ablation results with a longer training time (20 epochs)?\n\n\n---\n\nTypo:\nL55: objectors -> object detectors\nThe introduction section title is missing.\n\n---\n\n**Updates after rebuttal**\nI slightly raise my rating to 6 according to the authors' rebuttal. However, the authors should improve their connection between two contributions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not adequately addressed the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "SSr_TQxE7j", "reviewer": "Reviewer_vpe9", "summary": "This paper focuses on the task of 3D object detection from point cloud. Inspired by recent non-maximum suppression-free 2D object detection modes, this paper proposes a new 3D object detection architecture. Concretely, it models 3D object detection as message passing on a dynamic graph, generalizing the framework to predict a set of objects. Furthermore, it removes the necessity of post-processing via object confidence aggregation or non-maximum suppression. To facilitate object detection from sparse point clouds, the authors also propose a set-to-set distillation approach customized to 3D detection. Experiments are conducted on nuScenes dataset to demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1、The contribution of this paper is limited, and the motivation from the first section seems to be not solid. \n\nFirst, the authors just very simply introduce methods of PointPillars, PillarOD and CenterPoint in 3D object detection(L19-26), which is obviously not enough. Directions like voxel/point-based or one/two-stage are not discussed. \n\nSecond, the proposed method mainly involves technologies of set-to-set prediction [9], GCN [8] and knowledge distillation [10], all of them are popular algorithms published by others recently. However, the author didn’t not provide detailed and reasonable discussions/comparisons about why using these technologies, it seems to just simply combine them together. For the part of knowledge distillation, what is the difference or improvement comparing with [10, 56, 49]? What is the insightful design in sparse 3D object detection comparing with DETR?\n\nFor references, some sparse object detection methods [a,b] and graph-based 3d detection methods [c,d] are missing in the section of Related Work\n\n2、In Tab. 5, to compare the performance between DGCNN and multi-head self-attention, it would be better to list the computational cost, like FLOPs and #Parameters. Is the depth of multi-head self-attention the same as DGCNN (i.e., 6)?\n\n3、The ablation study is insufficient. In Sec. 6.4, most of them are about parameter tuning. It lacks the performance analysis of different modules (e.g., set-to-set prediction, DGCNN and knowledge distillation). For example, what are the advantages of the structural design of the proposed modules comparing with variations or similar competitors? Are there any insightful conclusions to be drawn from the optimal structural design or is the design consistent with the motivation or original idea?\n\n\n[a] Sun, Peize, et al. \"Sparse r-cnn: End-to-end object detection with learnable proposals.\" CVPR2021\n\n[b] Liu, Zili, et al. \"SparsePoint: Fully End-to-End Sparse 3D Object Detector.\"\n\n[c] Chen, Jintai, et al. \"A hierarchical graph network for 3D object detection on point clouds.\" CVPR2020\n\n[d] Shi, Weijing, and Raj Rajkumar. \"Point-gnn: Graph neural network for 3d object detection in a point cloud.\" CVPR2020\n\nAfter reading the authors' rebuttal and other reviewers' comments, they have solved part of my concerns, thus I would like to raise the rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has described the limitations of the work in Sec. 7. Please see comments above for more details", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Id8jTYyQK-r", "reviewer": "Reviewer_j8MH", "summary": "This paper proposes a method for object detection and velocity estimation in 3D LIDAR scans.  It improves previous voxel- or pillar-based frameworks by adding a graph network and by avoiding non-maximum suppression by teaching the network to not produce duplicates through bipartite result-GT-matching. Additionally, the bipartite matching is used to design a knowledge destillation framework to create a further improved network.\n\nThe proposed design is evaluated on a large standard dataset where it reaches and sometimes surpasses the performance of state of the art, without requireing an explicit NMS. Additional ablation studies show the effectiveness of the individual building blocks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Overview\n\nThe paper introduces a well though off architecture that introduces a NMS-free way for 3D object detection in LIDAR data. Though somewhat incremental in nature, the method is well motivated and convincingly evaluated with strong results. The end-to-end trainable architecture might open further research, as demonstrated with the proposed knowledge destillation.\n\n## Details\n\n* Overall the paper is very well written and and method is well described and can likely be reproduced based on the manuscript.\n\n  The only part I cannot quite follow are the formalities in Eq.(9) and (10). Shouldn't one of the 1_{...} expressions be c_j != empty set, instead of c_j == empty set? What is N in Eq.(10),(11),(12) (in \\Sum), is it the same N as line 131? Is the M in Eq.(9) the original number of ground-truth boxes, or the \"extended\" number M^*? To be honest it is rather difficult for me to follow these equations and their motivation and impact without looking into [7].\n\n* Though none of the building blocks used in the paper is novel ([4] and [60] as backbone, DGCNN for graph processing, [7] for the NMS-free bipartite matching, and a common student-teacher knowledge destillation), the combination is non-trivial and well thought off. The knowledge destillation would not be possible that easily without the NMS-free architecture, and the DGCNN built upon the Bird-Eye-View representation of the LIDAR data is well though off, aiding computational costs without much loss of performance, combining spatial relationships with feature-based relations through the dynamic graph. The method is quite generic and can use multiple different backbones. The NMS-free architecture might open up additional research.\n\n* The experiments are fair, extensive and overall convincing. A large standard dataset with a common evaluation protocoll is used, and the method is compared against state of the art. Several ablation studies are peformed to evaluate the impact of the different proposed extensions.\n\n* The results are good. In terms of absolute performance, the method is en par with or slightly exceeds state of the art. The results demonstrate that the proposed NMS-free architecture is a good replacement for NMS, i.e., the network learns not to produce duplicate results.\n\n  The only experiment that might be missing is to separately evaluate the impact of the Object DGCNN and the bipartite matching. For this, the last block of the network (the bipartite matching) could be replaced by a soft assignment followed by NMS, to compare \"Object DGCNN + NMS\" vs. \"Object DGCNN + Bipartite, no NMS\" (from what I understand, the methods that include NMS Table 1 evaluate \"Object DGCNN + Bipartite + NMS\").\n\n* An additional potentially interesting experiment is the distribution of the output scores w.r.t. overlapping boxes. The intuition is that the network learns the NMS and produces a single high-confidence prediction per object. Since a fixed number of predictions is used during inference, it is possible and even likely that these might include duplicate predictions, though with a lower confidence. If this intuition is correct, it could be demonstrated for example by performing a NMS and showing the confidence distributions of the removed boxes compared to the kept box; this should show a different curve than when performing NMS on [6], for example.\n\n\n## Further Notes\n\nTypos etc., no need to respond.\n\n* Eq.(3) and l.183: \\delta^0 ... \\delta^K is K+1 points, not K (compare L in l.173)\n\n* l.132: A more precise list of the 9 features of the bounding box would be interesting, or a reference to some paper that describes it.\n\n* The following paper might be interesting in the related work section. It is from this year's CVPR, i.e. published after the submission deadline, so it's not a must and completely at the authors' discretion:\n\n    [R1] Kumar, Abhinav, Garrick Brazil, and Xiaoming Liu. \"GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations, potential further work and potential societal impact are discussed in the conclusion. \n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After rebuttal, all three expert reviewers came to the same conclusion, namely that the paper would be a good contribution to NeurIPS. Having examined the reviews, rebuttal, and post-rebuttal discussion, the AC is in agreement with the reviewers. Given that the additional experiments (e.g., ablations) and promises of clarifications from the rebuttal were important for accepting the paper, the AC highly encourages the authors to incorporate these additional results into the final version of the paper. 
Decision: Accept (Poster)