Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the task of 3D object detection from point cloud. Inspired by recent non-maximum suppression-free 2D object detection modes, this paper proposes a new 3D object detection architecture. Concretely, it models 3D object detection as message passing on a dynamic graph, generalizing the framework to predict a set of objects. Furthermore, it removes the necessity of post-processing via object confidence aggregation or non-maximum suppression. To facilitate object detection from sparse point clouds, the authors also propose a set-to-set distillation approach customized to 3D detection. Experiments are conducted on nuScenes dataset to demonstrate the effectiveness of the proposed method.
main_review: 1、The contribution of this paper is limited, and the motivation from the first section seems to be not solid. 

First, the authors just very simply introduce methods of PointPillars, PillarOD and CenterPoint in 3D object detection(L19-26), which is obviously not enough. Directions like voxel/point-based or one/two-stage are not discussed. 

Second, the proposed method mainly involves technologies of set-to-set prediction [9], GCN [8] and knowledge distillation [10], all of them are popular algorithms published by others recently. However, the author didn’t not provide detailed and reasonable discussions/comparisons about why using these technologies, it seems to just simply combine them together. For the part of knowledge distillation, what is the difference or improvement comparing with [10, 56, 49]? What is the insightful design in sparse 3D object detection comparing with DETR?

For references, some sparse object detection methods [a,b] and graph-based 3d detection methods [c,d] are missing in the section of Related Work

2、In Tab. 5, to compare the performance between DGCNN and multi-head self-attention, it would be better to list the computational cost, like FLOPs and #Parameters. Is the depth of multi-head self-attention the same as DGCNN (i.e., 6)?

3、The ablation study is insufficient. In Sec. 6.4, most of them are about parameter tuning. It lacks the performance analysis of different modules (e.g., set-to-set prediction, DGCNN and knowledge distillation). For example, what are the advantages of the structural design of the proposed modules comparing with variations or similar competitors? Are there any insightful conclusions to be drawn from the optimal structural design or is the design consistent with the motivation or original idea?


[a] Sun, Peize, et al. "Sparse r-cnn: End-to-end object detection with learnable proposals." CVPR2021

[b] Liu, Zili, et al. "SparsePoint: Fully End-to-End Sparse 3D Object Detector."

[c] Chen, Jintai, et al. "A hierarchical graph network for 3D object detection on point clouds." CVPR2020

[d] Shi, Weijing, and Raj Rajkumar. "Point-gnn: Graph neural network for 3d object detection in a point cloud." CVPR2020

After reading the authors' rebuttal and other reviewers' comments, they have solved part of my concerns, thus I would like to raise the rating to 6.
limitations_and_societal_impact: The author has described the limitations of the work in Sec. 7. Please see comments above for more details
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for object detection and velocity estimation in 3D LIDAR scans.  It improves previous voxel- or pillar-based frameworks by adding a graph network and by avoiding non-maximum suppression by teaching the network to not produce duplicates through bipartite result-GT-matching. Additionally, the bipartite matching is used to design a knowledge destillation framework to create a further improved network.

The proposed design is evaluated on a large standard dataset where it reaches and sometimes surpasses the performance of state of the art, without requireing an explicit NMS. Additional ablation studies show the effectiveness of the individual building blocks.
main_review: 
## Overview

The paper introduces a well though off architecture that introduces a NMS-free way for 3D object detection in LIDAR data. Though somewhat incremental in nature, the method is well motivated and convincingly evaluated with strong results. The end-to-end trainable architecture might open further research, as demonstrated with the proposed knowledge destillation.

## Details

* Overall the paper is very well written and and method is well described and can likely be reproduced based on the manuscript.

  The only part I cannot quite follow are the formalities in Eq.(9) and (10). Shouldn't one of the 1_{...} expressions be c_j != empty set, instead of c_j == empty set? What is N in Eq.(10),(11),(12) (in \Sum), is it the same N as line 131? Is the M in Eq.(9) the original number of ground-truth boxes, or the "extended" number M^*? To be honest it is rather difficult for me to follow these equations and their motivation and impact without looking into [7].

* Though none of the building blocks used in the paper is novel ([4] and [60] as backbone, DGCNN for graph processing, [7] for the NMS-free bipartite matching, and a common student-teacher knowledge destillation), the combination is non-trivial and well thought off. The knowledge destillation would not be possible that easily without the NMS-free architecture, and the DGCNN built upon the Bird-Eye-View representation of the LIDAR data is well though off, aiding computational costs without much loss of performance, combining spatial relationships with feature-based relations through the dynamic graph. The method is quite generic and can use multiple different backbones. The NMS-free architecture might open up additional research.

* The experiments are fair, extensive and overall convincing. A large standard dataset with a common evaluation protocoll is used, and the method is compared against state of the art. Several ablation studies are peformed to evaluate the impact of the different proposed extensions.

* The results are good. In terms of absolute performance, the method is en par with or slightly exceeds state of the art. The results demonstrate that the proposed NMS-free architecture is a good replacement for NMS, i.e., the network learns not to produce duplicate results.

  The only experiment that might be missing is to separately evaluate the impact of the Object DGCNN and the bipartite matching. For this, the last block of the network (the bipartite matching) could be replaced by a soft assignment followed by NMS, to compare "Object DGCNN + NMS" vs. "Object DGCNN + Bipartite, no NMS" (from what I understand, the methods that include NMS Table 1 evaluate "Object DGCNN + Bipartite + NMS").

* An additional potentially interesting experiment is the distribution of the output scores w.r.t. overlapping boxes. The intuition is that the network learns the NMS and produces a single high-confidence prediction per object. Since a fixed number of predictions is used during inference, it is possible and even likely that these might include duplicate predictions, though with a lower confidence. If this intuition is correct, it could be demonstrated for example by performing a NMS and showing the confidence distributions of the removed boxes compared to the kept box; this should show a different curve than when performing NMS on [6], for example.


## Further Notes

Typos etc., no need to respond.

* Eq.(3) and l.183: \delta^0 ... \delta^K is K+1 points, not K (compare L in l.173)

* l.132: A more precise list of the 9 features of the bounding box would be interesting, or a reference to some paper that describes it.

* The following paper might be interesting in the related work section. It is from this year's CVPR, i.e. published after the submission deadline, so it's not a must and completely at the authors' discretion:

    [R1] Kumar, Abhinav, Garrick Brazil, and Xiaoming Liu. "GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

limitations_and_societal_impact: Limitations, potential further work and potential societal impact are discussed in the conclusion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces an NMS-free framework for 3D object detection in self-driving scenarios, following the 2D representative work DETR. Instead of using Transformers, it uses a dynamic graph CNN (DGCNN) to generate the set of detected objects. Besides, it proposes a set-to-set distillation loss to distill the knowledge of the teacher model, e.g., trained on multiple frames, for the student model, which will be tested on single frames. The experiments are only conducted on nuScenes.
main_review: Strongness:
- The writing is clear.

---

Weakness:
1. The major contributions are ambiguous. The connection between the NMS-free detector and the distillation is not tight. It confuses me when I read the introduction for the first time, as the knowledge distillation seems to suddenly appear. The authors seem to plan to equip the paper with two contributions, but it turns out that they are not connected well. Motivations, especially the connection of the two contributions, can be discussed more in the introduction.
2. The experiments are only done on one dataset (nuScenes). Most 3D object detection papers recently conduct experiments on at least two datasets (KITTI, Waymo apart from nuScenes), or conduct more comprehensive experiments on one dataset, in order to make the results more convincing. Can the authors show the results on any other dataset?
3. For the distillation experiments, the models are trained only for 1 epoch (mentioned in Sec 6.3). Can the authors report the results in Sec 6.3 and 6.4 with longer training time (like 20 epochs), so that they can be directly compared with the numbers in Table 1? 
4. As the authors have mentioned in Sec 6.4, dense graph networks, like Transformers, might be slower to converge compared to sparse graph networks, like DGCNN. Thus, can the authors also report the ablation results with a longer training time (20 epochs)?


---

Typo:
L55: objectors -> object detectors
The introduction section title is missing.

---

**Updates after rebuttal**
I slightly raise my rating to 6 according to the authors' rebuttal. However, the authors should improve their connection between two contributions.


limitations_and_societal_impact: The authors have not adequately addressed the limitations.
review_title: null

Meta-review: After rebuttal, all three expert reviewers came to the same conclusion, namely that the paper would be a good contribution to NeurIPS. Having examined the reviews, rebuttal, and post-rebuttal discussion, the AC is in agreement with the reviewers. Given that the additional experiments (e.g., ablations) and promises of clarifications from the rebuttal were important for accepting the paper, the AC highly encourages the authors to incorporate these additional results into the final version of the paper. 
Decision: Accept (Poster)