Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies knowledge distillation for point cloud object detection. Designing efficient architecture to process large scale point clouds is a long-standing challenge for autonomous driving. To that end, this paper tries to improve model efficiency by leveraging knowledge distillation. This paper conducts an extensive study of existing knowledge distillation methods under several settings. It proposes a new metric called Cost Performance Ratio (CPR) to measure the performance and efficiency trade-off. Also, this paper provides two systematical ways to compress both inputs and architectures respectively. In addition, this paper analyzes the feature responses in both teacher network and student network and proposes a novel way to select a sparse set of locations in the feature map to distill. The best performing method using the proposed knowledge distillation achieves very strong performance on the Waymo dataset and even outperforms its teacher counterpart while only requiring less than half of teacher flops. 
strengths_and_weaknesses: Overall, I think this paper makes good contributions to point cloud knowledge distillation. It provides an adequate study of existing knowledge distillation methods under 3D object detection settings. In addition, it introduces a novel metric to compare different knowledge distillation methods. Moreover, the analysis of feature response leads to a module to select a sparse set of features to distill. I summarize strengths and weaknesses as follows.

Strengths:

1. First, I like the adequate study of existing distillation methods conducted in the paper. These experiments show a clear overview of how knowledge distillation works in the point cloud object detection scenario. Also, the reimplementations of existing methods (e.g., PV-RCNN++) are faithful and very close to SOTA models. 

2. The proposed CPR is technically sound. This metric considers both cost and performance and can be potentially used in general knowledge distillation settings. Also, this metric reflects empirical results/observations of models. 

3. The analysis of feature response is interesting and intriguing. Based on the feature response, this paper comes up with a way to select features to distill rather than distilling the whole feature map to avoid focusing on feature noises. 

4. Empirically, the proposed method shows quite strong performance compared to existing knowledge distillation baselines. Its best performing student network even outperforms the teacher counterpart while only requiring 44% of teacher flops. What is more convincing is that the improvements are consist under almost all settings. 

Weaknesses:

1. My first concern is about the generalizability of the proposed method. It seems that this paper focuses on two specific architectures -- PointPillars/SparseConvs and PVRCNN. I am curious that whether this proposed method is applicable to Transformer-based model such as Object DGCNN and 3DETR. As those methods are already sparse, can the proposed sparse distillation still be applied in those cases?

2. I understand this paper focuses on object detection, which is a coarse prediction task compared to fine-grained prediction like semantic segmentation. So I am wondering, if this method is useful for dense tasks like semantic segmentation or flow prediction. My worry is that for dense prediction tasks, the feature selection module won't introduce too much improvement compared to dense feature map distillation. It would be great if authors can provide evidences on tasks other than object detection.  
questions: I would appreciate feedback/comments to the following questions:

1. What do authors think of sparse/dense distillation? If we work on sparse models such as Object DGCNN and 3DETR, can we still do similar distillation as this paper does? 

2. Can authors provide code to facilitate reproducibility? I think there are many details/hyperparameters still missing. It is not straightforward to reproduce the work. 
limitations: I don't have concerns about the potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use knowledge distillation to train more lightweight 3d detectors (which tend to suffer from heavy computational overhead). For given teacher networks (CenterPoint Pillar and Voxel variants), the authors experimentally determine a collection of student networks (e.g. via reducing width/depth, input size in certain ways) that achieve good efficiency and performance.

Then given these student architectures, the authors experiment with different knowledge distillation schemes (including a proposed scheme - called Pivotal Position Logit KD - that puts more emphasis on foreground regions). There are a few main findings, but one punchline (across a number of experiments mostly done on the Waymo dataset) is that the authors are able to realize a 3d detection model that outperforms the teacher model with ~2x less FLOPS.

strengths_and_weaknesses: The clear strength of this paper is that it is able to get a clear win over CenterPoint with the Waymo dataset and the paper also offers insights that would help others that would like to do knowledge distillation for a similar architecture.

On the other hand, the ideas in the paper are incremental — mostly this paper can be viewed as an empirical exploration of existing knowledge distillation ideas (with the exception of the pivotal position KD idea — but even the high level idea here of emphasizing foreground regions is not novel).  There is a running theme that some KD approaches are synergistic with each other — and these are discovered empirically, but there are no principles offered by which a reader can think about this at a higher level.  Finally, it is hard to know how general these findings are.  For example, how do we know that the choice of student models does not depend on the specifics of each KD approach?  And can we expect any of these findings to generalize beyond CenterPoint based models?  Would they generalize to non-Waymo datasets?

questions: * For the proposed CPR (cost performance ratio) - why not simply use actual measured runtime instead of activations?
* In the section on selecting student architectures, it’s not clear to me how these models are trained — what knowledge distillation approach was used, what dataset etc?
* A suggestion for Table 3: have a more descriptive caption --- explain that the “Ours” column is not re-explained until a later section
* On pivotal position logit KD
    * The authors mention “instance-aware local region imitation” in Sec 4.2 but don’t really talk about it until Sec 5.1 (at which point it’s called something else — pivotal position logit KD).  So am I correct that these are referring to the same idea?  
    * It still makes sense to include some background contribution in the loss function — how is this incorporated in pivotal position logit KD?
    * Focal loss is often used in 2d detection as a way to deal with foreground/background imbalance — could this idea be adapted for knowledge distillation for 3d detection?

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the task of efficient 3D object detection. It first studies how to obtain student models with good trade off between accuracy and efficiency. Then, it proposes an improved KD pipeline incorporating an enhanced logit KD method and a teacher-guided student model initialization to facilitate transferring teacher model’s feature extraction ability to students through weight inheritance. Extensive experiments on Waymo dataset show the efficiency of the proposed method.
strengths_and_weaknesses: $\textbf{Strength}$

1. The motivation of solid, and the paper is well organized.

2. Extensive experiments are conducted to analyze the designs of efficient student networks and the performance of benchmark knowledge distillation for 3D object detection.

3. Experiments on WOD demonstrate the effectiveness of the improved knowledge distillation.

$\textbf{Weakness}$

1. The designs of student network are limited to input resolution compression or model width/length compression. And each variant only involves in one compression method. Discussions on more designs of lightweight student network are encouraged, to see if the conclusions are still held. It will also increase the difficulty of knowledge distillation.

2. In Tab. 3, what is the performance of applying all three kinds of KDs (like Eq. 4). It seems that only the result of each KD (i.e., logit KD, feature KD and label KD) has been analyzed.

questions: Please see the comments above.
limitations: Yes, the limitations have been discussed in the supplementary file.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work comprehensively looks into the state of knowledge distillation as it applies to point cloud 3D object detection. In particular, the authors argue for knowledge distillation’s use case for model compression. They begin by analyzing how different compression techniques, both model- and input-level affect performance by looking into two CenterPoint based detectors. Next, they benchmark a strong suite of current knowledge distillation methods. The authors also propose their own KD method combining the strengths of prior KD paradigms (logit KD, label KD, and teacher-guided initialization), showing superior performance over the baselines on the Waymo dataset.
strengths_and_weaknesses: The paper is very well organized and clearly written, and each component of the research question seems thoroughly analyzed. Reading through the paper was a self-containing opportunity to learn about the field of KD in 3D object detection. The analysis section is thorough; the work not only benchmarks the different baselines, they also organize them according to the KD paradigms explained in the paper. Interestingly, the authors also explore the combination of different distillation paradigms. The method proposed was also convincing, obtaining superior performance over the baselines with a compressed model. The ablation section is thorough, showing that each part of the final method contributed to the performance. 

A minor criticism is that the study was only done on CenterPoint based models, and that there were no point-cloud based models evaluated (i.e., the conclusions were for pillar- and voxel-based inputs only). It would be interesting to see how point-based models (such as PointRCNN) would compress and hold up to these conclusions.

Miscellaneous:
L82 vs. L 85: "Suppl." vs "Appendix" term mismatch

questions: In Section 5.3, it is observed that the student model can outperform the teacher model. Is there intuition for why this may be? Intuitively, unless the student model has access to labels, it should not have enough information to correct these mistakes.
limitations: The authors do not explicitly address the limitations in their work.
review_title: null

Meta-review: metareview: In this paper, the authors propose a new method for knowledge distillation for 3D object detection in point cloud data. This problem is quite important for self-driving cars and 3D computer vision. The goal of their work is to compress models to achieve reasonable trade-offs in compute performance versus accuracy. The authors explore these questions using two popular forms of 3D detection: pillar-based and voxel-based architectures (a) and focus on extensive experimentation with the Waymo Open Dataset.

The authors first examined how to build student-teacher models with good trade-offs between accuracy and computational demand, introducing a new metric the Cost Performance Ratio (CPR). The authors then systematically explore a series of knowledge distillation methods (e.g.  logit KD, label KD, and teacher-guided initialization) to identify their best model. The end result of their search is to identify a student model that is able to outperform the teacher model but with ~2x less FLOPS.

The reviewers commented positively on the strength experiments and baselines as well as the selection and CPR metric. The main issues surfaced by the reviewers focused on the generalizability of the results outside of 3D object detection including whether the methods or results may be ported to other 3D detection architectures or other localization tasks. The authors responded with some discussion and new early experiments highlighting that the work may be ported to problems in semantic segmentation.

From my perspective, the paper comes across as technically sound with strong experiments and a solid overall result. I am concerned about the generality of these results. As this work largely focuses on 3D object detection of specific detection architectures, I could imagine that this this work would be better suited for conferences geared towards the topics of 3D object detection and self-driving cars (e.g. CVPR and associated workshops). In that spirit, I would consider this paper borderline. However, because this work shows promise for other architectures and problems, I view this work as potentially having more generality. It would be incumbent on the authors to revise their manuscript accordingly to include additional discussion on this generality as well as showcase encouraging results on other domains. This paper will be conditionally accepted assuming all of these changes are made to this manuscript.

(a) Note that this work seems to exclude the recently popular range-based methods [1] and it would be important for the authors to add discussion to their paper accordingly. E.g. 

[1] LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving
Gregory P. Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzalez, Carl K. Wellington (2019)

confidence: Less certain
Decision: Accept