Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a cross-modal knowledge distillation framework for mutli-view 3D object detection, with a 3D point cloud detector as teacher and an image detector as student.  The proposed method unifies different modalities in the BEV space and conducts knowledge distillation through the dense feature distillation and the sparse instance distillation. Extensive experiments demonstrate that the proposed method outperforms current KD approaches, and the method improves baseline model with a significant boost and achieves new state-of-the-art performance on nuScenes test leaderboard.
strength_and_weaknesses: Strength
+ The overall design is relatively novel,  it unifies the image and LiDAR features in the BEV space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm.
+ The method is well ablated and  achieves state-of-the-art performance on nuScenes.
+ The writing is good and easy to follow.

Weakness
- It would be better if some visualizations are provided to verify the effectiveness of proposed method.
- More experiments on comparison with BEVDepth will be interesting, which directly uses point clouds as depth ground truth and supervises the image detector. This comparison is necessary to clarify the motivation of the teacher-student paradigm, since it is also possible to supervise the image detector with LiDAR point cloud (more specifically, depth). 
clarity,_quality,_novelty_and_reproducibility: Good clarity and quality.
The novelty is accecptable.
The reproducibility is good since sufficient details (e.g., proofs, experimental setup) are described.
summary_of_the_review: The paper presents a cross-modal knowledge distillation framework for mutli-view 3D object detection which unifies the image and LiDAR features in the BEV space. The paper has good writing and extensive experiments. However, the paper is lack of more visualizations to verify the proposed method. 

My biggest concern is regarding the motivation of using the teacher-student distillation paradigm, since it is also possible to supervise the image detector with LiDAR point cloud (more specifically, depth). If authors can provide comparisons with BEVDepth and show the necessarity of the adopted distillation setting, I may raise my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores LiDAR-to-Monocular distillation methods for BEV object detection. Their method, BEVDistill, builds upon the work of BEVFusion and uses the intermediate BEV representation to ease the multi-modal alignment and feature imitation. This allows the use of an asymmetrical LiDAR detector, Object-DGCNN, which this paper chose to use. The method also uses a “dense feature distillation” and “sparse instance distillation”, which supervises both the feature level (with a soft-focus on the foreground region) and the instance level (focusing on high quality teacher predictions). This method is evaluated extensively on the nuScenes dataset, with a strong leaderboard performance (5th place on NDS score at the time of review). The authors also conduct extensive ablation studies to validate the components of the model.
strength_and_weaknesses: **Strengths**
The paper is very well written, and analytically thorough. The strongest point of this paper is the clear and complete experimental section, that exhaustively ablates the key components of the model. Reading over the tables, I observe that it is competitive as compared to a strong set of baselines (Table 1), is robust across backbones (Table 2), and is a well motivated distillation method with each component contributing to its performance (Table 3). Baseline wise, the authors compare against a strong set of methods, that – to the best of my knowledge – is comprehensive. Further ablations on the method’s distillation components and losses are interesting and informative, and convincingly beat a set of baseline comparisons.

One interesting point is that this method cleverly deals with different representations between the Student and Teacher models, which doesn’t limit the architecture of both (up until the distillation representation). This distillation process (foreground focusing) is intuitive and is validated experimentally (Table 4).


**Weakness**
One weakness is that the method appears to only be evaluated on a single dataset, nuScenes. It is strange to me, however, since the model BEVfusion from which they are building off of is evaluated both on nuScenes dataset and Waymo dataset. **This paper would be much stronger if there is a second dataset that they compare their method on**. This would ensure that their results – while excellent – are also general.

**Misc**
I am also curious if this distillation process (across BEV features) works on different BEV type detectors? It is a minor point, but does this distillation improve other Pseudo-LiDAR (Wang 2019) type methods, Lift-Splat (Philion, 2020) type methods, or Inverse Perspective Mapping (Reiher, 2020)? In theory, the method should improve upon all of the detectors that utilize intermediate BEV representations, and I would be curious to see if this is indeed the case.

clarity,_quality,_novelty_and_reproducibility: The method is clearly written, and would be reproducible by someone with some experience. At the time of the review, there does not seem to be any code released or promises of code release. The work is well motivated, of high quality, and easy to read. The greatest contribution of this work appears to be in the experimental results, and the strong performance on the nuScenes dataset shows the distillation proposed has potential. It appears to be built off of a few existing architectures, in particular the BEVFusion work, and novelty seems limited. The method of distillation is inspired by other works, but cleverly implemented by the authors. However, the strong experimental section is noteworthy and should be considered a strong contribution.
summary_of_the_review: In summary, this work builds upon the BEVFusion work and utilizes the intermediate BEV representation to ease the distillation process between detectors using LiDAR input to detectors using monocular camera input. By using their  “dense feature distillation” and “sparse instance distillation”, they are able to achieve very strong performance on the nuScenes dataset. The strong analysis section and ablations support their claim. However, their results are slightly limited by the fact that their results are only on a single dataset. Overall, the work is clear; I would be willing to increase the score if additional results on another dataset support their claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method named BEVDistill to distill knowledge from LiDAR-based 3D object detectors (teacher) to enhance multi-view 3D detectors (student). 
First, since both detectors project their inputs to the BEV space, the proposed method performs feature distillation between the BEV feature maps under the guidance of a ground-truth gaussian mask. 
Second, the teacher and student both adopt DETR-style prediction heads, the prediction-level distillation requires a matching between the two sets of instance predictions generated by the teacher and student models. Instance-level distillation is then performed between each pair of matched instances through a bounding box regression loss and a mutual information loss between the teacher and student’s hidden representations before logits. To suppress low-quality instance predictions, the authors also design an IoU-weighted quality score as the weight for each instance pair in the distillation loss.

The experiments on the nuScenes dataset proves that the proposed method works well for the chosen student model (BEVFormer) with various backbones. The ablation studies demonstrated the effectiveness of feature and instance distillation, as well as several specific designs within each component.

strength_and_weaknesses: Pros) 
Distilling knowledge from LiDAR-based detectors to multi-view image-based detectors is a novel attempt and can be a practical technique to enhance image-based detectors.
The effectiveness of the method and its components is shown by the experiments.

Cons) 
The designed IoU-based quality-score itself is not technically novel (explored in LiDAR-based methods [1,2]), and its effectiveness is not analyzed in the experiments.
Although effective, mutual information-based knowledge distillation is also studied in previous works [3]. This paper directly used an existing formulation.
Some necessary implementation details are missing, e.g., the overall loss and the hyper-parameters (\alpha and \beta in Eq.(13), \sigma_i in Eq(2), and \gamma in Eq(6)) are not given. These could affect the reproducibility of the proposed method.

[1] Zheng W, Tang W, Chen S, Jiang L, Fu CW. Cia-ssd: Confident iou-aware single-stage object detector from point cloud. AAAI 2021.
[2] Hu Y, Ding Z, Ge R, Shao W, Huang L, Li K, Liu Q. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. AAAI 2022.
[3] Tian Y, Krishnan D, Isola P. Contrastive representation distillation. ICLR 2020.

clarity,_quality,_novelty_and_reproducibility: The method is clearly presented and easy to understand, but some necessary implementation details are missing, which affects its reproducibility. The distillation setting is novel but the technical novelty of the proposed method is limited (see weaknesses above).
summary_of_the_review: Although this paper proposed an effective method for distilling knowledge from LiDAR-based detectors to multi-view image-based detectors, which is a novel attempt towards enhancing image-based detectors, the technical novelty of each proposed component is limited, since IoU reweighting and mutual information-based KD are previously studied. The experiments demonstrate reasonable improvements but some necessary implementation details are missing. Overall, this paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a cross-modality knowledge distillation framework for multi-view 3D object detection, called BEVDistill. It introduces two types of feature distillation, i.e., dense feature distillation and sparse instance distillation. Experiments prove the effectiveness of the proposed modules on the nuScenes dataset.
strength_and_weaknesses: Strength:
---------------
1. This work focus on cross-modality distillation. It's reasonable to transfer geometry-aware knowledge to images during training.
2. The experiments are sufficient and proven to be effective on the nuScenes dataset.
3. The overall writing is clear and easy to follow.

Weakness:
---------------
1. Compared with previous work that uses feature distillation like LIGA-stereo and UVTR, the main contribution of this work is the sparse instance distillation. However, target-level distillation is not a novel thing in the distillation domain. And it only contributes 0.6% mAP gain and 1.0% NDS gain on top of the dense feature distill baseline in Table 3. That means most of the improvements are still from the dense feature distillation. This could make the contribution of this work limited. 
2. The author claims "we postpone such supervision after the transformer encoder, which offers more opportunities for the network to align information across different modalities." in Section 3.2.1. But according to Figure 1, the dense feature distillation is still operated in the BEV representation. The reviewer is confusing the difference between it and UVTR, and why it offers more opportunities for the network to align information. Maybe the author can provide more details on the dense feature distillation.
3. Why the BEV feature in Figure 1 is 3D BEV? Is the height axis preserved here? It's better to add some illustrations in Section 3.2.1.
4. Writing: Section 4.2 can be moved to supplementary material. And it's better to report Table 10 in the main paper.
clarity,_quality,_novelty_and_reproducibility: The overall writing and clear and easy to follow. But the novelty could be limited.
summary_of_the_review: My main concern is the contribution of this work. It could be limited compared with previous cross-modality distillation methods. Please refer to the Weakness section for more details.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes BEVDistill, a cross-modal knowledge distillation framework for multi-view 3D object detection where all features are transformed into BEV space.  BEVDistill uses a LiDAR-based 3D point cloud detector as the teacher and an multi-view image detector as the student model.  The method support both a "dense feature distillation", with weights on foreground regions from the teacher (as LiDAR points will be only for part of the image), and "sparse instance distillation", where high quality instances from the teacher are weighed more.  Experiments on nuScenes show that the proposed approach outperforms prior distillation approaches and there are extensive ablation studies to justify the various model components.

Strengths
- The proposed method of knowledge distllation from LiDAR to image in BEV space is sufficiently novel and works well
- Experiments show the proposed approach outperforms prior approaches on nuScenes (and also Waymo after rebuttal)
- Paper is well written with clear description of the method, and careful experiments and ablations

Weaknesses
- Reviewers noted that the technical novelty of each component is limited
- Reviewers also noted that there may be some missing details that could make some parts of the method not completely reproducible


justification_for_why_not_higher_score: The proposed method is very specific and while effective, it is unclear whether it can be applied to other classes of problems.  Reviewers also noted that the technical novelty of the individual pieces are limited and have been explored in other work.
justification_for_why_not_lower_score: Reviewers all found the paper to be well-written and the proposed approach sufficient novel, and to work well, and all recommend accept.
Decision: Accept: poster