Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, authors propose a monocular image-based 3D object detection method based on knowledge distillation. Specifically, the monocular image-based 3D object detector [Ma et al 2019] is infused with the depth cues via distilling knowledge from LiDAR-based teacher model. At the test-time, the model detects 3D objects without any intermediate depth prediction and ranks 1st on the KITTI benchmark dataset. Hence, the proposed monocular-based 3D detector is end-to-end and more accurate than the other methods in this category, without adding extra depth-estimation overhead in between. 
main_review: Strengths:

(1) The end-to-end monocular 3D detection without intermediate depth estimation, which integrates depth cues relevant for 3D detection. Intermediate depth-estimation is a real bottleneck in monocular 3D object detection as shown in multiple previous work: [Ma et al 2021, Reading et al 2021, Lu et al 2021]. Hence, this contribution to infuse depth-cues directly to the monocular network via distillation is an important contribution.  

(2) The method proposed to infuse depth via distillation across structured-scene space, object feature-space, and object result-space. Through ablation study it allows to deduce that the design components are important for the 3D object detection. Furthermore, during distillation, authors mainly focus on the object-features that allow the network to focus on them and this argument is also supported by the ablation study. 

(3) Paper shows that the accuracy improvement compared to the baseline model mainly comes through improvement  in depth and dimension prediction of the objects. This has been shown via a valid cross-model experiment design. 

Weakness:

(1) The main claim of the paper is that the depth-cues can help in 3D object detection from monocular images. However, the results are only shown for only one teacher model [Ma et al, 2019]. This puts the generality of depth-cue based distillation in question. Also, from the section “3.2 Student Model” -- the choice of Ma et al. as a baseline is not clear. Can authors please explain what would happen with the other monocular-based methods as a teacher? Can you please show a small experiment?

(2) The motivation for the scene level distillation is not clear. The first line of Sec 3.3 “First, we believe that scene information is beneficial for our task” -- However, it is not clear why? A very similar method to the proposed scene level distillation via affinity map has already been proposed and used for semantic segmentation in the past [Hou et al 2020]. Can authors please explain why this would be beneficial for the 3D detection task? Also, please cite the relevant work. 

(3) “Besides, we further normalize the confidence of each predicted object using the estimated depth uncertainty (see Appendix A.1 for more details), which brings about 1 AP improvement.” -- while it is completely fine using the tricks that bring the accuracy up, this needs to be clearly stated in the results. It is not clear if the baseline model uses such normalization. If not, the comparison with the baseline model is not an egg-to-egg comparison. This needs to be clearly stated in the paper by adding accuracy of the current method without such normalization in table 3, table 4. And mentioning the accuracy improvement by infusing depth cues and confidence normalization separately Section 4.2 “Comparison with state-of-the-art methods”


Other minor points:

(1) Usually the projected LiDAR images will have empty regions (e.g. Fig. 2 have empty black regions on the top). How is this handled during training? Since, the LiDAR image in Fig.3 shows a complete image without any empty region. Also, do we need to do any pre-processing of the camera images at the inference time which requires the use of LiDAR information?

(2) Sec 4.2 Ablation Studies “3D detection performance by 3.34, 5.02, 2.98 and improve BEV performance by 5.16, 6.62, 3.87” -- here authors should compare their numbers with the confidence normalized baseline numbers, and not the raw baseline because improvements by confidence normalization are significant in this context.

(3) Sec 4.2 detailed design choice “which improves the accuracy by 0.7” → 0.7 in what? I think this number is only valid for 3D@IOU=0.7 moderate benchmark. Authors should mention that in the text.

(4) Sec 4.2 Comparison with State-of-the-art “By contrast, our method only takes 25ms to process a KITTI Image”. However, in Table 3, the runtimes show 40ms. Why this difference? I believe 40ms is the valid number since baseline (MonoDLE)  also reports 40ms runtime. 

(5) Table 3. → Authors should also put “*” superscript for their own method since it requires LiDAR data while training.

Typos:

4.2 Ablation Studies: “Specifically, we can found” →  “Specifically, we found” 

4.2 Comparison with state-of-the-art methods: “our method can only takes” → “our method only takes” 

4.3 What the model has learned “improvement of dimension part is also considerable (e → f)” → I think authors mean (c → f) ?

4.3 Do we need depth estimation “Here we qualitatively show the information loss” → I think it is “quantitatively”, since the table-7 are just accuracy numbers? 

References:
Hou, Yuenan, et al. "Inter-region affinity distillation for road marking segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

summary_of_the_review: I recommend this paper to be accepted for the ICLR conference (score: 8 good paper). Overall, the paper is well written and does extensive ablation studies for their design choices. In this paper authors proposed a monocular Image-based 3D object detection method that distills the knowledge from the LiDAR-based teacher. At the inference time  this method predicts at 25 FPS on GPU without any intermediate depth-prediction, which is critical for the real-time application such as autonomous driving. 

However, there are some minor issues in some of the claims (please see above). I would like to see the authors addressing “Weaknesses” in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to distill the features from a LiDAR teacher model to a monocular-based student model. To align the feature maps between the teacher and student model, the teacher model uses the same networks as the student and the only difference is that the teacher model takes the sparse/dense depth map as input. Several techniques include scene-level distillation, object-level distillation in feature/outputs, feature fusion. In the experimental part, this paper extensively ablates several key analyses including cross-modal evaluation between baseline model and full model, depth estimation. 
main_review: Pros:
1.	The paper is well written. The overall pipeline is simple and the key factor of KD is well illustrated. 
2.	The motivation and experiments look reasonable to me. I think this is the first paper that distills the features from the LiDAR model to the monocular model to learn depth cues, which gets good results and improvement. Experiments demonstrate the student model learns to predict better 3D location. Without inference cost, the improvement is indeed impressive despite KD being a well-known approach to improve model performance.

Cons:
1.	The main concern is about the technical improvement for distillation. As there are some papers [1] that demonstrates the effectiveness of cross-modal distillation, I think the paper should at least be compared with the baseline KD approaches and demonstrate the superiority of the proposed modules such as ``scene/object-level distillation'', whether to distill affinity map and discuss why the proposed methods work better than simple KD for monocular 3D object detection, or particularly depth estimation. Otherwise, the contribution is not enough as it cannot provide any new information for cross-modal KD. Also, the related works about KD for 2D object detection (e.g. [2]) as the main point of this paper can be used as the baseline and should be included at least. 
2.	From the ablation studies, it seems that the depth cues are the key factor to affect the learning of the model. Is it possible due to the teacher network interpolates the sparse depth map and thus the network transfers better the depth supervision? I think an ablation study of dense depth map supervision can help the analysis.
3.	The performance improvement for Pedestrian and Cyclist is not so obvious. The comparison with the baseline model in KITTI validation is missed and should be supplemented. 

[1] Cross Modal Distillation for Supervision Transfer. Gupta S, et al. CVPR 2016
[2] Learning Efficient Object Detection Models with Knowledge Distillation, Chen et.al. NeurIPS 2017


summary_of_the_review: The basic motivation is quite good and is worth studying. The current ablation studies also demonstrate the good point of cross-modal distillation. The performance seems good enough in the related works. But it’s known that cross-modal KD is helpful as shown in the Main Review (1), which is not a new enough idea. And the technical improvement of the proposed KD method over the baseline KD approach should be strengthened more. Overall, I would like to give the rating of borderline reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents MonoDistill, a way to enhance RGB-based 3D object detection through knowledge distillation from a LiDAR-based teacher.
A 3D detection pipeline (built around MonoDLE) is trained both on RGB images and densified LiDAR input. Three main mechanism enable teacher-student transfer: On feature level, scene-level distillation aligns affinity maps between teacher and student while masked features are trained on object-level, additional masked pseudo labels are leveraged around the teacher centre predictions.
Each component is carefully ablated and experiments are conducted on the KITTI benchmark.

main_review: In the following, strengths (S1-4) and weaknesses (W1-3) are detailed.

S1. Distillation pipeline. The way in which the student is trained through scene-level and object-level distillation together with the extended pseudo labels is an interesting approach to leverage the LiDAR input at training time which does not require architectural changes between student and teacher.

S2. Comparison to intermediate depth estimation tasks. Ablation experiments are conducted to question the usage of the additional task of depth estimation often used for 3D detection (also from stereo in the supplementary material). They give insight that this direction might be less effective. This might affect future network design choices.

S3. Results. The experimental evaluation of all involved components (loss functions, but also the densification) justifies their contributions. The overall accuracy of the pipeline is favourable over many state-of-the art approaches.

S4. Paper presentation. The paper is clearly motivated and easy to understand.

W1. Fair comparison. In the comparison (Tab. 3): The presented method should also get a "*" as the LiDAR signal is implicitly used for training.

W2. Minor notation information: In eqn. (3) information is missing on the index k.

W3. Some typos, grammar mistakes and minor issues make some sections of the paper hard to read. These are (in order of appearance):
"v.s."
"geometric constrain"
"then use LiDAR-based model"
"lots works"
"monocular v.s. stereo"
"results shows"
"baseline model adopt"
"and use several"
"this model achieve"
"models [are] mainly based on"
"the 2D bounding box are used"
"we can found that"
"our full model improve"
"can only takes"
point before Table 3
"Table 7 show"
"23,488 v.s. 3,712"
"depth maps v.s. noisy depth"
summary_of_the_review: While the paper has minor weaknesses, I believe that they can be corrected in the course of the review process.
The strenghts (in particulat S1-3) make this work worth sharing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach to leveraging knowledge distillation (KD) for training image-based monocular 3D detectors. Different from the ways to incorporate the LiDAR signal in prior works, this paper takes it as the input of a teacher network and further supplements the image-based student network in terms of spatial information. Three levels of distillation, scene-level, object-level in the feature space, and object-level in the result space, are devised. Experimental results show that this method can effectively boost the performance of image-based monocular 3D detection while still maintaining outstanding efficiency.
main_review: Strengths:
- The basic idea of this paper is easy to follow.
- The motivation and core contributions are clearly presented.
- The methodology is simple yet effective.
- Some experimental designs are novel and convincing. For instance, Table 5 for cross-model evaluation can validate that the localization accuracy has indeed improved a lot, and Table 6 provides a counterintuitive conclusion that the performance of the teacher model is not directly related to the performance improvement.
- The overall model achieves state-of-the-art on the KITTI benchmark while maintaining impressive efficiency.

Weaknesses:
- The main concern is that the technical novelty is limited. This paper is basically an application of knowledge distillation on monocular 3D detection while only different in terms of specific designs and conclusions due to different tasks.
- The generalization ability of this method is not discussed. For example, if the settings of LiDARs or cameras are changed, can the student network generalize to other scenarios well? If the consistency of these intrinsic and extrinsic settings is quite necessary, the application of this kind of method will also be much more limited.
- A minor problem is that the size of the KITTI dataset seems too small for now. It would be more convincing if the method can be validated on large-scale datasets, such as nuScenes and Waymo.

Minor comments:
- There are some small grammatical typos, such as “thus no extra computational cost is introduced” needs an “and” in front, and “Existing LiDAR-based models based on” needs an “are”. Please double-check, fix them and also polish the writing.
- Figure 4: The Gaussian-like mask means Gaussian weights in Eqn. (4) or only center-sampling regions with equal weights? Here it also lacks reference to FCOS [1] for 2D detection and FCOS3D [2] for monocular 3D detection.
- “Do we need depth estimation as an intermediate task”: This claim is closely related to another missing reference (it does not matter because it is really recently published), DD3D [3]. The difference in terms of method design is clear but it can be useful if an empirical comparison can be conducted.
- Table 7: It is not clearly presented the source of training data for DORN. Is it from KITTI-Depth or other datasets? Is there any data-leakage problem? Does it influence the result or conclusion in this part?
- Appendix A.3: How was the depth error computed? Is it done for top-k detection predictions or in other ways?

References:

[1] FCOS: Fully Convolutional One-Stage Object Detection. ICCV 2019

[2] FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection. ICCVW 2021

[3] Is Pseudo-Lidar needed for Monocular 3D Object detection? ICCV 2021
summary_of_the_review: This paper applies knowledge distillation to image-based monocular 3D detection and achieves state-of-the-art results. It can provide a potential path to better leverage LiDAR signals to boost the image-based 3D detectors while not much influencing the original design and efficiency. However, there are some concerns in terms of novelty and generalization ability. So I would vote for borderline accept temporarily.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper:  Aiming at accurately detecting objects in the 3D space from a single image, this paper proposes a new method. Experiments on KITTI dataset seems to show promising results.
main_review: 1. This paper claims to be a mono-based 3D object detection method. Although only RGB image data is needed in the test process, point cloud data is actually used in the training process. From this point of view, this claim is not appropriate.

2. Figure 3 is very confusing, please redraw it according to the actual process in order to show the actual process more clearly.

3. This paper claims in many places that the method based on monocular image is more valuable in application than the method based on pure point cloud, but it has to be admitted in the detection result that the result of 3D object detection based on image is far inferior to that based on pure point. So from this point of view, do the above remarks lack the measurement dimension of detection accuracy?

4. On the experimental results. (1) This paper choses this KITTI to conduct experiments. Although the good results are shown in Table 3, the method of this paper (anonymous) is not seen on KITTI's official website. The results on the KITTI test set should be submitted to the KITTI official website, so please submit your results anonymously on the KITTI official website, otherwise the authenticity of the experimental results in this article will be seriously questioned. (2) In the 3D object detection task, more data sets have emerged, such as Waymo, nuScences, etc. This article should try on more data sets.

5. The writings should be checked carefully. There are many singular and plural problems, symbol problems, tense problems and so on. Such as need->needs in Section 2, two repeated symbols f_j below formula 1.
summary_of_the_review: This paper has many problems in experiments, especially the authenticity of the experimental results, and the completeness and richness of the experiments. And there are many writing errors in this paper.
review_title: null

Meta-review: This paper received 5 quality reviews, with 3 of them rated 8, 1 rated 6, and 1 rated 5. In general, while there are minor concerns, the reviewers acknowledge the contribution of applying Knowledge distillation to the problem of monocular 3D object detection, and appreciate the SOTA performance on the KITTI validation and test sets. The AC concurs with these important contributions and recommends acceptance.
Decision: Accept (Poster)