Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces DeepInteraction, a new approach to fuse rgb and depth input to do 3D object detection. Prior works only add rgb features to the Lidar point cloud, while DeepInteraction treat them equally and fuse them in the transformer layers. Experiments are conducted on nuScenes benchmark and DeepInteraction achieves state-of-the-art performance.

strengths_and_weaknesses: Strengths:

- DeepInteraction achieves state-of-art performance on nuScenes benchmark. Under the same evaluation setting (without ensemble), it generally improves mAP by 2 points whether test-time augmentation is used or not.
- The experiments on nuScenes are extensive. It's interesting to see DeepInteraction has reasonable performance whether the 3D backbone is PointPillars or VoxelNet, and VoxelNet works better.

Weaknesses:

- How does the approach work on other automonous driving datasets? I'm convinced DeepInteraction helps recover tiny objects which are just several pixels in the Lidar points. But are such cases really general in other benchmarks? Or, are there any reasons it cannot be run on other benchmarks?

questions: See weaknesses.
limitations: Limitations are discussed in the paper.
review_title: null

final_score: rating: 1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations
review_content: summary: This work tackles the task of 3D object detection in autonomous driving scenarios. The claimed key contribution is the proper combination of two input modalities: RGB camera images and point clouds from LiDAR scanners (unlike prior methods which treat one of the modalities as auxiliary). The proposed method outperforms prior methods on the nuScenes benchmark. The method is analyzed on the same dataset.
strengths_and_weaknesses: - The paper makes a strong point that the bilateral fusion between camera and lidar modalities is the key part that contributes to the strong performance of the proposed model. From the ablation study alone it is not directly clear if the improvements come from the change from unilateral to bilateral modality information flow or from stronger backbones which seem to have the biggest impact according to Table (e).

- The method is evaluated only on a single dataset. There are multiple other popular 3D object detection benchmarks, e.g., [KITTI](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d).
Evaluation on more datasets would show the generalization capability of the model.

- Important related work on image depth fusion and transformer-based 3D object detection appears to be missing, e.g., [ImVoteNet](https://arxiv.org/pdf/2001.10692.pdf), [3DETR](https://arxiv.org/abs/2109.08141) [Misra et al. ICCV 2021].
questions: (1) Which experiment in the ablation study shows that the bilateral feature fusion is indeed the key component necessary for the improved scores?

(2) How does the model perform on other datasets e.g. KITTI 3D object detection?

**Minor Details**

- Fig.2: inconsistent notation \phi <-> \varphi in (a) and (b).
- l.59 ‘preform’ → ‘perform’
- reference [5] and [6] are the same

limitations: Potential negative societal impact is not discussed (there is a section in the supplementary titled as such but discusses only research impact). Limitations are only discussed on an abstract meta level. Overall, the two sections (E, F) in the supplementary are not adding any real value to the paper and seem to exist only to tick the required boxes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: (1) The paper reveals and gives an analysis of the unilateral limitation of feature aggregation on different modalities for existing methods.
(2) The paper proposes a new framework for bilateral feature interaction and association on different modalities.
(3) The paper achieves state-of-the-art results on nuScenes benchmark.
strengths_and_weaknesses: Strengths:
(1) The paper is well-written and easy to follow, especially for demonstrating its motivation in the introduction session.
(2) The experiments are sufficient and convincing for both comparing with previous methods and ablation studies.

Weaknesses:
(1) The paper lacks the comparison of speed and memory between different methods. For the autonomous driving scenarios whether the model can run in real-time is also important. I'm curious about whether the DETR-based framework can run as fast as, at least not slower too much compared with previous SOTAs.
(2) Although the proposed bilateral system is reasonable, it would be great to show some visualizations of the heat map of the features from the two modalities to help readers get a deeper understanding on how the visual information from the two sources is well selected and utilized. 

questions: Authors are encouraged to provide additional experiments for the two concerns I list in the weakness part:

(1) The comparison of speed and memory.
(2) Some visual results(e.g., heatmap of two sources) to convince the effect of the proposed bilateral framework.
limitations: The authors have presented the potential societal impacts and limitations in their supplementary material. Overall, they look good to me.
review_title: null

Meta-review: metareview: This work was overall positively technically evaluated with some concerns mainly related to limited experimental validation, the need to some additional justifications and explanation, and the missing computational cost analysis.
The provided rebuttal responded sufficiently well to these concerns and the overall evaluation is positive.


confidence: Certain
Decision: Accept