Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new multimodal architecture that receives images and a three-dimensional point cloud and detects 3D objects. The new architecture is based on fusing feature vectors both extracted from voxel spaces and feeding a transformer decoder with an MLP classifier. Besides being able to combine image and point cloud, experiments showed that the proposed strategy can also work in the presence of noise in the point cloud or lack of image data. Furthermore, the experimental results indicate that the proposed architecture is superior to several baselines in terms of NDS and mAP metrics.
strengths_and_weaknesses: - Strengths: 
    - While every single contribution taken separately seem limited, the integration of all idea appears to be an interesting contribution to field object detection. As far as robust multimodality fusion is concerned, the paper has shown a coherent and wise approach to provide good performance in the presence of noise and lack of data.

    - The experiments are done thoroughly, and the discussion section clearly interprets the observations from the results.

- Weakness:
    - Many important details for design choices and experiments are either missing or not properly addressed. These points are summarized and listed in the “Questions”.
 
    - Although the experimental results showed that the proposed approach is superior to the baselines, in several cases (multimodal approaches), the superiority is marginal, e.g., 71.1 versus 70.0, 67.1 versus 66.8. 
 
    - Many technical components (particularly the simple ones) are adopted without strong motivation.
questions: 1.  It is unclear how the “sample with prob” (Figure 3) is being done. According to the text, it seems that the features are being weighted by the occupancy grid store in the tensor D. Why is this a sample and with probability?
2.  Why is FPN being used? Why partial L2 distance? Why apply deformable attention and not a vanilla attention layer? Is there any property in the task that suggests those are the better choice?
3.  The voxel encoder is shallow. Please provide more details and motivation related to the design choices.
4.  The training is not clear. Is the network training in an end-to-end manner? Are the weights of the image and voxel backbone frozen? 
5.  Regarding the knowledge transfer and the training, it is not clear whether there is stage-wise training being applied or all the losses are used to train the networks in an end-to-end manner. Please kindly consider improving the description of the network training.
6.  Why is not used a linear layer or an MLP to learn how the weigh the features from the modalities before combining them?
limitations: Yes. The limitations were adequately addressed in the Discussion and Conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on 3D object detection in driving scenarios. It proposes to fuse image features with LiDAR point cloud features following a late-fusion pipeline. The fusion leverages a learned unified 3D voxel feature space. Moreover, it adopts a transformer decoder for object relationships	modeling. A feature mimicking based method is also proposed for knowledge transfer between two input modalities. This paper demonstrates 3D detection performance improvement over single-modality and other joint-modality competing methods on the nuScenes benchmark.
strengths_and_weaknesses: S-1) Image and LiDAR point cloud fusion is a challenging research problem with high real-world application values.

S-2) The cross-modality interaction is simple yet effective. It could be an easily applied method for knowledge transfer between different sensors. 

S-3) It is a good choice of using the transformer decoder as detection head because it can capture object-level relationships. Currently, most 3D object detection heads are based on center heat-map estimation, which lacks the ability to model interactions among different objects.

S-4) For the multi-modality fusion model training, it is a good strategy to first separately pre-train the branch of each modality and then conduct fine-tuning on the fusion model. Such a staged training pipeline can improve the training robustness and this is enabled by the proposed unified voxel feature space.

S-5) The ablation studies are extensive, which demonstrate the effectiveness of each proposed module and network architecture design. For example, Table 1 justifies the necessity of using 3D voxels instead of 2D pillars when encoding lifted image features.

S-6) Figure 5 is inspiring. It clearly demonstrates the robustness advantages of using multi-modality inputs.

W-1) In Eq. (1), how is the model trained to estimate the depth distribution $D_{I}$? For example, what losses are used? Moreover, it’s non-trivial to acquire dense depth annotation for each camera view.

W-2) For camera branch, the voxel resolution is $128 \times 128$ but $D$ is only set to 64. Thus, how do you lift camera features into the voxel space when determining the occupancy probability for voxel centers? Do you do some type of interpolation along the estimated occupancy rays?

W-3) For Table 2, how is 2D convolution used as encoders in a 3D voxel space?

W-4) In Table 5, for camera student and multi-modality teacher, where are the target/teacher features extracted from? Are they from the camera branch or the LiDAR branch, or a mixture of both?

W-5) Also in Table 5, for the best performing multi-modality fusion model, how is knowledge transfer used in its training recipe? It’s not quite clear how does knowledge transfer benefit the fused model. For example do you first train the LiDAR model, and then distill and train the camera model, which followed by the joint fine-tuning?

W-6) Does the transformer decoder jointly detect different types of objects (e.g. pedestrians, vehicles)?
questions: Please see the bullets mentioned in the weakness and limitation sections.
limitations: L-1) I’m wondering if the authors have also done experiments on comparing deformable transformer decoder with the widely adopted CenterPoint [1] detection head. This can help justify the importance and advantages of modeling object-level interactions.

L-2) Can the authors provide comparisons on the model inference speed and computation cost? We all know that latency control is critical for 3D object detection in autonomous driving scenarios.

L-3) Can the authors also provide results on other more widely adopted benchmarks such as Waymo Open Dataset and KITTI Dataset?

[1] Yin, Tianwei, Xingyi Zhou, and Philipp Krahenbuhl. "Center-based 3d object detection and tracking." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a 3D detection framework by projecting various sensor data, which are acquired at the same time, onto a voxel feature space. They first point out the limitation of conventional methods where input- or feature-level data are represented as 2D data with reduced dimensions (range image, predicted depth map, etc.). To overcome these types of issues, they introduce a unified voxel space without loss of spatial information. Additionally, the authors make the neural network more effective in extracting sensor fusion features by applying for knowledge transfer / Data augmentation without the complex aligning process.
strengths_and_weaknesses: (+) They improved the performance of the 3D detector by devising a sensor fusion method on the voxel representation that is richer than the bird-eye's view representation. Through sections 3 and 4 and the supplementary, the authors provide sufficient information to reconstruct and train the proposed network.

(-) The relationship between modality transfer learning techniques and cross-modality interaction feels awkward. As mentioned by the authors, Image and Lidar points are already complementary to each other, so just using the two features should improve performance. However, the knowledge transfer method seems to improve performance by making the features extracted from images similar to the features extracted from the lidar sensor data.

(-) The authors need to conduct performance evaluation in various environments to reveal the advantages of sensor fusion well.

questions: I think the authors' cross-modality interaction module should be validated in a more extreme environment. Rather than simply comparing whether features extracted from individual sensors are used, it is necessary to test results in an environment where one of the two sensors does not work at all. (For example, experiments on whether detection is possible only with camera data at a long distance where lidar sensor data does not exist, and experiments on cases where the camera does not work well at night or when passing through a tunnel entrance)

Introduction of deformable attention for cross attention: Unlike the image space, which contains various scales, the size of the object is determined in the actual 3D space. Therefore, it is not necessary to obtain information of various scales using deformable convolution. Please explain the reason for using deformable attention in unified voxel space. Additionally, can the authors attach a comparison with a network using local attention (basic conv.)?

limitations: The authors mentioned the shortcomings of their detector and the direction to be taken in the future. The disappointing point is that they restate known limitations, where voxel-based methods have a large increase in computation voxel-based methods have a large increase in the amount of computation compared to other representation methods (e.g., range view / bird’s eye view), in this field.
review_title: null

Meta-review: metareview: The paper proposes a multimodal system for 3d object detection and 3 expert reviewers vote for its acceptance, after rebuttal, based on their appreciation of the good improvements brought by multimodality, and due various interesting details of the system.

I agree with reviewer Bb8v that the writing should be polished, starting from the abstract, e.g. "Benefit from the unified manner, cross-modality interaction is then proposed to make full use of inherent properties from different sensors" -- this sentence reads very poorly.
confidence: Less certain
Decision: Accept