Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method to fuse two source of information for BEV detection namely multi-view images and LIDAR data in such a manner that any data defects in one source of information do not effect the network of the other method. Previous methods have combined the information from the two sources at different stages of the network pipeline, but they are prone to getting effected in the inference results when the data is corrupted from either source. This paper delays the combination of information to even later part of the pipeline thereby
mitigating the effect of bad/corrupted/unavailable data. They do so by generating a pseudo BEV point cloud just from multi-view cameras and combining that information with LIDAR BEV. The combination part is based on a dynamic fusion method which selects important fused features be it from camera based BEV or LIDAR based BEV. The results are shown where missing data in the LIDAR or camera image do not effect the detection unless both are missing for the same object in the scene e.g. car.
strengths_and_weaknesses: Strength:
The paper addresses a problem which could be a real life problem in LIDAR or image based data capture, where the LIDAR data is missing due to 3D scene material/reflectance properties or the image could be missing from a video stream. These problems can cause existing networks to fail. This paper addresses this problem which can make the commercial deployment of such systems more doable. The paper is well written with clear explanation of the previous work. The results are detailed and show scenarios where they are better than previous best results in challenging situations.

Weakness:
1. The dynamic fusion module should be explained in more detail as its one of the contributions of this paper. It should be explained with a scenario where data is missing from either of the streams and how the formulation in Eq.1 and Eq.2 will still be able to select the BEV information which exists to feed the final detection result.
2. Citations 44-45, 32-33, 57-58 are repeated citations. Please fix it.
4. grammar: #92 start->started, #3 discover->discovered
questions: For training the camera stream, a camera based BEV 2D point cloud is created. How is this camera stream trained as this may require a ground truth with camera based BEV 2D points and detections on that?

limitations: The reviewer didn't find any major limitations. The authors could discuss and show some failure cases of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduced a method for point cloud object detection based on LiDAR camera fusion.  The main contribution of this method is the fusion framework that combines the camera and Lidar stream. This fusion module is very simple because it mainly consists of the concatenation of LiDAR and camera streams and a typical feature selection with an average pooling and 1x1 convolution. The results show that this method slightly outperformed the other method for the comparison. Moreover, the robustness against camera or LiDAR malfunctions is shown in the results. The ablation study shows that each module employed in this method improves performance. 

strengths_and_weaknesses: Strength
- The performance is slightly improved. 
- The methodology is very simple. 

Weakness
- Considering the small performance improvement and the simple methodology, I would think that the contribution of this method is relatively limited. 
questions: I am wondering why this simple fusion method is better than the other methods compare in this paper. If there are some results and discussions that make sense would be helpful for the readers. 
limitations: I would suggest showing some failure cases and discussions about them because it will contribute to the community.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a framework for 3D detection from RGB and LiDAR inputs in autonomous driving scenes. The pipeline includes separate networks reasoning from RGB and LiDAR inputs independently, and uses a fusion network for refined detection when both sources are available. Also the paper considers situations of data corruption and proposed to boost the robustness in the model design. The paper is the first to identify and evaluate the problem that most existing methods do not consider situations where one or both sources are unavailable, and proposes a pipeline customized for this situation. The proposed method is evaluated in the standard settings of object detection and compared with baseline methods both qualitatively and quantitatively.
strengths_and_weaknesses: Strength:

[1] The task identification. As mentioned above, the paper is the first to identify the issue within the current literature and models, and proposes a pipeline accordingly which reasons from two sources independently and thus more robust when data unavailability occurs. In this sense, the task identification itself is valuable to the community in defining and bringing attention to the task.

[2] Extensive design choices and evaluation. Although the proposed pipeline is mostly based on existing methods, the paper is able to evaluate various design choices to demonstrate the flexibility of the proposed framework, as well as provide extensive evaluation into the results, yields SOTA results with both sources, and robust result when only one is available.

Weakness:

[1] Novelty and model design. The paper is novel in identifying the problem, which is legit and valuable. However for the proposed method itself, it is mostly a combination of existing methods utilization single sources without much modification, thus diminishing the merit of the proposed framework. Also the design to handle one or two sources in the framework is naive, basically running the first stage network only if only one source is available, and running both stages when two are available. A more sophisticated design could be, when for example camera stream is dropped for a few frames, is there a chance to stick to the fusion detector, but utilizing temporal information to compensate the missing RGB data, instead of simply drop the RGB branch and the fusion, running the LiDAR branch alone, which will likely result in a sudden drastic change to the detections?

[2] Simulation for data corruption. The paper proposes to augment the data to simulate possible data corruption scenarios, via dropping points and limiting FOV. However more effort can be done to boost the robustness: e.g. looking for real driving sequences in extreme weather or with bad data, and train/evaluate on those data.
questions: Please see the points in the Weakness section above.
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Most existing camera-lidar fusion work decorates lidar points with image features and then performs detection in 3D/BEV space. This work leverages recent Lift-Splat-Shoot work for cameras, which allows one to map both camera and lidar inputs to BEV space, before fusing and applying the detection head. 
strengths_and_weaknesses: Strengths: 
- The proposed idea and its realization makes sense, and I am not aware of such published work (even though there seems to be concurrent similar work, since this seems a logical next step given the existence of LSS [52]). 
- Details in the model seem well thought out. This include the extensions to LSS (Dual-Swin-Tiny architecture, ADP), as well as the layers in the dynamic fusion module. 
- The experimental results show that this work is close to SOTA on nuScenes and that it affords significant model robustness in the case of lidar information missing compared to existing methods. 
- The model details are pretty clearly explained. 

Weaknesses: 
- Related work section is confusing in a few places and can be streamlined further. Examples: 
1) The Camera detectors section contains a discussion of PointPillars, which is a purely Lidar method. 
2) Range images are not really Euclidean space (see line 88) 
3) 89: "Recently, people start to exploit these two feature modalities to increase the representation power" --> There is earlier work to do this, if I understand correctly the statement. E.g. [5] from the paper, or End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds, by Yin Zhou et al, CoRL 2019. 
4) 90: " Another line of work is to exploit the benefit of the bird’s eye view plane similar to the camera perception" --> a lot of this work came before camera started exploiting the BEV view. 

- Intuitive explanations are lacking in a couple of instances: 
1) Work does not explain the intuition why the model needs to be trained in two stages. What happens if it's trained in a single stage? 
2) 14: "Note that we do not conduct data augmentation when multi-view image input is involved, while data augmentation plays a critical part in other cutting edge methods." Is this a limitation of camera fusion methods in general or something specifically lacking in your case? Can you please clarify? 

- It is unclear whether the approach is SOTA on nuScenes or not. Can you please explicitly contrast your performance relative to the nuScenes leaderboard (at least for the published approaches). When exploring that leaderboard myself, I see mentions of a method called BEVFusion that is SOTA but seems to be a different method? Assuming that method is different and already on the leaderboard, your naming may be confusing / too generic. 

- nuScenes is a dataset with particularly poor lidar (compared to other public datasets, such as Waymo Open Dataset, Argoverse2.0 etc). Results on at least one more dataset with high quality and longer-range lidar are highly desirable. The core issue of missing lidar points may be a lot less pertinent for more modern lidars. Also, as range increases beyond ~40m to 70-200m, the approach here may actually underperform lidar-painting approaches, since BEV view can start containing errors > 10m in the camera case making fusion in BEV space difficult. To this effect, analysis of the method performance as a function of object distance, relative to SOTA fusion methods for long distances will help. 


Language: 
There are minor language issues and typos in the paper, it would benefit from another proofreading pass. 

questions: - Is the approach SOTA on nuscenes or not?  Can you please explicitly contrast your performance relative to the nuScenes leaderboard (at least for the published approaches). 

- Can you provide an analysis of performance as a function of distance to object, and compare to a standard lidar approach and TransFusion/DeepFusion etc? 

- Can you please provide results on at least one more dataset with high quality Lidar such as the Waymo Open Dataset? 

- What is the latency of the approach and how does it compare to the baselines? 
limitations: See comment on weaknesses. Some core potential limitations of the existing method have not been fully explored. 

My current rating is predicated on the assumption that a similar idea has not been published yet (not completely certain) and that I will receive reasonable responses to my questions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Towards the problem of current methods tend to fail at situations where hardware malfunctions, this paper presents a simple yet effective LiDAR-Camera fusion framework, namely BEVFusion. By disentangling camera pipeline from LiDAR network and using a dynamic fusion module, BEVFusion achieves SOTA performance and shows robustness against LiDAR or camera malfunction at the same time. An effective modification on the camera pipeline is also proposed to boost the final performance.
strengths_and_weaknesses: ## Strength
1. The paper is well written and easy to read.
2. Robustness of autonomous driving algorithms should be paid more attention to. This paper raises the issue and makes the attempt to addressing it.
3. Thorough experiments are performed. Claims are well-supported. SOTA performance is achieved on both normal and robust settings of nuScenes.
4. The clean design of the framework makes it easy to use any camera or LiDAR framework.


## Weaknesses
1. It is nice to see a simple yet effective module (dynamic fusion module) being proposed. But it would be nicer to provide some insights and analysis into the design itself. For example, by analyzing how would the fusion module work when facing incomplete LiDAR or camera inputs, we might gain some insights into the module design of CSF and AFS.
2. The experiments section does not provide runtime analysis, like inference time and memory footprint, and its comparison with other methods.
questions: 1. How does the baseline method in Table 7 fuse the features?
2. This does not affect my rating to the paper. Just out of curious, would BEVFusion still works when facing both camera and LiDAR malfunction?
limitations: The potential negative social impact is well discussed. But the limitation should be discussed more. For example, would the late-fusion style misses the opportunity to fuse intermediate LiDAR and camera features, and thus makes the pipeline suffer potential performance drop?
review_title: null

Meta-review: metareview: The paper proposes a method to fuse two sources of information for Bird’s Eye View (BEV) detection, namely multi-view images and LIDAR data, in a way that any data defects in one source of information does not affect the other. Most existing camera-lidar fusion works decorate lidar points with image features and then perform detection in 3D/BEV space. This work leverages recent Lift-Splat-Shoot work for cameras, which allows one to map both camera and lidar inputs to BEV space, before fusing and applying the detection head. The reviewers appreciate the identification of the problem of present fusion methods that are susceptible to damage in one of the two sources of information, the simplicity of the method and its good empirical performance. They raise concerns regarding its novelty, given  the obvious choices of the present method. The rebuttal submitted by the authors presents more empirical results and ablations. Most reviewers appreciate the contribution of the paper, and  the paper is suggested for publication. 
confidence: Certain
Decision: Accept