RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
BEVFusion: A General Framework for LiDAR-Camera Fusion
As shown in Fig. 2 , we present our proposed framework, BEVFusion, for the 3D object detection in detail. As our fundamental contribution is disentangling the camera network from the LiDAR features,
we first introduce the detailed architecture of the camera and LiDAR stream, then present a dynamic fusion module to incorporate features from these modalities.
LiDAR stream architecture: From point clouds to BEV space
Similarly, our framework can incorporate any network that transforms LiDAR points into BEV features, \mathbf{F_{\text{LiDAR}}}\in R^{X\times Y\times C_{\text{LiDAR}}} , as our LiDAR streams. A common approach is to learn a parameterized voxelization Zhou2018VoxelNetEL of the raw points to reduce the Z-dimension and then leverage networks consisting of sparse 3D convolution Yan2018SECONDSE to efficiently produce the feature in the BEV space.
In practice, we adopt three popular methods, PointPillars Lang2019PointPillarsFE , CenterPoint Yin2020Centerbased3O and TransFusion bai2022transfusion as our LiDAR stream to showcase the generalization ability of our framework.
Dynamic fusion module
To effectively fuse the BEV features from both camera ( \mathbf{F}_{\text{Camera}}\in R^{X\times Y\times C_{\text{Camera}}} ) and LiDAR ( \mathbf{F_{\text{LiDAR}}}\in R^{X\times Y\times C_{\text{LiDAR}}} ) sensors, we propose a dynamic fusion module in Fig. 3 .
Given two features under the same space dimension, an intuitive idea is to concatenate them and fuse them with learnable static weights. Inspired by Squeeze-and-Excitation mechanism HuSS18 , we apply a simple channel attention module to select important fused features. Our fusion module can be formulated as:

\mathbf{F}_{fused}=f_{adaptive}(f_{static}(\left[\mathbf{F}_{Camera},\mathbf{F_{LiDAR}}\right])), (1)

where \left[\cdot,\cdot\right] denotes the concatenation operation along the channel dimension. f_{\text{static}} is a static channel and spatial fusion function implemented by a 3\times 3 convolution layer to reduce the channel dimension of concatenated feature into C_{\text{LiDAR}} . With input feature \mathbf{F}\in R^{X\times Y\times C_{\text{LiDAR}}} , f_{\text{adaptive}} is formulated as:

f_{adaptive}(\mathbf{F})=\sigma\left(\mathbf{W}f_{avg}(\mathbf{F})\right)\cdot\mathbf{F}, (2)

where \mathbf{W} denotes linear transform matrix (e.g., 1x1 convolution), f_{\text{avg}} denotes the global average pooling and \sigma denotes sigmoid function.
Detection head
As the final feature of our framework is in BEV space, we can leverage the popular detection head modules from earlier works. This is further evidence of the generalization ability of our framework. In essence, we compare our framework on top of three popular detection head categories,
anchor-based Lang2019PointPillarsFE , anchor-free-based Yin2020Centerbased3O , and transform-based bai2022transfusion .
Experiments
In this section, we present our experimental settings and the performance of BEVFusion to demonstrate the effectiveness, strong generalization ability, and robustness of the proposed framework.
Experimental settings
Dataset. We conduct comprehensive experiments on a large-scale autonomous-driving dataset for 3D detection, nuScenes Caesar2020nuScenesAM .
Each frame contains six cameras with surrounding views and one point cloud from LiDAR. There are up to 1.4 million annotated 3D bounding boxes for 10 classes. We use nuScenes detection score (NDS) and mean average precision (mAP) as evaluation metrics. See Appendix B.1 for more details.

Implementation details. We implement our network in PyTorch using the open-sourced MMDetection3D mmdet3d .
We conduct BEVFusion with Dual-Swin-Tiny liang2021cbnetv2 as 2D bakbone for image-view encoder. PointPillars Lang2019PointPillarsFE , CenterPoint Yin2020Centerbased3O , and TransFusion-L bai2022transfusion are chosen as our LiDAR stream and 3D detection head.
We set the image size to 448 \times 800 and the voxel size following the official settings of the LiDAR stream Lang2019PointPillarsFE ; Yin2020Centerbased3O ; bai2022transfusion .
Our training consists of two stages: i) We first train the LiDAR stream and camera stream with multi-view image input and LiDAR point clouds input, respectively. Specifically, we train both streams following their LiDAR official settings in MMDetection3D mmdet3d ;
ii) We then train BEVFusion for another 9 epochs that inherit weights from two trained streams.
Note that no data augmentation (i.e., flipping, rotation, or CBGS Zhu2019ClassbalancedGA ) is applied when multi-view image input is involved.
In this version of paper, we additionally train the fusion phase with BEV-space data augmentation following huang2021bevdet ; liu2022bevfusion when comparing with the state-of-the-art methods.
In particular, we add the BEV-space augmentation implemented by liu2022bevfusion 2 2 2 https://github.com/mit-han-lab/bevfusion for a better result.
During testing, we follow the settings of LiDAR-only detectors Lang2019PointPillarsFE ; Yin2020Centerbased3O ; bai2022transfusion in MMDetection3D mmdet3d without any extra post-processing. See Appendix  Sec. B.1 for the detailed hyper-parameters and settings.
Generalization ability
To demonstrate the generalization ability of our framework, we adapt three popular LiDAR-only detector as our LiDAR stream and detection head, PointPillars Lang2019PointPillarsFE , CenterPoint Yin2020Centerbased3O and TransFusion-L bai2022transfusion , as described in Sec. 3 .
If not specified, all experimental settings follow their original papers.
In Table 1 , we present the results of training two single modality streams, followed by jointly optimized.
Empirical results show that our BEVFusion framework can significantly boost the performance of these LiDAR-only methods. Despite the limited performance of the camera stream, our fusion scheme improves PointPillars by 18.4% mAP and 10.6% NDS, and CenterPoint and TransFusion-L by a margin of 3.0% \sim 7.1% mAP. This evidences that our framework can generalize to multiple LiDAR backbone networks.

As our method relies on a two-stage training scheme, we nonetheless report the performance of a single stream in the bottom part of Table 1 . We observe that the LiDAR stream constantly surpasses the camera stream by a significant margin. We ascribe this to the LiDAR point clouds providing robust local features about object boundaries and surface normal directions, which are essential for accurate bounding box prediction.
Comparing with the state-of-the-art methods
Here, we use TransFusion-L as our LiDAR stream and present the results on the test set of nuScenes in Table 2 .
Without any test time augmentation or model ensemble, our BEVFusion surpasses all previous LiDAR-camera fusion methods and achieves the state-of-the-art performance of 69.2% mAP comparing to the 68.9% mAP of TransFusion bai2022transfusion .
Note that we do not conduct data augmentation when multi-view image input is involved, while data augmentation plays a critical part in other cutting-edge methods.
It is worth noticing that the original TransFusion bai2022transfusion is a two-stage detector, whose model consists of two independent detection heads. By contrast, our BEVFusion with TransFusion-L as LiDAR backbone only contains one detection head, yet still outperforms the two-stage baseline by 0.3% mAP.
As the only difference between our framework and TransFusion is the fusion mechanism, we ascribe this performance gain to a comprehensive exploration of the multi-modality modeling power of BEVFusion.
Robustness experiments
Here, we demonstrate the robustness of our method against all previous baseline methods on two settings, LiDAR and camera malfunctioning. See AnonymousBenchmark for more details.
Robustness against camera malfunctions
We further validate the robustness of our framework against camera malfunctions under three scenarios in AnonymousBenchmark : i) front camera is missing while others are preserved; ii) all cameras are missing except for the front camera; iii) 50% of the camera frames are stuck. As shown in Table 5 , BEVFusion still outperforms camera-only wang2022detr3d and other LiDAR-camera fusion methods Wang2021PointAugmentingCA ; sindagi2019mvxnet ; bai2022transfusion under above scenarios. The results demonstrate the robustness of BEVFusion against camera malfunctions.
Ablation
Here, we ablate our design choice of the camera stream and the dynamic fusion module.

Components for camera stream. We conduct ablation experiments to validate the contribution of each component of our camera stream using different components in Table 7 . The naive baseline with ResNet50 and feature pyramid network as multi-view image encoder, ResNet18 as BEV encoder following LSS Philion2020LiftSS , PointPillars Lang2019PointPillarsFE as detection head only obtains 13.9% mAP and 24.5% NDS. As shown in Table 7 , there are several observations: (i) when we replace the ResNet18 BEV encoder with our simple BEV encoder module, the mAP and NDS are improved by 4.0% and 2.5%. (2) Adding the adaptive feature alignment module in FPN helps improve the detection results by 0.1%. (3) Concerning a larger 2D backbone, i.e., Dual-Swin-Tiny, the gains are 4.9% mAP and 4.0% NDS. The camera stream equipped with PointPillars finally achieves 22.9% mAP and 31.1% NDS, showing the effectiveness of our design for the camera stream.

Dynamic fusion module. To illustrate the performance of our fusing strategy for two modalities, we conduct ablation experiments on three different 3D detectors, PointPillars, CenterPoint, and TransFusion and the LiDAR stream as baseline. As shown in 7 , with a simple channel& spatial fusion (left part in Fig. 3 ), BEVFusion greately improves its LiDAR stream by 16.5% ( 35.1\%\rightarrow 51.6\% ) mAP for PointPillars, 5.9% ( 57.1\%\rightarrow 63.0\% ) mAP for CenterPoint, and 2.4% ( 64.9\%\rightarrow 67.3\% ) mAP for TransFusion. When adaptive feature selection (right part in Fig. 3 ) is adopted, the mAP can be futher improved by 1.9%, 1.2%, and 0.6% mAP for PointPillars, CenterPoint, and TransFusion, respectively. The results demonstrate the necessity of fusing camera and LiDAR BEV features and effectiveness of our Dynamic Fusion Module in selecting important fused features.
Conclusion
In this paper, we introduce BEVFusion, a surprisingly simple yet unique LiDAR-camera fusion framework that disentangles the LiDAR-camera fusion dependency of previous methods. Our framework comprises two separate streams that encode raw camera and LiDAR sensor inputs into features in the same BEV space, followed by a simple module to fuse these features such that they can be passed into modern task prediction head architectures. The extensive experiments demonstrate the strong robustness and generalization ability of our framework against the various camera and LiDAR malfunctions. We hope that our work will inspire further investigation of robust multi-modality fusion for the autonomous driving task.
Broader Impacts Statement and Limitations
This paper studies robust LiDAR-camera fusion for 3D object detection.
Since the detection explored in this paper is for generic objects and does not pertain to specific human recognition, so we do not see potential privacy-related issues.
However, the biased-to-training-data model may pose safety threats when applied in practice.
The research may inspire follow-up studies or extensions, with potential applications in autonomous driving tasks. While our study adopts a simple camera stream as the baseline, we also encourage the community to expand the architecture, e.g, with temporal multi-view camera input and alignment between intermediate LiDAR and camera features . We leave the extension of our method towards building such systems for future work.
Acknowledgment
We thank the anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.
B.1 Implementation details.
Our training consists of two stages: i) We first train the LiDAR stream and camera stream with multi-view image input and LiDAR point clouds input, respectively. Specifically, we train both streams following their LiDAR official settings in MMDetection3D [ 8 ] . Data augmentation strategies, e.g., random flips along the X and Y axes, global scaling and global rotation as data augmentation strategies are conducted for LiDAR streams. CBGS [ 67 ] is used for class-balanced sampling for CenterPoint and TransFusion. The camera stream inherit weights of backbone and neck from nuimage pre-trained Mask R-CNN Dual-Swin-Tiny.
ii) We then train BEVFusion that inherit weights from two trained streams.
During fusion training, we set the learning rate to 1e^{-3} with a batch size of 32 on 8 V100 GPUs. For PointPillars, we train the detector for 12 epochs and reduce learning rate by 10\times at epoch 8 and 11. For CenterPoint and TransFusion, we train detectors for 6 epochs and reduce learning rate by 10\times at epoch 4 and 5, and we freeze the weights of the LiDAR stream for CenterPoint and TransFusion.
Note that no data augmentation (i.e., flipping, rotation, or CBGS [ 67 ] ) is applied when multi-view image input is involved. We only conduct BEV-space data augmentation when comparing with the state-of-the-art methods.
When we finetune LiDAR-camera fusion detectors on the augmented training set, we train detectors for 12 epochs where the initial learning rate is set to 1e^{-4} and reduced by 10\times at epoch 8 and 11 with a batch size of 32.
C.1 Robustness of Dynamic Fusion Module
To better show the effectiveness of each part of the Dynamic Fusion Module, we test BEVFusion equipped with TransFusion-L and show the result under robustness settings against LiDAR and camera malfunctions in
Sec. 4.4.1 and Sec. 4.4.2 .
We show the result in Table 8 . We can see that when LiDAR fails to receive points from the object, with a simple channel& spatial fusion (CSF), BEVFusion greatly improves its LiDAR stream by 15.5% mAP. When adaptive feature selection (AFS) is adopted, the mAP can be further improved by 0.2%. Under camera missing scenarios, AFS improves CSF-only by 0.5-1.6% mAP. The results show that our Dynamic Fusion Module is still able to select the BEV information which exists to feed the final detection result under input malfunctions.
C.2 Robustness under both modality malfunctions
We evaluate our BEVFusion equipped with TransFusion-L under the ’LiDAR fails to receive object reflection points’ (as in Table 4 ), ’missing front camera’, and ’preserving front camera’ (as in Table 5 ) malfunctions and rereport the result in Table 9 . The results show that BEVFusion remains effective in the face of a certain degree of two-modality malfunction. However, conceptually speaking, if one object is never captured by the camera or LiDAR, our framework will not be able to identify the object.
C.3 Robustness against Inferior Image Conditions
We demonstrate the robustness of our proposed framework against inferior image conditions on nuScenesvalidation set. We report the results of BEVFusion equipped with TransFusion-L under different lighting conditions in Table 10 . Compared with CenterPoint and TransFusion, BEVFusion shows the best robustness under different lighting conditions.
Figure 1: Comparison of our framework with previous LiDAR-camera fusion methods. Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism sindagi2019mvxnet ; vora2020pointpainting ; Wang2021PointAugmentingCA ; vora2020pointpainting ; Huang2020EPNetEP ; Yin2021MVP that project image features onto raw point clouds, and (b) feature-level fusion mechanism Chen2017Multiview3O ; Yoo20203DCVFGJ ; bai2022transfusion ; li2022deepfusion that projects LiDAR feature or proposals on each view image separately to extract RGB information.
(c) In contrast, we propose a novel yet surprisingly simple framework that disentangles the camera network from LiDAR inputs.; Comparison of our framework with previous LiDAR-camera fusion methods. Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism sindagi2019mvxnet ; vora2020pointpainting ; Wang2021PointAugmentingCA ; vora2020pointpainting ; Huang2020EPNetEP ; Yin2021MVP that project image features onto raw point clouds, and (b) feature-level fusion mechanism Chen2017Multiview3O ; Yoo20203DCVFGJ ; bai2022transfusion ; li2022deepfusion that projects LiDAR feature or proposals on each view image separately to extract RGB information.
(c) In contrast, we propose a novel yet surprisingly simple framework that disentangles the camera network from LiDAR inputs.
Figure 2: An overview of BEVFusion framework. With point clouds and multi-view image inputs, two streams separately extract features and transform them into the same BEV space: i) the camera-view features are projected to the 3D ego-car coordinate features to generate camera BEV feature; ii) 3D backbone extracts LiDAR BEV features from point clouds. Then, a fusion module integrates the BEV features from two modalities. Finally, a task-specific head is built upon the fused BEV feature and predicts the target values of 3D objects. In detection result figures, blue boxes are predicted bounding boxes, while red circled ones are the false positive predictions.; An overview of BEVFusion framework. With point clouds and multi-view image inputs, two streams separately extract features and transform them into the same BEV space: i) the camera-view features are projected to the 3D ego-car coordinate features to generate camera BEV feature; ii) 3D backbone extracts LiDAR BEV features from point clouds. Then, a fusion module integrates the BEV features from two modalities. Finally, a task-specific head is built upon the fused BEV feature and predicts the target values of 3D objects. In detection result figures, blue boxes are predicted bounding boxes, while red circled ones are the false positive predictions.
Figure 3: Dynamic Fusion Module.; Dynamic Fusion Module.
Figure 4: Visualization of predictions under robustness setting. (a) We visualize the point clouds under the BEV perspective of two settings, limited field-of-view (FOV) and LiDAR fails to receive object reflection points, where the orange box indicates the object points are dropped.
Blue boxes are bounding boxes and red-circled boxes are false-positive predictions.
(b) We show the predictions of the state-of-the-art method, TransFusion, and ours under three settings. Obviously,
the current fusion approaches fail inevitably when the LiDAR input is missing, while our framework can leverage the camera stream to recover these objects.; Visualization of predictions under robustness setting. (a) We visualize the point clouds under the BEV perspective of two settings, limited field-of-view (FOV) and LiDAR fails to receive object reflection points, where the orange box indicates the object points are dropped.
Blue boxes are bounding boxes and red-circled boxes are false-positive predictions.
(b) We show the predictions of the state-of-the-art method, TransFusion, and ours under three settings. Obviously,
the current fusion approaches fail inevitably when the LiDAR input is missing, while our framework can leverage the camera stream to recover these objects.
Figure 5: Adaptive Module in FPN.; Adaptive Module in FPN.
Figure 6: Visualization of Failure cases of BEVFusion equipped with PointPillars as LiDAR stream.; Visualization of Failure cases of BEVFusion equipped with PointPillars as LiDAR stream.
Generalization ability of BEVFusion. We validate the effectiveness of our fusion framework on nuScenes validation set, compared to single modality streams over three popular methods Lang2019PointPillarsFE ; Yin2020Centerbased3O ; bai2022transfusion . Note that each method here defines the structure of the LiDAR stream and associated detection head while the camera stream remains the same as in Sec. 3.1 .
Results on the nuScenes validation (top) and test (bottom) set.
Results on robustness setting of limited LiDAR field-of-view. Our method significantly boosts the performance of LiDAR-only methods over all settings. Note that, compared to the TransFusion with camera fusion, our method still achieves over 15.3% mAP and 6.6% NDS improvement, showcasing the robustness of our approach.
Results on robustness setting of object failure cases. Here, we report the results of baseline and our method that trained on the nuScenes dataset with and without the proposed robustness augmentation (Aug.). All settings are the same as in Table 3 .
Results on robustness setting of camera failure cases. F denotes front camera.
Ablating the camera stream.
Ablating the Dynamic fusion module in LiDAR and camera malfunctions.
Results when facing both camera and LiDAR malfunction.
Results on robustness setting under different lighting conditions.
Results on the distance between object center and ego vehicle in meters.
Distant regions performance comparison with TransFusion.