Introduction
Vision-based perception tasks, like detecting bounding boxes in 3D space, have been a critical aspect of fully autonomous driving tasks Yan2018SECONDSE ; Shi2019PointRCNN3O ; Yang20203DSSDP3 ; Shi2020PVRCNNPF . Among all the sensors of a traditional vision on-vehicle perception system, LiDAR and camera are usually the two most critical sensors that provide accurate point cloud and image features of a surrounding world. In the early stage of perception system, people design separate deep models for each sensor Qi2017PointNetDL ; qi2017pointnet++ ; Yin2020Centerbased3O ; huang2021bevdet ; wang2022detr3d , and fusing the information via post-processing approaches pang2020clocs . Note that, people discover that bird’s eye view (BEV) has been an de-facto standard for autonomous driving scenarios as, generally speaking, car cannot fly Lang2019PointPillarsFE ; abs-2203-17270 ; reading2021categorical ; huang2021bevdet ; abs-2204-05088 ; park2021pseudo .
However, it is often difficult to regress 3D bounding boxes on pure image inputs due to the lack of depth information, and similarly, it is difficult to classify objects on point clouds when LiDAR does not receive enough points.

Recently, people have designed LiDAR-camera fusion deep networks to better leverage information from both modalities.
Specifically, the majority of works can be summarized as follow:
i) given one or a few points of the LiDAR point cloud, LiDAR to world transformation matrix and the essential matrix (camera to world); ii) people transform the LiDAR points sindagi2019mvxnet ; vora2020pointpainting ; Wang2021PointAugmentingCA ; vora2020pointpainting ; Huang2020EPNetEP ; Yin2021MVP or proposals Chen2017Multiview3O ; Yoo20203DCVFGJ ; bai2022transfusion ; li2022deepfusion into camera world and use them as queries, to select corresponding image features.
This line of work constitutes the state-of-the-art methods of 3D BEV perception.

However, one underlying assumption that people overlooked is, that as one needs to generate image queries from LiDAR points, the current LiDAR-camera fusion methods intrinsically depend on the raw point cloud of the LiDAR sensor, as shown in Fig. 1 .
In the realistic world, people discover that if the LiDAR sensor input is missing, for example,
LiDAR points reflection rate is low due to object texture, a system glitch of internal data transfer, or even the field of view of the LiDAR sensor cannot reach 360 degrees due to hardware limitations AnonymousBenchmark , current fusion methods fail to produce meaningful results 1 1 1 See AnonymousBenchmark and Sec. 4.4 for more details. . This fundamentally hinders the applicability of this line of work in the realistic autonomous driving system.

We argue the ideal framework for LiDAR-camera fusion should be, that each model for a single modality should not fail regardless of the existence of the other modality, yet having both modalities will further boost the perception accuracy. To this end, we propose a surprisingly simple yet effective framework that disentangles the LiDAR-camera fusion dependency of the current methods, dubbed BEVFusion. Specifically, as in Fig. 1 (c), our framework has two independent streams that encode the raw inputs from the camera and LiDAR sensors into features within the same BEV space. We then design a simple module to fuse these BEV-level features after these two streams, so that the final feature can be passed into modern task prediction head architecture Lang2019PointPillarsFE ; Yin2020Centerbased3O ; bai2022transfusion .

As our framework is a general approach, we can incorporate current single modality BEV models for camera and LiDAR into our framework. We moderately adopt Lift-Splat-Shoot Philion2020LiftSS as our camera stream, which projects multi-view image features to the 3D ego-car coordinate features to generate the camera BEV feature. Similarly, for the LiDAR stream, we select three popular models, two voxel-based ones and a pillar-based one Yin2020Centerbased3O ; bai2022transfusion ; Lang2019PointPillarsFE to encode the LiDAR feature into the BEV space.

On the nuScenes dataset, our simple framework shows great generalization ability. Following the same training settings Lang2019PointPillarsFE ; Yin2020Centerbased3O ; bai2022transfusion , BEVFusion improves PointPillars and CenterPoint by 18.4% and 7.1% in mean average precision (mAP) respectively, and achieves a superior performance of 69.2% mAP comparing to 68.9% mAP of TransFusion bai2022transfusion , which is considered as state-of-the-art. Under the robust setting by randomly dropping the LiDAR points inside object bounding boxes with a probability of 0.5, we propose a novel augmentation technique and show that our framework surpasses all baselines significantly by a margin of 15.7% \sim 28.9% mAP and demonstrate the robustness of our approach.

Our contribution can be summarized as follow: i) we identify an overlooked limitation of current LiDAR-camera fusion methods, which is the dependency of LiDAR inputs; ii) we propose a simple yet novel framework to disentangle LiDAR camera modality into two independent streams that can generalize to multiple modern architectures; iii) we surpass the state-of-the-art fusion methods under both normal and robust settings.