Related Works
Here, we categorize the 3D detection methods broadly based on their input modality.

Camera-only. In the autonomous driving domain, detecting 3D objects with only camera-input has been heavily investigated in recent years thanks to the KITTI benchmark Geiger2012AreWR . Since there is only one front camera in KITTI, most of the methods have been developed to address monocular 3D detection lu2021geometry ; Roddick2019OrthographicFT ; liu2021autoshape ; kumar2021groomed ; zhang2021objects ; zhou2021monocular ; reading2021categorical ; wang2021progressive ; wang2021depth .
With the development of autonomous driving datasets that have more sensors, like nuScenes Caesar2020nuScenesAM and Waymo Sun2020ScalabilityIP , there exists a trend of developing methods wang2022probabilistic ; wang2021fcos3d ; wang2022detr3d that take multi-view images as input and found to be significantly superior to monocular methods. However, voxel processing is often accompanied by high computation.

As in common autonomous driving datasets, Lift-Splat-Shoot (LSS) Philion2020LiftSS uses depth estimation network to extract the implied depth information of multi-perspective images and transform camera feature maps into 3D Ego-car coordinate. Methods reading2021categorical ; huang2021bevdet ; abs-2204-05088 are also inspired by LSS Philion2020LiftSS and refer to the LiDAR for the supervision on depth prediction. A similar idea can also be found in BEVDet huang2021bevdet ; huang2022bevdet4d , the state-of-the-art method in multi-view 3D object detection.
MonoDistill chong2022monodistill and LiGA Stereo guo2021liga improve performance by unify LiDAR information to a camera branch.

LiDAR-only. LiDAR methods initially lie in two categories based on their feature modality: i) point-based methods that directly operate on the raw LiDAR point clouds qi2017pointnet++ ; Qi2017PointNetDL ; qi2018frustumPF ; Shi2019PointRCNN3O ; Yang20203DSSDP3 ; li2021lidar ; and ii) transforming the original point clouds into a Euclidean feature space, such as 3D voxels Zhou2018VoxelNetEL and feature pillar Lang2019PointPillarsFE ; Wang2020PillarbasedOD ; Yin2020Centerbased3O .
Recently, people started to exploit these two feature modalities in a single model to increase the representation power Chen2017Multiview3O ; Zhou2019EndtoEndMF ; Shi2020PVRCNNPF . Another line of work is to exploit the benefit of the bird’s eye view plane Lang2019PointPillarsFE ; fan2021rangedet ; Sun2021RSNRS .

LiDAR-camera fusion. As the features produced by LiDAR and camera contain complementary information in general, people started to develop methods that can be jointly optimized on both modalities and soon become a de-facto standard in 3D detection. As in Fig. 1 , these methods can be divided into two categories depending on their fusion mechanism, (a) point-level fusion where one queries the image features via the raw LiDAR points and then concatenates them back as additional point features Huang2020EPNetEP ; sindagi2019mvxnet ; Wang2021PointAugmentingCA ; Yin2021MVP ; (b) feature-level fusion where one firstly projects the LiDAR points into a feature space Yoo20203DCVFGJ or generates proposals bai2022transfusion , queries the associated camera features then concatenates back to the feature space Liang2018DeepCF ; focalsconv-chen . The latter constitutes the state-of-the-art methods in 3D detection, specifically, TransFusion bai2022transfusion uses the bounding box prediction of LiDAR features as a proposal to query the image feature, then adapts a Transformer-like architecture to fuse the information back to LiDAR features. Similarly, DeepFusion li2022deepfusion projects LiDAR features on each view image as queries and then leverages cross-attention for two modalities.

An overlooked assumption of the current fusion mechanism is they heavily rely on the LiDAR point clouds, in fact, if the LiDAR input is missing, these methods will inevitably fail. This will hinder the deployment of such algorithms in realistic settings. In contrast, our BEVFusion is a surprisingly simple yet effective fusion framework that fundamentally overcomes this issue by disentangling the camera branch from the LiDAR point clouds as shown in Fig. 1 (c). In addition, concurrent works liu2022bevfusion ; yang2022deepinteraction also address this problem and propose effective LiDAR-camera 3D perception models.

Other modalities. There exist other works to leverage other modalities, such as fusing camera-radar by feature map concatenation chadwick2019distant ; john2019rvnet ; nobis2019deep ; nabati2021centerfusion . While interesting, these methods are beyond the scope of our work.
Despite a concurrent work chen2022futr3d aims to fuse multi-modalities information in a single network, its design is limited to one specific detection head wang2022detr3d while our framework can be generalized to arbitrary architectures.