Progressive Multi-Modal Fusion for Robust 3D Object Detection

Authors: rohit  mohan
daniele  cattaneo
florian  drews
abhinav  valada
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:29:54

Abstract: Multi-sensor fusion is crucial for accurate 3D object detection in autonomous driving, with cameras and LiDAR being the most commonly used sensors. However, existing methods perform sensor fusion in a single view by projecting features from both modalities either in Bird's Eye View (BEV) or Perspective View (PV), thus sacrificing complementary information such as height or geometric proportions.
To address this limitation, we propose ProFusion3D, a progressive fusion framework that combines features in both BEV and PV at both intermediate and object query levels. Our architecture hierarchically fuses local and global features, enhancing the robustness of 3D object detection. Additionally, we introduce a self-supervised mask modeling pre-training strategy to improve multi-modal representation learning and data efficiency through three novel objectives. Extensive experiments on nuScenes and Argoverse2 datasets conclusively demonstrate the efficacy of ProFusion3D. Moreover, ProFusion3D is robust to sensor failure, showing strong performance when only one modality is available.

Main paper: 
 introduction: 
 Reliable object detection is crucial for automated driving, as it enables vehicles to safely navigate the environment by accurately identifying and locating objects in their surroundings [1, 2] . To enhance robustness [3] , LiDAR-camera fusion has emerged as the predominant paradigm, leveraging complementary information from different modalities. However, the different data distributions that stem from the inherently different nature of these modalities present a significant challenge. To address this limitation, various multi-modal fusion strategies [4, 5, 6, 7] have been explored, which mostly differ in the representation used for fusion: Bird's Eye View (BEV) or Perspective View (PV), and the stage at which fusion is performed: raw input, intermediate features, or object queries. While using a single representation eases the fusion by mapping the features of both modalities into a common space, it also leads to information loss such as height information in BEV and occlusions and perspective distortion in PV. Similarly, fusion at the raw input stage leads to the incorporation of sensor noise and irrelevant information. In intermediate feature fusion, modality-specific information can be lost, and object query-level fusion allows the integration of high-level semantic information but greatly relies on the quality of this information, potentially affecting robustness. In this paper, we propose a progressive fusion approach that fuses features from each modality in both BEV and PV at both intermediate features and object query levels. This incremental integration enables the model to effectively combine and leverage the strengths of different views and fusion stages. settings for 3D object detection? While some prior works [13, 14] have explored pre-training with LiDAR-camera fusion on unlabeled datasets, they are often designed for small and dense point clouds with uniform point densities typical of indoor environments. However, these methods are not adaptable for automotive settings, which involve larger, sparser, and more heterogeneous data [15] . As a solution to this problem, we introduce an effective multi-modal mask modeling pre-training framework tailored for sparse outdoor point clouds.In this paper, we propose the ProFusion3D framework, which performs progressive LiDAR-camera sensor fusion, and a novel self-supervised masked modeling pre-training strategy to enhance the performance of 3D object detection. Our model fuses features at two stages, the intermediate feature level and the object query level, across both BEV and PV. This comprehensive fusion strategy enables our model to effectively leverage both local and global features, significantly improving the accuracy and robustness of 3D object detection, even in the event of sensor failures. The main contributions of this work can be summarized as follows: (1) we propose ProFusion3D, a novel method for LiDAR-camera 3D object detection that performs fusion at different stages and in different views. (2) A novel inter-intra modality fusion module that performs joint self and cross window attention of view-specific features. (3) A self-supervised pre-training strategy for outdoor LiDAR-camera fusion methods. (4) Extensive evaluations and ablation study of ProFusion3D on nuScenes and Argoverse2 benchmarks.Inter-Intra Fusion: For each of the inter-intra base units, the corresponding channel dimension of the query, key, and value embeddings are set to 192. The following Feed-Forward Network (FFN) consists of two fully-connected layers with GeLU activation after the first one. The first layer expands the input channels to 1024 while the second layer condenses it back to the original embedding dimensions (192).The convolutional block of the fusion module uses two consecutive 3 × 3 depthwise separable convolutions, each with dilation rates of 2 and 4, respectively. The number of channels in these convolutions is twice the embedding dimensions (384). Decoder: We employ two DETR-style decoder layers in the BEV decoder, PV decoder, and Joint Decoder. We follow similar parameter settings as described in [1] for each of the decoder layers. The sequence of operations within each layer is as follows: self-attention, normalization, cross-attention, normalization, FFN, and normalization. The embedding dimensions are set to 256, and the FFN channels are set to 2048. Both the attention and FFN dropout rates are configured to 0.1. We set the number of queries to 600 for initializing Q 0 . The initial object queries Q 0 interact with the 3D position-aware BEV features F 3D bev in the BEV decoder to update their representations to Q bev . In parallel, Q 0 also interacts with the 3D position-aware PV features F 3D pv in the PV decoder to update their representations to Q pv . Following this, Q join = [Q bev ; Q pv ] interacts with the 3D position-aware joint features F 3D join = [F 3D bev ; F 3D pv ] in the joint decoder to generate the final updated query representations.We use two FFNs to predict the 3D bounding boxes and the classes using the updated queries in each of the decoder layers. The prediction for each decoder layer is then as follows:EQUATIONwhere ϕ reg and ϕ cls represent the FFNs for regression and classification, respectively. Q d i are the updated object queries of the i-th decoder layer of the d-th decoder, where d ∈ {bev, pv, joint}. We train ProFusion3D through set prediction by using bipartite matching for one-to-one assignment between predictions and ground truths. Specifically, we use the focal loss for classification and L1 loss for 3D bounding box regression:EQUATIONwhere λ 1 and λ 2 are the hyperparameters to balance the two loss terms.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 
related work: 
 Unimodal 3D Object Detection: For camera-based 3D object detection, some methods [16, 17, 18] lift 2D features into 3D space via depth estimation or using frustum mesh grids, while others employ detection Transformers [19, 20, 21] to map image views to 3D space. LiDAR-based detectors convert point clouds into representations like voxels [22, 23] , pillars [24] , or range images [25] , processing them akin to 2D detectors. Multi-modal 3D Object Detection: LiDAR and camera fusion in 3D object detection have gained significant attention due to their superior performance. Existing 3D detection methods typically perform multi-modal fusion at one of three stages: raw input, intermediate feature, or object queries. Input fusion [4, 5] incorporates the 3D point cloud with 2D features from the camera. For example, PointPainting [4] uses category scores or semantic features [26, 27] from a 2D instance segmentation network. On the other hand, fusion at the intermediate feature level [6, 28, 29] maps both modalities into a shared representation space. Lastly, object query fusion methods [7, 30, 31, 32] either aggregate multi-modal features via proposals or refine queries from high-response LiDAR regions through interaction with camera features in two stages at the detection head. In this work, we seek to leverage the benefits of both intermediate feature fusion and object query fusion through a progressive fusion approach. Mask Image/Point Modeling: Building on the success of masked language modeling methods [33, 34] , similar techniques have been employed in the image domain [8, 9] . Among these, MAE stands out as a straightforward approach for masking random image patches and using their pixel values as reconstruction targets. Similarly, several methods have adapted these techniques to the point cloud domain. Point-BERT [10] and Point-MAE [11] utilize a BERT-style pre-training for point clouds by masking and reconstructing parts of the input. However, these methods generally focus on small dense point clouds. Voxel MAE [15] and Occupancy-MAE [35] adapt the MAE approach to handle sparse point clouds for 3D object detection which are common in outdoor autonomous driving scenarios. Despite extensive work on unimodal masked autoencoders, to the best of our knowledge, no approaches integrate LiDAR and multi-view camera modalities through multi-modal mask modeling in automotive settings. Our work bridges this gap by developing a framework to learn rich multi-modal representations for improved data efficiency. In particular, we introduce novel pre-training objectives of denoising and cross-modal consistency, increasing robustness and enforcing cross-modal information sharing.For the multi-modal masked modeling, we introduce three pre-training objectives: masked token reconstruction, unmasked token denoising, and masked token cross-modal attribute prediction. To train each objective we employ the following losses: Masked Token Reconstruction: In this pre-training objective, we reconstruct each masked image patch from the PV branch and voxels from the BEV branch. For the image patch reconstruction, we employ L1 loss between the predicted values of the masked pixels and the corresponding RGB values as follows:EQUATIONwhere Îi and I i are the predicted and ground truth RGB value of the i-th masked pixel, respectively, and N mp is the total number of masked pixels.For the voxel reconstruction, let P gt,i = {x 1 , x where ∥ • ∥ 2 denotes the L2-norm. This loss function ensures that each point in the ground truth set P gt is close to some point in the reconstructed set P rec and vice versa. Unmasked Token Denoising: In this pre-training objective, we learn to predict noise for each unmasked image patch and voxel. For the unmasked image patches, we employ L1 loss between the predicted noise and the actual noise added as follows:EQUATION)where ni is the predicted noise and n i is the actual noise added to the i-th unmasked pixel, and N up is the total number of unmasked pixels.For the voxel denoising, let N a,i = {n 1 , n 2 , • • • , n N } be the noise added to the i-th unmasked voxel and N p,i = {ñ 1 , ñ2 Masked Token Cross-Modal Attribute Prediction: In this pre-training objective, we predict pixel intensities for points in masked voxels and depth values for masked image patches. To predict pixel intensities for points in masked voxels, we do this jointly with the masked voxel reconstruction. Hence, while predicting the points in the masked voxel, we also predict the corresponding pixel intensities. For pixel intensity prediction, we add a loss term to the Chamfer distance loss of masked token reconstruction, by replacing f in Eq. ( 4) with f :f (x, P ) = ∥x -P j ∥ 2 2 + λ|x I -P j I |, with j = arg minEQUATION)where λ is the loss balancing term, x I and P j I are the pixel intensities of x and of the j-th point in P . Following, for depth prediction for masked image patches, let d and d denote the ground-truth and the predicted depth, respectively. Our loss function for depth estimation is then defined by  
method: 
 In this section, we first present the architecture of our proposed ProFusion3D framework that utilizes progressive fusion for 3D object detection, as shown in Fig. 2a , and then we detail its main components: the inter-intra fusion module and the decoder. Finally, we introduce our multi-modal masked modeling formulation for ProFusion3D (Fig. 3 ), and detail the three pre-training objectives.Our ProFusion3D architecture takes LiDAR point cloud and multi-view camera images as input, encoding each with their corresponding encoders to compute F l bev BEV features for the LiDAR modality and F c pv PV features for camera modality. We refer to the modality native to a view as intra-modality. Thus, for the BEV space, F l bev is considered the intra-modality, whereas for the PV space, F c pv is the intra-modality. Next, we perform cross-view mapping by projecting F l bev to PV and F c pv to BEV, referring to these mapped features as the inter-modality features for each view space: F l pv = M BEV → P V (F l bev ) and F c bev = M P V → BEV (F c pv ). We then utilize our proposed inter-intra fusion block to fuse both modalities in each feature space. The resulting fused features F bev = IIF (F l bev , F c bev ) and F pv = IIF (F c pv , F l pv ) are fed to the decoder, which performs object query level fusion.LiDAR BEV to PV mapping M BEV → P V : For each pixel (i b , j b ) of the given LiDAR BEV features, we first compute the corresponding (x, y) metric location. Next, we retrieve the z-coordinate for all non-empty voxels whose BEV projection falls into (x, y). Finally, we project the resulting 3D points into the camera image coordinate frame (i c , j c ) using the camera extrinsic and intrinsic parameters.Camera PV to BEV mapping M P V → BEV : We "lift" [18] 2D camera PV features to BEV by predicting, for each camera pixel, a depth distribution over a fixed set of discrete depths. Using the camera intrinsic and extrinsic parameters, and the predicted depth distribution, we transform the image PV features into 3D to obtain the voxel grid V ∈ R X×Y ×Z×C . Following [6] , we further encode these voxel features into BEV space features by first reshaping V ∈ R X×Y ×(ZC) and then applying 3D convolutions to reduce the channel dimension.Inter-Intra Fusion: Our proposed inter-intra fusion block IIF (F intra , F inter ) takes F intra and F inter features as its input, which are represented by F l bev and F c bev for the BEV space, and F c pv and F l pv for the PV space, respectively. Namely, for the BEV space, the LiDAR represents the intra-modality, and the camera represents the inter-modality, and conversely for the PV space. Fig. 2b illustrates the architecture of our fusion block, which utilizes two inter-intra base units.The first inter-intra base unit applies two subsequent layers of window attention followed by a Feed-Forward Network (FFN) with residual connections, as shown in Fig. 2c . More precisely, it concatenates F inter and F intra to obtain F unified , and then partitions the feature map into non-overlapping windows of size M × M for both F unified and F intra . In this process, F unified is used to obtain the keys (K unified ) and values (V unified ) for the window attention, while F intra is used to get the queries (Q intra ). Following [36] , we add a relative position bias B ∈ R M 2 ×M 2 to provide positional information. Applying attention with these key, query, and value matrices between the corresponding partitions effectively constitutes window attention, and can be defined asEQUATIONwhere d is the embedding dimension. Afterward, the F unified features are shifted byEQUATIONto obtain the key and value for another layer of window attention. The output of the previous window attention, passed through an FFN layer, acts as the query for this new window attention layer. By shifting the key and value features, while keeping the query features fixed, we can create a cross-window connection, effectively increasing the cross-modal attention span. We then apply the attention mechanism defined in Eq. ( 1), followed by another FFN layer.The second inter-intra base unit is analogous to the first unit, but with the main difference being that the fused features F unified are first processed through a series of convolutional layers before being passed to the second unit. This approach allows the second unit to capture globally enriched spatial context, while the first unit focuses on capturing local context. We concatenate the outputs of these units to form the final output of the fusion module.Overall, our fusion block effectively integrates features from both modalities, conditioned on intramodal features, to yield comprehensive fused features. This hierarchical approach strikes a balance between capturing local patterns-crucial for identifying and distinguishing objects-and gaining a broader contextual understanding of the scene and the relationships between objects. Note that we employ an instance of our fusion block in both the BEV and PV to fuse the view-specific features.Decoder: The decoder of our network employs two parallel sets of standard transformer decoders [37] to interact with the fused features in BEV and PV individually. Each set consists of two decoder layers. The updated queries from these individual view-specific decoders are then merged and passed through two additional decoder layers, where the queries interact with both fused BEV and PV features simultaneously. 3D position-aware features: Before passing the features extracted in PV and BEV space to the decoder, we enhance them by encoding 3D positional information. We first generate an evenlyspaced meshgrid G i for each camera i, as well as a BEV meshgrid G bev . Similar to [20] , we defineG i ∈ R W f ×H f ×D fin the frustum space of the respective camera, where W f is the width, H f is the height, and D f is the depth of the frustum. Each point a in the meshgrid can then be expressed asG a i = (u a • d a , v a • d a , d a , 1)T , where (u a , v a ) represent pixel coordinates in the i-th camera's PV, and d a denotes the depth value. Finally, the corresponding 3D coordinates p 3d i can be computed using the extrinsic calibration between the respective camera and the LiDAR sensor. Similarly, any point b in the BEV meshgrid is expressed as G b bev = (i b , j b , h b , 1), where (i b , j b ) is the respective pixel location in the BEV feature space, and h b is the height value. The corresponding 3D point can be simply computed by multiplying the BEV pixel location with the BEV grid size (s x , s y ) asp 3d bev = (i b • s x , j b • s y , h b , 1). These 3D points, representing each view, are then fed into an MLP network and transformed into 3D Positional Embeddings (PE). The fused features F bev and F pv are processed by a 1 × 1 convolution layer and subsequently added to the 3D PE of the respective view to generate the 3D position-aware features F 3D bev and F 3D pv . Query Initialization: Similar to [20] , we first initialize a set of learnable anchor points in normalized 3D world coordinates with a uniform distribution. The coordinates of these 3D anchor points are then fed to a small MLP network with two linear layers to generate the initial object queries Q 0 . Query Refinement: We refine the initial object queries Q 0 independently in each view to obtain the refined queries Q bev and Q pv using two DETR-style decoder layers. We then concatenate the view-specific branches to obtain the joint queriesQ join = [Q bev ; Q pv ],and the joint featuresF 3D join = [F 3D bev ; F 3D pv ].We employ two additional decoder layers to predict the final object classes p and bounding boxes b. We describe further details in the supplementary material. For training, we utilize the Hungarian algorithm [38] to find the optimal bipartite matching between the predicted objects and the ground truth objects as described in [37] . We adopt the focal loss for classification and L1 loss for 3D bounding box regression, following the approach in [20] .For the multi-modal masked modeling pre-training, we first patchify the images from multi-view cameras and voxelize the point clouds from LiDAR. For simplicity, we refer to them as tokens, using a modality-agnostic term. We then apply an asymmetrical masking strategy to randomly mask the tokens in each modality, where one modality is randomly selected to have a higher masking rate than the other. Next, we add Gaussian noise to the unmasked tokens. We then add positional encoding to these tokens, and the encoder of each modality processes them. Following this, our inter-intra fusion module performs multi-modal fusion, and simple decoders are then added on top to learn three pre-training objectives. Fig. 3 illustrates our pipeline for multi-modal masked modeling.Masked Token Reconstruction: We reconstruct each masked image patch from the PV branch and voxels from the BEV branch to emphasize the learning of spatial correlations. For the former, we employ L1 loss between the predicted values of the masked pixels and the corresponding RGB values, similar to [9] . For the latter, we supervise it with Chamfer distance [11] , which measures the distance between two sets of points. Unmasked Token Denoising: We incorporate a denoising objective for each unmasked image patch and voxel to enhance the learning of high-frequency components, such as fine details and surface variations. For the unmasked image patches, we employ L1 loss between the predicted noise and the actual noise added. Similarly, for the unmasked voxels, we compute the denoising loss analogously to Chamfer distance, where the ground truth represents the actual noise added to each point. Masked Token Cross-Modal Attribute Prediction: We introduce a cross-modal attribute prediction for masked tokens to leverage complementary information from corresponding unmasked modality tokens. Using the aforementioned asymmetrical masking strategy, masked tokens in the modality with the high masking rate will have a higher probability of the corresponding points in the other modality being unmasked. This setup allows us to create an objective that explicitly encourages the model to leverage complementary cross-modal information from the unmasked tokens to predict the attributes of the masked token. Specifically, we add a prediction task for the corresponding pixel intensity of the points in the masked voxels. For masked image patches, we predict the depth values. This ensures more effective multi-modal representation learning by leveraging cross-modal synergies.For training depth, we use the depth loss defined in [39] . For pixel intensity prediction, we add an L1 loss term to the Chamfer distance loss of masked token reconstruction. We outline each of the loss terms for the pre-training objectives in detail in the supplementary material. During pre-training, we sum all the individual losses to compute the total pre-training loss. 
experimental evaluation: 
 Dataset and Evaluation Metrics: We perform comprehensive experiments on the large-scale nuScenes dataset [40] for 3D detection. This dataset provides point clouds from a 32-beam LiDAR and images with a resolution of 1600 × 900 from 6 surround cameras. We use mean average precision (mAP) and the nuScenes detection score (NDS) [40] as evaluation metrics. Additionally, we report mAP for the Argoverse2 dataset, which offers a long perception range of up to 200 m, covering an area of 400 m × 400 m, significantly larger than nuScenes. Implementation Details: We implement our network in PyTorch using the open-source MMDetec-tion3D [41] , with Swin-T [36] as the image backbone and VoxelNet [23] as the LiDAR backbone. FPN [42] fuses multi-scale camera features into a 1/8 resolution feature map. Camera images are downsampled to 448 × 800 and the LiDAR point cloud is voxelized at a resolution of 0.075 m. Our fusion module uses a window size of M = 7. Our training comprises two stages: pre-training and 3D object detection training. For pre-training, we utilize the AdamW optimizer [43] configured with β 1 = 0.95, β 2 = 0.99, and a weight decay of 0.01. The learning rate is initially set to 5 × 10 -5 , gradually increases to 5×10 -4 over the first 1000 iterations, and then decreases to 1×10 -7 according to a cosine annealing schedule. We use a batch size of 4 and run for 200 epochs, employing an asymmetric masking ratio of 0.7 and 0.3. For 3D object detection, we train our network for 24 epochs with CBGS [44] and a batch size of 16. The AdamW optimizer is employed with an initial learning rate of 1.0 × 10 -4 following a cyclic learning rate policy [45] . For details on loss, please refer to the supplementary material. Lastly, on the Argoverse2 dataset, we directly train for 3D object detection for 6 epochs similar to [46, 47] .In Tab. 1, we compare our proposed approach with other state-of-the-art methods on the nuScenes (Tab. 1a) and Argoverse 2 (Tab. 1b) datasets for 3D object detection. Additionally, for the nuScenes dataset, we evaluate the robustness of ProFusion3D (Tab. 1c), investigating how the performance is affected when either the LiDAR or camera inputs are unavailable. Our method achieves 71.1% mAP on nuScenes and 37.7% mAP on Argoverse2, outperforming the best-performing baseline methods UniTR and CMT by 0.6% and 1.6%, respectively. This consistent improvement in performance across both datasets demonstrates that using progressive fusion minimizes the loss of information compared to single-stage or single-view methods, while multi-modal mask modeling enables learning of richer multi-modal representation. For the robustness evaluation presented in Tab. 1c, we train our model with the same robustness training strategy as described in [32] . Among fusion-based methods, in the event of a camera sensor failure, our method outperforms BEVFusion by 0.9%. In the case of LiDAR failure, it surpasses CMT by 0.7%. While baseline fusion methods show variable robustness with different sensor failures, our approach consistently maintains higher performance in each scenario and remains relatively closer to state-of-the-art unimodal methods.We first investigate the impact of our multi-modal masked modeling pipeline on the downstream 3D object detection task with regard to data efficiency and evaluate each pre-training objective. Subsequently, we discuss the impact of various components of our progressive fusion approach. Data Efficiency: One of the major benefits of using self-supervised learning is a reduced need for annotated data. To study the effects of various dataset sizes, we train two versions of our model with varying subsets of the annotated dataset held out, where one model is initialized randomly and the other has been pre-trained on the entire dataset using our self-supervised objectives. Please refer to the supplementary material for the strategy of compilation of the subsets of the dataset. The results of this experiment are shown in Tab. 2a. The model trained with 80% of the data using our multi-modal masking initialization outperforms the model trained with 100% of the data. The most significant performance increase is seen with the smallest subset of labeled data, demonstrating that low-data settings benefit the most from the learned multi-modal latent representations. Moreover, all pre-trained models consistently outperform their randomly initialized counterparts across all percentages of labels.To investigate the synergy of the pre-training objectives, we pre-trained our model with different combinations of these objectives, focusing on the subset with 20% annotated data. The results, presented in Tab. 2b show that training with the masked token reconstruction objective alone provides a solid baseline, demonstrating effective multi-modal representations through the fusion module. Adding the unmasked token denoising objective yields a 0.8% gain in mAP due to improved robustness to noise, resulting in better fine detail extraction. The cross-modal token attribute prediction results in a 1.1% gain in mAP, further enhancing the model's ability to leverage features of complementary modalities. Combining both objectives achieves a 1.5% gain in mAP, demonstrating that the synergy of these pre-training objectives improves model performance by enhancing both robustness and cross-modal understanding. Impact of Fusion in Both View Spaces: Tab. 2c presents the results of fusion in only the BEV space versus both BEV and PV spaces. Our inter-intra fusion (IIF) module in the BEV space achieves a 4.8% gain in mAP over the non-fusion counterpart, demonstrating its effectiveness. Performing fusion in both BEV and PV spaces results in an additional 1% gain in mAP, indicating that using both view spaces helps preserve more information compared to using a single view space.To study the impact of our inter-intra attention, we replaced the fusion module with various configurations, as presented in Tab. 2d. Initially, we concatenated inter-and intra-features in each view space as fusion. Next, we applied self-attention on intra-features and then used cross-attention with inter-features to merge them. Following, we used one inter-intra base unit (refer to Sec. 3) as the fusion block. We then experimented using two successive inter-intra base units without concatenating inter-and intra-features for computing attention, where the first unit performed window self-attention on intra-features, and the second performed window cross-attention on inter-and intra-features. Finally, we used our inter-intra fusion (IIF) module. Our results show that the inter-intra base as the fusion block outperforms other variations while the IIF module achieves the best performance, underscoring the importance of both local and globally enriched spatial context. Need for View-Specific Decoders: Tab. 2e compares the performance of a view-agnostic decoder that simultaneously interacts with fused view space features against our proposed configuration of View-agnostic decoder 69.9 View-specific decoder (ours) 70.5 view-specific decoders followed by a view-agnostic decoder. The higher mAP in our configuration indicates that direct fusion via object queries causes an imbalance, with BEV dominating over PV. 
conclusion: 
 In this work, we presented our ProFusion3D architecture for 3D object detection, which not only outperforms existing baselines on the nuScenes and Argoverse2 datasets but also demonstrates robustness in sensor failure scenarios. Our approach incorporates a progressive fusion strategy that integrates features from each modality in both BEV and PV space at the intermediate feature level and object query level. Furthermore, we introduced a self-supervised multi-modal mask modeling pretraining framework that significantly enhances ProFusion3D performance on small annotated datasets, thereby improving data efficiency. This framework employs masked token reconstruction with two novel objectives to facilitate robust multi-modal representation learning. Extensive evaluations demonstrate the efficacy of our ProFusion3D.Limitations: Our method involves mapping from one view to another and thereby, relies on good calibration of sensors. To alleviate this dependency, we will explore learnable methods for alignment between views [54, 55] . Moreover, our model currently does not consider efficiency. In the future, we aim to develop a more advanced framework that uses uncertainty estimation [56] to dynamically select the most effective combination of different views and fusion levels, optimizing the balance between performance, efficiency, and robustness. 
 
 Appendix: In this supplementary material, we provide additional details on various aspects of our work. First, we provide further architecture details for our fusion module and decoder in Sec. 1. Next, we present the loss details of the three pre-training objectives for our proposed LiDAR and Multi-View Camera Mask Modeling in Sec. 2. In Sec. 3 we describe the strategy for selecting the subsets of the dataset used in our data efficiency experiment (Sec. 4.2 of the main manuscript). Lastly, we provide additional qualitative results for ProFusion3D in Sec. 4 and Sec. 5. 


Decision: Strengths:
- a novel idea;
- detailed ablations;
- clear presentation.

Weaknesses:
- limited discussion on real-world applications;
- marginal performance gain;
- some missing technical details.

Post-rebuttal: The authors did an impressive job during the rebuttal, with a significant number of additional experimental results.  All three reviewers recommended acceptance, and the AC agreed.  The authors are encouraged to include the content from the rebuttal in the camera-ready version.

Reviews: review: title: title: Progressive Multi-Modal Fusion for Robust 3D Object Detection
paper_summary: paper_summary: The paper introduces ProFusion3D, a novel framework for LiDAR-camera fusion in 3D object detection for automated driving, which employs a progressive fusion approach across multiple stages and views.
main_review: main_review: Strengths 
* This paper proposes to fuse features at both intermediate and object query levels in both Bird's Eye View (BEV) and Perspective View (PV), aiming to leverage the strengths of different fusion stages and viewpoints.
* It's novel to propose multi-modal masked modeling in pre-training stage as it can leverage self-surpervised learning and information from the other modality to further boost the performance. To the best of my knowledge, it's the first time multi-modal masked modeling is used in 3d object detection
* This paper is clearly written, presenting a well-structured approach to addressing the challenges of LiDAR-camera fusion for 3D object detection in autonomous driving scenarios. The authors articulate their methodology logically, progressing from the problem statement through their proposed solution and its implementation. 
* The ablation study is very thorough, covering all the technical contributions studied in this paper. It systematically examines each component of the ProFusion3D framework, including the progressive fusion approach, the inter-intra modality fusion module, and the self-supervised pre-training strategy. This comprehensive analysis helps to validate the effectiveness of each proposed element and provides insights into their individual and combined impacts on the overall performance of the 3D object detection system

Weaknesses
* The performance gaps between ProFusion3D and other SOTA methods are limited, especially comparing to UniTR.
* I am having trouble understand why the performance of ProFusion3d in Table 1a doesn't match that of Table 2a
* In the single sensor setting, ProFusion is worse than BEVFormer and TransFusion-L even with pre-training.
questions: questions: * For the data efficiency experiment in section 4.2, it's not very cleary that the model are 1. pre-trained on subset of training data and finetuned on subset  or 2. pre-trained on full set of training data and and finetuned on subset. 
* It would be interesting to see how the performance change if the model is pretrained on single modelity
limitations: limitations: 3
review_summary: review_summary: Even though the performance increase on top of SOTA methods is not significant, the novelty of this paper, especially multi-modal mask modeling, lead to my recommendation of this paper
title: title: Innovative Fusion Approach for 3D Object Detection in Autonomous Driving
paper_summary: paper_summary: This paper introduces ProFusion3D, a novel approach for 3D object detection using progressive fusion of LiDAR and camera data for autonomous driving. The method fuses features in both Bird's Eye View and Perspective View spaces at multiple levels, introduces a new inter-intra fusion module, and proposes a self-supervised pre-training strategy. The approach achieves state-of-the-art results on nuScenes and Argoverse2 datasets.
main_review: main_review: Strengths:

1. Novel progressive fusion approach addressing limitations of single-view or single-stage fusion methods.
2. Innovative inter-intra fusion module capturing both local and global spatial context.
3. Promising self-supervised pre-training strategy improving data efficiency.
4. Comprehensive ablation studies providing valuable insights.
5. State-of-the-art results on nuScenes and Argoverse2 datasets.
6. Improved robustness in scenarios with sensor failures.

Weaknesses:

1. Limited discussion on real-time performance and computational requirements.
2. Absence of real-world testing on actual autonomous vehicles.
3. Insufficient exploration of temporal aspects in autonomous driving scenarios.
4. Limited discussion on safety implications of improved detection capabilities.
5. Insufficient analysis of generalization to diverse driving environments.
6. While consistent, performance improvements are rather small.

The paper presents a significant contribution to 3D object detection for autonomous driving. The ProFusion3D architecture effectively leverages complementary strengths of LiDAR and camera data, crucial for robust and accurate perception in autonomous driving.
The self-supervised pre-training strategy addresses the challenge of data efficiency, particularly relevant given the cost and time involved in collecting and annotating large-scale autonomous driving datasets.
The robustness analysis for sensor failure scenarios is commendable, addressing a critical challenge in deploying learning-based systems for safety-critical applications.

However, the paper has notable limitations:
* Lack of real-time performance analysis: The paper should discuss computational requirements and inference time, comparing speed-accuracy trade-offs with existing methods.
* Absence of real-world testing: Including results from actual autonomous vehicle tests would provide stronger evidence of practical applicability.
* Limited temporal considerations: Exploring how ProFusion3D could leverage temporal information would enhance its relevance to real-world scenarios.
* Insufficient safety discussion: The paper should elaborate on how improved detection translates to safer decision-making in autonomous driving.
* Limited generalization analysis: A discussion on performance in driving environments not represented in the datasets would be valuable.
* Minor performance improvements might not warrant the added complexity.

Addressing these limitations would significantly strengthen the paper's impact and practical relevance to autonomous driving applications.
questions: questions: 1. What is ProFusion3D's computational complexity and real-time performance compared to existing methods?
2. Have you conducted tests on actual autonomous vehicles? If so, how do the results compare to dataset evaluations?
3. How might ProFusion3D be extended to leverage temporal information?
4. How does the improved detection performance translate to safer decision-making?
5. How well does ProFusion3D generalize to driving conditions not represented in the datasets?
limitations: limitations: 2
review_summary: review_summary: ProFusion3D presents innovative ideas for multi-modal fusion in 3D object detection and demonstrates impressive results on automotive datasets. The progressive fusion approach, inter-intra fusion module, and self-supervised pre-training strategy are valuable contributions. However, the paper would be strengthened by addressing real-time performance, including real-world testing results, and exploring temporal aspects. I recommend minor revisions to address these points, after which the paper would make a strong contribution to the field of perception in autonomous driving.
title: title: ProFusion3D:  Multi-View Sensor Fusion and 3D Positional Awareness for 3D Object Detection
paper_summary: paper_summary: The paper introduces ProFusion3D, a framework that performs progressive LiDAR-camera sensor fusion. The model fuses features at two stages: the intermediate feature level and the object query level. Fusion occurs across both Bird’s Eye View (BEV) and Perspective View (PV), enabling the model to leverage both local and global features effectively.
main_review: main_review: **Overview**

The architecture takes both LiDAR point clouds and multi-view camera images as input. It encodes these inputs using their corresponding encoders, resulting in BEV features for the LiDAR modality and PV features for the camera modality. These BEV features $F_{I_{bev}}$ for LiDAR and PV features $Fc_{pv}$ for the camera are termed intra-modality features in their respective spaces. The $F_{I_{bev}}$ features are projected to PV, resulting in inter-modality features $F_{I_{pv}}$ using a mapping function $\mathcal{M}_{BEV} \rightarrow PV$.

Similarly, the camera PV features $Fc_{pv}$ are mapped to BEV features $Fc_{bev}$ using the mapping function $\mathcal{M}_{PV} \rightarrow BEV$. This involves a 2D-to-3D view projection to construct camera BEV features from the camera PV features. This is the cross-view mapping stage.

For the fusion process, the inter-intra fusion block fuses intra-modality and inter-modality features in each feature space. This involves two units: the first focuses on capturing local context, while the second captures global spatial context. The fusion block applies a window attention mechanism, where queries $Q_{intra}$ are obtained from intra-modality features, and keys $K_{unified}$ and values $V_{unified}$ are derived from the concatenated intra and inter-modality features. By shifting the key and value features while keeping the query features fixed, the architecture creates cross-window connections, enhancing cross-modal attention.


**Strengths**:
-  The progressive fusion approach, involving both intermediate and object query levels across BEV and PV, allows effective leveraging of both local and global features.
- The use of joint self and cross window attention within the inter-intra fusion block is a novel method for combining multi-modal features.
- Overall it is a well-written paper and the results look good.

**Weaknesses**:
- The writing syle of the paper can be improved. The detailed description of the fusion mechanism, including cross-view mapping and hierarchical attention, described in section 3.1 ProFusion3D Architecture is complex and hard to parse. Simplifying or providing more intuitive explanations could help ease understanding. 
- Figures 2(a) are helpful but incredibly small. I prefer to see a connection in notation in Section 3 to the diagram to better understand the components.
-  Potentially a missing baseline[1] ?



[1] DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection -  https://openaccess.thecvf.com/content/CVPR2022/papers/Li_DeepFusion_Lidar-Camera_Deep_Fusion_for_Multi-Modal_3D_Object_Detection_CVPR_2022_paper.pdf
questions: questions: - Can you elaborate on the process of generating 3D positional embeddings from the camera and BEV meshgrids?
 
- How does the Hungarian algorithm contribute to the matching process between predicted and ground truth objects during training?

- Could you highlight the difference between ProFusion3D and DeepFusion referenced above? Since its also fusing LiDAR and Camera features.
limitations: limitations: 3
review_summary: review_summary: This is a good paper, with decent contributions and the paper is well-written. Because I'm not super up to date with SOT in the field, I am recommending weak accept, but happy to revise based on other reviewers recommendations.