Progressive Multi-Modal Fusion for Robust 3D Object Detection
Authors: Rohit Mohan
Daniele Cattaneo
Florian Drews
Abhinav Valada
Venue: CoRL 2024 (2024)
Keywords: 3D Object Detection
Multimodal Learning
Self-Supervised Learning

Abstract: Multi-sensor fusion is crucial for accurate 3D object detection in autonomous driving, with cameras and LiDAR being the most commonly used sensors. However, existing methods perform sensor fusion in a single view by projecting features from both modalities either in Bird's Eye View (BEV) or Perspective View (PV), thus sacrificing complementary information such as height or geometric proportions.
To address this limitation, we propose ProFusion3D, a progressive fusion framework that combines features in both BEV and PV at both intermediate and object query levels. Our architecture hierarchically fuses local and global features, enhancing the robustness of 3D object detection. Additionally, we introduce a self-supervised mask modeling pre-training strategy to improve multi-modal representation learning and data efficiency through three novel objectives. Extensive experiments on nuScenes and Argoverse2 datasets conclusively demonstrate the efficacy of ProFusion3D. Moreover, ProFusion3D is robust to sensor failure, showing strong performance when only one modality is available.

Progressive Multi-Modal Fusion for Robust 3D Object Detection
Rohit Mohan', Daniele Cattaneo', Florian Drews Abhinav Valada Department of Computer Science, University of Freiburg, Germany Bosch Research, Robert Bosch GmbH, Renningen, Germany attp://profusionad ca. uni ceii-freiburg.c.
Abstract: Multi-sensor fusion crucial for accurate 3D object detection in au- tonomous driving, with cameras and LiDAR being the most commonly used sensors. However, existing methods perform sensor fusion in a single view by projecting features from both modalities either in Bird's Eye View (BEV) or Perspective View (PV), thus sacrificing complementary information such as height or geometric proportions. To address this limitation, we propose ProFusion3D, progressive fusion framework that combines features in both BEV and PV at both intermediate and object query levels. Our architecture hierarchically fuses local and global fea- tures, enhancing the robustness of 3D object detection. Additionally, we introduce a self-supervised mask modeling pre-training strategy to improve multi-modal representation learning and data efficiency through three novel objectives. Exten- sive experiments on nuScenes and Argoverse2 datasets conclusively demonstrate the efficacy of ProFusion3D. Moreover, ProFusion3D is robust sensor failure, demonstrating strong performance when only one modality is available.
Keywords: 3D Object Detection, Multimodal Learning, Sape-uupeeeeed Learning
Introduction
Reliable object detection is crucial for automated driving, as it enables vehicles to safely navigate the environment by accurately identifying and locating objects in their surroundings [1, 2]. To enhance robustness [3], LiDAR-camera fusion has emerged as the predominant paradigm, leveraging complementary information from different modalities. However, the different data distributions that stem from the inherently different nature of these modalities present significant challenge. To address this limitation, various multi-moda fusion strategies [4, 5, 6, 7] have been explored, which mostly differ in the representation used for fusion: Bird's Eye View (BEV) or Perspective View (PV), and the stage at which fusion is performed: raw input, intermediate features, or object queries. While using a single representation eases the fusion by mapping the features of both modalities into common space, also leads to information loss such as height information in BEV and occlusions and perspective distortion in PV. Similarly, fusion the raw input stage leads to the incorporation of sensor noise and irrelevant information. In intermediate feature fusion, modality-speciin information can be lost, and object query-level fusion allows the integration of high-level semantic information but greatly relies on the quality of this information, potentially affecting robustness. In this paper, we propose a progressive fusion approach that fuses features from each modality in both BEV and PV at both intermediate features and object query levels. This incremental integration enables the model to effectively combine and leverage the strengths of different views and fusion stages.
Recently, self-supervised learning approaches such as Mask Image Modeling (MIM) [8, 9] and Mask Point Modeling (MPM) [10, 11] have shown great success in extracting strong representations from large-scale unlabeled data for images and point clouds, respectively. These methods enhance performance on fine-tuned downstream tasks and improve data efficiency [12]. This raises an important question: can similar techniques be applied to multi-modal sensor fusion automotive
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

LEAC
Eisodder
Facost
Lroce
Eroker
Fusion
Intermeiaate BEV PV
(d) Objectt Level
(1) Prog Fuskon
Figure I: Overview of multi-modal fusion strategies. (a) Raw input fusion: integrates sensor data directly, prone noise. (b) Intermediate fusion: combines features in BEV PV, may lose modality details. (c) Object Query level fusion: merges high-level semantic information, reliant its quality. (d) Progressive fusion (proposed); incrementally integrates features from BEV and PV at intermediate and object query levels.
settings for 3D object detection? While some prior works [13, 14] have explored pre-training with LiDAR-camera fusion on unlabeled datasets, they are often designed for small and dense point clouds with uniform point densities typical of indoor environments. However, these methods are not adaptable for automotive settings, which involve larger, spanser, and more heterogeneous data [15]. As solution to this problem, we introduce an effective multi-modal mask modeling pre-training framework tailored for sparse outdoor point clouds.
In this paper, we propose the ProFusion3I framework, which performs progressive LiDAR-camess sensor fusion, and novel self-super masked modeling pre-training strategy to enhance the performance of 3D object detection. Our model fuses features at two stages, the intermediate feature level and the object query level, across both BEV and PV. This comprehensive fusion strategy enables our model effectively leverage both local and global features, significantly improving the accuracy and robustness of 3D object detection, even in the event of sensor failures. The main contributions of this work be summarized follows: (1) propose ProFusion3D, novel method for LiDAR-camera 3D object detection that performs fusion at different stages and in different views. (2) A novel inter-intra modality fusion module that performs joint self and cross window attention of view-specific features. (3) A self-supervised pre-training strategy for outdoor LiDAR-camera fusion methods. (4) Extensive evaluations and ablation study ProFusion3D on nuScenes and Argoverse2 benchmarks.
2 Related Work
Unimodal 3D Object Detection: For camera-based 3D object detection, some methods [16, 17, 18] lift 2D features into 3D space via depth estimation or using frustum mesh grids, while others employ detection Transformers [19, 20, 21] to map image views to 3D space. LiDAR-based detectors convert point clouds into representations like voxels [22, 23]. pillars [24], or range images [25], processing them akin 2D detectors.
Multi-modal 3D Object Detection: LiDAR and camera fusion in 3D object detection have gained significant attention due to their superior performance. Existing 3D detection methods typically perform multi-modal fusion at one of three stages: raw input, intermediate feature, or object queries. Input fusion [4, 5] incorporates the 3D point cloud with 2D features from the camera. For example, PointPainting [4] uses category scores or semantic features [26, 27] from 2D instance segmentation network. On the other hand, fusion at the intermediate feature level [6, 28, 29] maps both modalities into shared representation space. Lastly, object query fusion methods [7, 30, 31, 32] either aggregate multi-modal features via proposals or refine queries from high-response LiDAR regions through interaction with camera features in two stages at the detection head. In this work, we seek to leverage the benefits of both intermediate feature fusion and object query fusion through progressive fusion approach.
Mask Image/Point Modeling: Building on the success of masked language modeling methods [33, 34], similar techniques have been employed in the image domain [8, 9]. Among these, MAE stands out as straightforward approach for masking random image patches and using their pixel values reconstruction targets. Similarly, several methods have adapted these techniques to the point

---

Panee
30
(a) ProFusion Architecture
(b) Inter- Fusion
(c) Inter-Intra Base Architecture
Figure 2: (a) Illustration of our proposed ProFusion3D architecture that employs progressive fusion. (b) The topology of proposed fusion module and (c) The core component of the aforementioned fusion module.
cloud domain. Point-BERT [10] and Point-MAE [11] utilize a BERT-style pre-training for point clouds by masking and reconstructing parts of the input. However, these methods generally focus on small dense point clouds. Voxel MAE [15] and Occupancy-MAF, [35] adapt the MAE approach to handle sparse point clouds for 3D object detection which are common in outdoor autonomous driving scenarios. Despite extensive work on unimodal masked autoencoders, to the best of our knowledge, no approaches integrate LiDAR and multi-view camera modalities through multi-modal mask modeling in automotive settings. Our work bridges this gap by developing a framework to learn rich multi-modal representations for improved data efficiency. In particular, we introduce novel pre-training objectives of denoising and cross-modal consistency, increasing robustness and enforcing cross-modal information sharing.
3 Method
In this section, we first present the architecture of our proposed ProFusion3D framework that utilizes progressive fusion for 3D object detection, shown in Fig. 2a, and then detail its main components: the inter-intra fusion module and the decoder. Finally, we introduce our multi-modal masked modeling formulation for ProFusion3D (Fig. 3), and detail the three pre-training objectives.
3.1 ProFusion3D Architecture
Our ProFusion3D architecture takes LiDAR point cloud and multi-view camera images as input, encoding each with their corresponding encoders to compute Fieev BEV features for the LiDAR modality and Fe pv yv PV features for camera modality. We refer to the modality native to view as motra-modality. Thus, for the BEV space, Fi is considered the intra-modality, whereas for the PV space, Pe.pv is the antra-modality.. Next, we perform cross- view mapping by projecting Fibey to PV and Fe to BEV, referring to these mapped features as the inter-modality features for each view space: FL = MBEV- PV (Fisev) and FeLsev = MMy BEV (Fe.v). ). We then utilize our proposed inter-intra fusion block to fuse both modalities in each feature space. The resulting fused features Feev 1IFFFFFFoo Febee) and F'p- (Fe.,,.
fed to the decoder, which performs object
query level fusion.
LiDAR BEV to PV mapping MBEV PV: For each pixel (is.jo) of the given LiDAR BEV features, we first compute the corresponding (I, y) metric location. Next, we retrieve the z-coordinate for all non-empty voxels whose BEV projection falls into (x, y). Finally, we project the resulting 3D points into the camera image coordinate frame (fc.Je) using the camera extrinsic and intrinsic parameters.
Camera PV BEV mapping MpV BEV: We "lift" [18] 2D camera PV features to BEV by predicting, for each camera pixel, depth distribution over a fixed set of discrete depths. Using the camera intrinsic and extrinsic parameters, and the predicted depth distribution, we transform the image PV features into 3D to obtain the voxel grid V E Xx xYx2xx Following [6], we further encode these voxel features into BEV space features by first reshaping V E R XxXx(( and then applying 3D convolutions to reduce the channel dimension.
Our proposed inter-intra fusion block IIF((mmaa Finte)) takes Fintra and Finee features as its input, which are represented by FlRev and Fe.bev for the BEV space, and Fegy and Figg for the PV space, respectively. Namely, for the BEV space, the LiDAR represents the intra-modality, and the camera represents the inter- modaltty,,,,,, and conversely for the PV space. Fig. 2b illustrates the architecture of our fusion block, which utilizes two inter-intra base units.
Inter-Intra Fusion:
3

---

The first inter-intra base unit applies two subsequent layers of window attention followed by Feed-Forward Network (FFN) with residual connections, as shown in Fig. 2c. More precisely, concatenates Finter and Fintra to obtain Fumilied, and then partitions the feature map into non-overlapping windows of size M x M for both Funified and Fintr- this process, Funified used to obtain the keys (Kunities) and values (Viiited for the window attention, while Fintra used get the queries (Qintra.. Following [36], we add a relative position bias B I x to provide positional information. Applying attention with these key, query, and value matrices between the corresponding partitions effectively constitutes window attention, and can be defined
(lintaainntee K unitied
InmatioonQQntaa. Kunified Vunitica) = Softmax
BBB
Vanitied
(I)
Vd
where d is the embedding dimension. Afterward, the Funified features are shifted by [- [-| A4 to obtain the key and value for another layer of window attention. The output of the previous window attention, passed through an FFN layer, acts as the query for this new window attention layer. By shifting the key and value features, while keeping the query features fixed, we can create a cross-window connection, effectively increasing the cross-modal attention span. We then apply the attention mechanism defined Eq. (1), followed by another FFN layer.
The second inter-intra base unit is analogous to the first unit, but with the main difference being that the fused features F Fuminied are first processed through series of convolutional layers before being passed to the second unit. This approach allows the second unit to capture globally enriched spatial context, while the first unit focuses on capturing local context. We concatenate the outputs of these units to form the final output the fusion module.
Overall, our fusion block effectively integrates features from both modalities, conditioned intra- modal features, to yield comprehensive fused features. This hierarchical approach strikes a balance between capturing local patterns crucial for identifying and distinguishing objects- and gaining broader contextual understanding of the scene and the relationships between objects. Note that we employ an instance of our fusion block in both the BEV and PV to fuse the view-specific features.
Decoder: The decoder of our network employs two parallel sets of standard transformer decoders [37] to interact with the fused features in BEV and PV individually. Each set consists of two decoder layers. The updated queries from these individual view-specific decoders then merged and passed through two additional decoder layers, where the queries interact with both fused BEV and PV features simultaneously.
3D position-aweeee aawere features: Before passing the features extracted in PV and BEV space to the decoder, we enhance them by encoding 3D positional information. We first generate an evenly- spaced meshgrid G, for each camera i, as well as BEV meshgrid Gber. Similar to [20], we define G e W DD in the frustum space of the respective camera, where Wy is the width, Hy the height, and Dy is the depth of the frustum. Each point in the meshgrid can then be expressed as G7 (u,- day Vo da, da, 1)z² where (uay va) represent pixel coordinates in the i-th camera's ' PV, and du denotes the depth value. Finally, the corresponding 3D coordinates pad can be computed using the extrinsic calibration between the respective camera and the LiDAR sensor. Similarly, any point b in the BEV meshgrid is expressed as Ghe = (is. hg. 1), where (1s. is the respective pixel location the BEV feature space, and ha the height value. The corresponding 3D point can be simply computed by multiplying the BEV pixel location with the BEV grid size (sx, Sy) as Pōe. (i6- Szz jo Sy, hy, 1). These 3D points, representing each view, are then fed into MLP network and transformed into 3D Positional Embeddings (PE). The fused features Feev and Po are processed by 1 x convolution layer and subsequently added to the 3D PE of the respective view to generate the 3D position-aware features E&D and pr
Query Initialization: Similar to [20], we first initialize a set of learnable anchor points in normalized 3D world coordinates with a uniform distribution. The coordinates of these 3D anchor points are then fed to a small MLP network with two linear layers to generate the initial object queries Qo- Query Refinement: We refine the initial object queries Qo independently in each view to obtain the refined queries Qeev and Qp using two DETR-style decoder layers. We then concatenate

---

Random Macking
Unmsssed
Vovels
Predecion
Cricce
Camen
leput
Rundon Masking
Pakiha
Addincsa
to
Figure 3: Illustration of our multi-moda mask modeling pipeline for learning multi-modal latent representations. It patchifies/voxelizes the input modalities into tokens, masks them asymmetrically, adds noise to the unmasked tokens, and trains the model with objectives of reconstruction, denoising, and cross-modal attribute prediction.
the view-specific branches to obtain the joint queries 2jenn = [Q Qpvl. and the joint features join = [ebr ppP). We employ two additional decoder layers to predict the final object classes ý and bounding boxes b. We describe further details in the supplementary material. For training, utilize the Hungarian algorithm [38] to find the optimal bipartite matching between the predicted objects and the ground truth objects as described in [37]. We adopt the focal loss for classification and LI loss for 3D bounding box regression, following the approach in [20].
F3D
3.2 LiDAR and Multi- View Camera Mask Modeling
For the multi-modal masked modeling pre-training, we first patchify the images from multi-view cameras and voxelize the point clouds from LiDAR. For simplicity, we refer to them as tokens, using a modality-agnostic term. We then apply an asymmetrical masking strategy to randomly mask the tokens in each modality, where one modality is randomly selected to have higher masking rate than the other. Next, we add Gaussian noise to the unmasked tokens. We then add positional encoding to these tokens, and the encoder of each modality processes them. Following this, our inter-intra fusion module performs multi-modal fusion, and simple decoders then added top to learn three pre-training objectives. Fig. 3 illustrates our pipeline for multi-modal masked modeling.
Masked Token Reconstruction: We reconstruct each masked image patch from the PV branch and voxels from the BEV branch to emphasize the learning of spatial correlations. For the former, we employ LI loss between the predicted values of the masked pixels and the corresponding RGB values, similar to [9]. For the latter, we supervise with Chamfer distance [1 I], which measures the distance between two sets of points.
Unmasked Token Denoising: We incorporate denoising objective for each unmasked image patch and voxel to enhance the learning of high-frequency components, such as fine details and surface variations. For the unmasked image patches, we employ LI loss between the predicted noise and the actual noise added. Similarly, for the unmasked voxels, we compute the denoising loss analogously to Chamfer distance, where the ground truth represents the actual noise added to each point.
Masked Token Cross-Modal Attribute Prediction: We introduce a cross-modal attribute prediction for masked tokens to leverage complementary information from corresponding unmasked modality tokens. Using the aforementioned asymmetrical masking strategy, masked tokens in the modality with the high masking rate will have a higher probability of the corresponding points in the other modality being unmasked. This setup allows us to create an objective that explicitly encourages the model to leverage complementary cross- modal information from the unmasked tokens to predict the attributes of the masked token. Specifically, we add prediction task for the corresponding pixel intensity of the points in the masked voxels. For masked image patches, we predict the depth values. This ensures more effective multi-modal representation learning by leveraging cross-modal synergies. For training depth, loss term to the Chamfer distance loss of masked token reconstruction. We outline each of the loss
the depth loss defined in [39]. For pixel intensity prediction, add an L1
5

---

terms for the pre- training objectives in detail in the supplementary material. During pre-training, we sum all the individual losses to compute the total pre-training loss.
4 Experimental Evaluation
Dataset and Evaluation Metrics: We perform comprehensive experiments on the large-scale nuScenes dataset [40] for 3D detection. This dataset provides point clouds from a 32-beam LiDAR and images with resolution of 1600 x 900 from surround cameras. We use mean average precision (mAP) and the nuScenes detection score (NDS) [40] as evaluation metrics. Additionally, we report mAP for the Argoverse2 dataset, which offers a long perception range of up to 200 m, covering an area of 400 400 m, significantly larger than nuScenes.
Implementation Details: We implement network in PyTorch using the open-source MMDetec- tion3D [41]. with Swin-T [36] the image backbone and VoxelNet [23] as the LiDAR backbone. FPN [42] fuses multi-scale camera features into a 1/8 resolution feature map. Camera images are downsampled to 448 x 800 and the LiDAR point cloud is voxelized at resolution of 0.075 m. Our fusion module uses a window size of M = 7. Our training comprises two stages: pre-training and 3D object detection training. For pre-training, we utilize the Adam W optimizer [43] configured with 31 0.95, B2 0.99, and a weight decay of 0.01. The learning rate is initially set gradually increases 5 over to cosine annealing schedule. We use a batch size of and run for 200 epochs, employing an asymmetric masking ratio of 0.7 and 0.3. For 3D object detection, we train our network for 24 epochs with CBGS [44] and batch size of 16. The AdamW W optimizer is employed with an initial learning rate of 1.0 x 10 following a cyclic learning rate policy [45]. For details on loss, please refer to the supplementary material, Lastly, on the Argoverse2 dataset, we directly train for 3D object detection for 6 epochs similar to [46, 47].
10 over the first 1000 iterations, and then decreases to 1 x 10-7 according
4.1 Benchmarking Results
In Tab. 1, compare our proposed approach with other state-of-the-ar methods on the nuScenes (Tab. Ia) and Argoverse 2 (Tab. 1b) datasets for 3D object detection. Additionally, for the nuScenes dataset, we evaluate the robustness of ProFusion3D (Tab. Ic), investigating how the performance is affected when either the LiDAR or camera inputs are unavailable. Our method achieves 71.1% % mAP on nuScenes and 37.7% mAP on Argoverse2, outperforming the best- rest-performing baseline methods UniTR and CMT by 0.6% and 1.6%, respectively. This consistent improvement performance across both datasets demonstrates that using progressive fusion minimizes the loss of information compared to single-stage or single-view methods, while multi-modal mask modeling enables learning of richer multi-modal representation. For the robustness evaluation presented in Tab. Ic, we train our model with the same robustness training strategy as described in [32]. Among fusion-based methods, in the event of a camera sensor failure, our method outperforms BEVFusion by 0.9%. In the case of LiDAR failure, it surpasses CMT by 0.7%. While baseline fusion methods show variable robustness with different sensor failures, our approach consistently maintains higher performance in each scenario and remains relatively closer state-of-the-ane unimodal methods.
4.2 Ablation Study
We first investigate the impact of multi-modal masked modeling pipeline the downstream 3D object detection task with regard to data efficiency and evaluate each pre-training objective. Subsequently, we discuss the impact of various components of our progressive fusion approach.
Data Efficiency: One the major benefits of using self-supervised learning is a reduced need for annotated data. To study the effects various dataset sizes, we train two versions our model with varying subsets of the annotated dataset held out, where one model initialized randomly and the other has been pre-trained on the entire dataset using our self-supervised objectives. Please refer the supplementary material for the strategy of compilation of the subsets of the dataset. The results of this experiment shown in Tab. 2a. The model trained with 80% of the data using

---

Table 1: Performance comparison with state-of-the-ar methods on nuSeenes and Argoverse2 datasets.
'C" denotes Camera, and 'I'' denotes LiDAR
(a) Evaluation on the nuScenes validation set.
(b) Benchmarking on the Argoverse2 validation set.
Methods
modality
mAP
NDS
Methods
Modality
mAP
CDS
DETR3D [19]
C
34.9
43.4
CenterPoint [49]
L.
22.0
17.6
BEVDet4D [48]
C
42.1
54.5
FSD [50]
28.2
22.7
BEVFormer [21]
41.6
51.7
VoxeINeXt [46]
30.5
23.0
SECOND [23]
L
52.6
63.0
FSF [47]
33.2
25.5
CenterPoint [49]
L
59.6
66.8
CMT [32]
LC
36.1
27.8
FSD [50]
I.
62.5
68.7
VoxeINeXt [46]
L
63.5
68.7
ProFusion3D (ours)
LC
37.7
29.1
LargeKemel3D [51]
L
63.3
Transfusion-L- [7]
L 64.9
69.9
(c) Robust performance comparison the nuScenes
FUTR3D [31]
LC
64.5
68.3
val set with missing modality in mAP.
UVTR [28]
LC
65.4
70.2
TransFusion [7]
LC
67.5
71.3
Methods
Modality at Inference Both only LiDAR only Cams
BEVFusion [6]
LC
68.5
71.4
DeepInteraction [30]
LC
69.9
72.6
BEVFormer [21]
41.6
DAL [52]
LC
70.0
73.4
TransFusion-L
64.9
CMT [32]
LC
70.3
72.9
FSF [47]
I.C
70.4
72.7
BEVFusion [6]
68.5
63.0
32.0
UniTR [53]
L.C
70.5
73.3
CMT [32]
70.3
61.7
38.3
ProFusion3D (ours)
L.C
71.1
73.6
ProFusion3D (ours)
71.1
63.9
38.9
multi-modal masking initialization outperforms the model trained with 100% of the data. The most significant performance increase is seen with the smallest subset of labeled data, demonstrating that low-data settings benefit the most from the learned multi-modal latent representations. Moreover, all pre-trained models consistently outperform their randomly initialized counterparts across all percentages of labels.
Role of Pre- Training Objectives: To investigate the synergy the pre-training objectives, we pre-trained our model with different combinations of these objectives, focusing on the subset with 20% annotated data. The results, presented in Tab. 2b show that training with the masked token reconstruction objective alone provides solid baseline, demonstrating effective multi-modal rep- resentations through the fusion module. Adding the unmasked token denoising objective yields 0.8% gain in mAP due to improved robustness to noise, resulting in better fine detail extraction. The cross-modal token attribute prediction results in 1.1% gain in mAP, further enhancing the model's ability to leverage features of complementary modalities. Combining both objectives achieves 1.5% gain in mAP, demonstrating that the synergy of these pre-training objectives improves model performance by enhancing both robustness and cross-modal understanding.
Impact of Fusion in Both View Spaces: Tab. 2c presents the results fusion in only the BEV space versus both BEV and PV spaces. Our inter-intra fusion (IIF) module in the BEV space achieves 4.8% gain in mAP over the non-fusion counterpart, demonstrating its effectiveness. Performing fusion in both BEV and PV spaces results in an additional I % gain in mAP, indicating that using both view spaces helps preserve more information compared to using a single view space.
Importance of Inter-Intre Fusion: To study the impact of our inter-intra attention, we replaced the fusion module with various configurations, as presented in Tab. 2d. Initially, we concatenated inter- and intra-features in each view space fusion. Next, applied self-attention on intra-features and then used cross-attention with inter-features merge them. Following, used one inter-intra base unit (refer to Sec. 3) as the fusion block. We then experimented using two successive inter-intra base units without concatenating inter- and intra-features for computing attention, where the first unit performed window self-attention on intra-features, and the second performed window cross-attention on inter- and intra-features. Finally, we used our inter-intra fusion (IIF) module. Our results show that the inter-intra base as the fusion block outperforms other variations while IIF module achieves best performance, underscoring the importance of both local and globally enriched spatial context.
Need for View-Specific Decoders: Tab. 2e compares the performance of view-agnostic decoder that simultancously interacts with fused view space features against proposed configuration of
7

---

Table 2: Ablation results on the nuScenes dataset.
(a) 3D object detection performance for randomly and pre-trained initialization with varying labeled data.
(c) Fusion single BEV vs. both BEV and PV view space.
(Voxel Size = 0.15m, Image Size = 256 x 704)
Fusion
mAP
Labeled Data
Initialization
mAP
BEV LiDAR only
64.7
Random
52.3
20%
IIF in BEV IIF in BEV + PV
69.5
Multi-Modal Masking
56.8
70.5
Random
59.5
40%
Multi-Modal Masking
63.2
(d) Effect of different fusion variations.
Random
62.4
60%
Fusion
mAP
Multi-Modal Masking
Concatenation
68.3
Random
64.1
80%
Self- + Cross- Attention
68.7
Multi-Modal Masking
65.6
Inter-Intra Base (ours)
69.6
Random
65.2
Inter-Intra Base
100%
69.1
Multi-Modal Masking
66.2
(Self- Cross- Attention
IIF (ours)
70.5
(b) Impact pre-training objectives initialization for 3D detection with 20% labeled data.
(c) View-agnostic decoder View-specific decoder.
Mask
Unmasked Cross- Modal Token
Token
Attribute Reconstruction Denoising Prediction
mAP
Fusion
mAP
View-agnostic decoder
69.9
55.3
View-specific decoder (ours)
70.5
56.4
56.8
Figure Qualitative object detection results of our proposed ProFusion3D network versus the state-of-the-an CMT architecture on the nuScenes val set. Red: incorrect detections, Green: correct detection.
view-specific decoders followed by a view-agnostic decoder. The higher mAP in our configuration indicates that direct fusion via object queries causes imbalance, with BEV dominating over PV.
5 Conclusion
In this work, presented ProFusion3D architecture for 3D object detection, which not only outperforms existing baselines the nuScenes and Argoverse2 datasets but also demonstrates robustness in sensor failure scenarios. Our approach incorporates progressive fusion strategy that integrates features from each modality in both BEV and PV space at the intermediate feature level and object query level. Furthermore, we introduced a supervupe multi-modal mask modeling pre- training framework that significantly enhances ProFusion3D performance on small annotated datasets, thereby improving data efficiency. This framework employs masked token reconstruction with two novel objectives to facilitate robust multi-modal representation learning. Extensive evaluations demonstrate the efficacy of our ProFusion3D.
Limitations: Our method involves mapping from one view to another and thereby, relies on good calibration of sensors. To alleviate this dependency, we will explore learnable methods for alignment between views [54, 55). Moreover, our model currently does not consider efficiency. In the future, we aim develop more advanced framework that uses uncertainty estimation [56] dynamically select the most effective combination of different views and fusion levels, optimizing the balance between performance, efficiincy, and robustness.

---

Acknowledgments
This research was funded by Bosch Research as part of a collaboration between Bosch Research and the University of Freiburg Al-based automated driving.
References
[1] C. Lang, A. Braun, and A. Valada. Robust object detection using knowledge graph embeddings. In DAGM German Conference Pattern Recognition, pages 445-461. Springer, 2022.
[2] C. Lang, A. Braun, L. Schillingmann, and A. Valada. On hyperbolic embeddings in object detection. In DAGM German Conference on Pattern Recognition, pages 462-476. Springer, 2022.
[3] R. Mohan, K. Kumaraswamy. J. V. Hurtado, K. Petek, and A. Valada. Panoptic cut-of-distribution segmentation. IEEE Robotics and Automation Letters, 2024.
[4] S. Vora, A. H. Lang, B. Helou, and O. Beijhom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4604 4612, 2020.
[5] C. Wang, Ma, M. Zhu, and X. Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11794-11980 11803, 2021.
[6] T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y. Wang. T. Tang, B. Wang, and Z. Tang. Bevfusion: A simple and robust lidar-camera fusion framework, Advances in Neural Information Processing Systems, 15411-1114444, 2022.
[7] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1090-1099, 2022.
[8] K. He, X. Chen, S. Xie, Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[9] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9653-96663 9663, 2022.
[10] X. Yu, L. Tang, Y. Rao, T. Huang. Zhou, and Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 19313-19322, 2022.
[11] Y. Pang, W. Wang, E. Tay, W. Liu, Y. Tian, and L. Yuan. Masked autoencoders for point cloud superviie learning. European conference on computer vision, pages 604 621. Springer, 2022.
[12] C. Lang, A. Braun, L. Schillingmann, and A. Valada. A point-based approach to efficient lidar multi-task perception. arXiv preprint arXiv:2404.1279 2024.
[13] A. Chen, K. Zhang, R. Zhang, Z. Wang, Y. Lu, Y. Guo, and S. Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5291-5301 2023.
[14] R. Zhang, L. Wang, Y. Qiao, P. Gao, and H. Li. Learning representations from 2d pre-trained models via image-to toogge--to-oontttt point masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21769-21780 2023.

---

[15] G. Hess, J. Jaxing, E. Svensson, D. Hagerman, C. Petersson, and L. Svensson. Masked autoen- coder for self-supervised pre-training on lidar point clouds. Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 350-359, 2023.
[16] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2019.
[17] C. Reading, A. Harakeh, J. Chae, and S. L. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 8555-8564, 2021.
[18] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision, pages 194-210. Springer, 2020.
[19] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191. PMLR, 2022.
[20] Y. Liu, T. Wang, X. Zhang, and J. Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference Computer Vision, pages 531-548. Springer, 2022.
[21] 7. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and Dai. Bevformer: Learning bird's v-ye representation from multi-camera images via spatiotemporal transformers. In European conference computer vision, pages 1-18. Springer, 2022.
[22] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 4490-4499, 2018
[23] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018.
[24] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pages 12697-12705, 2019.
[25] A. Bewley, Sun, T. Mensink, D. Anguelov, and C. Sminchiseseu. Range conditioned dilated convolutions for scale invariant 3d object detection. In Proceedings of the Conference on Robot Learning, volume 155, pages 627-641. PMLR, 2021.
[26] R. Mohan, T. Elsken, A. Zela, J. H. Metzen, B. Staffler, T. Brox, A. Valada, and F. Hutter. Neural architecture search for dense prediction tasks in computer vision. International Journal of Computer Vision, 131(7):18441887,, 2023.
[27] R. Mohan and A. Valada. Perceiving the invisible: Proposal-free amodal panoptic segmentation. IEEE Robotics and Automation Letters, 7(4):9302-930) 2022.
[28] Y. Li, Y. Chen, X. Qi, Z. Li, J. Sun, and J. Jia. Unifying voxel-based representation with transformer for 3d object detection, Advances in Neural Information Processing Systems, 35: 18442-18455, 2022.
[29] Y. Li, A. W. Yu, T. Meng. Caine, J. Ngiam, D. Peng, Shen, Y. Lu, D. Zhou, Q. V. Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17182-17191,2011 2022.
10

---

[30] Z. Yang, J. Chen. Z. Miao, W. Li, X. Zhu, and L. Zhang. Deepinteraction: 3d object detection via modality interaction, Advances in Neural Information Processing Systems, 35:1992-2005, 2022.
[31] X. Chen, T. Zhang, Y. Wang, Y. Wang, and H. Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 1722181, 2023.
[32] J. Yan, Y. Liu, J. Sun, F. Jia, S. Li, T. Wang, and X. Zhang. Cross modal transformer: Towards fast and robust 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18278, 2023.
[33] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 4171-4188 2019.
[34] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training. OpenAl, 2018. URL http::////ww..eemaiiiccchoooooo/ org/ Saardd-Naaaimmmaaa inarnn
[35] C. Min, L. Xiao, D. Zhao, Y. Nie, and B. Dai. Occupancy-man maa: Self-supervised pre-training large-scale lidar point clouds with masked occupancy autoencoders. IEEE Transactions on Intelligent Vehicles, 2023.
[36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierar- chical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[37] N. Carion, Massa, G. Synnacve, N. Usumier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[38] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics (NRL), 52(1):7-21, 2005.
[39] S. Qiao, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen. Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3997-4008, 2021.
[40] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[41] M. Contributors. Mmdetection3d: Open-mmlab next-generation platform for general 3d object cow/open satten-mmlbble mmlaab/mmettettionn 2020.
detection. https:
[42] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[43] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[44] B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv: 09492, 2019.
[45] L. N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference applications of computer vision (WACV), pages 464-472. IEEE, 2017.
11

---

[46] Y. Chen, J. Liu, X. Zhang, X. Qi. and J. Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21674-21683, 2023.
[47] Y. Li, L. Fan, Y. Liu, Z. Huang, Y. Chen, N. Wang, and Z. Zhang. Fully sparse fusion for 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[48] Huang and G. Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203.1703. 2022.
[49] T. Yin, X. Zhou, and P. Krahenbuhl. Center- based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784-11793, 2021.
[50] L. Fan, F. Wang, N. Wang. and Z.-X. Zhang. Fully sparse 3d object detection. Advances in Neural Information Processing Systems, 35:351-363, 2022.
[51] Y. Chen, J. Liu, X. Zhang, X. Qi, and J. Jia. Largekernel3d: Scaling up kernels in 3d sparse cnns. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13488-13331 13498, 2023.
[52] Huang. Y. Ye. Z. Liang, Y. Shan, and D. Du. Detecting as labeling: Rethinking lidar-camera fusion in 3d object detection. arXiv preprint arXiv:2311.07155 2023.
[53] H. Wang. H. Tang, S. Shi, A. Li, Z. Li, B. Schiele, and Wang. Unitr: A unified and efficient multi-modal transformer for bird's-eye s-eee-view representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6792-6802, 2023.
[54] D. Cattaneo and A. Valada. Cmrnext: Camera to lidar matching in the wild for localization and extrinsic calibration. arXiv preprint arXiv:2402.00129, 2024.
[55] K. Petek, N. Vödisch, J. Meyer, D. Cattaneo, A. Valada, and W. Burgard. Automatic target-less camera-lidar calibration from motion and deep point correspondences. IEEE Robotics and 2024.
Automation Letters, 9(1 11:99))--99999
[56] M. R. Nallapareddy, K. Sirohi, P. L.. Drews, W. Burgard, C.-H. Cheng, and A. Valada. Eveen- ternet: Uncertainty estimation for object detection using evidential learning. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5699-5706 IEEE, 2023.
12

---

Progressive Multi-Modal Fusion for Robust 3D Object Detection Supplementary Material
In this supplementary material, provide additional details on various aspects of our work. First, provide further architecture details for our fusion module and decoder in Sec. 1. Next, we present the loss details of the three pre-training objectives for our proposed LiDAR and Multi-View Camera Mask Modeling in Sec. 2. In Sec. 3 we describe the strategy for selecting the subsets the dataset used in our data efficiency experiment (Sec. 4.2 of the main manuscript). Lastly, we provide additional qualitative results for ProFusion3D Sec. 4 and Sec.
Additional Architectural Details of ProFusion3D
For each of the inter-intra base units, the corresponding channel dimension of the query, key, and value embeddings are set to 192. The following Feed-Forward Network (FFN) consists of two fully-connected layers with GeLU activation after the first one. The first layer expands the input channels 1024 while the second layer condenses back to the original embedding dimensions (192).
Inter-Intra Fusion:
The convolutional block of the fusion module uses two consecutive 3 x 3 depthwise separable convolutions, each with dilation rates and 4, respectively. The number of channels in these convolutions is twice the embedding dimensions (384).
DETR-style decoder layers in the BEV decoder, PV decoder, and Joint Decoder. We follow similar parameter settings described in [1] for each of the decoder layers. The sequence of operations within each layer as follows: self-attention, normalization, cross-attention, normalization, FFN, and normalization. The embedding dimensions are set to 256, and the FFN channels are set to 2048. Both the attention and FFN dropout rates are configured to 0.1. We set the number of queries to 600 for initializing Qo- The initial object queries Qo interact with the 3D position-aware BEV features bbv in the BEV decoder to update their representations to Quev- parallel, Qu also interacts with the 3D position-aware PV features F&D in the PV decoder update their representations to Qpv- Following this, Qjein (Qrev; Qpv] interacts with the 3D position-aware joint features E&3 [1 F3P] in the joint decoder to generate the final updated query representations.
Decoder: We employ
We use two FFNs to predict the 3D bounding boxes and the classes using the updated queries each of the decoder layers. The prediction for each decoder layer then as follows:
8 " (Q.).
p (Q)
(1)
where te and Seh represent the FFNs for regression and classification, respectively, Qf are the updated object queries the i-th decoder layer the d-th decoder, where d E jointt.
We train ProFusion3D through prediction by using bipartite matching for one-to-one assignment between predictions and ground truths. Specifically, use the focal loss for classification and L1 loss for 3D bounding box regression:
L 9) = Le + >g
(2)
where X₁ and d are the hyperparameters to balance the two loss terms.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

2 Loss Functions for Pre-Training Objectives
For the multi-modal masked modeling, we introduce three pre-training objectives: masked token reconstruction, unmasked token denoising, and masked token cross-modal attribute prediction. To train each objective we employ the following losses:
Masked Token Reconstruction: In this pre- training objective, we reconstruct each masked image patch from the PV branch and voxels from the BEV branch. For the image patch reconstruction, we employ LI loss between the predicted values of the masked pixels and the corresponding RGB values follows:
Nmp
LLI
(3)
Nmp
where i, and I; are the predicted and ground truth RGB value of the i-th masked pixel, respectively, and Nmp is the total number of masked pixels.
For the voxel reconstruction, let Pg.: = (21, T2, END be the i-th masked voxel where N the number of fixed points in voxels and Pree, = (ī1,ā2. .. ; be the corresponding reconstruction.
Then the Chamfer loss defined as follows:
LChamfer (PaLi Prec.) =
Σ PEP
f(z, Preci) +
Σ
f(i, Pgii,,
PPtui
|Prci
f(x, =1|x p 12. with = arg min || 121
where || ll2 denotes the L2-norm. This loss function ensures that each point in the ground truth set Pg. is close to some point in the reconstructed set Pree and vice versa.
Unmasked Token Denoising: In this pre-training objective, we learn to predict noise for each unmasked image patch and voxel. For the unmasked image patches, we employ loss between the predicted noise and the actual noise added as follows:
Nup
Ledeniise jinage
Nup
where ú; is the predicted noise and n, is the actual noise added to the i-th unmasked pixel, and Nup the total number of unmasked pixels.
For the voxel denoising, let Na,; (n1, .., nny be the noise added the i-th unmasked voxel and Np (īe, u2 nN) be the corresponding noise prediction.
Ldenoise.voxel (N Np.))
min 1nn- Σ NEN
Σ min || n//3.
(6)
Naiil
nEN,
NNNil
xEN,
ReN,
Masked Token Cross-Modal Attribute Prediction: In this pre-training objective, we predict pixel intensities for points in masked voxels and depth values for masked image patches. To predict pixel intensities for points in masked voxels, do this jointly with the masked voxel reconstruction. Hence, while predicting the points in the masked voxel, we also predict the corresponding pixel intensities. For pixel intensity prediction, we add loss term to the Chamfer distance loss of masked token reconstruction, by replacing f in Eq. (4) with f:
f(x,, P) = || pi 12 + A|r, - P/1, with arg min ||a pk113
where À is the loss balancing term, æy and Pf- the pixel intensities of æ and of the j-th point in P.
Following, for depth prediction for masked image patches, let and denote the ground-truth and the predicted depth, respectively. Our loss function for depth estimation is then defined by

---

2
Nmp
Nmp
T.dopti(d, d)
Σ
log - log di)
2
1
Σ
log d4 log d.)
New
ma
(8)
1
Nmp
dg
di
2
Nmp
di
This loss term is a combination the scale-invariant logarithmic error and the relative squared error.
3 Strategy for Subset Selection in Data Efficiency Experiments
In Sec. 4.2 of our main manuscript, we perform data efficiency experiment that utilizes subsets of 20%, 40%, 60%, and 80% from the 100% training annotated data of nuScenes. To select scenes for each subset of the training dataset, we sorted the dataset based on scene timestamps and then used systematic sampling method. Specifically, we divided the scenes into five groups based on their indices and selected scenes according to these groups:
For a 20% subset, we included all scenes from one group (i.e., those with indices where i mod 5 = -).
For a 40% subset, we included scenes from two groups (i.e., those with indices where i
mod 5 E {0, 2)).
For a 60% subset, we included scenes from three groups (i.c., those with indices where mod 5 E {0, 2)).
For an 80% subset, we included scenes from four groups (i.e., those with indices where mod 5 E {0, 2,2,
This systematic sampling method helps to minimize temporal dependencies between frames and ensures that the reduced datasets retain a similar level of diversity as the complete dataset.
Visualization
In Fig. provide additional qualitative results of our proposed ProFusion3D on the nuScenes dataset. The dataset encompasses urban road scenes with objects ranging from cars, trucks, bicycles, motorbikes, people, barriers, and cones in very cluttered environment with occlusions. Despite these challenging conditions, our ProFusion3D consistently and accurately detects all these objects.
Generalization in Real-World Scenarios
In this experiment, perform a real-world zero-shot generalization experiment evaluate the performance of our proposed ProFusion3D model using our in-house autonomous vehicle depicted in Fig. 2. We evaluate the multi-modal ProFusion3D model under the assumption that camera sensor fails during operation, meaning that the model operated without camera images. For this experiment, we utilized the model trained on the nuScenes dataset. It is important to note that our setup includes an Ouster 128 LiDAR. while nuScenes uses a Velodyne HDL32E. Thus, this experiment introduces additional challenges for object detection due to variations in sensor suites, domain differences, and the absence of camera input. Fig. presents qualitative results from this experiment. Despite these challenges, proposed ProFusion3D model demonstrates promising results.

---

cm
0
(d)
MRREE
1100
(c)
Figure 1: Visualization of 3D object detection prediction of our proposed ProFusion3D on the validation set of nuScenes. Classes are color-coded follows:
barrier, truck,
cone, bicycle, person.

---

CM1
Figure 2: Illustration our in-house autonomous driving vehicle used to demonstrate the robustness of ProFusion3D architecture on real-world scenes
References
[1] Y. Liu, T. Wang, X. Zhang, and J. Sun. Petr: Position embedding transformation for multi-view
3d object detection. In European Conference on Computer Vision, pages 531-548. Springer, 2022.
5

---

(a)
(b)
(d)
Figure 3: Visualization of 3D object detection predictions by our proposed ProFusion3D on real-world scenes, demonstrating its robustness under challenging conditions such as variations in sensor suites, domain differences. and the absence of camera images.

---

3D Position
Embedding
BEV
LiDAR
Inter-Intra Fusion (IIF)
Encoder
bey
BEV
Decoder
View Mapping
Query Generator
Joint Decoder
class
bbox
Camera Encoder
F
MAEv PM
Inter-Intra Fusion (IIF)
PV Decoder
PV
3D Position
Embedding
Figure Illustration our proposed ProFusion3D architecture that employs progressive fusion.


Reviews:
[{"review_id": "5YBzW8saSr", "reviewer": "Reviewer_9BjW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces ProFusion3D, a novel framework for LiDAR-camera fusion in 3D object detection for automated driving, which employs a progressive fusion approach across multiple stages and views.", "summary_of_recommendation": "Even though the performance increase on top of SOTA methods is not significant, the novelty of this paper, especially multi-modal mask modeling, lead to my recommendation of this paper", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "* For the data efficiency experiment in section 4.2, it's not very cleary that the model are 1. pre-trained on subset of training data and finetuned on subset  or 2. pre-trained on full set of training data and and finetuned on subset. \n* It would be interesting to see how the performance change if the model is pretrained on single modelity", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAaocoTBij", "reviewer": "Reviewer_anjQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces ProFusion3D, a novel approach for 3D object detection using progressive fusion of LiDAR and camera data for autonomous driving. The method fuses features in both Bird's Eye View and Perspective View spaces at multiple levels, introduces a new inter-intra fusion module, and proposes a self-supervised pre-training strategy. The approach achieves state-of-the-art results on nuScenes and Argoverse2 datasets.", "summary_of_recommendation": "ProFusion3D presents innovative ideas for multi-modal fusion in 3D object detection and demonstrates impressive results on automotive datasets. The progressive fusion approach, inter-intra fusion module, and self-supervised pre-training strategy are valuable contributions. However, the paper would be strengthened by addressing real-time performance, including real-world testing results, and exploring temporal aspects. I recommend minor revisions to address these points, after which the paper would make a strong contribution to the field of perception in autonomous driving.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. What is ProFusion3D's computational complexity and real-time performance compared to existing methods?\n2. Have you conducted tests on actual autonomous vehicles? If so, how do the results compare to dataset evaluations?\n3. How might ProFusion3D be extended to leverage temporal information?\n4. How does the improved detection performance translate to safer decision-making?\n5. How well does ProFusion3D generalize to driving conditions not represented in the datasets?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrC1oj7t6c", "reviewer": "Reviewer_nzZg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces ProFusion3D, a framework that performs progressive LiDAR-camera sensor fusion. The model fuses features at two stages: the intermediate feature level and the object query level. Fusion occurs across both Bird’s Eye View (BEV) and Perspective View (PV), enabling the model to leverage both local and global features effectively.", "summary_of_recommendation": "This is a good paper, with decent contributions and the paper is well-written. Because I'm not super up to date with SOT in the field, I am recommending weak accept, but happy to revise based on other reviewers recommendations.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can you elaborate on the process of generating 3D positional embeddings from the camera and BEV meshgrids?\n \n- How does the Hungarian algorithm contribute to the matching process between predicted and ground truth objects during training?\n\n- Could you highlight the difference between ProFusion3D and DeepFusion referenced above? Since its also fusing LiDAR and Camera features.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- a novel idea;
- detailed ablations;
- clear presentation.

Weaknesses:
- limited discussion on real-world applications;
- marginal performance gain;
- some missing technical details.

Post-rebuttal: The authors did an impressive job during the rebuttal, with a significant number of additional experimental results.  All three reviewers recommended acceptance, and the AC agreed.  The authors are encouraged to include the content from the rebuttal in the camera-ready version.
Decision: Accept