RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection
Authors: Jingtong Yue
Zhiwei Lin
Xin Lin
Xiaoyu Zhou
Xiangtai Li
Lu Qi
Yongtao Wang
Ming-Hsuan Yang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Vision， Radar Camera 3D Object Detection
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: While recent low-cost radar-camera approaches have shown promising results in
multi-modal 3D object detection, both sensors face challenges from environmen-
tal and intrinsic disturbances. Poor lighting or adverse weather conditions de-
grade camera performance, while radar suffers from noise and positional ambigu-
ity. Achieving robust radar-camera 3D object detection requires consistent perfor-
mance across varying conditions, a topic that has not yet been fully explored. In
this work, we first conduct a systematic analysis of robustness in radar-camera de-
tection on five kinds of noises and propose RobuRCDet, a robust object detection
model in bird’s eye view (BEV). Specifically, we design a 3D Gaussian Expan-
sion (3DGE) module to mitigate inaccuracies in radar points, including position,
Radar Cross-Section (RCS), and velocity. The 3DGE uses RCS and velocity priors
to generate a deformable kernel map and variance for kernel size adjustment and
value distribution. Additionally, we introduce a weather-adaptive fusion module,
which adaptively fuses radar and camera features based on camera signal confi-
dence. Extensive experiments on the popular benchmark, nuScenes, show that
our RobuRCDet achieves competitive results in regular and noisy conditions. The
source codes and trained models will be made available.

Published as a conference paper at ICLR 2025
ROBURCDET: ENHANCING ROBUSTNESS OF RADAR- CAMERA FUSION IN BIRD'S EYE VIEW FOR 3D OB- JECT DETECTION
Jingtong Yue
Zhiwei Lin-*
Xin Lin
Xiaoyu Zhou'
Xiangtai Li!
Lu Qi
Yongtao Wang'
Ming-Hsuan Yang
Peeinng University Sichuan University
3 UC Merced
IIsta3600 Research
ABSTRACT
While recent low-cost radar-camera approaches have shown promising results in multi-modal 3D object detection, both sensors face challenges from envi- ronmental and intrinsic disturbances. Poor lighting adverse weather condi- tions degrade camera performance, while radar suffers from noise and positional ambiguity. Achieving robust radar-camera 3D object detection requires consis- tent performance across varying conditions, a topic that has not yet been fully explored. In this work, we first conduct a systematic analysis of robustness in radar-camera detection on five kinds of noises and propose RobuRCDet, a robust object detection model bird's eye view (BEV), Specifically, we de- sign a 3D Gaussian Expansion (3DGE) module to mitigate inaccuracies in radar points, including position, Radar Pross-Section (RCS), and velocity. The 3DGE uses RCS and velocity priors to generate deformable kernel map and vari- ance for kernel size adjustment and value distribution. Additionally, we intro- duce a weather-adaptivve fusion module, which adaptively fuses radar and cam- era features based on camera signal confidence. Extensive experiments on the popular benchmark, nuScenes, show that our RobuRCDet achieves competitive results in regular and noisy conditions. The source code will be released at https: //tp:///gttttt// com Jingtong0527 RobuRCDet.
INTRODUCTION
Multi-modal 3D object detection crucial computer vision, as leverages the complementary signals captured by cameras and 3D sensors. Due to its accurate 3D depth information and robust- ness, radar has emerged as a promising and cost-effective 3D signal, benefiting applications such as autonomous driving (Qi et al., 2019; Liu et al., 2024) and robotic navigation (Amoold et al., 2019; Wan et al., 2024). Despite the significant success of previous radar-camera 3D object detection methods (Kim et al., 2024; Lin al., 2024: Zhou et al.. 2023; Jiang al., 2024), they often neglect the importance of model robustness, which limits the practical applicability of these methods.
Several works (Zhou et al., 2023; Kim al., 2023b; Lin et al., 2024) have designed practical feature encoders and multi-modal fusion modules to enhance model robustness in challenging scenarios, such as fewer input sweeps, single sensor failure, extended radar perception range. However, they usually neglect the interference caused by adverse weather and lighting conditions on camera signals, the impact of internal and external factors on radar signals, and the cooperative effects between the two sensors, as shown in Figure 1.
This motivates us to systematically analyze various types of noise in radar-camera 3D object de- tection and then propose robust method to counteract interference. However, there are very few datasets that encompass all possible scenarios. Take Radiate dataset (Sheeny et al., 2021) as an ex- ample; its camera data is limited to two views (left and right) and is mainly provided as radar images rather than point clouds. Additionally, the partial point clouds in the dataset only contain æ, y, and
equal contribution. Correspondence Xiangtai

---

Published as a conference paper at ICLR 2025
Pale Poeltiv
Reley
Sunmy
25
30
19.2
44.8
distansc (rs)
Compamsion point positive rainy and surnyy days.
Comparasion image visihiity rainy sunny days.
Figure 1: Illustration of radar and camera noise sunny and rainy days. Radar noise increases with distance from the radar sensor and is greater in rainy weather.
intensity dimensions, lacking many key characteristics of radar points, such Radar Cross-Section (RCS) and Doppler speed.
To address the lack of radar-camera corruption datasets, first simulate radar corruptions on the widely used and large-scale multi-modal dataset, nuScenes (Caesar et al., 2020). In particular, we focus on the graphic characteristics of corruption instead of the natural causes of corruption, explor- ing the optimal classification method for different noise patterns rather than being preoccupied with their causes. Our method can reduce overlaps between categories. For example, ground reflections or reflections caused by rainy or snowy weather, which are obviously different causes, may all result radar echo disappearance, They fall into our first category of factors. As long as we can address the noise with the same pattern under all scenarios, the exact cause of the noise becomes less criti- cal. This is because different interference conditions, such as multi-path effects and reflections, may change the distribution of radar point clouds in the same way, i.e., point loss or the generation of false detection points. Additionally, building different types of noise distributions is more practical than using one specific noise source to enhance the model's robustness. Specifically, we consider four distinct noise patterns often occurring in radar sensor deployment and autonomous driving sys- tems: 1) Key-point missing, which manifests as the loss of radar points related or unrelated the target. 2) Spurious points, which refers to the condition with false- false-posite positive radar points. 3) Point shifting, representing radar points with deviations in the x, y, and 3-axis due to interference, and 4) Non-positional disturbance, referring to the situation where the position of radar points remains unchanged, but other characteristics such as RCS and Doppler speed deviate.
To address these issues, we introduce robust 3D object detection framework named RobuRCDet, containing two critical designs for the robustness of both radar and camera signals. In this work, we propose a 3D Gaussian Expansion (3DGE) module to filter spatially inaccurate radar points through point expansion in the voxel field. We analyze the distribution pattern differences between the noisy and target point clouds. As shown in Figure 2, the noisy point is more randomly and sparsely dis- tributed in space. We thus leverage the sparsity to sum all radar voxels and conduct normalization to enhance the dense part and reduce the sparse area according to the amplitude difference. Moreover, we design a Confidence-guided Multi-modal Cross-Attention (CMCA) module to enhance camera robustness by dynamically evaluating camera signal confidence. Since the confidence of the cam- era and radar signal varies significantly in different conditions, for instance, in adverse weather, the radar is much more robust than the camera. We exploit CMCA learn reliable and accurate radar features from raw signals in proper conditions. Additionally, since the map is learned to have high camera signal confidence on high-quality camera images such as images on sunny days, it can preserve the original performance on clean data. With this module, we preserve the original per- formance on clean data while ensuring robustness against noise. We conduct extensive experiments on original and augmented data and demonstrate the effectiveness and robustness of our method, especially under interfering conditions.
The main contributions of our works are follows:
To the best of our knowledge, we are the first to conduct systematic analysis on the bustness of radar-camera 3D object detection. We summarize the radar corruptions and establish benchmark by simulating radar noises for robust 3D object detection evaluation.
2

---

Published as a conference paper at ICLR 2025
We propose robust 3D object detector, RobuRCDet, to perform robust 3D object detection in various noise conditions. We design a 3D Gaussian Expansion module to highlight the key points and reduce the impacts from noisy points and confidence-guidee Multi-modal Cross-Attention module to learn the robust multi-modal fusion.
Extensive experimental results on nuScenes have shown the effectiveness of the proposed method. Our method achieves 19.4% improvement in NDS and 25.7% improvement in mAP under conditions of simultancous radar signal interference and camera signal inter- ference, compared to the baseline with only radar backbone and camera backbone.
2 RELATED WORK
Camera-based 3D Object Detection. The success of 2D object detection (Zhi al., 2019; Xingyi et al., 2019), with the growing demand for 3D perception in fields like autonomous driving and robotics, prompts the development of 3D object detection technology. Early works (Li et al., 2023; Huang et al., 2021; Huang & Huang, 2022) are based on multi-view cameras which leverage multi- view information through cross-view interaction to improve 3D object detection performance. The multi-view 3D object detection methods can be briefly divided into two categories, i.e., dense BEV- based and sparse query- based methods.
Numerous dense BEV-based methods adopt Lift-Splat-Shoot (LSS) (Philion & Fidler, 2020) to transform 2D features into BEV features, such as BEVDet (Huang et al., 2021). On the other hand, BEVDepth (Li et al., 2023) designs trustworthy depth estimation module for better view transfor- mation the BEV space. For sparse query-based methods, PERT series (Liu al., 2022; 2023a; Wang et al., 2023a;b) incorporate the position information of 3D coordinates into image features and integrate long-term temporal fusion.
While these methods achieve advanced 3D object detection performance, they overlook robustness, a key factor in real-world applications. Unlike millimeter-wave radar, camera images are prone to interference in darkness and bad weather, leading to poor detection performance. To this end, RobuRCDet incorporates the camera signal confidence map to effectively enhance network robust- ness along with radar modality under various conditions.
Radar-Camert 3D Object Detection. The camera sensor inherently lacks 3D depth information, limiting its 3D detection accuracy. To alleviate this issue, researchers propose incorporating the cost-effective radar sensor into the 3D detection framework. The radar sensor provides the 3D depth prior and additional Doppler velocity, compensating for the camera sensor's weaknesses.
Specifically, CenterFusion (Nabati & Qi, 2021) uses key point detection network to obtain center points and then associates key points with the corresponding radar detection results in a pillar-based manner. After that, CRAFT (Kim et al., 2023a) further considers the spatial properties of radar and camera sensors and designs a proposal-Ieve early fusion framework. RCBEV (Zhou et al., 2023) introduces the feature- level fusion in the BEV space for unified feature representation. Meanwhile, RCM-Fusion (Kim et al., 2024) is proposed to combine radar and camera features at both the feature and instance levels, further improving the detection performance. More recently, CRN (Kim al., 2023b) transforms PV image features to BEV with radar occupancy to compensate for the depth information in images. RCBEVDet (Lin al., 2024) specifically customizes a feature extractor for radar and uses RCS as the object size prior. It further designs a Cross- Multi-layer Fusion feature alignment and fusion.
module for robust
By contrast, focusing on the framework robustness, our RobuRCDet proposes a 3DGE module to decrease the impact of potential noisy points in the radar voxels.
Robust 3D Object Detection. Sensor noise one of the most significant factors causing the de- crease in detection performance during inference for 3D object detection. Several methods (Kong et al., 2023a;b; Xie et al., 2023; Kong et al., 2023c) attempt to benchmark the common corruptions in 3D perception tasks from different angles. For instance, RoboDepth (Kong et al., 2023c; Ren et al., 2022) sets up benchmark to assess the robustness monocular depth estimation in the presence of comuptions. On the other hand, RoboBEV (Xie et al., 2023) presents extensive benchmark aimed at evaluating the robustness across four BEV perception tasks, i.e., 3D object detection (Liu al., 2023b; Liang al., 2022) and semantic segmentation (Zhou & Krähenbühl, 2022). At the
3

---

Published as a conference paper at ICLR 2025
(a) Key-point Missing
(b) Spurious Points (random)
(c) Spurious Points (related)
(d) Point Shifting
Figure 2: Point cloud visualization of radar signals with noise. The blue points refer to the ground truth radar points, and the red points represent noisy radar points in various conditions. Additionally, the light blue points in the key-point missing part denote the eliminated ground truth radar points.
same time, Robo3D (Kong et al., 2023b) evaluates the resilience of 3D detectors and segmenters when exposed to LiDAR-related corruptions. However, these benchmarks mostly focus on camera or lidar perceptions, but the radar corruptions are almost ignored.
Furthermore, the efforts (Jiang et al., 2024: Wu al., 2024) are made address the above menttoed corruptions and achieve robust 3D object detection under noisy conditions. However, these methods only model partial radar noise types. Further, they only consider radar or LiDAR degradation sce- narios and overlook camera failure cases. In contrast, our RoboRCDet addresses these issues and designs a module for the robust fusion of radar and camera features in the BEV view.
3 CORRUPTION TAXONOMY
Given the challenges of collecting real-world corruption data, we generate our training and valida- tion dataset by synthesizing noise for radar and image signals. Since many existing methods focus on the robustness of image data, we focus more on exploring four noise types in radar signals: key-point missing, spurious points, point shifting, and non-positional disturbance.
3.1 RADAR SIGNAL
For clear illustration, a radar point p E I55 with coordinates (Ips Up εp), Radar -ecsi-n (RCS), and Doppler Speed (v).
Key-point missing. The sparsity of radar point clouds poses a significant challenge to 3D detection. Interferences, such reflections from lost radar beams, worsen this issue, further increasing the sparsity of the point cloud. In our setting, we simulate key-point missing in two scenarios. Specifically, based on the existing clean radar points p. we either randomly or selectively remove points (p) across the entire set or the region of the target object, which are represented by = and 7
Pr. - R])) M1,
where R denotes the random process to delete points from The k is the number of points we should delete, ranging from 1 to M.The M is set to half of the number of p or eight when y is or 1. Finally, the kept final radar points defined as Pn-
Spurious points. Due to intrinsic issues, noisy environments, or even artificial interference, spurious points appear alongside the original radar point clouds. Similar missing key points, spurious points are categorized into two types. The first type consists of noisy points superimposed on the original radar points, correlated with their positions, and randomly distributed around them. The second type contains completely random points originating from complex external environments and unrelated to the target point cloud.
Pn PUpp' D' N(6,o), a~ U(1,0))
4

---

Published as a conference paper at ICLR 2025
Clean
Rainy
Snowy
Foggy
Low-light
Figure 3: Visualization of image signals under adverse weather conditions.
8 pl Up. Ep, RCS, v), point related,
R(Ip Up, Zp, RCS, v), random,
where p' is the added noisy points. As shown in equation 2, the selection of p' follows the normal distribution, where is defined in equation Under the first circumstance, is set to p. while the is decided by random process R in the second situation.
Point Shifting. We refer to point shifting as the misalignment of 3D information where radar points deviate from their original locations. To simulate this corruption, we apply distortions directly to the radar points using a normal distribution N(0, o).
Pn. p + Ap,
where Ap o(0 and (1,5
Non-positional disturbance. External interference can also affect the values of RCS and v, rather than introduce noise to the spatial coordinates. Although this scenario is less common than the previous three cases, we include it in our benchmark for completeness and refer to it as non- positional disturbance.
As shown in equation 5, only the RCS and v dimensions are disturbed in the normal distribution manner, and the values of and are consistent.
Pr Fp Up: Zp. RCS ARCS, + An),
(5)
where ARCS ~ N(O, o), Au ~ o),, and ~ U(1,50), Visualization of the point cloud is shown in Figure 2.
3.2 IMAGE SIGNAL
We simulate images under adverse weather conditions, including rainy, snowy, foggy, and low light, according to the simulation methods in basic low-level tasks. The visual results of the synthetic degraded images illustrated in Figure 3. 1) Adverse weather. We leverage the composition method in (Han et al., 2022) to synthesize the images in various conditions. The rain, snow, and fog maps provided, and synthesize the normalized clean images. 2) Low light. The low-light images are simulated through a classical gamma correction algorithm, where the gamma factor is randomly elected.
METHODOLOGY
Different from the concurrent works (Nabati & Qi, 2021; Zhou al., 2023; Kim et al., 2023b), propose RobuRCDet that focuses more on robust 3D object detection. As shown in Figure 4, our framework includes two separate branches for processing the image and radar point clouds and fusion module. In the following subsections, we will overview the whole pipeline of RobuRCDet. Then, the proposed two modules will be elaborately described.
4.1 OVERVIEW OF ROBURCDET
As shown in Figure 4, first pass the radar point cloud through voxelization process. Then, 3DGE applied to spread the RCS and Doppler speed dimensions to surrounding voxels according to Gaussian distribution. The expanded radar voxels and the original radar voxels then fed into
5

---

Published as a conference paper at ICLR 2025
Image Backbone
3D Detection Head
BEV Pooling
Canfidence-gul guneeeeeggggeeeeeeeeeeeee Mutti-modal Cros Afteation
Image Feature(PV)
Image Festure(BEV)
Multi-view Imanges
Radur Camera Encoder
Rodar Feature(BEV)
Radar Backbone
BEV Pooling
Radar Encoder
Rader Voxel
Share
Rodor Point Clond
Radar knceder
Radar Fenture(PV)
3DGE
Figure 4: Overall pipeline of the proposed RobuRCDet. First, we extract the image features from multi-views and transform them into BEV space. Concurrently, 3DGE employed on the radar voxels, and we put the original voxels and expanded voxels into the Radar Encoder to obtain radar features after summation. Finally, we fuse the image and radar features in the confidence oonidence-guided multi-modal cross cross-attention in the BEV space for 3D object detection.
radar encoder with shared weights, resulting in the radar feature after feature summation. Next, both multi-view images and the radar point cloud are fed into the image backbone to extract image features guided by radar information. After transforming the radar and image features into the BEV space, CMCA is applied to fuse the features from both modalities. Finally, the fused features are used for 3D object detection tasks.
4.2 3D GAUSSIAN EXPANDING
RCS
To mitigate the impact of noisy radar points, we introduce the 3D Gaussian Expanding (3DGE) module to filter radar points in the voxel space. illustrated in Figure 2, radar point distri- bution typically follows a pattern: points are dense within the target range, while false pos- itive points are usually sparse. Here, pro- posed 3DGE module leverages this semantic information in radar density, enhancing key points and suppressing false positives to handle extensive noise from radar corruption.
Deformable Kernel Map
EX
Figure 5: Illustration of the 3D Gaussian Ex- panding. first utilizes a projector to learn the deformable kernel map and from the RCS and velocity prior. After that, the two parameters are used conduct expansion each radar voxel.
We note that even if interference causes key points become sparser, radar points in the tar- get region remain denser than in false positive areas, shown in Figure 2. This is because false detection points exhibit strong randomness and are generally isolated or small clusters of scat- tered points. In contrast, target objects have certain area, and multiple radar beams are typically reflected from this area. As long as the interference does not cause all beams from the same target object to vanish completely, the resulting point cloud of high probability be denser than that of the false detection interference. By applying 3DGE, these sparse key points complement each other within the voxel, helping to restore the target area as much as possible and thereby maintaining the robustness of the radar branch.
Figure illustrates the details of the proposed 3DGE module. First, we input the RCS and velocity information into parameter encoder (Proj), generating the deformable kemnel map and determining the variance of the Gaussian kernel. Next, we apply 3D Gaussian expansion to each radar point. Specifically, the RCS and velocity values are spread the surrounding voxels of each radar point,

---

Published as a conference paper at ICLR 2025
with the spreading range determined by the kernel size Ap provided by the deformable kernel map. For instance, if Ap is 3, the RCS and velocity are expanded into a x 3 x 3 region according to the normal distribution. To balance efficiency and accuracy, expansion, the RCS and velocity values are summed within each voxel, followed by normalization to restore the values to their original range, as described by the following equation:
restrict that Ap 1,3,5. After
(
my)
|x Fpl E Ap:
V(RCS, V (2. 2,, RCS, e) 2
exp
20
ly
Up
App
(6)
otherwise,
where 2 x X o² is the coefficient of the Gaussian function, V represents the radar voxel, Tp and Vp are x-coordinate and y-coordinate of the radar point, and we perform the expansion on the RCS and velocity dimensions. Next, normalization is applied in each expanded space to maintain the summation of the expanding kernel and prepare the voxel for feature extraction. Finally, the 3DGE result V3DGE (ar, U. 2, RCS, u) combines the original radar voxel in res-block manner.
4.3
CONFIDENCE-GUIDEER MULTI-MODAL
In this section, we intro- duce Confidence-Guide Multi-Modal Cross-Attention (CMCA) module to address the challenges of low performance in degraded camera signals under adverse weather condi- tions and low-light scenarios. Radar demonstrates superior robustness compared to camera signals in many challenging sce- narios, such as rainy and foggy days with very low visibility. As discussed in Section 1, in these situations, the confidence in radar signals is higher than that of camera signals. Therefore, using the fixed fusion method in clear weather is unreasonable. We need an adaptive approach to adjust for both scenarios where the sensor confidences are similar (clear weather, i.e., sunny) and where there is a significant confidence disparity (low visibility conditions, i.e., foggy, rainy, snowy).
I-M
KY
Aggregation
Deform
Degnddttion sware head
Deform
fuE
M,
KY
Figure 6: Architecture the Confidence-guided Multi-modal Cross-Attention module. It considers the signal confidence of the camera and adaptively fuses image features and radar features to maintain robustness in various conditions.
To ensure the inference speed of the model and reduce training time costs, we do not apply specific evaluation or constraint mechanisms, such prompts, loss functions, or image quality assessment methods, to the CMCA module. Instead, we utilize the existing nighttime and rainy scenes in the nuScenes training dataset to guide the degradation-awan head in dynamically learning optimal performance strategies. As shown in Figure 6, we leverage the image feature fi as the input of the degradation aware head to generate the camera signal confidence map M. follows:
Mc- Softmax (MLmmmxxXMM(()))..
which is decided by the degradation level of the camera signal. Next, we multiply Si by Ma to get fÿ and multiply fp by 1 Me to get fü. allowing the information from Mc to guide the adaptive fusion of radar and image features thoroughly. Next, we concatenate fi and fp. then conduct aggregation to obtain fa Q while obtaining fim
fA W (CConaat( LN (F),
(8)
fmm (N x LN) (M) x fp)).
where W indicates the linear projection, LN refers to the LayerNorm and Concat refers to the concatenation process.
This adaptive adjustment mechanism calculates the confidence of the signals for each scene. In adverse weather conditions, the map will generate low confidence, guiding the fuser to integrate radar features while deeply minimizing interference from image signals. In addition, since the detection accuracy of radar camera-bar based methods primarily comes from the camera, we rely more on the camera signal when it has high confidence, with the radar signal serving adaptive support.

---

Published as a conference paper at ICLR 2025
Table 1: 3D Object Detection on nuScenes val set. 'C' and 'R' represent camera and radar, respectively. Some results borrowed from RCBEVDet (Lin al., 2024).
Methods
Input
Backhone
Image Size
NDST
mAPt
mATES
mASEt
mADET
mAVEL
mAAET
ConterFusion
C+R
DLA34
400
45.3
33.2
0.649
0.263
0.535
0,540
0.142
CRAFT
448
51.7
RCBEV4d
C+R
Swin-T
256x 704
49.7
38.1
0,526
0.272
0.445
0,465
0.185
CRN
C+R
RI8
206
54.2
44.9
0.518
0.283
0.552
0.279
0.180
RCBEVDet
C+R
R18
256 704
54.8
42.9
0.502
0.291
0.432
0.210
0.178
RobuRCDet
C+R
R18
256x 704
55.0
45.5
0,516
0.287
0.521
0
0.184
BEVDet
R50
206
39.2
31.2
0.691
0.272
0.523
0.909
0.247
BEVDeptt
C
R50
256 704
47.5
35.1
0.639
0.267
0.479
0.428
0.198
SOLOFusion
C
R50
256 704
53.4
42,7
0.567
0.274
0.411
0.252
0.188
StreamPETR
54.0
CRN
C+R
R50
256 704
56.0
49.0
0.487
0.277
0.542
0.344
0.197
RCBEVDet
C+R
R50
256x
56.8
45.3
0.486
0.285
0.404
0.220
0.192
RehuRCDet
C+R
R50
x56
56.7
51.2
0.481
0.273
0.499
0.317
0.193
Finally, we apply the multi-scale deformable cross-attention (Deform CA) to generate the BEV feature. For the original feature fusion, we concatenate fr and fp to be the key and Value, while for the confidence-award feature fusion, we concatenate ff and fü to be the key and Value. Then, the summation and convolution are conducted the outputs of Deform CA modules to form the BEV feature fBEV. The overall equation can be depicted as:
fBEV = Conv(Deform CA(JA, (oncat(f1, Deform CA(JAs
(9)
where fA refers to the features obtained from the sparse aggregation module. Through CMCA, we invite adaptability into the radar feature and image feature fusion pro- cess, which maintains the robustness of camera signals in multiple conditions.
Table 2: Corruption results on nuScenes val set. Cl to C3 represent the Spurious Points, Non-positional Distur- bance, and Key-point Missing, respectively. In addition, for the first two corruptions, the level refers to a while the level of C3 refers to the number of missing beams.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Commption
CRN
RCBEVDet
RobeRCDer
Datasets and Evaluation Metrics.
Type
level
NDS1
mAP
NDS1
mAPt
NDS1
mAPt
We train and evaluate our method on the widely used benchmark nuScenes (Caesar et al., 2020). We add the simulated radar and camera corruption to form the noisy dataset on nuScenes. We use the official metrics for the 3D object detection task, including NDS and mAP.
44.6
39.0
47.4
39.3
47.0
41.2
C1
39.2
36.0
44.2
38.9
44.6
40.5
37.3
35.4
41.7
39.6
42.2
40.6
C2
36.5
32.7
10
50.1
41.9
52.4
42.7
52.7
43.8
40.5
Rain
42.1
31.2
45.6
32.9
45.9
33.6
Fog
51.9
Snow
41.6
30.1
44.1
31.7
44.7
32.8
Implementation Details. For the camera stream, we adopt the image encoder in CRN (Kim et al., 2023b) with several modifications: That is, we add a confidence map projector to extract confidence map from image feature the BEV space. For the radar, we accu- mulate eight previous radar sweeps and use normalized RCS and Doppler speed as input features following GRIF (Kim et al., 2020) Net and CRN (Kim et al., 2023b). Our model is trained for 24 epochs with AdamW W (Loshchilov, 2017) optimizer. We apply image and BEV data augmenta- tion (Li et al., 2023) to prevent overfitting. In addition, we randomly drop sweeps and points for radar input following (Leng et al., 2023).
Night
42.1
35.0
5.2 MAIN RESULTS
Clean Results. We compare RobuRCDet with previous state-of-the-an 3D detection methods on the val set, shown in Table 1. The results show that RobuRCDet achieves competitive perfor-
8

---

Published as a conference paper at ICLR 2025
Key-point
NDS
NDS
4-135
A-125
34
33,00
CRN
31.00
31
-5-1.33
30.4
CRN
66
RebuRCDe
posiiiaall
(5) Comparasion CRN Rob degradatio types
Comparasion of and
Shifiing with
Figure 7: Validation various corruptions. All the metrics tested models with ResNet- 50 backbone. In figure (a), 'Clean' and 'Rainy' are tested on the real-world dataset, while the radar corruptions are augmented with sigma 1. In figure (b), the A refers the RobuRCDet's performance growth over CRN.
Table Ablation of the main components of RobuRCDet. We progressively integrate compo- nents into BEVDepth (Li et al., 2023) and the PointPillar encoder, forming RobuRCDet. Addition- ally, IB and RB refer to the Image Backbone and Radar Backbone, respectively.
Normal Condition
Comupttin Condition
IB
RB
3DGE
CMCA
ND81
mAPt
mATB)
mAP (Carit
NDS
mAPt
mATE)
mAP (Carif
43,9
0.716
54.3
42.4
0.536
68.4
28.5
23.9
0.709
39.5
54.910.6
46.1
71.533.1
29.415.5
0.677,0.032
47.3-7.8
55.210.9
45.813.4
0.501_0.005
70.7-2.3
33.1
28.6/1.7
0.681,0.028
46.7/7.2
55.010.7
45.513.1
0.516_0.020
70.7
34.15.66
30,0776.14
0.635[0.074
48.7'9.2
mance compared to previous methods. Specifically. with ResNet-18 18 as the image backbone, RobuR- CDet increases mAP by 2.6 and 0.6 compared to RCBEVDet and CRN, respectively. Notably, for the ResNet-50 backbone, method surpasses CRN and RCBEVDet mAP by 4.5% and 13.0%, respectively, showing the effectiveness RobuRCDet on detection tasks.
Corruption Results. In Table 2, illustrate the performance of RobuRCDet and another two state-of-the-ae models CRN and RCBEVDet on various augmented corruptions. Specifically, we achieve 44.7 NDS and 32.8 mAP snowy test sets, surpassing CRN by 3.1 NDS and 2.7 mAP.
In addition, as shown in Figure 7, we illustrate the comparison CRN and our proposed method on real-scenario data and each radar corruption mentioned in Section 3.1. Figure (a) shows that our method performs more robustly in all scenes than CRN. Furthermore, in Figure 7 (b), CRN has 5.24 NDS performance drop from a 2 to while RobuRCDet merely decreases 4.58 NDS, which improves robustness by 12.6%.
5.3 ABLATION STUDIES
We perform ablation studies on the nuScenes val set to evaluate the effectiveness of each configu- ration of RobuRCDet. The baseline model uses RobuRCDet with ResNet-18 backbone, an image size of 256x704, and a BEV size of 128x128.
Main Components. As shown in Table 3. we integrated 3DGE and CMCA into the baseline to enhance the detector's robustness. All results are obtained from models trained on the clean dataset, and we evaluate them on both the clean validation set (referred as the Normal Condition) and the synthesized noisy validation set (referred to the Corruption Condition). Under Normal Condi- tions, notable that 3DGE and CMCA improve NDS by 0.6 and 0.9, respectively, with 3DGE achieving 3.1 increase in mAP for cars.
Furthermore, under the Corruption Condition, 3DGE improves NDS by 17.67% and mAP by 22.82%. which indicates that 3DGE achieve favorable performance under strong radar inter- ference. Additionally, the higher performance increase over that under Normal Conditions demon- strates the robustness of the proposed components.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation of 3DGE. Uniform refers the kernel size remaining fixed during expansion, while Ada indicates an adaptively sized kernel map. Additionally, wxyz signifies that the input to the parameter encoder includes RCS and along with æ, y, and
Clean
Key-point Missing
Method
NDS
mAPt
mAOE!
mAP (Car)t
NDS1
mAPt
mAOE
mAP
Baseline
53.7
44.2
0.563
70.1
53.6
44.0
0.563
70.1
+uniform 3DGF
52.9
44.0
0.551
70.1
51.4
43.1
0.562
68.6
+Ada 3DGE (wxyz)
50.7
42.7
0.607
49.5
41.9
0.622
68.5
3DGE (ours)
54.8
45.5
0.523
70.7
54.7
45.3
0.522
70.5
Table 5: Validation on real word interference. We selected data from the nuScenes dataset under challenging lighting and weather conditions to validate the effectiveness of RobuRCDet in real- world scenarios.
Night
Rainy
Method
input
NDS1
mAPt
mAP (Car)f
NDS1
mAPt
mAP (Carjf
CRN (Kim et al., 2023b)
C+R
33.3
25.2
73.0
56.1
47.3
76.3
CRN+CMCA (Kim al., 2023b)
C+R
33.610.3
25.910.7
73.110.1
57.511.4
48.010.7
76.710.4
RCBEVDet (Lin al., 2024)
C+R
34.4
25.3
73.8
59.4
47.1
76.9
RobuRCDet (ours)
C+R
35.5
28.2
73.4
58.4
49.2
77.8
3DGE module. In Table 4, we conduct the ablation experiments of 3DGE design, especially for the part of the deformable kernel map. Notably, utilizing uniform kernel map or inviting position information (z,, y, z) to learn the kernel map and a is not beneficial to the detection performance. This is because the key to determining whether radar point is false positive lies not in its position but rather its RCS and Doppler speed. In addition, uniform kernel size may blur the boundaries of close in distance, resulting in detection difficulty and performance loss.
the target objects, which Specifically, our adaptive 3DGE increases the NDS and mAP by 1.1, 1.3, and 1.9, 1.5 compared to the baseline and uniform 3DGE the clean dataset. In addition, 3DGE decreases the mAOE by 7.3% under the Key-point Missing conditions compared to the baseline model. To this end, adaptive 3DGE the most effective design compared the uniform 3DGE and adaptive 3DGE with position information. Furthermore, adaptive 3DGE achieves higher performance than the baseline in both interference and non-interrereece environments.
5.4 ANALYSIS OF ROBUSTNESS
Table 5 illustrates the performance of RobuRCDet and previous state-of-the-art methods under real- world challenging lighting and weather conditions, i.e., rainy conditions and night conditions. In rainy conditions, we achieve 1.8 NDS, and 1.4 mAP over CRN while at night time RobuRCDet sur- replace the MDCA
passes RCBEVDet by 3.20% in NDS and 11.46% in mAP. In addition, module in CRN with our CMCA module. CRN achieves 56.1 NDS and 47.3 mAP in rainy scenarios and the incorporation of CMCA yields 1.4 NDS and 0.7 mAP improvement, which demonstrates the effectiveness and transferable characteristics of CMCA.
Furthermore, enable RobuRCDet achieve better performance under challenging conditions and to demonstrate the robustness and effectiveness of the proposed method, we also conducted training and testing on the noisy dataset. The results can be found in the supplementary materials.
CONCLUSION
We introduce RobuRCDet, radar-camera fusion method designed enhance the robustness of 3D object detection. Our approach addresses the challenges of strong interference and suboptimal per- formance in diverse perception conditions by designing two key modules: 3DGE and CMCA. Ex- perimental results demonstrate that RobuRCDet outperforms previous state-of-the -heth-rrtt camera 3DD object detection methods in challenging conditions.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. This work was supported by National Key R&D Program of China (Grant No. 2022ZD0160305).
REFERENCES
Eduardo Amold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis. A survey 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportation Systems, 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 8
MMCV Contributors. Mmev: Openmmlab computer vision foundation, 2018. 13
Junlin Han, Weihao Li, Pengfei Fang, Chunyi Sun, Jie Hong, Mohammad Ali Armin, Lars Petersson, and Hongdong Li. Blind image decomposition. In ECCV, 2022. 5
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detec- tion. arXiv preprint arXiv:2203. 17054, 2022. 3
Junjie Huang. Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performanee multi- camera 3d object detection bird-eye-view. arXiv preprint a2.iv:2 / 2021.
Shengyin Jiang, Shaoqing Xu, Li Liu, Ziying Song, Yang Bo, Zhi-Xin Yang. et Sparseinteraction: Sparse semantic guidance for radar and camera 3d object detection. In ACMMM, 2024. 1,
Jisong Kim, Minjae Seong, Geonho Bang. Dongsuk Kum, and Jun Won Choi. Rem-fusion: Radar- camera multi-level fusion for 3d object detection. In ICRA, 2024. 1,
Youngseok Kim, Jun Won Choi, and Dongsuk Kum. Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image. In IROS, 2020. 8
Youngseok Kim, Sanmin Kim, Jun Won Choi, and Dongsuk Kum. Craft: Camera-radar 3d object detection with spatio- conte--contextual fusion transformer. In AAAI, 2023a. 3
Youngscok Kim, Juyeb Shin, Sanmin Kim, In-Jac Lee, Jun Won Choi, and Dongsuk Kum. Cmn: Camera radar net for accurate, robust, efficient 3d perception. In ICCV, 2023b. 3, 10
Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking 3d perception robustness common corruptions and sensor failure. In ICLRW, 2023a. 3
Lingdong Kong, Youquan Liu, Xin Li. Rumnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In ICCV, 2023b. 3,
Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng. Benoit R. Cottereau, and Wei Tsang Ooi. Robodepth: Robust out-of-distribution depth estimation under corruptions. In NeurIPS, 2023c.
Zhaoqi Leng, Guowang Li, Chenxi Liu, Ekin Dogus Cubuk, Pei Sun, Tong He, Dragomir Anguelov, and Mingxing Tan. Lidar augment: Searching for scalable 3d lidar data augmentations. In ICRA, 2023. 8
Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. AAAI, 2023. 3, 8
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: simple and robust lidar-camera fusion framework. In NeurIPS, 2022.
11

---

Published as a conference paper at ICLR 2025
Zhiwei Lin, Zhe Liu, Zhongyu Xia, Xinhao Wang, Yongtao Wang. Shengxiang Qi, Yang Dong, Nan Dong, Le Zhang. and Ce Zhu. Rebevdet: Radar-camera fusion bird's eye view for 3d object detection. In CVPR, 2024. 3, 8, 10
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for mue 3d object detection. In ECCV, 2022. 3
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang, Petrv2: A unified framework for 3d perception from multi-camera images. In ICCV, 2023a.
Yuheng Liu, Xinke Li, Xueting Li, Lu Qi, Chongshou Li, and Ming-Hsuan Yang. Pyramid diffusion for fine 3d large scene generation. In ECCV, 2024. 1
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's s-ey' view representation. In ICRA, 2023b. 3
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0510// 2017.
Ramin Nabati and Hairong Qi. Centerfusion: Center-based radar and camera fusion for 3d object
detection. In WACV, 2021. 3,5
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020. 3
Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In CVPR, 2019. 1
Jiawei Ren, Liang Pan, and Ziwei Liu. Benchmarking and analyzing point cloud classification under corruptions. ICML, 2022.
Marcel Sheeny, Emanuele De Pellegrin, Saptarshi Mukherjee, Alireza Ahrabian, Sen Wang, and Andrew Wallace. Radiate: A radar dataset for automotive perception in bad weather. In ICRA,
2021. 1
Zhaoliang Wan, Yonggen Ling, Senlin Yi, Lu Qi, Wang Wei Lee, Minglei Lu, Sicheng Yang, Xiao Teng, Peng Lu, Xu Yang, al. Vint-6d: A large-scale object-in-hal -nnd dataset from vision, touch and proprioception. In ICMI,, 2024. 1
Shihao Wang, Xiaohui Jiang, and Ying Li. Focal-petr: Embracing foreground for efficient multi- camera 3d object detection. IEEE Transactions on Intelligent Vehicles, 2023a. 3
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV, 2023b. 3
Zizhang Wu, Yunzhe Wu, Xiaoquan Wang, Yuanzhu Gan, and Jian Pu. A robust diffusion modeling framework for radar camera 3d object detection. In WACV, 2024.
Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei
Liu. Robobev: Towards robust bird's eye view perception under corruptions. arXiv preprint arXiv:2304.06719, 2023. 3
Zhou Xingyi, Wang Dequan, and Kr "hana""n" "uhl Philipp. Objects points. arXiv preprint arXiv: 2019. 3
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018. 13
Tian Zhi, Shen Chunhua, Chen Hao, and He Tong. Fully convolutional one-stage object detection. In ICCV, 2019. 3
Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR, 2022. 3
Taohua Zhou, Junjie Chen, Yining Shi, Kun Jiang, Mengmeng Yang, and Diange Yang. Bridging the view disparity between radar and camera features for multi-modal fusion 3d object detection. IEEE Transactions Intelligent Vehicles, 2023. 3,
12

---

Published as a conference paper at ICLR 2025
A OVERVIEW
This supplementary material provides additional details of architecture, and qualitative and quanti- tative experimental results. We describe implementation details for experiments in the main paper (Section B). We further provide additional experimenta results on noisy training sets (Section C) and qualitative results (Section D).
B IMPLEMENTATION DETAILS
This section provides some experimental settings and network details in the main paper.
First, for the network detail and hyper-parameters, we employ SECONDFPN (Yan et al., 2018) to concatenate output feature maps stride 16 and let the output depth bins of the depth distribution network to be 112 with a depth range of [2.0, 58.0Jm and bin size to be 0.5m. For the CMCA module, we use the multi-scale deformable attention implementation from MMCV (Contributors, 2018) and set the number of attention heads to and sampling points to 2. We set the radar point range [-51.2, 51.2]m and make the BEV feature map 128 128.
Second, for the noisy training set, we set the ratio of clean images to noisy images in the training set to 8:2. For the noisy images, randomly synthesize corruptions, with the intensity of the noise also being random. Additionally, we synthesize harsh weather or low-light conditions the images corresponding to the timestamps with noise.
of the four types of proposed radar
In addition, the corruption levels of Spurious Points, Point Shifting, and Non- Non-positional Disturbance are determined by σ, while the number of missing beams determines the noise level of Key-point Missing. Furthermore, weather degradation is also classified into levels; for example, rain and fog can be divided into light or heavy, whereas snow does not have classification. Additionally, the low-light level at night is determined by the gamma coefficient, with a mild low-ligh coefficient set 1.0-2.0 and a heavy set 2.0-3.0.
C ANALYSIS ON THE NOISY TRAINING SET.
Table 6 illustrates the results of CRN and RobuRCDet which are trained the noisy dataset and tested on each corruption. ResNet-18 used as the backhone. In this table, CO denotes the clean testing set and it is notable that RobuRCDet achieves the performance (54.4 NDS and 44.9 mAP) as the model trained on the clean dataset and surpasses CRN by 1.6 NDS and 1.4 mAP. Furthermore, Cl C4 represent Spurious Point, Non-positional Disturbance, Key-point Missing. and Point Shifting.
According to Table and the comparison in Table 2, performance is improved when processing noisy radar point clouds after training with the corruption training set. Additionally, to ensure faimesss in the experiments,
train and test both the RobuRCDet and CRN methods on the disturbed dataset and compared their performance. It is clear that our method still demonstrates better robustness than CRN with 2.4 NDS and 1.9 mAP improvement in C1 with 10.
D ADDITIONAL VISUAL RESULTS.
In this section, present
synthesized noisy images. Notably, better simulate real scenarios, we ensure that the degradation types and levels for multiple cameras at the same timestamp are the same. Figure 8 and Figure 9 showcase the synthesized images of rainy days, snowy days, and nighttime mentioned in the main paper.
E SIMULATION OF THE EFFECT OF 3DGE ON THREE TYPES OF NOISE.
In this section, we included simulations of the effects of 3DGE on various types of data, which are shown in Figure 10. These simulation results visually demonstrate the functioning and effectiveness of 3DGE. For instance, although the patterns of the three types of noise differ, the surrounding noise points consistently appear deep blue, indicating that, after processing, their impact the
13

---

Published as a conference paper at ICLR 2025
Light Rain
Hcavy Rain
Snow
Figure 8: Visualization of synthesized challenging weather images. Two levels of rainy images and set of snowy images are displayed.
14

---

Published as a conference paper at ICLR 2025
Table 6: Validation of models trained with noisy training sets, We augment the image data and radar data to form the training set. denotes that the model is retrained.
Comuuttin
CRN
RobuRCDet
Type
NDS1
mAPt
mATE
mAP (Car)
NDS1
mAPt
mATE
mAP (Cur)t
52.8
43.5
0.550
69.6
54.4
44.9
0.517
70.9
52.1
42.9
0.553
54.0
44.7
0.524
70.6
CI
51.2
42.4
0.560
68.4
53.6
44.3
0.532
70.1
51.6
42.6
0.557
69.2
53.4
44.1
0.539
70.1
C2
50.5
41.1
0.568
68.1
52.6
42.9
0.547
69.7
52.3
43.0
0.551
69.2
54.1
44.6
0.537
70.2
C3
10
52.0
42.3
0.569
68.8
53.8
44.0
0.642
69.9
41.6
35.2
0.668
55.4
45.1
36.9
0.637
56.8
C4
10
33.4
28.0
0.750
41.5
36.7
31.2
0.699
47.0
Night (1.0-2.0)
Night (2.0-3.0)
Figure 9: Visualization of synthesized challenging light conditions. Two levels of low-light im- ages are displayed.
15

---

Published as a conference paper at ICLR 2025
600
1.00
600
509
60
Clean Data
Spurious Points
Key-point Missing
Point Shifting
Figure 10: Simulation of the effect of 3DGE on three types of noise.
recognition target is minimal. Furthermore, even though the shapes of the heatmaps around the target vary after processing, the deep red regions, representing the peak positions the targets, remain entirely consistent. Notably, the spurious points appearing around the target region can even contribute strengthening the target area and diminishing the influence of surrounding points.
16


Reviews:
[{"review_id": "skb8dxeHH6", "reviewer": "Reviewer_uUpc", "summary": "This paper conducts a systematic analysis of radar-camera detection robustness under five types of noise and proposes RobuRCDet, a robust object detection model in bird’s-eye view (BEV). To address radar point inaccuracies, including position, Radar Cross-Section (RCS), and velocity, this work introduces a 3D Gaussian Expansion (3DGE) module. This module uses RCS and velocity priors to create a deformable kernel map, adjusting kernel size and value distribution. Additionally, this paper proposes a weather-adaptive fusion module that dynamically merges radar and camera features based on camera signal confidence. Experiments show the effectiveness of the proposed RobuRCDet.", "questions": "please see the weakness section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1)The figures in this paper are well-crafted.\n(2)The proposed 3D Gaussian Expanding method is both novel and effective, as demonstrated by the experiments.", "weaknesses": "1) The CMCA module seems to be a standard method, how does the degradation-aware head assess the confidence of the camera and radar features?  \n2) Although Pepper is designed as a robust fusion method between radar and camera, its performance is not much stronger than that of RCBEVDet in tab.2, particularly regarding the NDS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFaLxpzJxD", "reviewer": "Reviewer_MsLQ", "summary": "The paper addresses the problem of robustness in radar-camera fusion techniques for 3D object detection. The authors point out that adverse weather conditions, poor lighting, and sensor noise often cause existing methods to fail specifically because of the \"flat\" fusion approach that is usually used. The paper introduces RobuRCDet to overcome the shortcomings of existing approaches by using confidence-based fusion.\nKey contributions:\nAnalysis of common noise types affecting radar data in real-world scenarios (key-point missing, spurious points, point shifting, non-positional disturbance) and create a benchmark by simulating noise patterns for evaluating robustness.\nA model with 2 key contributions: 3D Gaussian Expansion (3DGE) for filtering out noisy radar points based on the sparsity distribution. And a Confidence-guided Multimodal Cross Attention (CMCA) for dynamically and reliably fusing the radar and camera features based on the confidence in the camera signal.\nAblation studies to corroborate the effectiveness of the added contributions on noisy radar and camera signals.", "questions": "In the voxelization and kernel generation approaches there are unclarities or unanswered questions:\nwhat voxel size is used and how does it affect the quality of the detection?\nif there are multiple targets in the same voxel, does that affect the computation of the 3DGE? equation 6 indicates otherwise but this does not make sense since a voxel with more targets should have more influence than a voxel with a single target.\nWhile nuScenes radar data includes the \"z\" value of the radar, it is unclear how accurate that value is and in reality, most of the radars on market are 2.5D and not 3D, aka they do not have a proper way to measure the elevation (no resolution in elevation) and thus the cartesian z value. How would the results of the 3D detection change if no \"z\" values were used?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is tackling a relevant problem hindering the reliability of machine learning approaches for sensor fusion in challenging scenarios. The related work is comprehensive and detailed. The approach presented is simple yet effective and builds on top of proven concepts. Although \"confidence-based fusion\" is not a new concept in itself and has been used for a long time in classical fusion approaches (e.g. Kalman Filters), the approach presented by the authors seems to be effective while not overly complicated. The authors combine multiple tried and proven ideas to achieve their results.", "weaknesses": "While the idea is presented clearly, there seem to be some missing definitions of parameters used in equations that are possible to infer but could be clearer. The diagrams, although readable, they can be more detailed to reflect the equations and information in the text. One of the methods mentioned in Table 1 is not cited (StreamPETR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yg4bXhh1iP", "reviewer": "Reviewer_B47b", "summary": "The paper proposes a BEV space object detector using camera and radar data that uses a 3D Gaussian to mitigate radar noise and adaptively fuses camera and radar features based on camera feature quality. The 3D Gaussian Edge module learns to spread the RCS and velocity values to surrounding voxels and the Confidence-Guided Multi-modal cross attention module learns to adaptively fuse radar and camera features by learning to detect degradation of the image features. Training and evaluation is done using simulated nuScenes dataset and shows improvement over CRN and RCBEVDet.", "questions": "Line 69: What is meant by “focus on the corruption graphic characteristics instead of the natural causes of the corruption”? If the noise distribution of the corruption doesn’t match the noise characteristics of the radar then the resulting model doesn’t add much benefit in practice.\n\nFigure 2: how were the noise parameters of the plots determined? How were the ground truth points determined in the captured data, i.e., they can already have the 4 types of radar noise.\n\nHow is the camera signal confidence reliably learned in practice with imbalance data?\n\nThe need for the learned 3D Gaussian Expanding component is unclear especially given that the set of lambda_p is small, how does the model perform without learning the sigma and simply performing deformable convolution on a 5x5x5 grid?\n\nHow well does the proposed approach work on real adverse weather and noisy data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The main strength of the paper is that it considers different types of sensor degradation and proposes a radar data expansion and camera-radar fusion approach to mitigate those degradations.", "weaknesses": "The proposed 3DGE seems to be densifying the radar data based on the reasoning in lines 306-316. How do the other noise types, i.e., spurious points, point shifting and non-positional disturbances get mitigated? Furthermore, spurious points may get worse due to being spread across multiple voxels, and there’s no discussion on how non-positional disturbances are addressed through 3DGE. Furthermore, the ablation experiments only include keypoint noise and missing the other 3 types of radar noises as mentioned in the paper.\n\nThere’s no discussion on the training process. How does the model learn M_c from the data? Adverse data are rare and what prevents the network from always choosing image features?\n\nAll the results are based on simulated data therefore the conclusions from the results would depend a lot on the fidelity of the simulation. There isn’t much discussion on how the radar noise simulations were done. As for the image signal, the performance will be limited by [Han et al. 2022] for adverse weather. For low-light, cameras have signal dependent noise characteristics which cannot be modeled by random gamma factor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gEvFq7W6aq", "reviewer": "Reviewer_YH6m", "summary": "The paper introduces RobuRCDET, a novel approach to effectively fuse radar and camera features for 3D object detection. The core idea is to suppress false radar points by predicting Gaussian kernel variance. The approach demonstrates promising results on the nuScenes validation set.", "questions": "Please see the weakness. I will need nuscenes leaderboard results on the 640x1600 resolution and comparison against SparseBEV 640x1600 resolution.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and easy to comprehend.\n+ The core idea of suppressing false radar points by predicting Gaussian kernel variance is nice and leverages the information bottleneck principle.\n+ The proposed method of weighting camera and radar streams leads to robust feature representation.\n+ The approach demonstrates promising results on the nuScenes validation set.", "weaknesses": "- The idea of predicting Gaussian variance in 3DGE module like decomposition and re-combining is one of the ways to denoise radar points. Another way to denoise is using self-attention blocks [A]. How does the method work when you replace the 3DGE module by multiple self-attention blocks.\n\n- The claim of \"Extensive Experiments\" in the abstract is exaggerated. It would be beneficial to quantitatively include results from the nuScenes leaderboard, particularly comparing against a strong camera baseline like SparseBEV with 640x1600 resolution.\n\n- The experiments are conducted with super small backbones (ResNet18 and ResNet50). It would be insightful to quantitatively evaluate the approach on higher resolutions (512x1508 and 640x1600) to assess its performance.\n\n- The paper focuses on mid-level feature fusion. A quantitative comparative analysis with end-level fusion, as employed in RADIANT [B], would provide valuable insights. To further strengthen the argument for mid-level fusion, incorporating the radar association module from RADIANT should be considered.\n\n\nReferences:\n- [A] Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis, Teo et al, NeurIPS 2024\n- [B] RADIANT: Radar Image Association Network for Radar-Camera Object Detection, Long et al, AAAI 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)