CRT-Fusion: Camera, Radar, Temporal Fusion Using  Motion Information  for 3D Object Detection
Authors: Jisong Kim
Minjae Seong
Jun Won Choi
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Object Detection
Sensor Fusion
Temporal Fusion
Radar
Camera
Primary area: deep_learning_architectures
TL;DR: CRT-Fusion is a novel framework that significantly improves the accuracy and robustness of 3D object detection by effectively integrating radar-camera information and temporal cues, explicitly considering the motion of dynamic objects.

Abstract: Accurate and robust 3D object detection is a critical component in autonomous vehicles and robotics. While recent radar-camera fusion methods have made significant progress by fusing information in the bird's-eye view (BEV) representation, they often struggle to effectively capture the motion of dynamic objects, leading to limited performance in real-world scenarios. In this paper, we introduce CRT-Fusion, a novel framework that integrates temporal information into radar-camera fusion to address this challenge. Our approach comprises three key modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module fuses radar and image features within both the camera view and bird's-eye view, thereby generating a more precise unified BEV representation. The MFE module conducts two simultaneous tasks: estimation of pixel-wise velocity information and BEV segmentation. Based on the velocity and the occupancy score map obtained from the MFE module, the MGTF module aligns and fuses feature maps across multiple timestamps in a recurrent manner. By considering the motion of dynamic objects, CRT-Fusion can produce robust BEV feature maps, thereby improving detection accuracy and robustness. Extensive evaluations on the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-the-art performance for radar-camera-based 3D object detection. Our approach outperforms the previous best method in terms of NDS by +1.7%, while also surpassing the leading approach in mAP by +1.4%. These significant improvements in both metrics showcase the effectiveness of our proposed fusion strategy in enhancing the reliability and accuracy of 3D object detection.

CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection
Jisong Kim! Minjae Seong Jun Won Choi
Hanyang University "Soul National University (jskim, mjseong) hanyang....................aa..a ac kr (junwchoi) 0anuu.iissss...mm ac.kt kr
Abstract
Accurate and robust 3D object detection is a critical component in autonomous vehi- cles and robotics. While recent radar-camera fusion methods have made significant progress by fusing information in the bird's-eye view (BEV) representation, they often struggle effectively capture the motion of dynamic objects, leading to lim- ited performance in real-worke scenarios. In this paper, we introduce CRT-Fusion, novel framework that integrates temporal information into camera fusion address this challenge. Our approach comprises three key modules: Multi- View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module fuses radar and image features within both the camera view and bird's s-eye view, thereby generating a more precise unified BEV representation. The MFE module conducts two simultaneous tasks: estimation pixel-wise velocity information and BEV segmentation. Based on the velocity and the occupancy score map obtained from the MFE module, the MGTF module aligns and fuses feature maps across multiple timestamps in recurrent manner. By considering the motion of dynamic objects, URT-Fusion feature maps, thereby improving detection accuracy and robustness. Extensive evaluations the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-the-arr performance for radar-camera-basee 31) object detection. Our approach outperforms the previous best method in terms of NDS by +1.7%, while also surpassing the leading approach mAP by +..4%. These significant improvements in both metrics showcase the effectiveness of our proposed fusion strategy in enhancing the reliability and accuracy of 3D object detection.
produce robust BEV
Introduction
3D object detection plays a crucial role in autonomous vehicles and roboties, leveraging sensors such as lidar, cameras, and radar to localize and classify objects in the environment. Extensive research has been conducted to explore various strategies for improving detection accuracy and robustness. One prominent approach is the integration of data across multiple timestamps, which aims to mitigate the inherent limitations associated with relying solely on instantaneous data. By incorporating historical information, this approach provides more robust perception of the environment, addressing the challenges of incomplete data caused by occlusions, sensor failures, and other factors.
Numerous studies have investigated the utilization of temporal information to enhance the perfor- mance of LiDAR-based and camera-based 3D object detection methods. Recent works have also explored the incorporation of temporal cues in radar-camera fusion methods [ 2 These methods generated seed's view (BEV) feature maps for each frame by fusing radar and camera data into a
"qqual contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Midian Catggn Dargenaatiinn
Performano
frame
Frame
Co Conraborntion
25
2.25
2.0
1.5
I- Trame
I- frame
0-1
2-1
Object elaciiy
(a) Previous Methods
(b)) CRR--FusiiO
(c) Performance vs abject velocity
Figure 1: Comparison of temporal fusion methods: (a) Previous methods concatenate BEV feature maps without considering object motion. (b) CRT-Fusion estimates and compensates for object
motion before concatenation. (c) Performance gain of CRT-Fusion over the direct concatenation method, showing CRT-Fusion's superior accuracy across different object velocity ranges.
unified BEV representation. The resulting BEV feature maps are then concatenated across frames to create comprehensive spatio-temporal representation, illustrated in Figure a However, these approaches face limitations in effectively capturing object motion, as they merge data from different time intervals without explicitly considering the dynamics of moving objects. Consequently, the performance accuracy for dynamic objects is compromised.
To address these challenges, we propose a motion-aware approach, as illustrated in Figure | (b), which goes beyond simple concatenation of BEV feature maps. Our method first estimates the locations of dynamic objects with their corresponding velocity vector for each timestamped BEV feature map. Subsequently, we leverage this predicted information to rectify the motion of dynamic objects in each feature map and fuse them in temporally consistent manner. Figure [[c) presents graph depicting the performance gain achieved by our proposed method over the direct concatenation of temporal BEV feature maps for different object velocity ranges. It is evident that approach consistently outperforms the existing method across all velocity ranges, with a notable performance improvement for objects moving medium velocities. This demonstrates the effectiveness of our motion-aware fusion strategy in capturing and compensating for object motion, leading to superior performance in 3D object detection.
In this paper, we introduce CRT-Fusion, novel approach for integrating temporal information into radar-camera fusion. Our framework comprises three modules: Multi- Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF), The MVF module generates radar-camera fused BEV feature maps for each timestamp. The MVF enhances image features with radar BEV features, achieving more precise depth predictions through Radar-Cameds Azimuth Attention (RCA). The enhanced camera BEV features and radar BEV features are integrated through gating operation. The MFE module predicts velocity information and performs BEV segmentation for each pixel in the fused BEV features identify object regions and provide values for shifting the feature map spatially. Finally, the MGTF module generates the final feature map by leveraging the fused BEV feature maps, segmentation results, and velocity predictions. The MGTF module begins with the BEV features from the (t - N yh time step and aligns them with those from each subsequent time step. These aligned features are then aggregated one-by-one across all N timestamps in a recurrent manner. Consequently, CRT-Fusion achieves state-of-the-a performance on the nuScenes 3D object detection benchmark for radar-camera fusion methods, with improvements of +1.7% in NDS and +1.4% in mAP compared to existing state state-of-hee of-the--rttt approaches.
In summary, the main contributions of this work are follows:
We introduce CRT-Fusion, a novel framework that effectively integrates temporal informa- tion into radar-camera fusion for 3D object detection. By considering the motion of dynamic objects, CRT-Fusion significantly improves detection accuracy and robustness in complex real-world scenarios.
We design a Multi- Vieew Fusion module that enhances depth prediction by leveraging radar features to improve image features before fusing them into unified BEV representation.

---

We introduce an effective temporal fusion strategy through MFE and MGTF modules. MFE estimates pixel-wise velocity information, and MGTF iteratively aligns and fuses feature maps across multiple timestamps using the motion information obtained from MFF.
CRT-Fusion achieves state-of-the-art performance on the nuScenes dataset for radar-camera- based 3D object detection, surpassing previous best method by +1.7% in NDS and + +1.4% in mAP.
2 Related Works
2.1 Camera-Radar 3D Object Detection
Due to the high cost of LiDAR sensors, research on 3D object detection using radar-camera sensor fusion has gained significant traction in recent years. These studies aim to utilize radar sensors as auxiliary sensors overeome the fundamental limitation camera-based 3D object detection research, which is the lack of depth information.
CenterFusion 3 adopts frustum- based approach to establish connections between radar point clouds and image features, enabling the refinement of 3D proposals. CRAFT [1] captures the interactions between radar and camera data within polar coordinate system using a cross-attention mechanism, effectively integrating information from both modalities. RCM--uuion 4 combines radar and image features both feature-level and instance-level achieve
precise 3D object detection. RADIANT [ fuses radar and image features within image pixel coordinates to provide more accurate 3D location estimation. CRN [6] employs 3D object detection method that strikes a balance between speed and performance by leveraging radar information to enhance camera BEV features and fusing multi-modal BEV features. RCBEVDet 2 introduces a novel radar backbone network that utilizes point-based feature extraction techniques for radar and fuses radar and image features using deformable aotssnattention. CRKD 7 employs a method transfer the knowledge possessed by the LiDAR-Cames fusion detector to the Camera-Rade fusion detector using the Cross-modality Knowledge Distillation technique.
2.2 Temporal fusion in 3D Object Detection
The approach to utilizing temporal information in 3D object detection varies depending on the type of sensors employed. In LiDAR-bases methods, relying solely on single-frame point cloud data introduces challenges such occlusion and partial views. To mitigate these issues, several studies have integrated temporal information at the feature level g 9 I0 IT]. Another approach involves extracting proposals using an object detector and subsequently leveraging temporal information at A third method focuses on fusing temporal information within the query
the object level [12 13
14]
representation
15
On the other hand, camera-based approaches have exploited temporal information to overcome the inherent limitations of image data, such as inaccuracies in depth prediction. One common methodology in camera-based perception research is to fuse temporal information with BEV-based Another approach has centered on enhancing the accuracy of depth estimation through temporal-stre -teeroo methods [21 22 20).
methods [66 17
18
19 20
In the context of radar-camera fusion methods, the utilization of temporal information remains less explored, with most studies following the strategies employed in BEV-based camera-only methods. However, our proposed CRT-Fusion deviates from existing research methods by introducing novel temporal fusion method. Our approach incorporates a temporal BEV fusion mechanism that explicitly considers the movement of objects, thereby enhancing object detection performance.
3 CRT-Fusion
In this work, we introduce CRT-Fusion, a novel framework for 3D object detection that efficiently fuses information from radar, camera, and temporal domains. The overall architecture of CRT-Fusion illustrated in Figure Our approach first extracts the features from radar and camera data separately using their respective backbone networks. Subsequently, employ the MVF module to generate
3

---

frame data
Baclhone
MVF
MFF
3D Letectinn Havedd
Backhone
tframe Fused BEV Features
NCTI
frame
Camera Backhone
MVF
MFE
Radar Backhone
t-N Erame Fineel BEV I' Feotures
Mulli-View Fusion
Motion Feature Estimation
Camera fraturo
Velocity
Head
RCA Attention
Radar Indiase
Fosed Insture
Ubject Onc
Head
Fou
Figure 2: Overall architecture of CRT-Fusion: Features are extracted from radar and camera data using backbone networks at each timestamp. The MVF module combines these features generate fused BEV feature maps. The MFE module predicts the location and velocity of dynamic objects from these maps. The MGTF module then uses the predicted motion information create the final feature map for the current timestamp, which is fed into the 3D detection head.
a fused BEV feature map for each timestamp by combining the radar and camera features. The sequence of fused feature maps is then utilized by the MFE module predict the location and velocity information of dynamic objects. The predicted motion information is exploited by the MGTF module align the BEV feature maps spatially within time window. Then, the aligned features maps are aggregated to obtain the final feature maps. Finally, 3D object detection is performed using the detection head proposed by CenterPoint 23
3.1 Multi-View Fusion
Recent advancements in BEV-based camera-only approaches have significantly improved perfor- mance, leading to an increased focus on radar-camera fusion approaches within BEV representations. These studies [6 4 2 primarily aimed address the inherent limitations of camera-only approaches, particularly the challenge of accurate depth prediction, by leveraging radar data. The existing state- of-the-art model 66) enhanced this process by combining occupancy information from radar point clouds with camera frustum features, effectively incorporating radar positional data. While this method facilitates the direct integration of radar positional information, noise in radar point clouds can adversely affect depth prediction accuracy. To mitigate this issue, we propose novel fusion strategy that incorporates radar and camera features in both bird's eye view and perspective view. Unlike the existing method [6], our approach enhances camera features using radar information prior to depth prediction, enabling more accurate camera BEV features. Subsequently, we employ CNN-based gated fusion network to obtain the final fused features.
Perspective view fusion. As illustrated in Figure 3 (a), the Radar-Camer Azimuth attention (RCA) module takes the camera perspective view features F. E RNxCx HXW XW and the radar BEV features F, E XCX
inputs, where H and W represent the height and width of the camera features, and X and Y denote the size of the radar BEV features along the x-axis and y-axis, respectively. For the i-th image, the camera feature FL HC compressed along both height and width dimensions using max pooling and MLP layers, resulting in W E R CxxXX and H: e IR Cx Let Wi( be the value of the W features the j-th position along the width direction. We associate the radar feature element F, y) at the position (æ. y) with W2() through Azimuth Grouping. The azimuth angle values corresponding W)( and F, (x. ae.p y) are denoted as (%()) and (, y). respectively. A set of M radar features Rj associated with ()( is obtained using:
R3 = F.
arg min xe(00
(18:6) - 0, v)
(I)

---

Hulli frone REY Fonurr
MGTF
CKN
Firalrry Futeres
Fl
F
B
OERRI
X N
Asdeuth Creeping
Fied wade
B
M/-h
Comprrosio
B'
B
Comcioantion
Nailig Migiciicaa
Metsory
(a) Radar-CCamer Azimath attention (RCA)
(b) Motion Guided Temporal Fusion (MGTF)
Figure 3: Core components of CRT-Fusion: (a) RCA module enhances image features with radar features for accurate depth prediction. (b) MGTF module compensates for object motion across multiple frames, producing the final BEV feature map for 3D object detection.
where arg min(2 (M' M) returns the indices of the smallest M elements in X, and Rj represents M radar features F, (2, v) whose azimuth angles are closest (.()). Pixel-wise fusion module is then applied to R3 to obtain an enhanced feature ()) as
(m) = ML..2 (concat (LM. (m))) (m))))
(2)
W Σ softmax Sofmax(MLP334(((mm)) -M))
where (m) denotes the m-th element of Rj. The concatenation (i)) and (m) passed through two MLP layers to obtain an intermediate feature (m). These intermediate features are then processed through another MLP layer followed by softmax function determine the relevance to ()). The weighted sum of these intermediate features yields the enhanced (i. Finally, we perform element-win multiplication of W RCxIx and Hi E Rxx to obtain the camera perspective view features Fi. The concatenation of Ft and Fs is then passed through convolution layer to obtain Fy. which is the perspective view features for the i-th image fused with the radar BEV features. These steps are depicted in Figure (a).
Bird's eye view fusion. The enhanced camera feature Fr in the perspective view is employed for depth prediction and camera view semantic segmentation. Inspired by SA-BEV 24] we utilize a Ixl CNN layer head structure to predict both depth map and segmentation scores in the perspective view. The network outputs D E (6+ xw where b denotes the number of depth bins and the additional dimension corresponds to the foreground score prediction. The predicted segmentation scores are thresholded using value Tp to identify the foreground regions only, which are then projected to the BEV domain. The resulting camera BEV features are fused with the radar BEV features obtained from the radar pipeline using a gated fusion network 25 26 to yield the final BEV features B. The gated fusion network assigns weights to each feature according to their significance, effectively boosting the effect of feature fusion.
3.2 Motion Feature Estimation
Temporal fusion methods that consider object motion have been extensively studied in the context of 3D object detection [12 13 14). These methods typically predict object locations and velocities each timestamp using an object detector, and then aggregate object information from the past timestamps to the current timestamp at the object level. However, this approach has a drawback as the overall model performance heavily depends on the initial detector's performance. Moreover, object-level temporal fusion methods have not utilized motion information of objects effectively. To address these limitations, we propose a simple yet effective solution: by predicting velocity information and object presence for each pixel in the BEV features, the model can produce the aligned BEV features, which can be temporally fused the feature level in the BEV domain rather than at the object level.
5

---

Suppose that we obtain a set of BEV features B = {Bi-slk E JI,,.,,HH from the preceding Multi- Fusion module, where B,- represents the BEV feature at timestamp k. Each feature B & is then processed in parallel by two distinct heads: velocity head and an object occupancy head, both composed of 3x3 and 1xl convolutions. The velocity head extracts motion information M, E TR2xXxx XxY containing velocity estimates along the and axes for each pixel in the feature map B Simultaneously, the object occupancy head produces an occupancy score map -,- E R1xXxY where each element indicates whether the corresponding pixel belongs to an object. To facilitate training of these modules, we generate ground truth targets for each timestamp using the following equations. The IoU ratio r(z. y) for pixel at location (I: y) is defined as:
r(z, v) = [H, y) (Pg))
(4)
[1. v)1
where (x, y) is the physical box in the BEV domain corresponding to one pixel located at (a, y). g is the set of ground truth 3D object boxes, and P(G) is the projection of these boxes onto the BEV domain. The loU ratio calculates the overlap between the physical box and the projected ground truth boxes, helping to determine positive samples. The ground truth values then given by:
{ (ugl :uy)
if r(, v) 2 Tiou otherwise
(2, v)
(5)
(0,00
{
if rf v) 2 Teou
0 v)
(6)
otherwise
where M,TT(,,, y) and 0 (I., v) are the ground truth velocity and occupaney scores for pixel (xt, v) in the BEV feature Be-k. respectively. The velocity vector of the corresponding ground truth object is denoted by (vg*, og ) and Tiou is a predefined threshold set to 0.5. Pixels with an IoU ratio exceeding Trou are classified as positive and assigned the GT velocity and GT occupancy state for supervision.
3.3 Motion-Guided Temporal Fusion
Figure (b) (b) presents the Motion-Guider Temporal Fusion (MGTF) module, which integrates BEV features to construct a dynamic representation of object motion across multiple timestamps. Our model utilizes a memory bank structure, where previously computed BEV features generated through the MGTF process are stored in the buffer, effectively minimizing redundant computations. This memory-efficient design significantly reduces the computational overhead during temporal fusion.
At each timestep - k: the BEV feature map By_k is retrieved from the memory bank, representing the previous result processed through the MGTF. For each coordinate (I, y) in B,-k, the corresponding velocity vector M1- (z. y) = [Uz, vy] is used to compute the positional shift Az Ux ty and Ay By t., where f, denotes the duration of single frame. These positional shifts are applied to the feature values if the velocity magnitude exceeds a predefined threshold T,. The shifted feature maps are then obtained as
(2,
Σ
Bin(i
S( v)1
(L.)))((..))
where
5(x, = ((:)) : : | = (M > T.)-
(8)
Here |8(x, y) denotes the cardinality of S(x, 3) and [-1 denotes the rounding operation. The shifted feature map Bi is then concatenated with the feature map BI at the next timestamp. To filter out any irrelevant features resulting from the shifting process, the concatenated feature map element- wise multiplied with the occupancy score map 0z-k+1 as
B (o O O
This process iteratively applied N times, generating the BEV feature maps B BI-N+I...- B, sequentially. The final B, is then passed through a Ixl convolution to obtain the final BEV feature map. This sequential nature of the process enhances overall comprehension of the MGTF module.
By incorporating motion information and occupancy score maps, the MGTF module captures the dynamics of moving objects while filtering out irrelevant features, resulting in a more robust BEV representation. Compared to traditional methods, MGTF captures trajectories of moving objects to enhance 3D object detection performance.

---

Table 1: Performance comparisons with 3D object detector on the nuScenes val set. 'L', 'C', and 'R' represent LiDAR, camera, and radar, respectively. t: trained with CBGS. f: use TTA.
Method
Input Backbone
Image Size
NDS
mAP
mATH
mASH
mAOR
mAVE
mAAE
FPS
CenterPoint-P
23
I.
Pillars
59.8
49.4
0.320
0.262
0.377
0.334
0.198
CenterPoint- V
23
Voxel
65.3
56.9
0.285
0.253
0.323
0.272
0.186
CenterFusion
3
C+R
DLA34
448 448x800
45.3
33.2
0.649
0.263
0.535
0.540
0.142
BEVDepth
R50
256x704
47.5
35.1
0.639
0.267
0.479
0.428
0.198
11.6
RCBEV4d
C+R
R50
256x704
49.7
38.1
0,526
0.272
0.445
0.465
0.185
CRAFT' 1
C+R
DLA34
448x800
51.7
41.1
0.494
0.276
0.454
0.486
0.176
4.1
RCM-Fusion
4
C+R
R50
450x800
52.8
44.4
0.527
0.272
0.450
0.515
0.180
SOLOFusion
20
C
R50
256x704
53.4
42.7
0.567
0.274
0.411
0.252
0.188
11.4
CRN 6
C+R
R50
256x704
56.0
49.0
0.487
0.277
0.542
0.344
0.197
20.4
RCBEVDet P
C+R
R50
256x
704
56.8
45.3
0.486
0.285
0.404
0.220
0.192
21.3
CRT- ORT-Fusiin-light
C+R
R50
256x
57.8
48.8
0,480
0.265
0.480
0.248
0.189
20.5
CRT-Fusion
C+R
R50
256x 704
57.2
50.0
0.499
0.277
0.531
0.261
0.192
14.5
CRT-Fusion"
C+R
R50
256x704
59.7
50.8
0.461
0.264
0.419
0.234
0.186
14.5
MVFusion'
32
C+R
R101
900x 1600
45.5
38.0
0.675
0.258
0.372
0.833
0.196
BEVFormer
16
R101
900 1600
51.7
41.6
0.673
0.274
0.372
0.394
0.198
BEVDepth
17
C
R101
1408 542x1408
53.5
41.2
0.565
0.266
0.358
0.331
0.190
5.0
SOLOFusion
C
R101
512x 1408
54.4
47.2
0.518
0.275
0.604
0.310
0.210
SOLOOFsion
20
C
R101
512x 1408
58.2
48.3
0.503
0.264
0.381
0.246
0.207
CRN 6
C+R
R101
512x 1408
59.2
52.5
0.460
0.273
0.443
0.352
0.180
7.2
CRN 6
C+R
R101
512x 1408
60.7
54.5
0.445
0.268
0.425
0.332
0.180
C+R
R101
512x 1408
62.1
55.4
0.425
0.264
0.433
0.237
0.193
4.9
4 Experiment
4.1 Experimental setup
Dataset We evaluated our proposed method using the nuScenes dataset 27 a popular public dataset for autonomous driving. This dataset consists of 1,000 scenes, divided into 700 scenes for training. 150 scenes for validation, and 150 scenes for testing. Each scene contains approximately 20 seconds of data. The nuScenes dataset provides comprehensive 360-degree coverage with data from six cameras, and five radars. Keyframes are annotated at frequency of 2Hz, covering 10 object classes. We use the official evaluation metrics provided by the nuScenes benchmark, which are mean Average Precision (mAP) and nuScenes Detection Score (NDS).
Implementation details We adopted BEVDepth [7 as our baseline model. For fair comparison with existing methods, we employed ResNet 28 and ConvNeXt (29] as backbone encoders in the camera branch. the radar branch, we accumulated the past 6 radar sweeps to obtain the input point clouds and used PointPillars (30] with randomly initialized weights as the backbone network. Our proposed CRT-Fusion model performed temporal fusion using the BEV features from the past 6 frames. We also introduce CRT-Fusion-I
lightweight version of CRT-Fusion, where the 2D-CNN backbone is removed from the radar pipeline. CRT-Fusion-Light performs temporal fusion using the BEV features from the past frames. Detailed values of hyperparameters including learning rate, optimizer, and data augmentation methods are provided Appendix.
4.2 Comparison to the state of the art
Table Table[[compare our proposed CRT-Fusion method with state-of-the-art 3D object detection methods on the nuScenes validation set. CRT-Fusion consistently outperforms existing radar-camera fusion models and camera-only models across various camera backbone configurations. With the ResNet-50 backbone, CRT-Fusion achieves an improvement of 12.2% in NDS and 15.7% mAP compared to the baseline model BEVDepth [17]. Additionally. CRT-Fusion achieves 1.2% higher NDS and 1.0% higher mAP than the sfate-of-the-arr model CRN under the same configurations without class- balanced grouping and sampling (CBGS) 337 Furthermore, when CBGS is applied, our approach outperforms the current best model, RCBEVDet 2 by 2.9% in NDS and 5.5% in mAP. When using the ResNet- 101 backbone, CRT-Fusion surpasses CRN by 1.4% in NDS and 0.9% in mAP without
7

---

Table 2: Performance comparisons with 3D object detector the nuScenes test set. 'L', 'C', and 'R'
represent LiDAR, camera, and radar, respectively. f: use Test Time Augmentation.
Method
Input
Backbone
NDS
mAP
mATE
mASE
mAOE
mAVE
mAAE
PointPillars
30
L
Pillars
55,0
40.1
0.392
0.269
0.476
0.270
0.102
CenterPoint
23
L
Voxel
67.3
60.3
0.262
0.239
0.361
0.288
0.136
KPConvPillars
R3
R
Pillars
13.9
4.9
0.823
0.428
0.607
2.081
1.000
RadarDistill
R
Pillars
43.7
20.5
0.461
0.263
0.525
0.336
0.072
CenterFusion 3
C+R
DLA34
44.9
32.6
0.631
0.261
0.516
0.614
0.115
RCBEV 31
C+R
Swin-T
48.6
40.6
0.484
0.257
0.587
0.702
0.140
MVFusion 62
C+R
V2-99
51.7
45.3
0,569
0.246
0.379
0.781
0.128
CRAFT 11
C+R
DLA34
52.3
41.1
0.467
0.268
0.456
0.519
0.114
BEVFormer 16
C
V2-99
56,9
48.1 1
0,582
0.256
0.375
0.378
0.126
BEVDepth 17
C
ConvNeXI-B
60.9
$2.0
0.44S
0.243
0.352
0.347
0.127
SOLOFusion 20
C
ConvNeXI-B
61.9
54.0
0.453
0.257
0.376
0,276
0.148
CRN 6
C+R
ConvNeXi-B
62.4
57.5
0,416
0.264
0.456
0.365
0.130
SparseBEV 35
C
V2-99
63.6
55.6
0.485
0.244
0.332
0.246
0.117
StreamPETR
36
C
V2-99
63.6
55.0
0.493
0.241
0.343
0.243
0.123
RCBEVDet
C+R
V2-99
63.9
55.0
0.390
0.234
0.362
0.259
0.113
CRT-Fusion
C+R
ConvNeXt-B
64.9
58.3
0.365
0.261
0.405
0.262
0.132
CRT-Fusion
C+R
ConvNeeI-B
65.6
58.9
0.358
0.258
0.392
0.248
0.130
test time augmentation (TTA). Our lightweight version, CRT-Fusion-light, maintains a comparable FPS while delivering better performance compared existing radar-camera 3D object detectors, demonstrating the efficiency and effectiveness of our approach.
the performance of CRT-Fusion on the nuScenes test set. Our method outperforms all existing radar-camera fusion models, achieving titt--of-hh-art performance both settings with and without TTA. Note that the V2-99 backbone is pre-trained on the external depth dataset DDAD (38
4.3 Ablation studies
We conducted comprehensive ablation studies on the nuScenes validation set to evaluate the effec- tiveness of the key components in CRT-Fusion. Throughout these experiments, unless otherwise specified, we used a ResNet-50 backbone and an image size of 256 x 704 for the camera branch, and BEV size of 128 x 128. All models are trained for 24 epochs without applying CBGS.
Component analysis. To assess the effect of each component, we gradually added each to our baseline model and analyze the performance improvements, as shown in Table The first row shows performance of the BEVDepth model as reported in the paper, achieving an NDS of 47.5% and an mAP of 35.1 15.1%. Our baseline model, represented in the second row, reproduces BEVDepth without CBGS and incorporates long-term temporal fusion 200). achieving an NDS of 47.4% and an mAP of 37.8%. By fusing radar and camera features at the BEV stage, including gating fusion, observe a significant improvement, reaching an NDS of 55.4% and an mAP of 47.8%. RCA, which fuses radar features in the frustum view, contributes to additional 1.2% and 1.1% improvement in NDS and mAP, respectively. Finally, by using MFE and MGTF, we achieve an additional gain of 1.1 % in both NDS and mAP, resulting in the highest performance with NDS of 57.2% and mAP of 50.0%. These results demonstrate the effectiveness of each key component in CRT-Fusion, with our proposed modules providing significant improvements
the baseline model.
Robustness to diverse weather and lighting conditions. In Table 4 we analyze the performance of our model under varying weather and lighting conditions. For fair comparison, we the same settings as CRN, with ResNet-101 backbone and an input size of 512 x 1408. The baseline model BEVDepth shows low mAP scores due to the impact of weather and light on camera sensors. However, by incorporating radar sensors, which are less affected by these factors, CRT-Fusion achieves over 15% higher mAP in all scenarios. Compared to CRN, the state-of-the--e camera- radarr 3D object detection model, CRT-Fusion achieves higher mAP in all conditions except Sunny, offering particularly notable gains in night environments.
Impact accurate velocity estimation on MFF and MGTE, Table demonstrates the importance of accurate velocity prediction for each BEV grid. When the MFE and MGTF modules are applicd to

---

Table Ablation study of the main components of CRT-Fusion.
Table 4: Performance comparison under dif- ferent weather and lighting conditions.
Model Configuration Input NDS mAP
mATE mAOE
Method
Input Rainy Day Night
BEVDepth [17]
47.5
35.1
0.639
0.479
BEVDepth [7
C
39.0
39.0 39.3 16.8
Baseline
C
47.4
37.8
0.676
0.654
RCBEV 331
C+R
36.1
38.5
37.1 15.5
BEV Fusion
C+R
55.4
47.8
0.528
0.584
RCM-Fusion
C+R
49.4
51.4
50.1 25.6
+ RCA
C+R
56.1
48.9
0.516
0.574
CRN [6]
C+R
54.8
57.0
55.1 30.4
+ MFE & MGTF
C+R 57.2 50.0
0.497
0.532
CRT-Fusion
C+R
54.7
57.9
55.8 33.0
Table 5: Ablation study of the MFE and MGTF module applied to both the camera-bases model ((17) and our proposed model.
Table 6: Comparison of radar- view trans- formation methods. RGVT: Radar-Guided View Transformer. RVT: Radar-assisted View Transformation.
Method
MFE & MGTF
NDS
mAP
mAVE
Method
NDS mAP mATE mAOE mAVE
X
47.4
37.8
0.312
BEVDepth
46.9
37.3
0.349
RGVT 39
48.6
41.3
0.571
0.603
0.522
X
56.1
48.9
0.278
RVT 6
48.2
41.4
0.576
0.577
0.560
CRT-Fusion
57.2
50.0
0.265
RCA
50.5
41.7
0.573
0.541
0.432
the camera-bart based baseline BEVDepth, performance degrades, as shown the second row the table. The NDS and mAP scores both drop by 0.5%, highlighting the challenge of accurately estimating velocities solely from camera information. In contrast, CRT-Fusion leverages radar information to achieve more precise velocity predictions. By incorporating the MFF and MGTF modules, CRT- Fusion achieves an improvement of 1.1% both NDS and mAP, demonstrating the effectiveness of these modules in enhancing the performance. These results indicate that the successful operation of the MFE and MGTF modules is highly dependent on accurate velocity estimation. In the absence of radar data, velocity estimation accuracy significantly compromised, suggesting that MFE and MGTF modules are optimally suited for radar-camera fusion frameworks.
Comparison of radar-based view transformation methods. Table compares various view transformation methods leveraging radar information. For a fair comparison, we used CRN as the baseline model and conducted experiments in single-frame setting. The LSS approach predicts 3D) depth from camera features and transforms them into BEV features, but can generate inaccuracies due imprecise depth information. RGVT 339 projects radar points encodes radar depth features using a lightweight CNN, which are then combined with image features predict 3D depth. RVT (6) refines the depth distribution predicted from camera features using radar occupancy information. Our proposed RCA outperforms existing methods in almost all metrics, demonstrating its effectiveness in utilizing radar information transform camera features into accurate BEV features. The superior performance of RCA demonstrates that leveraging radar information for depth prediction significantly enhances BEV perception.
the image plane and
Qualitative results. Figure presents qualitative comparison between our proposed CRT-Fusion model and the previous state-of-the-art CRN model across various real-world scenarios. CRT-Fusion demonstrates enhanced detection capabilities, accurately identifying objects and showing superior precision in predicting orientations and center positions. Furthermore, CRT-Fusion maintains high accuracy across diverse conditions, effectively capturing multiple targets. These results underscore the robustness and enhanced spatial perception of CRT-Fusion. Additional qualitative results available in the Supplemental Materials.
5 Discussion and Conclusion
Limitations and future work. While CRT-Fusion achieves significant performance gains over the baseline, the computational cost increases with the number of previous frames used for temporal fusion, limiting the number of frames that can be incorporated due to hardware constraints. This

---

CRT-Fusion
CRN
a
(b)
Figure 4: Qualitative results comparing CRT-Fusion and CRN: Green boxes indicate CRT-Fusion prediction boxes, blue boxes denote CRN prediction boxes, and red boxes represent ground truth (GT) boxes.
issue likely due to the parallel fusion structure used for combining BEV features. To address this, potential solution is adopt recurrent fusion structure, which fuses BEV features temporally in a sequential manner. This approach could maintain computational feasibility while incorporating long-term historical BEV features. Future work will explore this recurrent fusion architecture for CRT-Fusion to further reduce its computational complexity.
Conclusion. In this paper, we introduced CRT-Fusion, a novel framework that integrates temporal information into radar-camera fusion for 3D object detection. By explicitly taking the motion of dynamic objects into account through our proposed Motion Feature Estimator and Motion Guided Temporal Fusion modules, CRT-Fusion significantly improves detection accuracy and robustness in complex real-world scenarios. Our Multi-Vice Fusion module enhances depth prediction by leverag- ing radar features to improve image features before fusing them into a unified BEV representation. Extensive experiments on the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-ther -frtte-rr performance in the radar-camera-baat existing methods. Additionally, weather and lighting conditions, highlighting its potential for real-world deployment. We believe that work will inspire further research the fusion of temporal and multi-modal information for robust perception in adverse environments.
3D object detection category, surpassing all approach demonstrates remarkable robustness under diverse
Acknowledgement
This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2022-0-00957, model PIM (Processor in Memory) semiconductor technology
Distributed on-chip memory-procest development for edge applications): the IITP grant funded by the Korea government (MSIT) (No. RS-2021-11211 1343, Artificial Intelligence Graduate School Program, Seoul National University); and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2020R 1220R1A2C2011 A2C2012146).
10

---

References
[1] Youngseok Kim, Sanmin Kim, Jun Won Choi, and Dongsuk Kum. Craft: Camera-radar 3d object detection with spatio-contextual fusion transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1 1160-1166 2023.
[2] Zhiwei Lin, Zhe Liu, Zhongyu Xia, Xinhao Wang, Yongtao Wang, Shengxiang Qi, Yang Dong, Nan Dong, Le Zhang. and Ce Zhu. Rebevdet: Radar-camera camera fusion in bird's eye view for 3d object detection. arXiv preprint arXiv:2403. 16440, 2024.
[3] Ramin Nabati and Hairong Qi. Centerfusion: Center-based radar and camera fusion for 3d object detection. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1527-1536, 2021.
[4] Jisong Kim, Minjae Seong, Geonho Bang, Dongsuk Kum, and Jun Won Choi. Rem-fusion: Radar-camera multi-level fusion for 3d object detection. arXiv preprint arXiv:2307.10249, 2023.
[5] Yunfei Long, Abhinav Kumar, Daniel Morris, Xiaoming Liu, Marcos Castro, and Punarjay Chakravarty, Radiant: Radar-image association network for 3d object detection. 2023.
[6] Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, and Dongsuk Kum. Crn: Camera radar net for accurate, robust, efficient 3d perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17615-17626, 2023.
[7] Lingjun Zhao, Jingyu Song. and Katherine A Skinner. Crkd: Enhanced camera-radar object detection with cross- modality knowledge distillation. arXiv preprint arXiv:2403. 19104, 2024.
[8] Junho Koh, Junhyung Lee. Youngwoo Lee, Jackyum Kim, and Jun Won Choi. Mgtanet: Encoding sequential lidar points using long short-term motion temporal attention for 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1179-1188 2023.
[9] Zhenxun Yuan, Xiao Song, Lei Bai, Zhe Wang, and Wanli Ouyang. Temporal-channel trans- former for 3d lidar-based video object detection for autonomous driving. IEEE Transactions Circuits and Systems for Video Technology, 32(4):2068-22 -078, 2021.
[10] Zhenyu Zhai, Qiantong Wang, Zongxu Pan, Zhentong Gao, and Wenlong Hu. Muti-frame point cloud feature fusion based on attention mechanisms for 3d object detection. Sensors, 22(19):7473, 2022.
[11] Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser, and Alireza Fathi. An Istm approach to temporal 3d object detection in lidar point clouds. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 266-282. Springer, 2020.
[12] Zetong Yang. Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d network for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1863-1872 1872, 2021.
attention
[13] Charles Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 6134 6144, 2021.
[14] Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal object detection. In European Conference Computer Vision, pages 680-697. Springer, 2022.
[15] Jinghua Hou, Zhe Liu, Zhikang Zou, Xiaoqing Ye, Xiang Bai, et al. Query-based temporal fusion with explicit motion for object detection. Advances Neural Information Processing Systems, 36, 2024.
11

---

[16] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning eeye representation from multi nulei-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[17] Yinhao Li. Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1477-1485, 2023.
[18] Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203. 17054, 2022.
[19] Zhuofan Zong, Dongzhi Jiang, Guanglu Song, Zeyue Xue, Jingyong Su, Hongsheng Li, and Yu Liu. Temporal enhanced training of multi-view 3d object detector via historicalobject prediction. arXiv preprint arXiv:2304.00967, 2023.
[20] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection, arXiv preprint arXiv:2210.02443; 2022.
[21] Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang, and Di Huang. Sts: Surround-view temporal stereo for multi-view 3d detection. arXiv preprint arXiv:2208. 10145, 2022.
[22] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1486-1494, 2023.
[23] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and track- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784-11793, 2021.
[24] Jinqing Zhang, Yanan Zhang, Qingjie Liu, and Yunhong Wang. Sa-bev: Generating semantic- aware bird's s-eee--w feature for multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3348-3351, 2023.
[25] Jackyum Kim, Junho Koh, Yecheol Kim, Jachyung Choi, Youngbae Hwang, and Jun Won Choi. Robust deep multi-modal learning based on gated information fusion network, In Asian Conference on Computer Vision, pages 90-106. Springer, 2018.
[26] Jin Hycok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pages 720-736. Springer, 2020.
[27] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-1163) 2020.
[28] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11986 2022.
[30] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12697-12705, 2019.
2019,
12

---

[31] Taohua Zhou, Junjie Chen, Yining Shi, Kun Jiang, Mengmeng Yang. and Diange Yang. Bridging the view disparity between radar and camera features for multi-modal fusion 3d object detection. IEEF Transactions on Intelligent Vehicles, 8(2):1523-1533, 2023.
[32] Zizhang Wu, Guilian Chen, Yuanzhu Gan, Lei Wang, and Jian Pu. Myfusion: Multi-view 3d object detection with semantic-aligned radar and camera fusion. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 2766-2773 IEEE, 2023.
[33] Michael Ulrich, Sascha Braun, Daniel Köhler, Daniel Niederlöhner, Florian Faion, Claudius Gläser, and Holger Blume. Improved orientation estimation and detection with hybrid object detection networks for automotive radar. In 2022 IEEE 25th International Conference on Intelligent Transporiation Systems (ITSC), pages 111-117. IEEE, 2022.
[34] Geonho Bang, Kwangjin Choi, Jisong Kim, Dongsuk Kum, and Jun Won Choi. Radardistill: Boosting baser object detection performance via knowledge distillation from lidar features. arXiv preprint arXiv:2403.0 000600 2024.
[35] Haisong Liu, Yao Teng, Tao Lu. Haiguang Wang, and Limin Wang. Sparsebev: High- performance sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18580-18590, 2023.
[36] Shihao Wang, Yingfei Liu, Tiancai Wang. Ying Li, and Xiangyu Zhang. Exploring object- centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3631, 2023.
[37] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv: 09492, 2019.
(38] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2485-2499 2020.
[39] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird s-eee view representation. In 2023 IEEE international conference robotics and automation (ICRA), pages 2774-2781. IEEE, 2023.
[40] MMDetection3D Contributors. MMDetection33:: Open next-generation platform for general 3D object detection. https://github. hwtpggittttb..ooooooooooooot.t aaa.eennnnmmlmmmmmmettttt.o..t.a 2020.
[41] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High perforreefrm multi-cament 3d object detection in huddeye-view. arXiv preprint 12. 2021.
[42] Zhaoqi Leng, Guowang Li, Chenxi Liu, Ekin Dogus Cubuk, Pei Sun, Tong He, Dragomir Anguelov, and Mingxing Tan. Lidar augment: Searching for scalable 3d lidar data augmentations. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7039-7045. IEEE, 2023.
[43] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
13

---

Supplementary Materials for CRT-Fusion
In this Suppiementary Material, we provide additional details that were not covered in the main paper. We organize the content as follows: comprchensive formulation of loss functions (Section A architectural specifications and training protocols (Section BB) extensive ablation studies and com- putational efficiency analysis (Section C. in-depth qualitative evaluation on the nuScenes dataset (Section D) and discussion of societal implications (Section
A
Loss Function
The total loss function used in CRF-Fusion is composed of five components: a standard 3D object detection loss and four additional losses derived from different head networks within our model. The total loss Lioral is given by
Leotal = Ldet + Adeeth + Xeeg + AvelLvve + Aoce
(10)
where Ldet the 3D object detection loss, Leepth is the loss from the Depth Prediction Head in MVF. Leeg is the loss from the Perspective- View Semantic Segmentation Head in MVF, Luei is the loss from the Velocity Prediction Head in MFE, and Loce is the loss from the Object Occupancy Prediction Head in MFE. The parameters Adepth. Aseg, Avet, and Aace are the weights for the corresponding loss terms. The Depth Prediction Loss uses binary cross-entropy loss for depth estimation, with a weight of Adepth 3.0, following the approach used in BEVDepth. For the Perspective View Segmentation Loss, we also employ the binary cross-entree coos--nntropy loss, with a weight of Aseg 25, inspired by SA-BEV. The Velocity Prediction Loss, which handles velocity (Vay (yn) and orientation prediction, utilizes Mean Squared Error (MSE) with weight of Awet = 1. Finally, the BEV Object Occupancy Loss uses Binary Focal Loss for foreground and background segmentation, with weight of Xnce 30.
B Implementation Details
The nuScenes datasets 27 are publicly available to use under CC BY-NC-SA 4.0 license and can be downloaded from httph::/tt/:ttt http://wwsceess..o org/ We implemented our model using the MMDetection3D [40] codebase and trained it for a total of 24 epochs. The training process consists of two phases. In the initial phase, the model is trained for 6 epochs without the MGTF module. Then, the entire model is trained for the remaining 18 epochs. For the ResNet50-bass basedd model, we used 4 NVIDIA RTX 3090 GPUs for training, while for ResNet101 and ConvNeXt-B. we used NVIDIA A100 GPUs Table summarrzess the training settings for different camera backbone networks
In the perspective view, we apply data augmentation techniques consistent with previous studies 41 including horizontal random flipping, random scaling ([-0.06, 0.11]), and random rotation (+..44). For the bird's-eye view, employ random flipping along the and y axes, random scaling ((0.95, 1.05]), and random rotation (+..3925 rad). Additionally, we use technique that randomly drops radar sweeps and points (42) To ensure fair comparison with other models, we do not utilize ground-truth sampling augmentation (GT-AUG) 43), which is commonly used in LiDAR-based models.
C
Additional Experimental Results
C.1 Analysis of the Motion Feature Estimation Module.
The Motion Feature Estimation (MFE) module generates a motion-aware BEV feature map by estimating velocity for each grid and using it to refine the BEV representation. While MFE can be integrated into various architectures, its effectiveness is highly reliant on accurate velocity predictions. To demonstrate this, we applied MFE to both the camera-based model BEVDepth and our proposed CRT-Fusion, which incorporates radar data.
Figure 5 shows the results of applying the MFE module to each model. The red boxes represent Ground Truth (GT) boxes, the red arrows indicate GT velocity, and the white arrows show the predicted velocity from the MFE module. The size and direction of the arrows reflect the velocity's s
14

---

Table Training settings for different backbone networks.
Configs
ResNet-50
ResNet-101
ConvNext-B
Image size
256 704
512 1408
512 1408
BEV size
128 x 128
256 256
256 x 256
Optimizer
AdamW
AdamW
AdamW
Base Learning Rate
2e-4
le-4
le-4
Weight Decay
1e-7
1e-7
1e-2
Optimizer Momentum
0.9, 0.999
0.9, 0.999
0.9, 0.999
Batch Size
16
12
12
Training Epochs
24
24
24
LR Schedule
Step Decay
Step Decay
Step Decay
Gradient Clip
(a) BEVDepth
(b) CRT-Fusion
Figure 5: Comparison of velocity prediction using the MFF module BEVDepth and CRT- Fusion. Red boxes are the Ground Truth (GT) boxes, red arrows show GT velocity, and white arrows represent predicted velocity. Yellow highlights indicate areas where CRT-Fusion predicts velocity more accurately, while orange highlights show static objects correctly identified by CRT-Fusion but misclassified by BEVDepth.
magnitude and direction. The yellow and orange highlights illustrate the differences in velocity prediction accuracy and the ability to distinguish static objects. Specifically, in the yellow-highlighted areas, CRT-Fusion predicts velocities closely aligned with GT, whereas BEVDepth shows inaccuracies. In the orange- saange-iighlihhted areas, CRT-Fusion accurately identifies static objects, while BEVDepth misclassifies them as dynamic. In conclusion, the effectiveness of the MFE module closely tied to the accuracy of velocity estimation. Integrating radar data in CRT-Fusion significantly enhances the module's ability to generate reliable motion-aware BEV features, underscoring the value of sensor fusion for robust 3D object detection.
15

---

Table 8: Ablation study of temporal frames.
of prev frames
NDS
mAP
mATE
mAOE
mAVE
0
50,4
44.4
0.515
0.598
0.567
1
55.1
47.1
0.508
0.570
0.295
2
56.0
48.1
0.506
0.542
0.280
3
56.6
48.8
0.498
0.540
0.269
4
56.7
49.3
0.505
0.556
0.268
5
56.6
49.2
0.506
0.557
0.262
6
57.2
50.0
0.494
0.557
0.265
7
57.3
49.8
0.497
0,532
0.265
8
57.3
49.7
0.499
0,531
0.261
Table 9: Ablation study of the hyperparameters of CRT-Fusion.
TP
NDS
mAP
NDS
mAP
NDS
mAP
of orid
NDS
mAP
0.15
56.8
49.4
0.00
56.7
49.2
0.0
57.0
49.2
32
56.1
48.8
0.20
56.6
49.0
0.05
57.2
50.0
0.5
56.6
48.9
64
56.8
49.3
0.25
57.2
50.0
0.10
50.0
50.0
0.30
56.7
49.4
0.15
55.8
48.4
1.5
56.8
49.3
56.7
49.4
(a) Perspective view segmentation threshold
(b) TB: Bird eye's view segmentation threshold
(c) Motion estimation threshold
(d) Number of radar grid
in RCA
C.2 Hyperparameter Analysis.
The ablation studies on the hyperparameters used in CRT-Fusion are shown in Tables and Table examines the optimal number of previous frames consider for achieving the best performance. The results show that the performance of CRT-Fusion improves as the number of frames increases, with 6 frames yielding the best results when considering computational cost and performance. Tables 9 (a) and (b) investigate the optimal segmentation thresholds for the perspective view and bird's eye view (BEV), respectively. In the perspective view, segmentation threshold Tp of 0.25 achieves the best performance, while in the BEV, threshold of 0.05 yields the highest performance. Increasing the threshold may lead the removal foreground regions, resulting a decline performance. Table (c) explores the velocity threshold т, for considering BEV grid dynamic object in the MFE module. The results demonstrate that considering BEV grids with velocities above m/s achieves highest performance. Finally, Table (d) examines the number of radar BEV grids to match with each image feature pixel the RCA module. The experiments reveal that matching 128 grids yields the best performance.
C.3 Evaluating Model Efficiency.
Table T0 presents the analysis of performance, latency, and GPU memory usage of CRT-Fusion and CRN as the number of past frames increases. We reproduced the CRN model following the official implementation. For fair comparison, both models employ identical camera and radar backbones with consistent input image dimensions. All experimental evaluations were conducted on a single NVIDIA RTX 3090 GPU and an Intel Xeon Silver 4210R CPU.
Our experimental results demonstrate that CRT-Fusion consistently achieves superior performance in both NDS and mAP metrics across all temporal configurations. Notably, as we extend the temporal context by incorporating additional past frames, CRT-Fusion exhibits stable resource usage with minimal degradation. The memory efficiency of our approach is evidenced by its peak consumption of 3.754 GB frames, while CRN reaches 4.342 GB under identical conditions. In terms of computational latency, CRT-Fusion demonstrates robust scalability, with inference time increasing
16

---

Table 10: Quantitative results comparing CRT-Fusion and CRN. Comparison of accuracy (NDS. mAP) and efficiency (GPU memory, Latency) of CRT-Fusion and CRN with increasing number of history frames. Mem. represents the GPU memory consumption at inference phase.
FuT-Fusion
CRN
of prev frumes
NDS
mAP
Mem (GB)
Latency (ms)
NDS
mAP
Mem (GB) Latency (ms)
50.35
41.36
4.232
55.6
1
55.08
47.07
3.692
62.0
53.14
43.
4.244
85.0
2
56.03
48.07
3.698
62.8
54.79
45.88
4.270
120.2
3
56.60
48.82
3,706
64.2
55.57
46.87
4,286
138.6
4
56.66
49.29
3.714
64.9
56.01
47.10
4.302
187.1
5
56.55
49.19
3.724
66.2
55.83
47.18
4,310
212.0
57.15
50.01
3.744
67.1
55.55
47.40
4.326
243.5
7
57.30
49.75
3.754
69.6
56.30
47.61
4.342
273.0
moderately from 57.1 ms to 69.6 ms when expanding from O to 7 frames. CRN also benefits from substantial increase in computational
increased temporal information, although it experiences overhead, with latency reaching 273 ms at 7 frames. These empirical results highlight the efficiency of our proposed architecture in maintaining real-time inference capabilities while utilizing temporal information effectively.
C.4 Latency Analysis of CRT-Fusion and CRT-Fusion-light.
Table 11 presents the component-wise latency analysis of our CRT-Fusion variants. CRT-FUsion-light,, which incorporates lightweight radar backbone and processes fewer temporal frames, achieves an overall latency of 48.8 ms compared to 67.1 of the original architecture. The efficiency gains primarily originate from the radar processing pipeline, where the radar backbone latency decreases from 13.9 ms 3.9 ms, and the MGTF module reduces from 15.2 8.6
Table 11: Ablation study of Inference Time.
Methods
C.B.
R.B.
MVF
MFE
MGTF
Head
Total
CRT-Fusion
13.4 ms
13.9 ms
15.0 ms
0.7 ms
15.2 ms
8.9 ms
67.1 ms
CRT FrT-Fusiin-l
13.4 ms
3,9 ms
13.3 ms
0.7 ms
8.6 ms
8.9 ms
48.8 ms
D Qualitative results of CRT-Fusion.
Figure 6 (showcases qualitative results of our proposed CRT-Fusion method on the nuScenes validation set. We compare the object detection performance of CRT-Fusion with the baseline model, BEVDepth, by visualizing the predicted bounding boxes in the BEV representation. Both models employ ResNet- 101 as the camera backbone for feature extraction. The examples demonstrate the effectiveness of CRT-Fusion in various driving scenarios, such urban streets, intersections, and highways. Our model consistently produces more accurate and well-aligned bounding boxes compared to baseline model.
E Discussions of potential societal impacts.
CRT-Fusion has the potential to enhance the accuracy and robustness of 3D object detection in autonomous vehicles and roboties systems by fusing radar, camera, and temporal information. Despite its benefits, the reliance on sophisticated technology and data fusion could lead to increased costs and complexity, potentially limiting accessibility and widespread adoption.
17

---

(a) BEVDepth
(b) CRT-Fusion
Figure 6: Qualitative results under different scenarios on the nuScenes validation set. Red boxes represent ground truth annotations, while blue and green boxes indicate the predicted bounding boxes from BEVDepth and CRT-Fusion, respectively. The white points represent the radar point cloud.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contribution aims to enhance object detection performance through temporal fusion considering dynamic objects, as shown in Figure and Tables and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
This is discussed in Supplementary Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
This is shown in Figures and 4, and Table 5.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our model, novel radar-camera model, is thoroughly detailed in the CRT- Fusion section.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification:
Our code will made publicly available the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
www.SSubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details of our experiments are provided in both the main text and Supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our performance evaluation on the nuScenes dataset adheres to standard practice in the field, which does not include probabilistic values or error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is mentioned in Supplementary Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///eerrrppp.cc//aal///eaea
Answer: [Yes]
Justification: This research fully adheres to the NeurIPS Code of Ethics, ensuring ethical methodologics, privacy protections, and consideration of societal impacts throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is mentioned in Supplementary Section E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
This is mentioned in Supplementary Section A.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "0BkxAXKwwk", "reviewer": "Reviewer_NhQh", "summary": "The paper talks about multi sensor fusion using Motion Information for Bird’s Eye View Object\nDetection. Authors have presented a multiple step approach to improve the object detection on nuScenes dataset. The proposed approach comprises of three parts; Multi-View Fusion (MVF) that enhances depth prediction by leveraging radar features to improve image features before fusing them into a single BEV vector; a Motion Feature Estimator (MFE) step that estimates pixel-wise velocity information and Motion Guided Temporal Fusion (MGTF) step that iteratively aligns and fuses feature maps across multiple timestamps. The proposed approach is tested on the latest nuScenes dataset for object detection and obtains state-of-the-art results on the dataset surpassing many publicly listed approaches.", "questions": "1. Can authors provide a small description of how this model can be made a bit efficient such that this can be deployed in the real world?\n2. Figure 2 is not very clear. For the MVF (bottom left side) figure, what does the middle gray box represent?\n3. Line 135 states an LSS approach, but that hasn’t been mentioned anywhere. What is LSS?\n4. Line 137 states “existing state-of-the-art…”, can authors please cite which work they are referring to here?\n5. The calculation of GT for MFE is not very clear. Why do we use the input for MVF to compute the GT for MFE (line 199)?\n6. For line 201, how did the authors determine the value for T_{iou}? Why did they consider 0.5? Was there some ablation study conducted?\n7. For Equation 7, it’s not very clear what B^_{t-k} is? Also how was the value for T_{v} chosen here?\n8. Table 1 here refers to TTA. What is TTA?\n9. Can the authors provide some qualitative results? And compare the existing work with the proposed approach to show what are the strengths of their approach? Just listing down metrics may not be motivating enough to see where the proposed approach does best.\n10. Line 283 states that CRT does not do well in Sunny conditions which is a bit strange since sunny weather conditions should be ideal for object detection. Can the authors provide some explanation here?\n11. For table 6, RCA has worse performance as compared to RGVT on mATE. Can authors address why is that the case?", "limitations": "Authors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to read and follow. Authors have described each step carefully and mentioned proper mathematical formulation for each step.\n2. The motivation and improvements in the performance is clear. In many cases the proposed framework archives state of the art results. \n3. Ablation study is also very well supported. Authors have included results for baseline modules and how each of their component improves the performance of object detection.", "weaknesses": "1. I believe that the technical novelty of the paper is somewhat limited. The idea of MVF fusion, i.e. Radar and Camera image feature fusion has been studied well in the past.\n2. The paper lacks qualitative results completely. No where in the main paper have the authors presented any qualitative analysis. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails.\n3. Overall approach with multiple components, although produces state-of-the art results, but is nearly impractical to scale or even productionize in a real world setting. Authors haven’t mentioned anywhere in the paper about this limitation (only described it briefly in the Supplementary section). The limitations of the current approach should be more broadly presented in the main paper, i.e. if the number of frames increase, the model will have high computational cost and hence not scalable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The novelty of the paper is limited.**\n\nWe are sorry that the reviewer does not appreciate the contribution of our study, to which we have dedicated much effort. We believe that our work can make a substantial contribution to enhancing the effectiveness of temporal fusion in the field of radar-camera based 3D object detection. While several radar-camera fusion methods have been previously studied, our research primarily focuses on an effective temporal fusion approach for camera-radar integration. Unlike existing methods that merely concatenate features from previous frames, our method achieves spatial feature alignment using predicted velocity information. This novel approach has not been proposed in previous methods such as CRN and BEVFusion-R. Our method performs feature-level temporal alignment and fusion, resulting in significant performance gains over existing techniques.\n\n**W2: The paper lacks qualitative results.**\n\nWe have presented qualitative results in Figure 1 of the attached PDF file. In the final version, we will add this comprehensive qualitative analysis.\n\n**W3: The approach is impractical for real-world deployment and lacks sufficient limitations analysis.**\n\nAlthough our model achieves state-of-the-art performance, its latency and memory usage are comparable to other methods, making it suitable for practical implementation. We have also presented a lightweight version, CRT-Fusion-Light, which meets real-time requirements for deployment in real-world settings. \n\nWe have discussed the limitations of our model in the Author Rebuttal section  Q3.\n\n**Q1: Can authors provide a small description of how this model can be made more efficient for real-world deployment?**\n\nThe computation time of our model can be optimized using model compression techniques or reducing the model capacities. CRT-Fusion-light, presented in Table 1 of the manuscript, is an optimized version of CRT-Fusion that yields improved FPS with minimal performance drop. Please note that CRT-Fusion-light achieves 20.5 FPS, potentially satisfying real-time requirements.\n\n**Q2: Figure 2 is not very clear. For the MVF (bottom left side) figure, what does the middle gray box represent?**\n\nIn Figure 2, the middle gray box in the MVF represents the intermediate feature, $\\hat{F}_c$, as shown in Figure 3\n\n**Q3: Line 135 states an LSS approach, but that hasn’t been mentioned anywhere.**\n\nWe apologize for not providing the introduction to LSS. The LSS (Lift, Splat, Shoot) method is a widely used operation for transforming multi-view 2D images into BEV features in camera-based 3D perception. This method was introduced in the ECCV 2020 paper `Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D'. LSS predicts depth values for each pixel in 2D images and converts multi-view camera features into BEV feature  map through view transformation. We will add detailed information about LSS in the final version.\n\n**Q4: Line 137 states “existing state-of-the-art…”, can authors please cite which work they are referring to here?**\n\nThe 'existing state-of-the-art' mentioned in Line 137 refers to the CRN method. We will mention CRN in the final version.\n\n**Q5: The calculation of GT for MFE is not very clear. Why do we use the input for MVF to compute the GT for MFE (line 199)?**\n\nWe apologize for any confusion regarding the calculation of GT for MFE. We did not use the BEV feature map $B_{t-k}(x,y)$ in generating GT velocity.  In equation 4, we calculated the IoU ratio $r(x,y)$ between the physical box whose size corresponds to the pixel at $(x,y)$ and the 3D object boxes projected onto the BEV domain. If this IoU ratio was higher than $\\tau_{iou}$, we considered this pixel as positive and assigned the GT velocity for supervision. \n\n**Q6: For line 201, how did the authors determine the value for $\\tau_{iou}$ Value? Why did they consider 0.5? Was there some ablation study conducted?**\n\nThe threshold $\\tau_{iou}$ was determined empirically. Below are the performance of CRT-Fusion obtained with several values of $\\tau_{iou}$. \n\n| $\\tau_{iou}$ | NDS  | mAP  |\n|------------|------|------|\n| 0.3       | 56.9 | 49.3 |\n| 0.5   | **57.2** | **50.0** |\n| 0.7       | 57.0 | 49.7 |\n\n**Q7: For Equation 7, it’s not very clear what $B_{t-k}$ is? Also how was the value for $\\tau_{v}$ chosen here?**\n\nThe notation $B_{t-k}$ is the BEV feature map obtained from the MVF module. \n\nThe threshold $\\tau_{v}$ was determined experimentally. Our ablation study for $\\tau_{v}$ was provided in Table 9 (c) of the Supplementary Materials.\n\n**Q8: What is TTA?**\n\nTTA stands for Test Time Augmentation, a method that applies various augmentations to test data to improve model prediction.\n\n**Q9: Can the authors provide some qualitative results? And compare the existing work with the proposed approach to show what are the strengths of their approach?**\n\nWe have added qualitative results of our CRT-Fusion in Q2 of the Author Rebuttal. Our detailed comparison with CRN was provided in Figure 1 of the PDF file attached.\n\n**Q10: Line 283 states that CRT does not do well in Sunny conditions which is a bit strange since sunny weather conditions should be ideal for object detection.**\n\nIn sunny conditions, our model achieves 54.7\\% mAP, nearly identical to CRN with only a 0.1\\% difference. In contrast, we see notable improvements in more challenging conditions like rainy weather and nighttime. The slight variation in sunny conditions is within the margin of normal experimental variance and does not indicate a significant trade-off. Instead, it demonstrates that our model maintains strong performance in good conditions while offering substantial improvements in challenging scenarios.\n\n**Q11: For table 6, RCA has worse performance as compared to RGVT on mATE.**\n\nOur RCA module result in only 0.002 drop in mATE compared to RGVT. Unfortunately, we were unable to figure out why RCA did not improve mATE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epBxfOJzlt", "reviewer": "Reviewer_6YHD", "summary": "The paper proposes CRT-Fusion that fuses radar and camera inputs for 3D object detection in BEV space. A radar-camera azimuth attention module is proposed for improving image features for better motion awareness, which further improves the quality of image BEV features. A motion-guided temporal fusion is introduced to align dynamic objects in history BEV feature maps with their positions in the current BEV feature map, using predicted object presence information and predicted velocities. The authors conduct experiments on the nuScenes dataset and prove the effectiveness of the proposed model.", "questions": "In Table 1, I notice that CRT-Fusion-light has lower mATE, mASE, mAOE, mAVE, mAAE compared to CRT-Fusion, which is unexpected. It would be beneficial if the authors could provide a detailed explanation or revisit the experimental setup to clarify these results.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed framework is concise and straightforward, while showing good performance on nuScenes benchmark\n2. The proposed RCA first decouples the image feature map to row and column features, and only applies cross-attention on column features, which enhances image features with a relatively low computation cost.\n3. The motion guided temporal fusion mitigates the BEV feature misalignment issue of moving objects across different timestamps, and improves the 3D object detection performance.", "weaknesses": "1. The proposed temporal fusion requires aggregating multiple feature maps of different timestamps in a recurrent way. The inference latency and required GPU memory will grow linearly with the number of past frames.\n2. The multi-view fusion section needs more details. It's unclear how the radar BEV feature map is obtained and how the gated fusion is performed. It's better to include some equations for illustration.\n3. The equation 7 for motion-guided temporal fusion seems to be incorrect. Further details for generating the shited feature map are needed. For example, let's consider this situation: the velocity predictions at (x1, y1) and (x2, y2) are (Δx1, Δy1) and (Δx2, Δy2) respectively, and x1+Δx1 = x2+Δx2, y1+Δy1=y2+Δy2, what will the shifted feature map be at (x1+Δx1, y1+Δy1)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The inference latency and required GPU memory will grow linearly with the number of past frames.**\n\nOur model utilizes a memory bank structure where previous BEV features obtained through a cascade of backbone, MVF, and MFE are stored in the buffer to reduce redundant computations. Thus, only the computation of MGTF increases  as the number of frames increases. This design significantly reduces the computational overhead for temporal fusion.\n\nAs illustrated in Figure 2 of our PDF file, our model uses only 3.744 GB of GPU memory, and achieves a latency of 67.1ms, which demonstrates superior performance to the existing methods.\n\n**W2: The multi-view fusion section needs more details. It's unclear how the radar BEV feature map is obtained and how the gated fusion is performed.**\n\nWe apologize for the lack of details in the MVF (multi-view fusion) section.\n\nFirst, the radar BEV feature was obtained by applying a PointPillars-based backbone network.\n\nMVF combines feature maps obtained from each modality using the Gated Fusion Network. The Gated Fusion Network computes the combining weights that adaptively adjust the contributions of the two modalities.\nLet $F_r$ be the radar BEV feature and $F_c^{BEV}$ be the camera BEV feature map. The equations for obtaining the combining weights are given by \n\n$W_r = \\sigma(\\text{Conv}_r(F_r \\oplus F_c^{BEV}))$\n\n$W_c =  \\sigma(\\text{Conv}_c(F_r \\oplus F_c^{BEV})),$\n\nwhere  $\\sigma$ denotes the sigmoid function, and $\\text{Conv}_r$ and $\\text{Conv}_c$ are convolutional layers.\nThe final fused BEV feature map is obtained by\n\n$F_{fused} = (W_r \\times F_r) \\oplus (W_c \\times F_c^{BEV}),$\n\nwhere  $\\times$ denotes element-wise multiplication operation and $\\oplus$ indicates element-wise summation operation.\nWe will add this explanation in the final version. \n\n**W3: The equation 7 for motion-guided temporal fusion seems to be incorrect. Further details for generating the shifted feature map are needed.**\n\nAs the reviewer mentioned, there may be cases where feature maps overlap when shifted. To address this, we apply averaging operation on the overlapping features. We will comment on this point in the final version of our paper.\n\nWe can revise the equation 7 as folows:\nFor each coordinate $(x, y)$ in the BEV feature map $B_{t-k}$, the feature value $B_{t-k}(x, y)$ is shifted based on the corresponding velocity vector $M_{t-k}(x, y) = [\\Delta x, \\Delta y]$ if the magnitude of the velocity exceeds a specified threshold $\\tau_v$. The shifted feature maps are obtained as\n\n$B'_{t-k}(x, y) = \\frac{1}{|S(x,y)|}\\sum_{(i,j) \\in S(x,y)} {B}_{t-k}(i, j)$\n\nwhere \n\n$S(x,y) = \\{(i,j): x= i+\\lfloor \\Delta x \\rceil, y= j+ \\lfloor \\Delta y \\rceil , |M_{t-k}(i, j)|> \\tau_v \\}$\n\nwhere  $|S(x,y)|$ denotes the cardinality of $S(x,y)$ and $\\lfloor \\cdot \\rciel$ denotes the rounding operation.\n\n**Q1: An explanation is needed for CRT-Fusion-light's performance compared to CRT-Fusion in Table 1.**\n\nIt is not fair to compare CRT-Fusion with CRT-Fusion-light&#8224; since CRT-Fusion did not employ CBGS augmentation. \nPlease compare CRT-Fusion&#8224; with CRT-Fusion-light&#8224; since CRT-Fusion&#8224; employed CBGS augmentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UeLMyno0V5", "reviewer": "Reviewer_dkEf", "summary": "In this study, the authors present CRT-Fusion, an innovative framework designed to incorporate temporal information into radar-camera fusion, thereby enhancing the robustness of 3D object detection. The CRT-Fusion framework is composed of three integral modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF).", "questions": "1. The paper lacks a detailed description of the loss function used in the proposed framework.\n\n2. The authors claim that the Multi-View Fusion (MVF) module enhances depth prediction. However, no experimental evidence is provided to substantiate this assertion.\n\n3. The Motion Feature Estimator (MFE) module is responsible for the pixel-wise velocity estimation task, yet the paper does not clarify how the velocity estimation is supervised during the learning process.\n\n4. The overall improvement achieved by the proposed method is limited. Given that the method incorporates temporal information, velocity estimation, and depth estimation tasks, its performance enhancement compared to the CRN framework is relatively modest.", "limitations": "Please consult the question section for further information.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a novel framework capable of integrating temporal information into radar-camera fusion for enhanced 3D object detection.\n\nOverall, the paper is well-written and clearly articulated. Experimental results demonstrate that this method outperforms previous approaches in 3D object detection.", "weaknesses": "Please consult the question section for further information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The paper lacks a detailed description of the loss function used in the proposed framework.**\n\nThe total loss function used in CRF-Fusion consists of five loss terms: a standard 3D object detection loss term and four additional loss terms derived from different head networks within our model. The total loss $ L_{total}$ is given by\n\n$ L_{total} = L_{det} + \\lambda_{depth} L_{depth} + \\lambda_{seg} L_{seg} + \\lambda_{vel} L_{vel} + \\lambda_{occ} L_{occ}, $\n\nwhere $L_{det}$ is the 3D object detection loss, $L_{depth}$ is the loss from the Depth Prediction Head in MVF, $L_{seg}$ is the loss from the Perspective-View Semantic Segmentation Head in MVF, $L_{vel}$ is the loss from the Velocity Prediction Head in MFE, and $L_{occ}$ is the loss from the Object Occupancy Prediction Head in MFE. The parameters $\\lambda_{depth}$, $\\lambda_{seg}$, $\\lambda_{vel}$, and $\\lambda_{occ}$ are the  weights for the corresponding loss terms.\n\nThe specific descriptions for each loss component are as follows. The Depth Prediction Loss uses binary cross-entropy loss for depth estimation with a weight of $\\lambda_{depth} = 3.0$ following the approach used in BEVDepth. For the Perspective View Segmentation Loss, we also employ the binary cross-entropy loss with a weight of $\\lambda_{seg} = 25$. The Velocity Prediction Loss, which handles velocity $(v_x, v_y)$ and orientation prediction, utilizes Mean Squared Error (MSE) with a weight of $\\lambda_{vel} = 1$. Finally, the BEV Object Occupancy Loss uses Binary Focal Loss for foreground and background segmentation with a weight of $\\lambda_{occ} = 30$.\n\nWe will add the detailed information on the loss function in the final version. \n\n\n**W2: No experimental evidence is provided that the MVF module enhances depth prediction.**\n\nWe conducted additional experiments for analyzing the impact of the RCA component in MVF module on depth prediction accuracy.  Our experimental results show that utilizing RCA significantly improves depth prediction accuracy in Mean Square Error (MSE). The table below provides the detailed results.\n\n| Methods       | Depth MSE (m) |\n|---------------|:-------------:|\n| Ours w/o RCA  |      4.3      |\n| Ours w RCA    |      3.8      |\n\n\n**W3: The paper does not clarify how the velocity estimation in the MFE module is supervised during the learning process.**\n\nThe velocity values predicted by the MFE module are supervised with the GT velocity values through MSE loss. Details on the loss function used in CRT-Fusion are provided in our response to the comment W1. The MSE loss measures errors in both the magnitude of velocity in the x and y axes, as well as the orientation value of the velocity.\n\n**W4: The overall improvement by the proposed method is limited compared to the CRN framework. Given that the method incorporates temporal information, velocity estimation, and depth estimation tasks, its performance enhancement compared to the CRN framework is relatively modest.**\n\nTable 2 in our manuscript shows that our CRT-Fusion achieves a performance gain of 3.2\\% in NDS and 1.4\\% in mAP over CRN. These gains can be considered substantial in the highly competitive nuScenes BEV object detection benchmarks! Please note that like our CRT-Fusion, the CRN method also used both depth estimation and utilized temporal information. Therefore, the performance gains achieved by our proposed method are attributed to both radar-assisted depth prediction and velocity-driven feature alignment. This highlights the significant impact of our main ideas.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ytOt2IER7", "reviewer": "Reviewer_EC1R", "summary": "This paper presents a method, called CRT-Fusion, for 3D object detection that fuses temporal information with radar-camera features. The CRT-Fusion captures the object motion with three modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module uses Radar-Camera Azimuth Attention (RCA) to enhance the depth information in camera BEV features and combines them with radar features through a gated fusion network. The MFE module estimates pixel-wise velocity and performs BEV segmentation, which guides the temporal feature alignment and fusion in the MGTF module across multiple timestamps. The evaluation on the nuScenes dataset indicates that CRT-Fusion achieves state-of-the-art performance.", "questions": "In line 212, should it be $M_{t-k}(x, y)$? In the inference, the model does not have access to ground truth.", "limitations": "The paper poorly discusses limitations. Failure cases and potential improvements of the approach in the future should be discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow. While there is prior work on fusing temporal and camera-radar features to improve 3D detection, the presented method appears novel as it learns how to estimate object pixel-wise velocity and occupancy to guide the fusion of object-level temporal features. The proposed Radar-Camera Azimuth Attention (RCA) is shown to effectively enhance image features with depth information from radar features. The experiments are well-designed, and the results on the nuScenes dataset show a significant performance gain over SOTA.", "weaknesses": "The paper could improve clarity in the following:\n1.\tAs mentioned in lines 181-185, CRT-Fusion uses object-level pixel-wise velocity and occupancy to integrate temporal information with BEV features. CRT-Fusion's performance relies on the accuracy of both detection and velocity estimation. It is not clear how to address error propagation across frames.\n\n2.\tRegarding inference latency, some approaches (e.g., CRN[6]) use only 2 adjacent frames as input. According to the results in Table 8 of the supplementary material, the 2-frame version of CRT-Fusion performs similarly to (or worse than) CRN in terms of NDS and mAP metrics. This raises concerns that the performance gains of CRT-Fusion may be a trade-off involving longer temporal cues, leading to increased latency and complexity. The paper should discuss this trade-off.\n\n3.\tThe batch size used during inference is not specified; Table 7 only presents the batch size used during training. Additionally, it appears that the authors directly used speed results from other papers for comparison, which may not be meaningful as the machines used for different models can vary significantly (Unfortunately, some other papers also directly cite and compare the speed) .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: It is unclear how to address error propagation across frames.**\n\nError propagation in CRT-Fusion has been addressed by (1) taking a weighted sum of aligned BEV feature maps through the Gated Fusion Network (GFN) and (2) applying MGTF only for absolute velocity predictions above a threshold ($\\tau_v$). The gating mechanism of GFN allows the model to reduce the contribution of misaligned features during feature aggregation. Additionally, velocity thresholding enables the model to be less affected by noisy velocity predictions. We will include these points in the final version.\n\n**W2: This raises concerns that the performance gains of CRT-Fusion may be a trade-off involving longer temporal cues, leading to increased latency and complexity.**\n\nTo address the review's concern, we evaluated the performance, latency, and GPU memory usage of both CRT-Fusion and CRN as the number of past frames increases.  Figure 1 in the attached PDF file shows that our model consistently outperforms CRN in terms of NDS and mAP across all frame settings. This confirms that the performance gains are not solely attributed to using longer temporal information. Furthermore, CRT-Fusion demonstrates superior efficiency in terms of GPU memory usage and latency compared to CRN.\n\n**W3: Inference batch size is not specified.**\n\nWe used a batch size of 1 for inference. \n\n**Q1: In line 212, should it be $M_{t-k}(x, y)$?**\n\nWe thank the reviewer for finding the typo.  $M_{t-k}^{GT}(x, y)$ should be replaced with $M_{t-k}(x, y)$, which will be corrected in the final version of the paper. \n\n**L1: The analysis of limitations is insufficient.**\n\nHere is our discussion about the limitations of our study, which will be added in the final version:\nWhile our model has achieved significant performance gains over the baseline, the computation time increases with the number of previous frames used for temporal fusion. This limitation means our method cannot accommodate as many previous frames as desired due to hardware constraints. This issue might stem from the parallel fusion structure used to combine BEV features. One potential remedy is to adopt a recurrent fusion structure that combines BEV features temporally in a recurrent fashion. By doing so, the computational feasibility can be maintained while incorporating BEV features from long-term horizon. In the future, we will explore the recurrent fusion architecture for temporal fusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQ0AoACvmF", "reviewer": "Reviewer_mr4N", "summary": "This paper proposes a radar-camera and temporal fusion method (CRT) for the 3D object detection task. CRT design Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF) modules. MVF employs radar information for 2D-to-3D projection. MGTF utilizes velocity and occupancy predictions from MFE to fuse multi-frame BEV features. The proposed method achieves state-of-the-art 3D object detection results on nuScenes.", "questions": "1. Training details. How is the model trained? Is the multi-frame multi-modal model trained in one step?\n2. What about the training cost?\n3. What TTA is used in Table 2?\n4. In Line 153, a set of M radar features are associated with W. What M does the model use? What if radar features are less than M?\n5. What is B_{t−k}(x, y) in eq.4？Please give more details for object occupancy ground truth generation.\n6. What do O and X mean in Table 5?", "limitations": "The authors address the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The MGTF module is interesting in decoupling and fusing multi-frame object features on dense BEV features.\n2. The paper achieves a good balance between performance and computational efficiency, making it a promising advancement for real-time radar processing in autonomous driving.\n3. The paper achieves a SOTA for radar-camera 3D object detection", "weaknesses": "1. The proposed method requires velocity supervision, while some datasets, such as Waymo, do not provide velocity labels.\n2. Table 1 shows that the proposed has well speed-accuracy trade-offs. It is better to provide the inference time of each part in the model.\n3. It seems that hyperparameters have a significant impact on model accuracy (Table 9).\n4. The paper does not show the model generalization of the proposed method. It is only applied to BEVDepth.\n5. BEV segmentation in MFE is misleading. It is better to use terms like object occupancy or foreground segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The proposed method requires velocity supervision, while some datasets, such as Waymo, do not provide velocity labels.**\n\nThe velocity ground truth (GT) for objects is provided as state information in the nuScenes dataset. Without this velocity information, we may need to calculate the derivatives from the object tracks over multiple frames.\n\n**W2: Table 1 shows that the proposed has well speed-accuracy trade-offs. It is better to provide the inference time of each part in the model.**\n\nAs per the reviewer's suggestion, we provide the breakdown of the inference time for CRT-Fusion and CRT-Fusion-light below\n\n|                    | Camera Backbone | Radar Backbone |  MVF  |  MFE  | MGTF | Detection Head |  Total  |\n|--------------------|:---------------:|:--------------:|:-----:|:-----:|:----:|:-----------------:|:-------:|\n| CRT-Fusion         |     13.4 ms     |    13.9 ms     | 15 ms | 0.7 ms|15.2 ms|      8.9 ms       | 67.1 ms |\n| CRT-Fusion-light   |     13.4 ms     |     3.9 ms     |13.3 ms| 0.7 ms| 8.6 ms|      8.9 ms       | 48.8 ms |\n\nCRT-Fusion-light uses a radar backbone network with lower capacity and combines fewer past frames compared to CRT-Fusion. Therefore, the main differences in inference time are due to the radar backbone, MVF, and MGTF components. \n\n**W3: It seems that hyperparameters have a significant impact on model accuracy (Table 9).**\n\nThe hyperparameters might impact our model's accuracy, requiring tuning to optimize performance. This is a common behavior observed in other 3D object detection models.\n\n**W4: The paper does not show the model generalization of the proposed method. It is only applied to BEVDepth.**\n\nMany BEV-based 3D object detectors employ the LSS structure presented in BEVDepth. Therefore, our method is applicable to any BEV frameworks employing LSS. We have not yet applied our method to Transformer-based BEV frameworks such as BEVFormer, which will be explored in future work.\n\n**W5: BEV segmentation in MFE is misleading. It is better to use terms like object occupancy or foreground segmentation.**\n\nAs the reviewer suggested, we will rename this as \"object occupancy\" to avoid confusion in the final revision of the paper.\n\n**Q1: Training details. How is the model trained? Is the multi-frame multi-modal model trained in one step?**\n\nAs detailed in Supplementary A, we trained the model using single-frame images for the initial 6 epochs, and then included temporal fusion for the subsequent 18 epochs.\n\n**Q2: What about the training cost?**\n\nTraining times without using Class Balanced Grouping and Sampling (CBGS) are provided below:\n\nResNet 50: 15 hours on 4 3090 GPUs\n\nResNet101: 26 hours on 3 NVIDIA A100 GPUs\n\nConvNeXt-B: 35 hours on 3 NVIDIA A100 GPUs\n\n**Q3: What TTA is used in Table 2?**\n\nWe applied TTA using  flipping in both BEV and image domains.\n\n**Q4: In Line 153, a set of $M$ radar features are associated with $W$. What $M$ does the model use? What if radar features are less than $M$?**\n\nWe use the setting $M=128$, which was determined empirically. If there are fewer radar points to cluster than $M$, we pad with zeros. Our ablation study  for $M$ are provided in Table 9 (d) of the Supplementary Materials.\n\n**Q5: What is $B_{t-k}(x, y)$ in eq. 4？ Please give more details for object occupancy ground truth generation.**\n\nWe apologize for any confusion regarding the calculation of GT for MFE. It is confusing to use $B_{t-k}(x,y)$ in Equation 4. \n\nIn Equation 4, $r(x,y)$ represents the Intersection over Union (IoU) ratio between the physical box corresponding to the pixel at a position $(x,y)$ and the 3D object boxes projected onto the BEV domain. Pixels with an IoU ratio exceeding $\\tau_{iou}$ are classified as positive and are assigned the GT velocity and GT occupancy state for supervision. We revised the equation 4 by replacing $B_{t-k}(x,y)$ with $H(x,y)$:\n\nDefine the IoU ratio $r(x, y)$ for pixel $(x, y)$ as\n\n\n$r(x, y) = \\frac{|H(x, y) \\cap \\mathcal{P}(\\mathcal{G})|}{|H(x, y) \\cup \\mathcal{P}(\\mathcal{G}) |}\n$\n\nwhere \\( H(x, y) \\) is the physical bounding box whose size corresponds to the pixel at $(x, y)$, $\\mathcal{G}$ is the set of 3D object boxes, and $\\mathcal{P}(\\mathcal{G})$ is the projection of these boxes onto the BEV domain.\n\n**Q6: What do O and X mean in Table 5?**\n\nIn Table 5 of the manuscript, 'O' indicates the use of the MFE and MGTF modules, while 'X' indicates non-use. Some mistakes in Table 5 have been corrected below\n\n| Method      | MFE & MGTF | NDS  | mAP  | mAVE  |\n|-------------|:----------:|:----:|:----:|:-----:|\n| BEVDepth    |      X     | 47.4 | 37.8 | 0.312 |\n| BEVDepth    |     O     | 46.9 | 37.3 | 0.349 |\n| CRT-Fusion  |      X     | 56.1 | 48.9 | 0.278 |\n| CRT-Fusion  |     O     | 57.2 | 50.0 | 0.265 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your thoughtful reviews. We have provided responses to the common questions raised by the reviewers below. Additionally, we have attached PDF files presenting qualitative results, as well as the analysis of GPU memory usage, latency, and performance relative to the number of frames.

**Q1: A comparative analysis with existing research is needed as the number of frames increases.**

Our model utilizes a memory bank structure where previous BEV features obtained through a cascade of backbone, MVF, and MFE are stored in the buffer to reduce redundant computations. Thus, only the computation of MGTF module increases  as the number of frames increases. This design significantly reduces the computational overhead for temporal fusion.

Table 1 and Figure 2 in the attached PDF file present the analysis of performance, latency, and GPU memory usage of CRT-Fusion as the number of past frames increases. (The hardware configuration included an NVIDIA RTX 3090 GPU and an Intel Xeon Silver 4210R CPU processor.) The trend of CRT-Fusion is compared with that of CRN.  For a fair comparison, we used the same camera backbone, radar backbone, and input image size. Our model achieved better performance in terms of mAP and NDS across all frames. Figure 2 also demonstrates that our model's GPU memory usage is consistently lower, and the latency of CRT-Fusion increases more slowly with the number of past frames compared to CRN. We will add these results in the final version. 

**Q2: The paper lacks qualitative results.**


We conducted a qualitative analysis of CRT-Fusion compared to CRN. Figure 1 in the attached PDF file shows the qualitative results in different scenarios. In Figure 1(a), CRT-Fusion accurately detects objects that CRN misses. In Figure 1(b), CRT-Fusion better predicts vehicle orientations and centers. Figure 1(c) also highlights CRT-Fusion's improved accuracy over CRN. We will include this qualitative analysis in the Supplemental Materials of the final version.


**Q3: The analysis of limitations is insufficient.**

Here is our discussion about the limitations of our study, which will be added in the final version:
While our model has achieved significant performance gains over the baseline, the computation time increases with the number of previous frames used for temporal fusion. This limitation means our method cannot accommodate as many previous frames as desired due to hardware constraints. This issue might stem from the parallel fusion structure used to combine BEV features. One potential remedy is to adopt a recurrent fusion structure that combines BEV features temporally in a recurrent fashion. By doing so, the computational feasibility can be maintained while incorporating BEV features from a long-term historical frames. In the future, we will explore the recurrent fusion architecture for CRT-Fusion.

**Q4: What is $B_{t-k}(x, y)$ in equation 4?**

We apologize for any confusion regarding the calculation of GT for MFE. It is confusing to use $B_{t-k}(x,y)$ in Equation 4. 

In Equation 4, $r(x,y)$ represents the Intersection over Union (IoU) ratio between the physical box corresponding to the pixel at a position $(x,y)$ and the 3D object boxes projected onto the BEV domain. Pixels with an IoU ratio exceeding $\tau_{iou}$ are classified as positive and are assigned the GT velocity and GT occupancy state for supervision. We revised the equation 4 by replacing $B_{t-k}(x,y)$ with $H(x,y)$:

Define the IoU ratio $r(x, y)$ for pixel $(x, y)$ as

$r(x, y) = \frac{|H(x, y) \cap \mathcal{P}(\mathcal{G})|}{|H(x, y) \cup \mathcal{P}(\mathcal{G}) |}$

where \( H(x, y) \) is the physical bounding box whose size corresponds to the pixel at $(x, y)$, $\mathcal{G}$ is the set of 3D object boxes, and $\mathcal{P}(\mathcal{G})$ is the projection of these boxes onto the BEV domain.

Meta-review: Details TBD, needs discussion.

Scores are very borderline, 4, 5, 4, 5, 6.
Decision: Accept (poster)