CRT-Fusion: Camera, Radar, Temporal Fusion Using  Motion Information  for 3D Object Detection

Authors: jisong  kim
minjae  seong
jun  won choi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:13:16

Abstract: Accurate and robust 3D object detection is a critical component in autonomous vehicles and robotics. While recent radar-camera fusion methods have made significant progress by fusing information in the bird's-eye view (BEV) representation, they often struggle to effectively capture the motion of dynamic objects, leading to limited performance in real-world scenarios. In this paper, we introduce CRT-Fusion, a novel framework that integrates temporal information into radar-camera fusion to address this challenge. Our approach comprises three key modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module fuses radar and image features within both the camera view and bird's-eye view, thereby generating a more precise unified BEV representation. The MFE module conducts two simultaneous tasks: estimation of pixel-wise velocity information and BEV segmentation. Based on the velocity and the occupancy score map obtained from the MFE module, the MGTF module aligns and fuses feature maps across multiple timestamps in a recurrent manner. By considering the motion of dynamic objects, CRT-Fusion can produce robust BEV feature maps, thereby improving detection accuracy and robustness. Extensive evaluations on the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-the-art performance for radar-camera-based 3D object detection. Our approach outperforms the previous best method in terms of NDS by +1.7%, while also surpassing the leading approach in mAP by +1.4%. These significant improvements in both metrics showcase the effectiveness of our proposed fusion strategy in enhancing the reliability and accuracy of 3D object detection.

Main paper: 
 introduction: 
 3D object detection plays a crucial role in autonomous vehicles and robotics, leveraging sensors such as lidar, cameras, and radar to localize and classify objects in the environment. Extensive research has been conducted to explore various strategies for improving detection accuracy and robustness. One prominent approach is the integration of data across multiple timestamps, which aims to mitigate the inherent limitations associated with relying solely on instantaneous data. By incorporating historical information, this approach provides a more robust perception of the environment, addressing the challenges of incomplete data caused by occlusions, sensor failures, and other factors.Numerous studies have investigated the utilization of temporal information to enhance the performance of LiDAR-based and camera-based 3D object detection methods. Recent works have also explored the incorporation of temporal cues in radar-camera fusion methods [1, 2] . These methods generated bird's-eye view (BEV) feature maps for each frame by fusing radar and camera data into a unified BEV representation. The resulting BEV feature maps are then concatenated across frames to create a comprehensive spatio-temporal representation, as illustrated in Figure 1 (a). However, these approaches face limitations in effectively capturing object motion, as they merge data from different time intervals without explicitly considering the dynamics of moving objects. Consequently, the performance accuracy for dynamic objects is compromised.To address these challenges, we propose a motion-aware approach, as illustrated in Figure 1 (b), which goes beyond simple concatenation of BEV feature maps. Our method first estimates the locations of dynamic objects with their corresponding velocity vector for each timestamped BEV feature map. Subsequently, we leverage this predicted information to rectify the motion of dynamic objects in each feature map and fuse them in a temporally consistent manner. Figure 1 (c) presents a graph depicting the performance gain achieved by our proposed method over the direct concatenation of temporal BEV feature maps for different object velocity ranges. It is evident that our approach consistently outperforms the existing method across all velocity ranges, with a notable performance improvement for objects moving at medium velocities. This demonstrates the effectiveness of our motion-aware fusion strategy in capturing and compensating for object motion, leading to superior performance in 3D object detection.In this paper, we introduce CRT-Fusion, a novel approach for integrating temporal information into radar-camera fusion. Our framework comprises three modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module generates radar-camera fused BEV feature maps for each timestamp. The MVF enhances image features with radar BEV features, achieving more precise depth predictions through Radar-Camera Azimuth Attention (RCA). The enhanced camera BEV features and radar BEV features are integrated through a gating operation. The MFE module predicts velocity information and performs BEV segmentation for each pixel in the fused BEV features to identify object regions and provide values for shifting the feature map spatially. Finally, the MGTF module generates the final feature map by leveraging the fused BEV feature maps, segmentation results, and velocity predictions. The MGTF module begins with the BEV features from the (t -N )th time step and aligns them with those from each subsequent time step. These aligned features are then aggregated one-by-one across all N timestamps in a recurrent manner. Consequently, CRT-Fusion achieves state-of-the-art performance on the nuScenes 3D object detection benchmark for radar-camera fusion methods, with improvements of +1.7% in NDS and +1.4% in mAP compared to existing state-of-the-art approaches.In summary, the main contributions of this work are as follows:• We introduce CRT-Fusion, a novel framework that effectively integrates temporal information into radar-camera fusion for 3D object detection. By considering the motion of dynamic objects, CRT-Fusion significantly improves detection accuracy and robustness in complex real-world scenarios.• We design a Multi-View Fusion module that enhances depth prediction by leveraging radar features to improve image features before fusing them into a unified BEV representation.• We introduce an effective temporal fusion strategy through MFE and MGTF modules. MFE estimates pixel-wise velocity information, and MGTF iteratively aligns and fuses feature maps across multiple timestamps using the motion information obtained from MFE.• CRT-Fusion achieves state-of-the-art performance on the nuScenes dataset for radar-camerabased 3D object detection, surpassing previous best method by +1.7% in NDS and +1.4% in mAP.2 Related WorksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our contribution aims to enhance object detection performance through temporal fusion considering dynamic objects, as shown in Figure 1 and Tables 3 and 5 .Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
camera-radar 3d object detection: 
 Due to the high cost of LiDAR sensors, research on 3D object detection using radar-camera sensor fusion has gained significant traction in recent years. These studies aim to utilize radar sensors as auxiliary sensors to overcome the fundamental limitation of camera-based 3D object detection research, which is the lack of depth information.CenterFusion [3] adopts a frustum-based approach to establish connections between radar point clouds and image features, enabling the refinement of 3D proposals. CRAFT [1] captures the interactions between radar and camera data within a polar coordinate system using a cross-attention mechanism, effectively integrating information from both modalities. RCM-Fusion [4] combines radar and image features at both feature-level and instance-level to achieve more precise 3D object detection. RADIANT [5] fuses radar and image features within image pixel coordinates to provide more accurate 3D location estimation. CRN [6] employs a 3D object detection method that strikes a balance between speed and performance by leveraging radar information to enhance camera BEV features and fusing multi-modal BEV features. RCBEVDet [2] introduces a novel radar backbone network that utilizes point-based feature extraction techniques for radar and fuses radar and image features using deformable cross-attention. CRKD [7] employs a method to transfer the knowledge possessed by the LiDAR-Camera fusion detector to the Camera-Radar fusion detector using the Cross-modality Knowledge Distillation technique.The approach to utilizing temporal information in 3D object detection varies depending on the type of sensors employed. In LiDAR-based methods, relying solely on single-frame point cloud data introduces challenges such as occlusion and partial views. To mitigate these issues, several studies have integrated temporal information at the feature level [8, 9, 10, 11] . Another approach involves extracting proposals using an object detector and subsequently leveraging temporal information at the object level [12, 13, 14] . A third method focuses on fusing temporal information within the query representation [15] .On the other hand, camera-based approaches have exploited temporal information to overcome the inherent limitations of image data, such as inaccuracies in depth prediction. One common methodology in camera-based perception research is to fuse temporal information with BEV-based methods [16, 17, 18, 19, 20] . Another approach has centered on enhancing the accuracy of depth estimation through temporal-stereo methods [21, 22, 20] .In the context of radar-camera fusion methods, the utilization of temporal information remains less explored, with most studies following the strategies employed in BEV-based camera-only methods. However, our proposed CRT-Fusion deviates from existing research methods by introducing a novel temporal fusion method. Our approach incorporates a temporal BEV fusion mechanism that explicitly considers the movement of objects, thereby enhancing object detection performance.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This is discussed in Supplementary Section C. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
crt-fusion: 
 In this work, we introduce CRT-Fusion, a novel framework for 3D object detection that efficiently fuses information from radar, camera, and temporal domains. The overall architecture of CRT-Fusion is illustrated in Figure 2 . Our approach first extracts the features from radar and camera data separately using their respective backbone networks. Subsequently, we employ the MVF module to generate a fused BEV feature map for each timestamp by combining the radar and camera features. The sequence of fused feature maps is then utilized by the MFE module to predict the location and velocity information of dynamic objects. The predicted motion information is exploited by the MGTF module to align the BEV feature maps spatially within a time window. Then, the aligned features maps are aggregated to obtain the final feature maps. Finally, 3D object detection is performed using the detection head proposed by CenterPoint [23] .Recent advancements in BEV-based camera-only approaches have significantly improved performance, leading to an increased focus on radar-camera fusion approaches within BEV representations. These studies [6, 4, 2] primarily aimed to address the inherent limitations of camera-only approaches, particularly the challenge of accurate depth prediction, by leveraging radar data. The existing stateof-the-art model [6] enhanced this process by combining occupancy information from radar point clouds with camera frustum features, effectively incorporating radar positional data. While this method facilitates the direct integration of radar positional information, noise in radar point clouds can adversely affect depth prediction accuracy. To mitigate this issue, we propose a novel fusion strategy that incorporates radar and camera features in both bird's eye view and perspective view. Unlike the existing method [6] , our approach enhances camera features using radar information prior to depth prediction, enabling more accurate camera BEV features. Subsequently, we employ a CNN-based gated fusion network to obtain the final fused features.Perspective view fusion. As illustrated in Figure 3 (a), the Radar-Camera Azimuth attention (RCA) module takes the camera perspective view features F c ∈ R N ×C×H×W and the radar BEV features F r ∈ R C×X×Y as inputs, where H and W represent the height and width of the camera features, and X and Y denote the size of the radar BEV features along the x-axis and y-axis, respectively. For the i-th image, the camera feature F i c ∈ R C×H×W is compressed along both height and width dimensions using max pooling and MLP layers, resulting in W i c ∈ R C×1×W and H i c ∈ R C×H×1 . Let W i c (j) be the value of the W i c features at the j-th position along the width direction. We associate the radar feature element F r (x, y) at the position (x, y) with W i c (j) through Azimuth Grouping. The azimuth angle values corresponding to W i c (j) and F r (x, y) are denoted as θ i c (j) and θ r (x, y), respectively. A set of M radar features R i j associated with W i c (j) is obtained using: where arg min(X , M ) returns the indices of the smallest M elements in X , and R i j represents M radar features F r (x, y) whose azimuth angles are closest to θ i c (j). Pixel-wise fusion module is then applied to R i j to obtain an enhanced feature W i c (j) asEQUATIONEQUATIONEQUATIONwhere R i j (m) denotes the m-th element of R i j . The concatenation of W i c (j) and R i j (m) is passed through two MLP layers to obtain an intermediate feature M i j (m). These intermediate features are then processed through another MLP layer followed by a softmax function to determine the relevance to W i c (j). The weighted sum of these intermediate features yields the enhanced W i c (j). Finally, we perform an element-wise multiplication of W i c ∈ R C×1×W and H i c ∈ R C×H×1 to obtain the camera perspective view features F i c . The concatenation of F i c and F i c is then passed through a convolution layer to obtain F i c , which is the perspective view features for the i-th image fused with the radar BEV features. These steps are depicted in Figure 3 (a) .Bird's eye view fusion. The enhanced camera feature F i c in the perspective view is employed for depth prediction and camera view semantic segmentation. Inspired by SA-BEV [24] , we utilize a 1x1 CNN layer head structure to predict both depth map and segmentation scores in the perspective view. The network outputs D i c ∈ R (b+1)×H×W , where b denotes the number of depth bins and the additional dimension corresponds to the foreground score prediction. The predicted segmentation scores are thresholded using a value τ P to identify the foreground regions only, which are then projected to the BEV domain. The resulting camera BEV features are fused with the radar BEV features obtained from the radar pipeline using a gated fusion network [25, 26] to yield the final BEV features B. The gated fusion network assigns weights to each feature according to their significance, effectively boosting the effect of feature fusion.Temporal fusion methods that consider object motion have been extensively studied in the context of 3D object detection [12, 13, 14] . These methods typically predict object locations and velocities at each timestamp using an object detector, and then aggregate object information from the past timestamps to the current timestamp at the object level. However, this approach has a drawback as the overall model performance heavily depends on the initial detector's performance. Moreover, object-level temporal fusion methods have not utilized motion information of objects effectively. To address these limitations, we propose a simple yet effective solution: by predicting velocity information and object presence for each pixel in the BEV features, the model can produce the aligned BEV features, which can be temporally fused at the feature level in the BEV domain rather than at the object level.Suppose that we obtain a set of BEV features B = {B t-k |k ∈ {0, 1, ..., N }} from the preceding Multi-View Fusion module, where B t-k represents the BEV feature at timestamp t -k. Each feature B t-k is then processed in parallel by two distinct heads: a velocity head and an object occupancy head, both composed of 3x3 and 1x1 convolutions. The velocity head extracts motion information M t-k ∈ R 2×X×Y , containing velocity estimates along the x and y axes for each pixel in the feature map B t-k . Simultaneously, the object occupancy head produces an occupancy score map O t-k ∈ R 1×X×Y , where each element indicates whether the corresponding pixel belongs to an object. To facilitate training of these modules, we generate ground truth targets for each timestamp using the following equations. The IoU ratio r(x, y) for a pixel at location (x, y) is defined as:EQUATIONwhere H(x, y) is the physical box in the BEV domain corresponding to one pixel located at (x, y), G is the set of ground truth 3D object boxes, and P(G) is the projection of these boxes onto the BEV domain. The IoU ratio calculates the overlap between the physical box and the projected ground truth boxes, helping to determine positive samples. The ground truth values are then given by:EQUATION)EQUATIONwhere M GT t-k (x, y) and O GT t-k (x, y) are the ground truth velocity and occupancy scores for pixel (x, y) in the BEV feature B t-k , respectively. The velocity vector of the corresponding ground truth object is denoted by (v gt x , v gt y ), and τ iou is a predefined threshold set to 0.5. Pixels with an IoU ratio exceeding τ iou are classified as positive and assigned the GT velocity and GT occupancy state for supervision.Figure 3 (b) presents the Motion-Guided Temporal Fusion (MGTF) module, which integrates BEV features to construct a dynamic representation of object motion across multiple timestamps. Our model utilizes a memory bank structure, where previously computed BEV features generated through the MGTF process are stored in the buffer, effectively minimizing redundant computations. This memory-efficient design significantly reduces the computational overhead during temporal fusion.At each timestep t-k, the BEV feature map Bt-k is retrieved from the memory bank, representing the previous result processed through the MGTF. For each coordinate (x, y) in Bt-k , the corresponding velocity vector M t-k (x, y) = [v x , v y ] is used to compute the positional shift ∆x = v x • t s and ∆y = v y • t s , where t s denotes the duration of a single frame. These positional shifts are applied to the feature values if the velocity magnitude exceeds a predefined threshold τ v . The shifted feature maps are then obtained asEQUATIONwhere S(x, y) = {(i, j) :EQUATIONHere |S(x, y)| denotes the cardinality of S(x, y) and ⌊•⌉ denotes the rounding operation. The shifted feature map B ′ t-k is then concatenated with the feature map B t-k+1 at the next timestamp. To filter out any irrelevant features resulting from the shifting process, the concatenated feature map is element-wise multiplied with the occupancy score map O t-k+1 asEQUATION)This process is iteratively applied N times, generating the BEV feature maps Bt-N+1 , . . . , Bt sequentially. The final Bt is then passed through a 1x1 convolution to obtain the final BEV feature map. This sequential nature of the process enhances overall comprehension of the MGTF module.By incorporating motion information and occupancy score maps, the MGTF module captures the dynamics of moving objects while filtering out irrelevant features, resulting in a more robust BEV representation. Compared to traditional methods, MGTF captures trajectories of moving objects to enhance 3D object detection performance. Dataset We evaluated our proposed method using the nuScenes dataset [27] , a popular public dataset for autonomous driving. This dataset consists of 1,000 scenes, divided into 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. Each scene contains approximately 20 seconds of data. The nuScenes dataset provides comprehensive 360-degree coverage with data from six cameras, and five radars. Keyframes are annotated at a frequency of 2Hz, covering 10 object classes. We use the official evaluation metrics provided by the nuScenes benchmark, which are mean Average Precision (mAP) and nuScenes Detection Score (NDS).Implementation details We adopted BEVDepth [17] as our baseline model. For a fair comparison with existing methods, we employed ResNet [28] , and ConvNeXt [29] as backbone encoders in the camera branch. In the radar branch, we accumulated the past 6 radar sweeps to obtain the input point clouds and used PointPillars [30] with randomly initialized weights as the backbone network.Our proposed CRT-Fusion model performed temporal fusion using the BEV features from the past 6 frames. We also introduce CRT-Fusion-Light, a lightweight version of CRT-Fusion, where the 2D-CNN backbone is removed from the radar pipeline. CRT-Fusion-Light performs temporal fusion using the BEV features from the past 3 frames. Detailed values of hyperparameters including learning rate, optimizer, and data augmentation methods are provided in Appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
comparison to the state of the art: 
 Table 1 compares our proposed CRT-Fusion method with state-of-the-art 3D object detection methods on the nuScenes validation set. CRT-Fusion consistently outperforms existing radar-camera fusion models and camera-only models across various camera backbone configurations. With the ResNet-50 backbone, CRT-Fusion achieves an improvement of 12.2% in NDS and 15.7% in mAP compared to the baseline model BEVDepth [17] . Additionally, CRT-Fusion achieves 1.2% higher NDS and 1.0% higher mAP than the state-of-the-art model CRN under the same configurations without classbalanced grouping and sampling (CBGS) [37] . Furthermore, when CBGS is applied, our approach outperforms the current best model, RCBEVDet [2] by 2.9% in NDS and 5.5% in mAP. When using the ResNet-101 backbone, CRT-Fusion surpasses CRN by 1.4% in NDS and 0.9% in mAP without test time augmentation (TTA). Our lightweight version, CRT-Fusion-light, maintains a comparable FPS while delivering better performance compared to existing radar-camera 3D object detectors, demonstrating the efficiency and effectiveness of our approach.Table 2 shows the performance of CRT-Fusion on the nuScenes test set. Our method outperforms all existing radar-camera fusion models, achieving state-of-the-art performance in both settings with and without TTA. Note that the V2-99 backbone is pre-trained on the external depth dataset DDAD [38] .We conducted comprehensive ablation studies on the nuScenes validation set to evaluate the effectiveness of the key components in CRT-Fusion. Throughout these experiments, unless otherwise specified, we used a ResNet-50 backbone and an image size of 256 × 704 for the camera branch, and a BEV size of 128 × 128. All models are trained for 24 epochs without applying CBGS.Component analysis. To assess the effect of each component, we gradually added each to our baseline model and analyze the performance improvements, as shown in Table 3 . The first row shows the performance of the BEVDepth model as reported in the paper, achieving an NDS of 47.5% and an mAP of 35.1%. Our baseline model, represented in the second row, reproduces BEVDepth without CBGS and incorporates long-term temporal fusion [20] , achieving an NDS of 47.4% and an mAP of 37.8%. By fusing radar and camera features at the BEV stage, including gating fusion, we observe a significant improvement, reaching an NDS of 55.4% and an mAP of 47.8%. RCA, which fuses radar features in the frustum view, contributes to an additional 1.2% and 1.1% improvement in NDS and mAP, respectively. Finally, by using MFE and MGTF, we achieve an additional gain of 1.1% in both NDS and mAP, resulting in the highest performance with an NDS of 57.2% and an mAP of 50.0%. These results demonstrate the effectiveness of each key component in CRT-Fusion, with our proposed modules providing significant improvements over the baseline model.Robustness to diverse weather and lighting conditions. In Table 4 , we analyze the performance of our model under varying weather and lighting conditions. For a fair comparison, we use the same settings as CRN, with a ResNet-101 backbone and an input size of 512 × 1408. The baseline model BEVDepth shows low mAP scores due to the impact of weather and light on camera sensors. However, by incorporating radar sensors, which are less affected by these factors, CRT-Fusion achieves over 15% higher mAP in all scenarios. Compared to CRN, the state-of-the-art camera-radar 3D object detection model, CRT-Fusion achieves higher mAP in all conditions except Sunny, offering particularly notable gains in night environments.Impact of accurate velocity estimation on MFE and MGTF. The NDS and mAP scores both drop by 0.5%, highlighting the challenge of accurately estimating velocities solely from camera information. In contrast, CRT-Fusion leverages radar information to achieve more precise velocity predictions. By incorporating the MFE and MGTF modules, CRT-Fusion achieves an improvement of 1.1% in both NDS and mAP, demonstrating the effectiveness of these modules in enhancing the performance. These results indicate that the successful operation of the MFE and MGTF modules is highly dependent on accurate velocity estimation. In the absence of radar data, velocity estimation accuracy is significantly compromised, suggesting that MFE and MGTF modules are optimally suited for radar-camera fusion frameworks.Comparison of radar-based view transformation methods. Table 6 compares various view transformation methods leveraging radar information. For a fair comparison, we used CRN as the baseline model and conducted experiments in a single-frame setting. The LSS approach predicts 3D depth from camera features and transforms them into BEV features, but can generate inaccuracies due to imprecise depth information. RGVT [39] projects radar points onto the image plane and encodes radar depth features using a lightweight CNN, which are then combined with image features to predict 3D depth. RVT [6] refines the depth distribution predicted from camera features using radar occupancy information. Our proposed RCA outperforms existing methods in almost all metrics, demonstrating its effectiveness in utilizing radar information to transform camera features into accurate BEV features. The superior performance of RCA demonstrates that leveraging radar information for depth prediction significantly enhances BEV perception.Qualitative results. Figure 4 presents a qualitative comparison between our proposed CRT-Fusion model and the previous state-of-the-art CRN model across various real-world scenarios. CRT-Fusion demonstrates enhanced detection capabilities, accurately identifying objects and showing superior precision in predicting orientations and center positions. Furthermore, CRT-Fusion maintains high accuracy across diverse conditions, effectively capturing multiple targets. These results underscore the robustness and enhanced spatial perception of CRT-Fusion. Additional qualitative results are available in the Supplemental Materials. 
discussion and conclusion: 
 Limitations and future work. While CRT-Fusion achieves significant performance gains over the baseline, the computational cost increases with the number of previous frames used for temporal fusion, limiting the number of frames that can be incorporated due to hardware constraints. This issue is likely due to the parallel fusion structure used for combining BEV features. To address this, a potential solution is to adopt a recurrent fusion structure, which fuses BEV features temporally in a sequential manner. This approach could maintain computational feasibility while incorporating long-term historical BEV features. Future work will explore this recurrent fusion architecture for CRT-Fusion to further reduce its computational complexity. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our model poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This is mentioned in Supplementary Section A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this paper, we introduced CRT-Fusion, a novel framework that integrates temporal information into radar-camera fusion for 3D object detection. By explicitly taking the motion of dynamic objects into account through our proposed Motion Feature Estimator and Motion Guided Temporal Fusion modules, CRT-Fusion significantly improves detection accuracy and robustness in complex real-world scenarios. Our Multi-View Fusion module enhances depth prediction by leveraging radar features to improve image features before fusing them into a unified BEV representation.Extensive experiments on the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-the-art performance in the radar-camera-based 3D object detection category, surpassing all existing methods. Additionally, our approach demonstrates remarkable robustness under diverse weather and lighting conditions, highlighting its potential for real-world deployment. We believe that our work will inspire further research on the fusion of temporal and multi-modal information for robust perception in adverse environments.The total loss function used in CRF-Fusion is composed of five components: a standard 3D object detection loss and four additional losses derived from different head networks within our model. The total loss L total is given byEQUATIONwhere L det is the 3D object detection loss, L depth is the loss from the Depth Prediction Head in MVF, L seg is the loss from the Perspective-View Semantic Segmentation Head in MVF, L vel is the loss from the Velocity Prediction Head in MFE, and L occ is the loss from the Object Occupancy Prediction Head in MFE. The parameters λ depth , λ seg , λ vel , and λ occ are the weights for the corresponding loss terms. The Depth Prediction Loss uses binary cross-entropy loss for depth estimation, with a weight of λ depth = 3.0, following the approach used in BEVDepth. For the Perspective View Segmentation Loss, we also employ the binary cross-entropy loss, with a weight of λ seg = 25, inspired by SA-BEV. The Velocity Prediction Loss, which handles velocity (v x , v y ) and orientation prediction, utilizes Mean Squared Error (MSE) with a weight of λ vel = 1. Finally, the BEV Object Occupancy Loss uses Binary Focal Loss for foreground and background segmentation, with a weight of λ occ = 30.The nuScenes datasets [27] are publicly available to use under CC BY-NC-SA 4.0 license and can be downloaded from https://www.nuscenes.org/. We implemented our model using the MMDetection3D [40] codebase and trained it for a total of 24 epochs. The training process consists of two phases. In the initial phase, the model is trained for 6 epochs without the MGTF module. Then, the entire model is trained for the remaining 18 epochs. For the ResNet50-based model, we used 4 NVIDIA RTX 3090 GPUs for training, while for ResNet101 and ConvNeXt-B, we used 3 NVIDIA A100 GPUs. Table 7 summarizes the training settings for different camera backbone networks.In the perspective view, we apply data augmentation techniques consistent with previous studies [41, 17, 6] , including horizontal random flipping, random scaling ([-0.06, 0.11]), and random rotation (±0.54 • ). For the bird's-eye view, we employ random flipping along the x and y axes, random scaling ([0.95, 1.05]), and random rotation (±0.3925 rad). Additionally, we use a technique that randomly drops radar sweeps and points [42] . To ensure a fair comparison with other models, we do not utilize ground-truth sampling augmentation (GT-AUG) [43] , which is commonly used in LiDAR-based models.C.1 Analysis of the Motion Feature Estimation Module.The Motion Feature Estimation (MFE) module generates a motion-aware BEV feature map by estimating velocity for each grid and using it to refine the BEV representation. While MFE can be integrated into various architectures, its effectiveness is highly reliant on accurate velocity predictions.To demonstrate this, we applied MFE to both the camera-based model BEVDepth and our proposed CRT-Fusion, which incorporates radar data.Figure 5 shows the results of applying the MFE module to each model. The red boxes represent Ground Truth (GT) boxes, the red arrows indicate GT velocity, and the white arrows show the predicted velocity from the MFE module. The size and direction of the arrows reflect the velocity's magnitude and direction. The yellow and orange highlights illustrate the differences in velocity prediction accuracy and the ability to distinguish static objects. Specifically, in the yellow-highlighted areas, CRT-Fusion predicts velocities closely aligned with GT, whereas BEVDepth shows inaccuracies.In the orange-highlighted areas, CRT-Fusion accurately identifies static objects, while BEVDepth misclassifies them as dynamic. In conclusion, the effectiveness of the MFE module is closely tied to the accuracy of velocity estimation. Integrating radar data in CRT-Fusion significantly enhances the module's ability to generate reliable motion-aware BEV features, underscoring the value of sensor fusion for robust 3D object detection. The ablation studies on the hyperparameters used in CRT-Fusion are shown in Tables 8 and 9 . Table 8 examines the optimal number of previous frames to consider for achieving the best performance. The results show that the performance of CRT-Fusion improves as the number of frames increases, with 6 frames yielding the best results when considering computational cost and performance. Tables 9 (a) and (b) investigate the optimal segmentation thresholds for the perspective view and bird's eye view (BEV), respectively. In the perspective view, a segmentation threshold τ P of 0.25 achieves the best performance, while in the BEV, a threshold τ B of 0.05 yields the highest performance. Increasing the threshold may lead to the removal of foreground regions, resulting in a decline in performance. Table 9 (c) explores the velocity threshold τ v for considering a BEV grid as a dynamic object in the MFE module. The results demonstrate that considering BEV grids with velocities above 1 m/s achieves the highest performance. Finally, Table 9 (d) examines the number of radar BEV grids to match with each image feature pixel in the RCA module. The experiments reveal that matching 128 grids yields the best performance.Table 10 presents the analysis of performance, latency, and GPU memory usage of CRT-Fusion and CRN as the number of past frames increases. We reproduced the CRN model following the official implementation. For a fair comparison, both models employ identical camera and radar backbones with consistent input image dimensions. All experimental evaluations were conducted on a single NVIDIA RTX 3090 GPU and an Intel Xeon Silver 4210R CPU.Our experimental results demonstrate that CRT-Fusion consistently achieves superior performance in both NDS and mAP metrics across all temporal configurations. Notably, as we extend the temporal context by incorporating additional past frames, CRT-Fusion exhibits stable resource usage with minimal degradation. The memory efficiency of our approach is evidenced by its peak consumption of 3.754 GB at 7 frames, while CRN reaches 4.342 GB under identical conditions. In terms of computational latency, CRT-Fusion demonstrates robust scalability, with inference time increasing moderately from 57.1 ms to 69.6 ms when expanding from 0 to 7 frames. CRN also benefits from increased temporal information, although it experiences a more substantial increase in computational overhead, with latency reaching 273 ms at 7 frames. These empirical results highlight the efficiency of our proposed architecture in maintaining real-time inference capabilities while utilizing temporal information effectively.C.4 Latency Analysis of CRT-Fusion and CRT-Fusion-light.Table 11 presents the component-wise latency analysis of our CRT-Fusion variants. CRT-Fusion-light, which incorporates a lightweight radar backbone and processes fewer temporal frames, achieves an overall latency of 48.8 ms compared to 67.1 ms of the original architecture. The efficiency gains primarily originate from the radar processing pipeline, where the radar backbone latency decreases from 13.9 ms to 3.9 ms, and the MGTF module reduces from 15.2 ms to 8.6 ms. E Discussions of potential societal impacts.CRT-Fusion has the potential to enhance the accuracy and robustness of 3D object detection in autonomous vehicles and robotics systems by fusing radar, camera, and temporal information. Despite its benefits, the reliance on sophisticated technology and data fusion could lead to increased costs and complexity, potentially limiting accessibility and widespread adoption.  


Decision: Details TBD, needs discussion.

Scores are very borderline, 4, 5, 4, 5, 6.

Reviews: review: paper_summary: paper_summary: The paper talks about multi sensor fusion using Motion Information for Bird’s Eye View Object
Detection. Authors have presented a multiple step approach to improve the object detection on nuScenes dataset. The proposed approach comprises of three parts; Multi-View Fusion (MVF) that enhances depth prediction by leveraging radar features to improve image features before fusing them into a single BEV vector; a Motion Feature Estimator (MFE) step that estimates pixel-wise velocity information and Motion Guided Temporal Fusion (MGTF) step that iteratively aligns and fuses feature maps across multiple timestamps. The proposed approach is tested on the latest nuScenes dataset for object detection and obtains state-of-the-art results on the dataset surpassing many publicly listed approaches.
strength_weakness: strength_weakness: 1. I believe that the technical novelty of the paper is somewhat limited. The idea of MVF fusion, i.e. Radar and Camera image feature fusion has been studied well in the past.
2. The paper lacks qualitative results completely. No where in the main paper have the authors presented any qualitative analysis. This makes it hard to interpret where the proposed approach performs better than current methods and more importantly where the proposed approach fails.
3. Overall approach with multiple components, although produces state-of-the art results, but is nearly impractical to scale or even productionize in a real world setting. Authors haven’t mentioned anywhere in the paper about this limitation (only described it briefly in the Supplementary section). The limitations of the current approach should be more broadly presented in the main paper, i.e. if the number of frames increase, the model will have high computational cost and hence not scalable.
questions: questions: 1. Can authors provide a small description of how this model can be made a bit efficient such that this can be deployed in the real world?
2. Figure 2 is not very clear. For the MVF (bottom left side) figure, what does the middle gray box represent?
3. Line 135 states an LSS approach, but that hasn’t been mentioned anywhere. What is LSS?
4. Line 137 states “existing state-of-the-art…”, can authors please cite which work they are referring to here?
5. The calculation of GT for MFE is not very clear. Why do we use the input for MVF to compute the GT for MFE (line 199)?
6. For line 201, how did the authors determine the value for T_{iou}? Why did they consider 0.5? Was there some ablation study conducted?
7. For Equation 7, it’s not very clear what B^_{t-k} is? Also how was the value for T_{v} chosen here?
8. Table 1 here refers to TTA. What is TTA?
9. Can the authors provide some qualitative results? And compare the existing work with the proposed approach to show what are the strengths of their approach? Just listing down metrics may not be motivating enough to see where the proposed approach does best.
10. Line 283 states that CRT does not do well in Sunny conditions which is a bit strange since sunny weather conditions should be ideal for object detection. Can the authors provide some explanation here?
11. For table 6, RCA has worse performance as compared to RGVT on mATE. Can authors address why is that the case?
limitations: limitations: Authors have presented the limitations of the work (which is a good thing), but this has been mostly discussed in the supplementary section (which is outside of the main paper). I would encourage the authors to include a brief summary of the limitations in the main paper as well.
paper_summary: paper_summary: The paper proposes CRT-Fusion that fuses radar and camera inputs for 3D object detection in BEV space. A radar-camera azimuth attention module is proposed for improving image features for better motion awareness, which further improves the quality of image BEV features. A motion-guided temporal fusion is introduced to align dynamic objects in history BEV feature maps with their positions in the current BEV feature map, using predicted object presence information and predicted velocities. The authors conduct experiments on the nuScenes dataset and prove the effectiveness of the proposed model.
strength_weakness: strength_weakness: 1. The proposed temporal fusion requires aggregating multiple feature maps of different timestamps in a recurrent way. The inference latency and required GPU memory will grow linearly with the number of past frames.
2. The multi-view fusion section needs more details. It's unclear how the radar BEV feature map is obtained and how the gated fusion is performed. It's better to include some equations for illustration.
3. The equation 7 for motion-guided temporal fusion seems to be incorrect. Further details for generating the shited feature map are needed. For example, let's consider this situation: the velocity predictions at (x1, y1) and (x2, y2) are (Δx1, Δy1) and (Δx2, Δy2) respectively, and x1+Δx1 = x2+Δx2, y1+Δy1=y2+Δy2, what will the shifted feature map be at (x1+Δx1, y1+Δy1)?
questions: questions: In Table 1, I notice that CRT-Fusion-light has lower mATE, mASE, mAOE, mAVE, mAAE compared to CRT-Fusion, which is unexpected. It would be beneficial if the authors could provide a detailed explanation or revisit the experimental setup to clarify these results.
limitations: limitations: N/A
paper_summary: paper_summary: In this study, the authors present CRT-Fusion, an innovative framework designed to incorporate temporal information into radar-camera fusion, thereby enhancing the robustness of 3D object detection. The CRT-Fusion framework is composed of three integral modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF).
strength_weakness: strength_weakness: Please consult the question section for further information.
questions: questions: 1. The paper lacks a detailed description of the loss function used in the proposed framework.

2. The authors claim that the Multi-View Fusion (MVF) module enhances depth prediction. However, no experimental evidence is provided to substantiate this assertion.

3. The Motion Feature Estimator (MFE) module is responsible for the pixel-wise velocity estimation task, yet the paper does not clarify how the velocity estimation is supervised during the learning process.

4. The overall improvement achieved by the proposed method is limited. Given that the method incorporates temporal information, velocity estimation, and depth estimation tasks, its performance enhancement compared to the CRN framework is relatively modest.
limitations: limitations: Please consult the question section for further information.
paper_summary: paper_summary: This paper presents a method, called CRT-Fusion, for 3D object detection that fuses temporal information with radar-camera features. The CRT-Fusion captures the object motion with three modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module uses Radar-Camera Azimuth Attention (RCA) to enhance the depth information in camera BEV features and combines them with radar features through a gated fusion network. The MFE module estimates pixel-wise velocity and performs BEV segmentation, which guides the temporal feature alignment and fusion in the MGTF module across multiple timestamps. The evaluation on the nuScenes dataset indicates that CRT-Fusion achieves state-of-the-art performance.
strength_weakness: strength_weakness: The paper could improve clarity in the following:
1.	As mentioned in lines 181-185, CRT-Fusion uses object-level pixel-wise velocity and occupancy to integrate temporal information with BEV features. CRT-Fusion's performance relies on the accuracy of both detection and velocity estimation. It is not clear how to address error propagation across frames.

2.	Regarding inference latency, some approaches (e.g., CRN[6]) use only 2 adjacent frames as input. According to the results in Table 8 of the supplementary material, the 2-frame version of CRT-Fusion performs similarly to (or worse than) CRN in terms of NDS and mAP metrics. This raises concerns that the performance gains of CRT-Fusion may be a trade-off involving longer temporal cues, leading to increased latency and complexity. The paper should discuss this trade-off.

3.	The batch size used during inference is not specified; Table 7 only presents the batch size used during training. Additionally, it appears that the authors directly used speed results from other papers for comparison, which may not be meaningful as the machines used for different models can vary significantly (Unfortunately, some other papers also directly cite and compare the speed) .
questions: questions: In line 212, should it be $M_{t-k}(x, y)$? In the inference, the model does not have access to ground truth.
limitations: limitations: The paper poorly discusses limitations. Failure cases and potential improvements of the approach in the future should be discussed.
paper_summary: paper_summary: This paper proposes a radar-camera and temporal fusion method (CRT) for the 3D object detection task. CRT design Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF) modules. MVF employs radar information for 2D-to-3D projection. MGTF utilizes velocity and occupancy predictions from MFE to fuse multi-frame BEV features. The proposed method achieves state-of-the-art 3D object detection results on nuScenes.
strength_weakness: strength_weakness: 1. The proposed method requires velocity supervision, while some datasets, such as Waymo, do not provide velocity labels.
2. Table 1 shows that the proposed has well speed-accuracy trade-offs. It is better to provide the inference time of each part in the model.
3. It seems that hyperparameters have a significant impact on model accuracy (Table 9).
4. The paper does not show the model generalization of the proposed method. It is only applied to BEVDepth.
5. BEV segmentation in MFE is misleading. It is better to use terms like object occupancy or foreground segmentation.
questions: questions: 1. Training details. How is the model trained? Is the multi-frame multi-modal model trained in one step?
2. What about the training cost?
3. What TTA is used in Table 2?
4. In Line 153, a set of M radar features are associated with W. What M does the model use? What if radar features are less than M?
5. What is B_{t−k}(x, y) in eq.4？Please give more details for object occupancy ground truth generation.
6. What do O and X mean in Table 5?
limitations: limitations: The authors address the limitations.

Comments: title: Response to rebuttal
The rebuttal successfully addresses my review comments
comment: Thank you authors for presenting the rebuttal documents. I have read through the comments and the attached PDF. Since most of my concerns are addressed, I am okay in changing my rating from "borderline reject" to "borderline accept". There are still some concerns on the complexity of the solution but given that most of the issues have been addressed, I am inclined towards borderline acceptance here.
The authors successfully address my questions. I would like to keep my original rating.
The authors successfully address all my questions. I would like to keep my original rating and am inclined to accept this paper.
**What does the physical bounding box mean?**

The term "physical bounding box" refers to an anchor box whose size in the real world corresponds to that of a pixel. During our encoding process, actual scenes in the physical world shrink to features of smaller size. Thus, the actual size of a single pixel corresponds to a 2D box of $0.8m \times 0.8m$ in the x and y axes of the BEV  domain. We computed the IoU (Intersection over Union) ratio between this anchor box and the 2D ground truth (GT) box projected onto the BEV to determine whether a pixel is positive.


**Besides, in the corrected Table 5 (Q6), it seems using MFE and MGTF modules for BEVDepth decreases the performance.**

In this experiment, we applied MFE and MGTF to BEVDepth, which does not utilize radar data for BEV perception. While CRT-Fusion employs MFE and MGTF to enhance fused BEV features derived from both radar and camera inputs, we applied these modules solely to the camera-derived BEV features of BEVDepth.

Our results indicate that MFE and MGTF do not enhance performance in the absence of radar data. This behavior demonstrates that the information provided by radar data is crucial for MFE and MGTF to achieve performance gains over the baseline.
During the editing process, our session closed, causing some parts of the equations in W3's response to be omitted. We apologize for this. We are now adding the explanation for the relevant parts:

The shifted feature maps are obtained as

$B' _ {t-k}(x, y) = \frac{1}{|S(x,y)|}\sum_{(i,j) \in S(x,y)} {B}_{t-k}(i, j)$

where 

$S(x,y) = \\{(i,j): x= i+\lfloor \Delta x \rceil, y= j+ \lfloor \Delta y \rceil , |M_{t-k}(i, j)|> \tau_v \\}$

where  $|S(x,y)|$ denotes the cardinality of $S(x,y)$ and $\lfloor \cdot \rceil$ denotes the rounding operation.
The rebuttal has addressed most of my concerns. However, I still have a question about the object occupancy ground truth generation (Q5). What does the physical bounding box mean? Besides, in the corrected Table 5 (Q6), it seems using MFE and MGTF modules for BEVDepth decreases the performance.
The rebuttal successfully addresses my review comments. The rebuttal presents additional results that address my concerns about the latency of the proposed method.  I would like to keep my original rating.