SparseInteraction: Sparse Semantic Guidance for Radar and Camera 3D Object Detection

Authors: shaoqing  xu
shengyin  jiang
fang  li
ziying  song
bo  yang
zhi-xin  yang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:50:24

Abstract: Multi-modal fusion techniques, such as radar and images, enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. However, existing fusion methods for surround-view images and radar are challenged by the inherent noise and positional ambiguity of radar, which leads to significant performance losses. To address this limitation effectively, our paper presents a robust, end-to-end fusion framework dubbed SparseInteraction. First, we introduce the Noisy Radar Filter (NRF) module to extract foreground features by creatively using queried semantic features from the image to filter out noisy radar features. Furthermore, we implement the Sparse Cross-Attention Encoder (SCAE) to effectively blend foreground radar eatures and image features to address positional ambiguity issues at a sparse level. Ultimately, to facilitate model convergence and performance, the foreground prior queries containing position information of the foreground radar are concatenated with predefined queries and fed into the subsequent transformer-based decoder. The experimental results demonstrate that the proposed fusion strategies markedly enhance detection performance and achieve new state-of-the-art results on the nuScenes benchmark. Source code is available at https://github.com/GG-Bonds/SparseInteraction.

Main paper: 
 introduction: 
 Perception of 3D obstacles via different types of sensors is a fundamental task in the field of computer vision and robotics. The fusion of LiDAR and camera technologies has achieved high accuracy performance [19, 26, 28, 29, [37] [38] [39] . However, cameras face limitations in low-visibility conditions like heavy fog or rain while the high cost of LiDAR poses further challenges.Radar sensors are notable for their affordability, resilience in all weather conditions and capacity for accurate speed estimations over considerable distances [3, 20, 30, 32, 33, 41] . Nevertheless, the direct usability of radar points is impeded by the inherent challenges of noise and positional ambiguity.Recent studies achieved convincing performance by transforming image features onto the BEV (Bird's Eye View) and fusing them with radar features. In particular, CRN [11] fuses image-bevfeatures and radar features based on transformer architecture to resolve the ambiguity in radar positioning. However, the method requires assistance from LiDAR point clouds during the training process. Additionally, the presence of noisy radar data before the fusion process leads to suboptimal fusion features. For filtering the noisy information of radar, the CramNet [8] , depicted in Figure 1-(a) directly eliminates noisy radar by leveraging radar features. Conversely, the approach illustrated in Figure 1-(b ), such as HVDetFusion [12] , utilizes detection bounding boxes from the image to filter noisy information. Additionally, methods(e.g., MVFusion [36] ) like in Figure 1-(c ), incorporate foreground classification from the image to filter irrelevant information by projecting 3D radar points onto 2D image pixels. However, a common limitation of these methods is their reliance on the characteristics of a single modality before filtering. This singular focus tends to overlook the potential benefits of integrating multiple modalities which leads to suboptimal outcomes. In this paper, we present a robust, transformer-based, end-to-end fusion framework named SparseInteraction. Firstly, considering the negative impact of noisy radar on feature fusion, we introduce the Noisy Radar Filter module composed of Semantic-enhanced Radar Encoder (SRE) and Foreground Radar Mask (FRM) to preserve useful radar information under the guidance of image semantic information. Subsequently, we leverage the sparse foreground radar features and their related effective image feature areas to enhance the interactive information of multimodal features through Sparse Cross-Attention Encoder(SCAE) which effectively addresses feature misalignment. This fusion method not only resolves the issue of radar's ambiguous positioning but also provides robust results even in the event of sensor failures. Furthermore, we extract foreground object queries from both foreground radar features and enhanced BEVusion features in an efficient way. Finally, the foreground prior queries containing position information of the foreground radar are concatenated with predefined queries and fed into the subsequent transformer-based decoder, for facilitating the model convergence and performance. Ultimately, our framework achieves impressive and robust 3D object detection results without LiDAR points cloud in either the training or testing phases.In general, the contribution of this work can be summarized as follows:• We introduce a novel Noisy Radar Filter module designed to efficiently address false positive issues by filtering out useless information under the guidance of image semantic features and radar features. This approach can be categorized into bottom-up methods, such as LSS [24] , BEVDet [7] , and BEVDepth [14] , which transform image features into BEV features through depth prediction. Conversely, top-down methods pre-define queries in 3D space and project onto images for sampling. For instance, DETR3D [35] uses learnable 3D queries for end-to-end detection pipelines without NMS postprocessing. BEVFormer [15] generates BEV features by taking gridlike queries to sample relevant features from images. 
multi-modal 3d object detection: 
 While FUTR3D [2] employs 3D reference points as queries to sample features from the projected view and BEVFusion [19] uses a liftsplat-shoot operation to project image features onto BEV space and then fuse them with LiDAR features. Similar to LiDAR, radar can offer potential position and precision velocity information. However, challenges often arise due to the inherently noisier, sparser points and vague height information while cost-effectiveness and functionality in various conditions make it an appealing option for enhancing camera robustness [27] . CRN [11] transforms image feature to BEV space and integrates them with radar BEV features via a transformer structure while requiring LiDAR data for depth supervision. To mitigate radar noise, some studies first process the foreground radar, which results in more significant outcomes. e.g., CramNet [8] and HVDetFusion [12] . However, these methods do not integrate features from the other modality before extracting foreground information, leading to suboptimal extraction and complexity.  
methodology: 
 An overview of the proposed framework SparseInteraction is illustrated in Figure 2 . We first extract features from radar points cloud and multi-view images, respectively. To address noisy radar information, the Noisy Radar Filter is proposed to filter useless features by combining both radar features and high-level semantic features of images. Then, the enhanced foreground radar features and image features are sent into the Sparse Cross-Attention Fusion Encoder, which utilizes deformable attention mechanism [44] . Compared to previous approaches such as BEVFusion [19] , our Sparse Cross-Attention Encoder effectively resolves radar localization ambiguity without LiDAR depth supervision. Finally, prior information from the foreground radar features is encoded as foreground prior queries concatenated with predefined queries and introduced as object queries in transformer decoder layers.3.1.1 Image Feature Extractor. Following BEVFormer [15] , we adopt ResNet101 [5] as the backbone for extracting image features from multi-view images and use a standard Feature Pyramid Network (FPN) [17] on top of the backbone to generate multi-scale image feature representations𝐹 𝑖 𝑐 , 𝑖 ∈ [1, 2, • • •, 𝑁 ],where i denotes each view of cameras.3.1.2 Radar Feature Extractor. Given the limitation of radar in providing reliable elevation, we utilize pillar-based representation for radar points cloud. Following FUTR3D [2] , we adopt Pillar-Net [25] and multi-layer perception (MLP) to extract radar features in BEV space, denoted as 𝐹 𝑟 ∈ R 𝐶 ×𝐻 ×𝑊 , where 𝐶 is the dimension of radar features and (𝐻,𝑊 ) represents the BEV resolution.Despite radar recording the velocity and location of objects, it also inadvertently captures information of non-target which leads to false positive detection issues. To overcome this limitation, we propose an innovative lightweight module Noisy Radar Filter (NRF), comprising two key components: Semantic-enhanced Radar Encoder and Foreground Radar Mask. The detailed architecture of NRF module is depicted in Figure 3 .To migrate noisy radar information, we take non-empty features in 𝐹 𝑟 as queries for aggregating neighboring features guided by semantic information from the image. More importantly, a substantial portion of features originating from the radar encoder is vacant owing to the inherent sparsity of radar points. Consequently, we employ a 3x3 bias-free convolution layer to disperse foreground radar features into vacant positions, thereby augmenting the richness of radar features. Subsequently, the valid radar features are onto corresponding image features to generate semantic-enhanced radar features, 𝐹 𝑠 𝑟 ∈ R 𝐶 ×𝐻 ×𝑊 . Specifically, considering radar's uncertain height information, for a valid radar features pixel 𝑄 𝑟 𝑥,𝑦 located at (𝑥, 𝑦), we lift it to 𝑁 𝑟𝑒 𝑓 3D points with different heights 𝑧 𝑗 and projecting them onto the 𝑖-th image at projection pixels (𝑢 𝑖 , 𝑣 𝑖 ). Then valid radar features are enhanced using deformable attention [44] with the image features 𝐹 𝑖 𝑐 . The entire enhanced process is represented as follows:EQUATIONwhere 𝑄 𝑠 𝑥,𝑦 is the semantic-enhanced radar features pixel located at (𝑥, 𝑦). 𝑃 𝑖 (𝑥, 𝑦, 𝑧 𝑗 ) represents projection point of 3D point (𝑥, 𝑦, 𝑧 𝑗 ) in the image coordinate system of the 𝑖-th camera.Acknowledging that radar points predominantly manifest near objects, only a few points exist within 3D bounding boxes due to the ambiguity in radar localization. We figured that radar points in proximity to 3D bounding boxes also contribute to the detection process. Therefore, we enlarge the 3D bounding boxes derived from ground truth labels to 1.5 times of original dimensions, projecting them onto the BEV view as binary mask ground truth 𝑀 𝑔𝑡 . We employ a 1x1 convolution layer paired with a sigmoid activation function to classify the semantic-enhanced radar features generated above. To ensure a superior recall, a lower foreground threshold 𝛾 is used. Consequently, radar features with foreground score 𝑠 𝑖 exceeding the threshold 𝛾 are classified as foreground radar features 𝐹 𝑓 𝑟 ; the reverse applies for lower scores as background radar features 𝐹 𝑏 𝑟 .While CRN [11] first generates sparse and discrete BEV features of the image under the LiDAR depth supervision [16] and fuses with radar features through the attention mechanisms which leads to suboptimal fusion results. To address this, we propose the Sparse Cross-Attention Encoder (SCAE), a novel Multi-modal fusion module based on transformers. As shown in Figure 2 , SCAE comprises six encoding layers, each consisting of Temporal Self-Attention (TSA) and Multi-modal Cross-Attention (MCA). First, a set of learnable FusionBEV queries interact with historical FusionBEV features through TSA. Specially, for optimizing computation and more precision information, only sparse fusion features of the foreground are utilized as valid queries in the subsequent module. To acquire more comprehensive feature information, the relevant key values from the image feature map and foreground radar features (generated by Section 3.2) will be queried in the MCA submodule. Subsequently, the FusionBEV queries are updated by a feedforward network as input for the next layer. After six encoding layers, we generate precise and robust FusionBEV features. Significantly, all the operations of our SCAE module are in sparse space which requires less computation compared to FusionFormer [6] .3.3.1 FusionBEV Queries. We defined a set of learnable parameters 𝑄 ∈ R 𝐶 ×𝑊 ×𝐻 called FusionBEV queries. Here, 𝑊 × 𝐻 represents the resolution of BEV view, and 𝐶 is the channel of FusionBEV queries. These queries are shared across all modalities, directly fusing images features 𝐹 𝑐 and foreground radar features 𝐹 𝑓 𝑟 .3.3.2 Temporal Self-Attention. Temporal information plays a crucial role in 3D object detection. Following BEVFormer [15] , Initially, we align historical FusionBEV features based on ego-motion. Subsequently, we employ Temporal Self-Attention to effectively fuse these historical FusionBEV features. The process is as follows expression:𝑇 𝑆𝐴 𝑄 𝑝 , 𝑄, 𝐵 ′ 𝑡 -1 = 𝑉 ∈ {𝑄,𝐵 ′ 𝑡 -1 } 𝐷𝑒 𝑓 𝐴𝑡𝑡𝑛 𝑄 𝑝 , 𝑝, 𝑉 .(2) where 𝑄 𝑝 denotes the FusionBEV query located at 𝑝 = (𝑥, 𝑦). 𝐵 ′ 𝑡 -1 represents the FusionBEV features at timestamp 𝑡 -1 after temporal alignment.Multi-modal Cross-Attention. As detailed in Section 3.2, the positions of the background have been established in the BEV space. To optimize computational efficiency, FusionBEV queries in these positions are excluded from the computation, which makes FusionBEV queries focus on the potential target object positions 𝑝 𝑡 = (𝑥 𝑡 , 𝑦 𝑡 ). More importantly, these queries are directly projected onto each modality feature map and enhanced with deformable attention [44] , eliminating the need to transform image features into BEV space beforehand. This process is represented by the following formula:EQUATIONwhere 𝑄 𝑝 𝑡 denotes the FusionBEV query located at 𝑝 𝑡 . 𝑃 2𝐷 (𝑝 𝑡 ) denotes the projected positions on the BEV space. As FusionBEV queries lack height information, we employed the method similar to described in Section 3.2, which lifting points 𝑝 𝑡 with different heights 𝑧 𝑗 to obtain 𝑁 𝑟𝑒 𝑓 3D projection points (𝑥 𝑝 , 𝑦 𝑝 , 𝑧 𝑗 ), where 𝑃 𝑖 3𝐷 (𝑥 𝑝 , 𝑦 𝑝 , 𝑧 𝑗 ) are the corresponding positions of these 3D projection points on image 𝐹 𝑖 𝑐 .Our 3D detection head adopts the decoder from BEVFormer [15] , similar to the decoder in Deformable DETR [44] . It is well known that DETR-style predefined queries struggle to converge due to the lack of prior knowledge [22, 40] . Inspired by two-stage methods [40] , we encode the prior information of foreground radar features near the region of interest directly into the decoder. To minimize the effect of low-quality foreground radar, a higher threshold r is adopted, ensuring only the most salient foreground radar features are considered. Each foreground radar feature pixel in the BEV plane only provides 2D position 𝑝 𝑓 = 𝑥 𝑓 , 𝑦 𝑓 , and the ap-proximate height 𝑧 𝑝𝑟𝑒𝑑 𝑓of each foreground radar feature pixel is predicted using the FusionBEV features 𝐹 𝑏 at the specific location 𝑝 𝑓 , expressed as:EQUATIONWhere 𝐹 𝑓 𝑏 is sampled feature from FusionBEV features at position 𝑝 𝑓 .After obtaining the foreground prior 3D positional information (𝑥 𝑓 , 𝑦 𝑓 , 𝑧), we encode it into foreground prior queries as follows:EQUATIONEQUATION𝑄 prior = 𝑄 sem + 𝑄 pos (7) where 𝑠 𝑖 represents the confidence score of the foreground radar as predicted in Section 3.2. 𝑄 𝑠𝑒𝑚 and 𝑄 𝑝𝑜𝑠 represent the semantic feature embedding and position embedding, respectively. 𝑃𝑜𝑠𝐸𝑛𝑐𝑜𝑑𝑒𝑟 consists of a sinusoidal transformation [31] , and another MLP, with 𝑆𝑒𝑚𝐸𝑛𝑐𝑜𝑑𝑒𝑟 also being an MLP. And 𝑄 𝑝𝑟𝑖𝑜𝑟 represents the final foreground prior queries.Finally, the encoded foreground prior queries and predefined queries are concatenated and then fed into the decoder.We adopt a Deformable DETR-like detection head that outputs the probability of object classes as well as the 3D detection boxes directly without the need for NMS post-processing. The total loss comprises two optimization terms: the mask loss L 𝑚𝑎𝑠𝑘 and the detection loss L 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 , as shown in Eq. 8. The mask loss L 𝑚𝑎𝑠𝑘 aims to optimize foreground radar binary classification in the mask module, and the detection loss L 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 optimizes for the 3D detection head, respectively.EQUATIONConsidering the scarcity of foreground radar features, we employ focal loss for training in the foreground radar mask module. For fair comparison with BEVFormer, we adopt focal loss for classification and L1 loss for box regression in the 3D detection head. 
experiments 4.1 experimental settings: 
 4.1.1 Dataset and Metrics. We evaluate our approach on the large-scale autonomous driving dataset, nuScenes [1] . It is divided into 700/150/150 scenes for training, validation, and testing, respectively. It provides a 360-degree panoramic view and includes data from multiple sensors: six cameras, one LiDAR, and five radars. For the nuScenes detection task, mean Average Precision (mAP) and NuScenes Detection Score (NDS) are used as the metrics. The mAP, calculated based on the BEV center distance between the predictions and ground truth, is averaged over threshold distances of 0.5, 1, 2, and 4 meters. The NDS is a weighted average of the mAP and additional true positive metrics, which include the Mean Average Translation Error (mATE), Mean Average Scale Error (mASE), Mean Average Orientation Error (mAOE), Mean Average Velocity Error (mAVE), and Mean Average Attribute Error (mAAE).. By default, in the camera branch, we use ResNet101 [5] pre-trained with FCOS3D as backbone. For neck, we utilize a standard FPN [17] , featuring a dimension of 256 and scales of 1/16, 1/32, and 1/64. In Sparse Cross-Attention Encoder, six fusion encoding layers are employed. For the radar branch, we accumulate five previous radar sweeps. Similar to BEVFormer-Base [15] , unless specified otherwise, we use three frames of historical BEV features in our Temporal Self-Attention module.Our model is trained end-to-end for 24 epochs on 8 A100 GPUs using the AdamW [21] optimizer, with learning rate of 2e-4 and batch size of 1. No class balancing strategy (CBGS) [43] or BEV data augmentation [14] is utilized during training.To evaluate the performance of our SparseInteraction, we conducted comparison using nuScenes [1] val and test set, with results presented in Table 1 and Table 2 . Notably, our SparseInteraction surpasses previous state-of-the-art radar-camera fusion methods, achieving 51.1% mAP and 59.5% NDS on the validation set, and 61.7% mAP and 53.8% NDS on the test set without LiDAR auxiliary supervision. Compared with our baseline BEVFormer [15] , SparseInteraction shows substantial improvements, achieving 7.8% and 9.5% performance gains in NDS and mAP, respectively. These superior metrics underscore the effectiveness of our method.Significantly, unlike CRN [11] , our method does not require extra depth supervision from LiDAR. Although CRN utilizes a more powerful backbone, ConvNextB [18] using extra LiDAR data source as depth supervision. V2-99 is pre-trained on external depth dataset DDAD [4] .competitive results in terms of NDS and mAVE., leveraging the lightweight backbone, V2-99 [4] .In this section, we present a comprehensive validation of our framework. For our ablation studies, we utilize a smaller version of SparseInteraction, which includes reducing image resolution from 900x1600 to 700x1260, decreasing frames of historical BEV features from 3 to 2 in our Temporal Self-Attention module, reducing Fusion-BEV queries from 200x200 to 150x150, reducing fusion encoding layers from 6 to 3, and employing R101 as the backbone with image features downsampled by a factor of 32. As shown in Table 3 , we start with BEVFormer-Small [15] as our baseline (in #1) and incrementally incorporate each module to assess its impact. result worsens when extracting the radar foreground features directly using radar information without the semantic context from the image. Ultimately, the Noisy Radar Filter achieves impressive enhancement across all settings, the results as shown in setting 4.To further validate the robustness and versatility of our method, we also conducted experiments on BEVDepth [14] . The results demonstrate consistent benefits, with an increase of 1.1% mAP and 1.2% NDS, shown in Table 5 , this also highlights the effectiveness of removing noisy radar.Sparse Cross-Attention Encoder. Incorporating the Sparse Cross-Attention Encoder (SCAE) yields an improvement of 1.0% in mAP and 1.2% in NDS presented in Table 3 . We compare various fusion strategies including addition, concatenation, and the Multimodal Feature Aggregation (MFA) method mentioned in CRN [11] to demonstrate the effect of our SCAE in shown in Table 3 . To evaluate the effect of initial positioning height in FPQ, we conducted experiments comparing the learning-based method with statistics-based approaches, shown in Table 7 . The statistics-based refers to use the average height of ground truth bounding boxes in the dataset as the initial height for FPQ. In contrast, the learning-based method acquires different heights on the BEV plane. Unlike the fixed value in the statistics-based approach, this method simplifies the learning complexity for object queries at heights, thereby enhancing performance.4.4.1 Weather and Lighting. We analyze the robustness and performance under different weather and lighting conditions in Figure 4 . It is noted that detection during nighttime poses challenges for camera-only method. Thanks to the stable performance of radar under adverse weather conditions, our fusion model achieves consistent improvements in sunny and rainy weather, as well as daytime and nighttime conditions, compared with baseline model. This demonstrates the superior robustness and performance of our proposed SparseInteraction in all conditions. We perform the analysis on the convergence of SparseInteraction in Figure 5 . Thanks to the aid of radar, Sparse Interaction without FPQ consistently achieves significantly better detection performance compared to BEVFormer. However, their rates of convergence are identical, which we believe is attributed to them utilizing the same decoder head. Furthermore, because FPQ provides prior location information, it enables SparseInteraction to converge ahead of schedule, approximately around epoch 16, while also enhancing detection performance.4.4.3 Performance and Latency. We compare the performance and latency using different frames in Table 8 . After observing that the utilization of multiple temporal frames significantly enhances NDS, mAP, and mAVE while the best performance is achieved with four frames. Consequently, we opted to use four frames, taking into consideration computation time and memory constraints during training. Since we pre-save the history fusionBEV features during testing, the use of temporal information does not increase latency. Furthermore, in the Multi-Modal Cross Attention module, the exclusion of FusionBEV queries at background positions from computations enables the FusionBEV queries to focus on potential target object positions. Consequently, our model achieves lower latency and better performance compared to baseline model.We provide a more straightforward visualization in Figure 6 . Comparing the predictions of BEVFormer, we note that enhanced fusion features can refine the bounding box by SCAE. However, this process is sometimes disrupted by noisy radar, leading to false positives. In contrast, when comparing BEVFormer to SparseInteraction, it becomes evident that our framework successfully eliminates noisy radar data before fusion, effectively preventing false positives. 
conclusion: 
 In this paper, we proposed a novel transformer-based foreground radar camera fusion framework with a light module that extracts foreground radar features under the guidance of semantic image. Our approach effectively addresses the inherent challenges of noise and positional ambiguity which achieves state-of-the-art performance. In future work, we plan to incorporate velocity prediction into the temporal module to further enhance the detection performance. 
 
 Appendix:  


Decision: The paper "SparseInteraction" introduces a novel fusion framework for integrating radar and camera data to improve 3D object detection. The key contributions include the Noisy Radar Filter (NRF) module, which utilizes semantic features from images to filter out noisy radar data, and the Sparse Cross-Attention Encoder (SCAE) that merges radar and image features to address positional ambiguities. The method achieves state-of-the-art performance on the nuScenes benchmark, demonstrating significant improvements in detection accuracy and robustness under various conditions.

While the reviewers have highlighted some limitations, such as the need for further validation on additional datasets and scenarios, the overall consensus leans towards the strength and innovation of the proposed framework. The authors have addressed most concerns in their rebuttal, showcasing the effectiveness and potential of SparseInteraction in advancing radar-camera fusion techniques.

Given the substantial performance improvements and the novelty of the approach, the paper makes a contribution to the field of multimodal perception and 3D object detection.

Reviews: review: paper_summary: paper_summary: This paper presents a robust end-to-end fusion framework that effectively filters noisy radar data using semantic features from images and integrates these with radar data using a Sparse Cross-Attention Encoder, achieving state-of-the-art performance on the nuScenes benchmark.
strength_weakness: strength_weakness: A new fusion strategy significantly improves detection performance and robustness across various conditions, demonstrating superior results compared to existing methods without relying on LiDAR data.
limitations: limitations: 1.How does the proposed Noisy Radar Filter (NRF) quantitatively compare to traditional radar filtering techniques in terms of noise reduction and feature enhancement?
2.Can the authors provide more detailed results on the performance of SparseInteraction across different weather and lighting conditions? How consistent are the improvements in these varying scenarios?
3.What is the computational overhead introduced by the NRF and SCAE modules, and how does it impact the overall processing time and feasibility of real-time application in autonomous driving?
4.Has the SparseInteraction framework been tested on datasets other than nuScenes? If not, how do the authors justify the generalization capability of their model across different datasets and real-world conditions?
5.The paper mentions ablation studies for various components. Can the authors provide more detailed insights into the individual contributions of each component (NRF, SCAE, FPQ) to the overall performance improvement?
paper_summary: paper_summary: This paper introduces an end-to-end fusion framework that enhances radar and camera 3D object detection by using a Noisy Radar Filter (NRF) to filter out noisy radar data with semantic image features and a Sparse Cross-Attention Encoder (SCAE) to efficiently merge these features, achieving state-of-the-art performance without requiring LiDAR supervision. 

This approach significantly improves detection accuracy and robustness, especially in adverse weather and lighting conditions, surpassing existing methods on the nuScenes benchmark.
strength_weakness: strength_weakness: 1. This paper enhances 3D object detection by introducing an innovative Noisy Radar Filter (NRF) module that uses semantic features from images to filter out noisy radar data, and an efficient Sparse Cross-Attention Encoder (SCAE) that merges radar and image features at a sparse level, improving precision and reducing computational complexity.
2. This paper effectively addresses the challenge of 3D object detection without relying on LiDAR data, utilizing only camera and radar inputs.
limitations: limitations: 1. This paper uses different backbones for evaluations on the validation set (ResNet101) and the test set (V2-99), which may affect the fairness and consistency of performance comparisons.
2. This paper only compares its results with RCM-Fusion’s camera-only (C) results, not its camera and radar (C+R) results. RCM-Fusion uses a single-frame approach, whereas SparseInteraction employs a multi-frame approach. Additionally, RCM-Fusion uses a multi-modal fusion pipeline, which could weaken the claim of focusing on the characteristics of a single modality before filtering.
paper_summary: paper_summary: The paper presents a framework called SparseInteraction , designed for 3D object detection using radar and camera data. This multi-modal fusion approach addresses the limitations of existing methods by mitigating noise and positional ambiguity inherent in radar data. The framework includes key components such as the Noisy Radar Filter (NRF) and Sparse Cross-Attention Encoder (SCAE), which enhance radar features using semantic information from images and blend these features effectively to improve detection performance. The approach is validated through experiments on the nuScenes benchmark, showing state-of-the-art results.
strength_weakness: strength_weakness: Firstly, the proposed NRF module effectively filters out noisy radar data by leveraging semantic information from images. This reduces false positives and enhances the overall quality of the radar features used for fusion, leading to more accurate detection.

Secondly, the Sparse Cross-Attention Encoder (SCAE) accurately fuses radar and image features by addressing radar data's positional ambiguity. This fusion method allows for precise feature alignment and interaction, boosting overall detection performance.

Thirdly, SparseInteraction achieves high detection performance on the nuScenes dataset without relying on LiDAR data for depth supervision. It excels in metrics such as mean Average Precision (mAP) and NuScenes Detection Score (NDS), outperforming existing radar-camera fusion methods.
limitations: limitations: 1.The proposed method relies on extracting high-precision semantic information from images. However, its effectiveness in maintaining high performance under conditions of poor image quality, such as adverse weather or low lighting, is uncertain and requires further validation.
2.The experiments in the paper are solely conducted on the nuScenes dataset. To robustly verify the effectiveness of the proposed method, additional comparisons on other mainstream 3D detection datasets like Waymo and KITTI are necessary.
3.The paper would benefit from including detailed visualizations of 3D detection results in the ablation studies. This would allow for a thorough analysis of the degradation effects caused by the absence of specific components, providing deeper insights into the contributions of each part of the model.
paper_summary: paper_summary: The paper introduces SparseInteraction, an end-to-end fusion framework, to enhance detection performance. It employs the Noisy Radar Filter (NRF) module to extract foreground features by filtering out noisy radar data using semantic image features. The Sparse Cross-Attention Encoder (SCAE) effectively combines foreground radar and image features to resolve positional ambiguities. To further improve model performance and convergence, foreground prior queries with positional information are integrated into the transformer-based decoder. Experimental results show that these fusion strategies achieve state-of-the-art results on the nuScenes benchmark.
strength_weakness: strength_weakness: 1. A novel Noisy Radar Filter was proposed to efficiently address false positive issues.
2. Sufficient experiments on nuScenes dataset demonstrate the superiority of the proposed method.
limitations: limitations: 1.Is Eq.1 the first time proposed? Please provide more references to highlight your contribution

2.Foreground Radar Mask seems like an interesting concept. Is there any previous work proposing similar ideas? Could it be visualized in a paper?

3.Some recent open-source works can be compared in the experiment.
[1] Wang S, Liu Y, Wang T, et al. Exploring object-centric temporal modeling for efficient multi-view 3d object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 3621-3631

4.Some recent works can be introduced in the related works.
[1] Jiang X, Li S, Liu Y, et al. Far3d: Expanding the horizon for surround-view 3d object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(3): 2561-2569.
[2] Yang C, Chen Y, Tian H, et al. BEVFormer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17830-17839.