Title: Availability-aware Sensor Fusion via Unified Canonical Space
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sensor Fusion

4D Radar

LiDAR

Camera

Sensor Degradation

Sensor Failure

Autonomous Driving
Abstract: Sensor fusion of camera, LiDAR, and 4-dimensional (4D) Radar has brought a significant performance improvement in autonomous driving. However, there still exist fundamental challenges: deeply coupled fusion methods assume continuous sensor availability, making them vulnerable to sensor degradation and failure, whereas sensor-wise cross-attention fusion methods struggle with computational cost and unified feature representation. This paper presents availability-aware sensor fusion (ASF), a novel method that employs unified canonical projection (UCP) to enable consistency in all sensor features for fusion and cross-attention across sensors along patches (CASAP) to enhance robustness of sensor fusion against sensor degradation and failure. As a result, the proposed ASF shows a superior object detection performance to the existing state-of-the-art fusion methods under various weather and sensor degradation (or failure)  conditions. Extensive experiments on the K-Radar dataset demonstrate that ASF achieves improvements of 9.7\% in $AP_{BEV}$ (87.2\%) and 20.1\% in $AP_{3D}$ (73.6\%) in object detection at IoU=0.5, while requiring a low computational cost. All codes are available at https://github.com/kaist-avelab/k-radar.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for a new angle on sensor fusion between photometric (e.g., image) and geometric (e.g., ToF including lidar/radar) modalities -- \"availability-aware\" sensor fusion, which they call ASF. \n\nThe method consists of two \"sub-modules\" which are (1) a unified canonical projection into a common space for all modalities, and the authors claim that this eliminates inconsistencies between modalities. Then, (2) the method addresses the problem of a missing sensor/modality by learning to estimate missing, or uncertain, modalities.\n\nQuantitative results on the K-Radar dataset are shown, and the proposed method is seen to outperform baseline methods of multi-sensor fusion.\n\nSTRENGTHS AND WEAKNESSES:\n(S1) First of all, the idea is well-motivated and the problem setting of dealing missing or damaged sensors is very interesting and relevant to real-world systems such as autonomous driving vehicles.\n\n(S2) Most parts of the method are described thoroughly with sufficient mathematical rigor, and importantly, are well-motivated. In other words, the authors provide a written justification of most of the design choices they made, e.g., removal of positional embedding and post-feature normalization, among others.\n\n(S3) The ablation and sensitivity choices are thorough and present quantitative backing for the design choices mentioned in S2.\n\n(W1) The authors claim that they only evaluate performance on the K-Radar dataset because it \"is the only dataset with data captured in adverse weather conditions.\" While I will not go so far as to call this assertion incorrect, it is important to note that \"adverse weather conditions\" is far from an objective term -- in my opinion, many datasets including the nuScenes dataset contain at least some amount of data captured in some degree of adverse conditions.\nImportantly, most autonomous driving works studying multi-modal (particularly, image-lidar) 3D object detection works are benchmarked on the canonical nuScenes dataset, such as UniTR (Wang et al., ICCV 2023) and SparseFusion (Xie et al., ICCV 2023). Can the authors present the results of their proposed sensor fusion for detection framework on this dataset, as has become the standard in this line of work? Besides nuScenes being the standard, it would be extremely valuable to see that the proposed method works well (enough) on more than one dataset. Currently, this is the main reason for the \"fair\" rating for Significance.\n\n(W2) In the introduction, the authors seem to equate a missing sensor and an uncertain sensor. While these concepts may seem tied together at face value, the concept of uncertainty is not brought up again in the rest of the paper. The authors should clarify whether their method actually addresses the problem of uncertainty, which in itself is studied in a large body of existing works.\n\n(W3) Minor weakness: Figure 3 would be more convincing if ground truth and prediction were overlaid, as it is currently hard to draw conclusions between many combinations who seem, at least to my eye, to show very similar performance. A form of error map or plot would be equally helpful. Relying on the captions containing quantitative results to tell the story means that it would have been more efficient to simply present another small table for the two examples shown.\n\n(W4) Very minor weakness: A few too many abbreviations that hinder the flow and readability of the paper. The authors could consider undoing some of the more unnecessary ones like autonomous driving (AD) and feature map (FM).\n\nQUESTIONS:\n(Q1) Can the authors present results on the nuScenes dataset, which is canonical for multi-modal multi-sensor 3D object detection? This would greatly help to motivate the method, see W1 for more details. Alternatively, if the authors would like to provide a rebuttal for why the nuScenes dataset would be inappropriate for this setting, I would be open to this argument, but in this case I would still like to see at least one more relevant experiment against existing approaches.\n\n(Q2) Can the authors expand upon the claim that this method details with uncertainty in certain sensors? Have the authors done the necessary literature search on methods that quantify uncertainty in data, i.e., aleatoric uncertainty?\n\n(Q3) Again, these are minor points, but can the authors provide a plan to improve clarity and convincingness in the aforementioned areas of the paper (figure, readability)?\n\nI am inclined to keep my score of Accept given that my few concerns are addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper proposes a pipeline for 3D object detection by exploiting the fusion of multimodal sensor data.
The pipeline works with 3 steps:
first, specific pre-trained feature extractors are used for each type of sensor (camera, LiDAR, radar).
This provides features from all sensors in the same BEV format.
Then, the fusion is done in two stages, one to unify the features from different source in a common canonical space, then use cross-attention model that is trained on available data, making it give more attention to available sensors; This improves the performance by filtering noisy data from bad weather or broken sensors. The patch based optimization and the single common loss for all modalities allow faster computation.
This provides features that can be used for 3D object detection.
The final results show SOTA performance across different weathers and sensor failures against multiple baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
- Proposes a method to leverage multimodal sensors for 3D detection that is able to automatically filter noisy data through attention weight.
- SOTA results in difficult situations (bad weather, broken sensors)
- Paper is clear and well-written with in-depth experiences and analysis

Weaknesses:
- Camera information still provides less reliable performance as it does not provide geometric information like the LiDAR or radar.

QUESTIONS:
How would stereo cameras impact the performance, as it would provide geometric information from the images?
Would any other representation that BEV be possible and what impact would it have on performance?
How much of the performance variation can be attributed to the feature extractor of each modality?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a novel approach to sensor fusion designed to enhance robustness and reliability in autonomous vehicle perception by integrating data from multiple sensors (cameras, LiDAR, radar) based on their availability. \nTo this end, authors propose a cross-attention method and a unified canonical projection.\nThe unified canonical space allows a unified and consistent of features from different sensors, while the cross-attention across sensors along patches dynamically adjusts the important of features from the sensors based on their current availability and reliability. The objective is to prevent adverse effects on the fused data quality when one sensor is unavailable or sends low quality data. A sensor combination loss is used to optimize the detection performance across sensors. \nThe method is evaluated through experiments on the K-Radar dataset. Results show improvements on object detection performance metrics compared to state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \nThe proposed method is technically sound, novel, and very useful to the field, as it tackles a real issue in perception. Sensor fusion drastically improves performances and robustness, but can have negative impacts when some sensors become unavailable or degraded. This works provides a solution to this issue by dynamically changing the impact of the contribution of features from each sensor based on cross-attention. This is a clever use of cross-attention that solves a real-world problem. The contribution of the Unified Canonical Space for more consistent feature representation across sensors is valuable, and its usage with the proposed cross-attention method seems to succeed in reducing issues linked to sensor availability. \n The paper is well written, the equations describing the cross-attention method are clear, and the method is well evaluated on the K-Radar dataset. Visualizations of t-SNE plots and attention maps are an interesting addition. The results of experiments are also quite impressive, proposing significant improvement over the state of the art on a key task: object detection. The code is also already available on GitHub.\n\n## Weaknesses\nFigure 3, while very useful and containing interesting visualizations, could be made clearer for the reader. It would also be interesting to evaluate the method on different datasets, although I understand adequate datasets are not that numerous.\n\nQUESTIONS:\nHow do you plan to address the limitations related to the camera features integration? Particularly in difficult weather conditions. \n\nThe method is evaluated on K-Radar. This is enough, but it would be interesting to see evaluations on other datasets or on real-world data. \n\nI would recommend to improve a bit the figures, which are very informative but slightly complex. Figure 3 would especially benefit from a simpler organization and more space for better readability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sensor fusion framework (Availability-aware Sensor Fusion) to address the degradation problem in autonomous driving sensors. ASF introduces two components: Unified Canonical Projection (UCP), which projects features from heterogeneous sensors into a unified representation space, and CASAP (Cross-Attention across Sensors Along Patches), which dynamically fuses sensor features based on availability. A Sensor Combination Loss (SCL) is adopted to improve robustness. Experiments on the K-Radar benchmark demonstrate significant improvements in both performance and efficiency over prior works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.ASF achieves state-of-the-art results under multiple weather and sensor failure conditions.\n2.Lower memory usage and higher inference FPS than existing methods. CASAP simplifies the standard SCF computation by reducing the cross-attention complexity.\n\nWeaknesses \n1.The camera-based object detection (particularly in adverse weather) remain unaddressed. Yet the performance on adverse weather is highlight in the paper.\n2.The experimental section primarily consists of straightforward performance tables under different sensor/weather configurations, but lacks in-depth diagnostic or interpretative analysis.\n\nQUESTIONS:\n1.How does CASAP actually infer sensor reliability? Is there any theoretical justification or formal analysis behind its attention mechanism, or is it purely learned empirically?\n2.Can the method generalize to datasets without 4D Radar (e.g., nuScenes, Waymo)? If not, how tightly is the approach bound to K-Radar’s sensor setup?\n3.The influence of camera features remains unclear. Given that the limitation section identifies camera as a weak modality, how much does it actually contribute to detection performance in normal vs. adverse conditions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}