Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion

Authors: Minkyoung Cho, Yulong Cao, Jiachen Sun, Qingzhao Zhang, Marco Pavone, Jeong Joon Park, Heng Yang, Zhuoqing Mao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal perception; Sensor fusion; Robustness; Uncertainty quantification

Abstract:
An important paradigm in 3D object detection is the use of multiple modalities to enhance accuracy in both normal and challenging conditions, particularly for long-tail scenarios. To address this, recent studies have explored two directions of adaptive approaches: MoE-based adaptive fusion, which struggles with uncertainties arising from distinct object configurations, and late fusion for output-level adaptive fusion, which relies on separate detection pipelines and limits comprehensive understanding. In this work, we introduce Cocoon, an object- and feature-level uncertainty-aware fusion framework. The key innovation lies in uncertainty quantification for heterogeneous representations, enabling fair comparison across modalities through the introduction of a feature aligner and a learnable surrogate ground truth, termed feature impression. We also define a training objective to ensure that their relationship provides a valid metric for uncertainty quantification. Cocoon consistently outperforms existing static and adaptive methods in both normal and challenging conditions, including those with natural and artificial corruptions. Furthermore, we show the validity and efficacy of our uncertainty metric across diverse datasets.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730308512256

thread_type:
official_review

<thread_object>
creator: reviewer_38s7

<content>
summary:
T​this paper propose a multimodal fusion framework called Cocoon for 3D object detection. The core innovation of Cocoon is to achieve object and feature level fusion through uncertainty quantification. The framework introduces a feature aligner and a learnable alternative truth value (called "Feature Impression") to fairly compare uncertainties between multimodalities and dynamically adjust the weight of each mode. Experimental results show that Cocoon outperforms existing static and adaptive fusion methods under standard and challenging conditions, and performs more robustly under natural and artificial perturbations.

strengths:
1. The paper is clearly presented and easy to understand.
2. The innovation of uncertainty quantification mechanism is feasible from the principles and experiments in Section 4.
3. What I care about more is that the Cocoon framework has good scalability and can be further applied to more types of modal fusion, such as combining camera, LiDAR and radar data for autonomous driving, or combining visual and language modalities for visual language models (VLM).

weaknesses:
1. The entire COCOON process, especially the structural comparison shown in Figure 1, and the "FEATURE IMPRESSION-BASED FEATURE ALIGNER" and "UNCERTAINTY QUANTIFICATION" parts in Sec 4.2 and 4.3, made me feel the huge computational overhead. Whether it is based on RGB, LiDAR, or multi-modal fusion target detection methods, I think they cannot be implemented quickly if the real-time performance cannot be guaranteed. The advantages can only be reflected at the numerical level of the paper.
2. Does COCOON rely too much on the quality of feature alignment and calibration in the data? Section 4.2 mentioned that since data from different modalities need to be projected into the same feature space for comparison, the accuracy of this part of the alignment directly affects the reliability of uncertainty quantification. If there are large differences in the quality or sampling frequency of the features of each modality, it will be difficult for the feature aligner to effectively align the data distribution, will this in turn affect the overall fusion effect?
3. At the end of Section 5.1, the value of the hyperparameter $ \alpha $ is set, but Table 4 directly assigns $ \alpha $ to 0.1. And Limitations mentions that the change of alpha will bring about a significant change in the algorithm effect. Is COCOON too dependent on hyperparameters?

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644805982

thread_type:
official_review

<thread_object>
creator: reviewer_9juv

<content>
summary:
The paper introduces Cocoon, a novel framework that addresses the challenge of multi-modal fusion in 3D object detection. The paper proposes a novel uncertainty-aware fusion framework that operates at both object and feature levels through the introduction of Feature Impression and a feature alignment mechanism. This approach leverages conformal prediction theory to quantify uncertainties in heterogeneous sensor data, enabling dynamic weight adjustment between different modalities such as cameras and LiDAR.

strengths:
1.Proposes a novel framework (Cocoon) that uniquely combines object-level and feature-level uncertainty awareness in multi-modal fusion, addressing a significant gap in existing approaches.
2.Thoroughly evaluates the model under diverse corruption scenarios, including both natural corruptions.
3.Provides solid mathematical foundation for the uncertainty quantification method, with clear derivations and validations across multiple datasets.

weaknesses:
1.The paper shows a substantial increase in computational latency - from 1.1s to 1.6s when integrating Cocoon with FUTR3D.
2.While the paper demonstrates consistent improvements over baselines, the actual gains are relatively modest in many scenarios.
3.Only evaluates camera and LiDAR fusion, missing other important sensors like radar and ultrasonic.
4. The lack of validation on other datasets raises concerns about the generalizability of the method. Validation can be performed on the Waymo and Argoverse 2 datasets.
5. The method provides marginal gains and performs worse than previous methods such as CMT, BEVFusion, and DeepInteraction.

questions:
1.While you propose using fewer decoder layers or queries to reduce latency: How does this affect the accuracy-speed tradeoff specifically?
2.Could you provide ablation studies on the importance of different components in your framework? Such as, how much does each component (feature aligner, Feature Impression, uncertainty quantification) contribute to the final performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649936812

thread_type:
official_review

<thread_object>
creator: reviewer_xhyu

<content>
summary:
This paper introduces an object- and feature-level uncertainty-aware fusion framework for robust multi-modal sensor fusion in 3D detection scene of autonomous driving. It designs a valid uncertainty estimator for heterogeneous representations, and then dynamically fuses multi-modal sensor features. Experiments indicate the effectiveness of fusing heterogeneous features in several unrobust scene.

strengths:
(1) It provides a very intuitive feature fusion analysis to derive the motivation of this paper
(2) All the designs that underpin the motivation of this article are very valid and reasonable.
(3) The results are rich and convincing.

weaknesses:
Although the writing logic and thinking are clear, the explanation of the method is too complicated, which will bring inconvenience to the reader.

questions:
No.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731885123105

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1 & 2-1): Computational Overhead Mitigation & Ablation Study

comment:
## [1] Computational Overhead Mitigation

First of all, we reduced inference latency through code refactoring: static fusion from 1.1 to 0.91 seconds and Cocoon from 1.6 to 1.27 seconds.

From the ablation study (described in next response), we found that adjusting the MLP layer count and the vector dimension of Feature Impression (FI) can reduce the overhead of 120 ms while maintaining higher accuracy than static fusion.

To further reduce the overhead, we devised a fine-tuning approach: dynamic fusion is applied only to the last decoder layer, and we fine-tune only the components in the later stages of the fusion process (the earlier parts must remain unchanged to preserve the validity of nonconformity score). 

Using this fine-tuning approach, we reduced Cocoon’s additional overhead (excluding the base model) to 50 ms—**a significant reduction of 86% from the previous overhead value of 360 ms**—while achieving the final accuracy shown below on an NVIDIA GeForce RTX 2080 (11 TOPS [1]). **Given the computational capability of current AV systems (254 TOPS) [2], this overhead is negligible, with total latency within the 100 ms threshold for real-time guarantees [3].**

| Fusion                      | No Corruption | Rainy Day | Clear Night | Rainy Night | Point Sampling (L) | Random Noise (C) | Avg. Latency  (sec)     |
|----------------------------|---------------|-----------|-------------|------------|---------------------|------------------|--------------------|
| Static       | 66.16         | 68.34     | 44.51       | 27.26      | 65.17               | 60.39            | 0.91 ± 0.02       |
|  Cocoon (original) | 66.80         | 68.89     | 45.68       | 27.98      | 65.89               | 61.83           | 1.27 ± 0.04        |
| Cocoon (fine-tuning) | 67.13      | 69.12     | 45.70       | 28.07      | 66.02               | 61.99            | 0.96 ± 0.03        |

***Reference***


[1] https://www.nvidia.com/en-us/geforce/news/geforce-rtx-20-series-super-gpus/

[2] https://www.nvidia.com/en-us/self-driving-cars/in-vehicle-computing/

[3] Lin, Shih-Chieh, et al. "The architectural implications of autonomous driving: Constraints and acceleration." Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems. 2018.

---
## [2] Ablation Study

Given that feature aligner (FA) and feature impression (FI) are designed for uncertainty quantification (based on conformal prediction), we conducted the following experiments to evaluate each component's impact.

### [2-1]   Uncertainty quantification (UQ) vs. Object-level dynamic weighting 

- When both are disabled, static fusion is performed via concatenation.
- When conformal prediction is disabled, a simple MLP model is used to output a weight value for each query.
- When dynamic weighting is disabled, the nonconformity (NC) score in UQ is measured for the entire input feature.

| Conformal Prediction-based UQ | Dynamic Weighting |  No Corruption | Rainy Day | Clear Night | Rainy Night  | Cam Blackout | Avg. Latency (sec)         |
| -------------------- | ----------------- | ----------- | -------- | ----------- | ---------- | -------- | ------------- |
|                |                  |  66.16         | 68.34     | 44.51       | 27.26      |  45.13            | 0.91 ± 0.02       |
|                | V                 | 63.01       | 63.51    | 42.01       | 21.91      | 37.01    | 0.91 $\pm$ 0.03   |
| V                    |            | 66.2        | 68.37    | 44.52       | 27.3       | 45.21    | 1.26 $\pm$ 0.04 |
| V                    | V                 | 66.8        | 68.89    | 45.68       | 27.98      | 51.87    | 1.27 $\pm$ 0.04 |


	Conclusion:  Dynamic weighting cannot function without UQ, and UQ is ineffective without dynamic weighting, making them both essential to each other.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new object- and feature-level uncertainty-aware multimodal fusion framework for 3D object detection tasks. The proposed framework adopts a feature aligner and a feature impression strategy to achieve uncertainty quantification for heterogeneous representations. The proposed approach is novel and effective and consistently outperforms existing static and adaptive methods. Reviewer concerns are about the complicated writing, increased computational latency, and some missing experiments and analysis. After rebuttal, most of these concerns were addressed, and two reviewers improved their score from 5 to 6. Finally, all reviewers achieve a consistent opinion of acceptance. After reading the paper and all the discussions, the AC confirmed the recommendation.