Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
Authors: Minkyoung Cho
Yulong Cao
Jiachen Sun
Qingzhao Zhang
Marco Pavone
Jeong Joon Park
Heng Yang
Zhuoqing Mao
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-modal perception; Sensor fusion; Robustness; Uncertainty quantification
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Cocoon is a new object- and feature-level uncertainty-aware multimodal fusion framework designed for 3D OD tasks. At the core of Cocoon are uncertainty quantification and comparison for heterogeneous representations.

Abstract: An important paradigm in 3D object detection is the use of multiple modalities to enhance accuracy in both normal and challenging conditions, particularly for long-tail scenarios. To address this, recent studies have explored two directions of adaptive approaches: MoE-based adaptive fusion, which struggles with uncertainties arising from distinct object configurations, and late fusion for output-level adaptive fusion, which relies on separate detection pipelines and limits comprehensive understanding. In this work, we introduce Cocoon, an object- and feature-level uncertainty-aware fusion framework. The key innovation lies in uncertainty quantification for heterogeneous representations, enabling fair comparison across modalities through the introduction of a feature aligner and a learnable surrogate ground truth, termed feature impression. We also define a training objective to ensure that their relationship provides a valid metric for uncertainty quantification. Cocoon consistently outperforms existing static and adaptive methods in both normal and challenging conditions, including those with natural and artificial corruptions. Furthermore, we show the validity and efficacy of our uncertainty metric across diverse datasets.

Published as a conference paper at ICLR 2025
CoCoon: ROBUST MULTI-MODAL PERCEPTION WITH UNCERTAINTY-ANEE SENSOR FUSION
Minkyoung Cho', Yulong Cao', Jiachen Sun', Qingzhao Zhang',
Marco Pavone Jeong Joon Park', Heng Yang* and Z. Morley Mao
'University of Michigan
"NVIDIA Research
Stannoor University
"Harvard University
https: htlpa///miinyyougchho.ooo wwt.tgtthhhh.................... wttluub..o/ooooo coconn/
ABSTRACT
An important paradigm in 3D object detection is the use of multiple modalities to en- hance accuracy in both normal and challenging conditions, particularly for long-tail scenarios. To address this, recent studies have explored two directions of adaptive approaches: MoE-based adaptive fusion, which struggles with uncertainties arising from distinet object configurations, and late fusion for output-level adaptive fusion, which relies on separate detection pipelines and limits comprehensive understand- ing. In this work, introduce Cocoon, object- and feature-level uncertainty- aware fusion framework. The key innovation lies in uncertainty quantification for heterogeneous representations. enabling fair comparison across modalities through the introduction of a feature aligner and Icarnable surrogate ground truth, termed feature impression. We also propose a training objective to ensure that their relation- ship provides valid metric for uncertainty quantification. Cocoon consistently out- performs existing static and adaptive methods across both normal and challenging conditions, derived from natural and artificial corruptions. Furthermore, we demon- strate the validity and efficacy of our uncertainty metric across diverse datasets.
INTRODUCTION
In the rapidly evolving field of intelligent systems, accurate and robust 3D object detection is fundamental and crucial functionality that significantly impacts subsequent decision-making and control modules. With the increasing importance of this field, there has been extensive research on vision-centii (Wang et al., 2022; Liu et al., 2022: 2023a; Wang al., 2023) and LiDAR based detection (Yan et al., 2018; Yin et al., 2021; Lang et al., 2019; Zhou et al., 2024), but both approaches have demonstrated weaknesses due to inherent characteristics in sensor data. For example, 2D data lacks geometric information, while 3D data lacks semantic information. To address these limitations, multi-modal approaches that integrate multi-sensory data are actively studied enhance capabilities in areas such as autonomous vehicles, robotics, and augmented/virtual reality, setting a new standard for state-of-the-art performance various perception tasks (Chen al., 2023; Prakash al., 2021; Bai et al., 2022; Li et al., 2022b; Yan et al., 2023; Li et al., 2022a; 2023; Xie et al., 2023).
The accuracy of as time and the composition of surrounding objects (Yu et al., 2023; Dong et al., 2023). Given the diverse (often complementary) nature of different modalities. maximizing their utility across various objects and contexts remains challenging, particularly for long-tai tail objects (Feng et al., 2020; Curran et al., 2024). Existing research efforts to address this issue mainly fall into two categories: Mixture of Experts (MoE)-based approaches (Mees al., 2016; Valada et al., 2017) and late adaptive fusion (Zhang et al., 2023; Lou et al., 2023; Lee & Kwon, 2019). Despite their contributions, existing methods often overlook the varying levels of informativeness among individual objects in the scene by applying weights uniformly across entire features (Fig. la) or limiting comprehensive understand- ing/decoding by simply combining outputs from separately trained detectors (Fig. lb), both resulting
perception is significantly influenced by the operating context, such

---

Published as a conference paper at ICLR 2025
Color
20 Enc
2D Dec
Uncertainty Estimator for Heleregeneous Representations
X++
Objects
GTY
Depth
Depth
Out
20 Enc
Color & Depeh
Enc
Objects
[e1.. Qt)
Dec
Out
y-
(b) Late Fusion: Each modality's ' detector is trained independently, limiting accuracy that of the better- perter-peroormin modality.
30 Ene
IP1.. Bx)
(a) MoE-Based Fusion: Whole feature fusion ignores object-level informativeness variation.
(c) Cocoon (Ours): yi indicates query potential object.
Figure 1: Existing Fusion Methods' and Our Approach.
Object Distinction Cross-Modal Feature Enrichment
MoE-Based Fusion (Mees el al., 2016; Valada el al., 2017)
Late Fusion (Lee Kwon, 2019; Zhang et al,, 2023; Lou al., 2023) Cocoon (Ours: object- and feature-level uncertainty-avee
fusioo)
Table 1: Comparison with Prior Works
in sub-optimal performance. To our knowledge, none of these approaches address both object-level uncertainty variation and feature-level (i.e., intermediate) fusion needs, as highlighted in Table 1.
To address these challenges, we introduce Cocoon (Conformity-consciin
sensor fusion), which tackles the aforementioned challenges by quantifying uncertainty at both the object and feature levels, and dynamically adjusting the weights of modality-peecific features (Fig. 1c). Cocoon comprises four key components: a base object detector, feature alignment, uncertainty quantification, and adaptive fusion. For object- level adaptive fusion, Cocoon uses a query-based architecture as the base object detector, where a fixed number of queries are predetermined, each representing a potential object (Carion et al., 2020a; Chen et al., 2023; Bai et al., 2022). When multi-modal data received, Cocoon first projects the per-query features, encoded by modality-specif detector, into a common representation space (Jing et al., 2021). Our uncertainty quantification is based conformal prediction (Angelopoulos & Bates, 2021; Teng et al., 2023) that is widely adopted in practical applications due to its low computational complexity and interpretability (Luo et al., 2022a; Su et al., 2024a; Yang & Pavone, 2023). We propose new conformal prediction method for heterogeneous representations by introducing the concept of a learnable surrogate ground truth, termed Feature Impression (FI), which enables feature-level uncertainty quantification. While traditional conformal prediction uses detection outputs and their ground truth labels in the output space-which are not available at intermediate stages of the model-our approach bypasses this limitation by employing FI. We demonstrate how FI can guide uncertainty quantification across heterogeneous representations, thus enabling fair comparisons. As such, Cocoon performs linear combination using uncertainty values, achieving object- and feature-level uncertainty-awar fusion.
encoders in the base
The main contributions are as follows:
We observed that ignoring object-level uncertainty in multi-modal object detection leads to suboptimal accuracy, particularly in long-tail conditions such adverse weather or nighttime, highlighting the need for object-level uncertainty aware fusion.
Cocoon achieves object- and feature-level uncertainty-awert -wwre fusion by introducing uncertainty quantification for heterogeneous representations. The combination of a feature aligner and feature impression enables efficient and effective measurement of uncertainty at intermediate stages of the model. The resulting weights adjust each modality's contribution, allowing the model to prioritize the modalities with higher certainty to deliver more robust performance.
In our evaluation on the nuScenes dataset (Caesar et al., 2019), Cocoon demonstrates notable improvement in both accuracy and robustness, consistently outperforming static and other adap- tive fusion methods across normal and challenging scenarios, including natural and artificial corruptions. For example, in cases of camera malfunction, Cocoon improves the base model's s mAP by 15% compared to static fusion.
Following the review of related work, present detailed analysis of the technical challenges in uncertainty-awart fusion within multi-modal perception systems. We then introduce the Cocoon framework and evaluate its effectiveness in addressing these challenges.
While most existing work focuses on 2D perception or the fusion of RGB and depth camera data, we adapt these concepts our situation.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Multi-Modal Perception. Multi-modal solutions aim to enhance the accuracy and robustness of perception systems by leveraging diverse data fusion designs, which are typically categorized into early, intermediate, and late fusion. Early fusion combines features at the input level (Malawade et al., 2022b; Putzar et al., 2010; Huang et al., 2020; Sindagi et al., 2019; Vora et al., 2020), while late fusion integrates decisions from independent models at the object level (Sun et al., 2020; Huang et al., 2022; Xiang et al., 2023; Yeong et al., 2021). Favored in recent methods, intermediate fusion enhances accuracy by addressing discrepancies among the inductive biases of different modalities (Chen et al., 2017: Qin et al., 2023; Malawade et al., 2022a). These biases, which vary based on each modality's 's characteristics, are harmonized via joint learning, ensuring a more cohesive integration of multi-modie data. For example, cameras (Lu et al., 2021; Huang & Huang, 2022: Huang et al., 2021) rely on color and 2D shapes, while LiDAR (Shi al., 2020; 2019) focuses on reflective properties and 3D shapes. Intermediate fusion combines these biases for a more robust environmental understanding. Techniques have evolved from M.DAR-centrii fusion approaches, such as PointPainting and PointAug- menting (Vora et al., 2020; Wang et al., 2021), to more sophisticated architectures like the Y-shaped model, which fuses multi-modaa features through concatenation, element-wiss addition, or an adaptive fusion method (Liu et al., 2023b; Liang et al., 2022; Jiao et al., 2023). Recently, Transformer-bass based approaches (Chen et al., 2023; Prakash et al., 2021; Bai et al., 2022; Li et al., 2022b; Yan et al., 2023; Li et al., 2022a; 2023; Xie et al., 2023), inspired by DETR (Carion et al., 2020b; Zhu et al., 2020b; Wang et al., 2022), have become popular. They offer efficient detection paradigms using the Hungarian algorithm (Kuhn, 1955) (i.e., the predefined number of queries matched objects), allowing flexible associations between modalities. Among these, FUTR3D (Chen et al., 2023) and TransFusion (Bai et al., 2022) stand out as the most foundational and representative transformer- based models, significantly influencing the current ofate-of-the-arr in 3D object detection (Prakash et al., 2021; Li et al., 2022b;a; Yan et al., 2023; Li et al., 2023; Xie et al., 2023).
Uncertainty Quantification. Uncertainty quantification in machine learning enhances the trust- worthiness and decision- pre-trained models (epistemic uncertainty) and observed data (aleatoric uncertainty). Bayesian tech- niques are well-known methods in this domain, as they incorporate prior knowledge and update it with new data to estimate posterior distributions for model weights or input data. However, Bayesian meth- ods are computationally intensive, making them less suitable for multi-sensory systems that require timely execution (Blundell al., 2015; Kendall & Gal, 2017). In such contexts, alternative methods are preferred. One such method is direct modeling, primarily for data uncertainty, which modifies de- tection models by adding output layers to directly predict output variance (Feng et al., 2021: Su et al., 2024b; 2023). Another approach is Double-M Quantification. which estimates variability for both data and model uncertainty through resampling (Su et al., 2023). Conformal prediction provides prediction intervals using nonconformity measure and a calibration dataset for addressing both data and model uncertainty (Angelopoulos & Bates, 2021; Teng et al., 2023; Javanmardi et al., 2024). Conformal prediction, with its low computational complexity, distribution--ree nature, provability, and inter- pretability, is most optimal for practical use (Luo et al., 2022a; Su et al., 2024a; Yang & Pavone, 2023).
capabilities of systems by leveraging uncertainties present in both
3 CHALLENGES IN UNCERTAINTY-WWAR FEATURE FUSION
Object-Level Uncertainty Variation Phenomenon. Assessing each modality's 's uncertainty, defined by how reliable its information about the input scene, challenging due to the variety of objects within the scene. Additionally, uncertainties are influenced by various factors, including data informa- tiveness and unusualness relative to a pre-trained perception model, making the assessment non-trivial.
We conduct an experiment to empirically demonstrate the impact of diverse uncertainties, as shown in Fig. 2. We apply baseline perception model (FUTR3D (Chen et al., 2023)), trained to fuse camera and LiDAR data in 1:1 ratio, to various traffic scenes. We tune the fusion ratio and observe its impact on model effectiveness, measured by the average confidence score. Interestingly, the 1:1 fusion ratio between 2D and 3D data proved to be suboptimal in most scenarios. In daytime scenarios, camera data on distant and small objects is more informative than LiDAR data, suggesting an optimal fusion ratio of 7:3. Conversely, for nearby objects, greater accuracy is achieved when the contribution of LiDAR is increased relative to the camera. Moreover, LiDAR sensors are more effective than cameras at night- time, indicating that more weight should be assigned to LiDAR inputs to improve perception accuracy.
Overall, this imbalance in the contribution of 2D and 3D data is common and related to various factors such as lighting conditions, object sizes, and distances to the ego agent. These observations
3

---

Published as a conference paper at ICLR 2025
fmals
Confiience Score (18)
Confidencc Score 19%)
80
far-large
mid-small
70
mid-med
60
mid-large
near-small
50
near-med
40
near-large
700
Pusion Ratio (CAR:LIDAR) (a) Daylime
Fusion Ratio (Cam:LIDAR) (b) Nighttime
Figure 2: Impact of fusion ratio average confidence scores under various lighting conditions and object configurations. The black stars denote the optimal camera-to- fusion ratios achieving the highest scores for each configuration. Object configurations are categorized based on two attributes: object size and distance to the ego agent. Object sizes are classified into three categories: small (< 2m), medium (2-4m), and large (> 4m). Similarly, distances are segmented into three ranges: near (< 20m), mid-distance (20-40m), and far (> 40m).
highlight the dynamic nature of object-level uncertainties and the need for specific dynamic weighting for each object based on its unique characteristics. To address this, Cocoon quantifies uncertainties for per-object features and dynamically adjusts the weights accordingly.
Undefined Consistent Basis for Comparison. Although existing methods like Cal-DETR (Munir al., 2023), which computes variance along transformer decoder layers, can quantify uncertainties within a single modality, projecting uncertainties from multi-modal data into a consistent and comparable space remains challenging. This step is crucial for effective multi-modal, uncertainty- aware fusion, where the alignment and comparability of data from different modalities are essential.
The challenge arises from the distinct representation spaces of multi-modal data: 2D data provides color and texture information, whereas 3D data offers depth and geometric insights. We address this challenge through Feature Impression-bes based feature alignment, motivated by Jing et al. (2021).
4 CoCOON
In this section, we introduce Cocoon, an object- and feature-level modality fusion solution. Cocoon employs a query-based architecture with fixed number of predefined queries, each representing a potential object, enabling to extract object-level features without relying on final detection outputs (Carion et al., 2020a; Chen et al., 2023; Bai et al., 2022). Specifically, we use FUTR3D (Chen et al., 2023) and TransFusion (Bai et al., 2 foundational and representative query-based models-as as our base models. Built on these architectures, Cocoon operates in two phases: an online phase that performs adaptive fusion on test data, and an offline phase that prepares for uncertainty quantification (i.e., conformal prediction) through training and calibration stages.
Fig. illustrates the online mechanism, where we first align per-object features into a common representation space (Sec. 4.2) and then quantify uncertainties for each pair of features (Sec. 4.3). These uncertainties are used as weights (a and (3) in the adaptive fusion. For the offline mechanism, first split the training set into proper training set and a calibration set. The proper training set is used to train both the base model components and the parameters for uncertainty quantification. Using the calibration set, we create comparison group containing task-related scores for each calibration sample, which are then used to assess how unusual the online input data is, thereby facilitating uncertainty quantification (Sec. 4.4).
We begin by revisiting the inspiring work, Conformal Prediction (Angelopoulos & Bates, 2021) and Feature CP (Teng al., 2023), and their limitations multi-modal setting. We then explain how we can fairly quantify the uncertainties of multi-modal features on consistent basis.
4.1 PRELIMINARIES
Conformal Prediction (CP). Conformal prediction, referred to as Basic CP, is an algorithm that outputs a prediction interval, statistically guaranteeing the inclusion of the ground truth (GT) for a given test input (Angelopoulos & Bates, 2021; Teng et al., 2023), rather than merely providing single prediction value. This prediction interval established with respect to a significance level (A), ensuring that the interval contains the GT with probability of at least 1-A, follows:
4

---

Published as a conference paper at ICLR 2025
(Sec. 4.4)
Results from 2D data corruption
(C(x)
Calbb
Feature Aligner
Unceriainty
Quantfication
2D data
sampling
(1)
query
Transformer
Task Head
Uncertaiintyso())
3D
Feature Aligner
Uncertainty Quantiication
Baseline fusion)
Modallty-paciffc
aligned
components
features
common
Pedestr
3D data
Motorcycle
Flow model
quantificatio
Figure 3: Cocoon Online Procedure (left) and Example Results (right): Cocoon operates on top of
base model components. The feature aligner projects per-object features representation space. Next, uncertainties are estimated for each feature pair into a common
and converted
into weights (a and B) for adaptive fusion. These weights amplifylattenuate the contribution of each modality's s original feature
to the fused feature, which then used in the basemodel's decoder.
Table 2: Summary of Main Symbols: Meaning and Purpose
Symbol
Meaning
Purpose
Section
a
Key symbols; represents the weight of one modality (e.g.. camera). and represents the weight another (e.g., LiDAR).
Uncertainty- based weights (Dynamic)
$4.3, 85.2, Fig. 3,
Fig. 6, Fig.
6,C.
Coeficcinnt in the training loss function. Significance value used to explain the concept of Analysis conformal prediction.
Hyperparameteer
$5.1, Ey 3
A
$4.1, $5.3, Table 4
P( (Z))
where Z (x=) represents the prediction interval for online test input x'.
The process begins with preparing training set, calibration set, a pre-trained model, and the establishment of nonconformity function. During the offline stage, the nonconformity score NC; Distance(p;, q:) is computed for each calibration instance, where p; and qi denote the predicted output and the GT of instance i, respectively. The pool of scores used to gauge how unusual online test data is. In the online stage, the algorithm uses 1 A to compute the tightest prediction interval that includes the GT with at least 1 confidence. This is achieved by calculating the A)-th quantile from the distribution derived from NC,, for E calibration set. Unlike Basic CP, which quantifies output-space uncertainty using GT labels on calibration samples, we address feature-space uncertainty without relying on poddality-specinfe GT labels.
Feature Conformal Prediction (Feature CP). Teng et al. (2023) extends conformal prediction to the feature level using a preitrri model M, structured into f (encoder) and g (decoder; task head), formulated as Y = M(X) (g o (X), with denoting the composition operator. This method measures nonconformity scores directly at the feature level via the encoder f. For nonconformity measure, surrogate ground truth v* is identified through an iterative search (see Fig. 4 (a)) to serve as a proxy for the normal GT within a norm-based nonconformity function: NC, = [[] v Here, e* is meticulously calibrated to closely resemble (wx) However, applying feature conformal prediction directly to problem poses challenges, primarily because assumes homogeneous representation across target features an assumption that does not hold in multi-modal settings.
4.2 FEATURE AMPRESSION-BASER FEATURE ALIGNER
Core Issue: Lack of Modality -Deiictte Decoder. As illustrated in Fig. 4, a dedicated decoder often does not exist in multi-modal detection models. The absence of g highlights a core issue, as 9 plays crucial role in facilitating the bidirectional movement between output space and feature space. To address this issue, we perform all operations directly at the feature level, eliminating the conventional move in/out mechanism. Our intuition is that effectively aggregating multi-modal
5

---

Published as a conference paper at ICLR 2025
Movable
Fixed
Mocally-speciti foalures
Alignad features
Burrogall ground buth
M
Single-Modal Model
100
M(x)
Multi-Modal Model
%
M(xe x)
R
nonconforsity score
dsf
de2
de1
(00
di2
geommtric mediaa
monconformitt
2D fealure space
20 feature spece
2D space
3D space
Common feature space
Ibrative Search
Join Touning on (ll))) Fasture Aligner (b)
(
move
(C)
(P
moving towards (a) Feature CP
converging
(b) Cocoon
Figure 4: Feature CP vs. Cocoon. In the offline stage with calibration data, Feature CP identifies the surrogate ground truth for each feature through iterative search. Each is derived using the real ground truth label in the output space and the decoder g (serving as classifier). However, in a multi-modal setting, each feature lacks a modality-specific To resolve this, Cocoon leverages joint training of the feature aligner (which projects heterogeneous features representation space) and the surrogate ground truth termed FI). Our proposed training objective (Eq. 3) ensures that each FI becomes the geometric median of aggregated features for valid uncertainty quantification. In the offline stage, the calibration samples' nonconformity scores (i.e., distance) are collected and used as a comparison target to gauge the uncertainty of online test inputs. In the online stage with test data, while Feature CP performs iterative search for Cocoon saves time by directly projecting input features through our feature aligner h and using a pre-trained
into common
features into a high-dimensional sphere substantially reduces the need for g. The following sections demonstrate how conformal prediction can operate efficiently with multi-modal features without
Based on our intuition, we initiate the process by aligning multi- features within unified representation space using a simple multi-layer perceptron (MLP), inspired by Jing et al. (2021). ). The key aspect of approach is the learnable surrogate GT and the joint training of both the surrogate GT and the feature aligner. For clarity, we introduce the new term, feature impressions (FIs; in Fig. 4 (b)), with each Fl representing class or of similar features and serving a surrogate GT.
Conditions for FI. As shown in Fig. 4, Feature CP identifies the surrogate GT ey as the nearest feature in Fig. 4 (a)) to the input sample x, ) obtained through iterative search. In the feature space, each has its unique e which eventually corresponds to in the output space. In other words, although they measure nonconformity scores feature space, their final uncertainty values (w.rtt yı) are quantified in the output space. Note that if this is classification task, y/ (e.g., class label) shared by multiple r;s belonging to the same class. Based on our intuition, we aim to remove bidirectional movement between feature and output spaces and perform uncertainty quantification directly in the feature space. To achieve this, our surrogate ground truth should satisfy two conditions: (I) we have to identify a common ground truth for the aligned multi-modal features; (2) given evidence Feature CP, each FI node should be as close as possible to the aligned feature to serve effective surrogate GT.
Assuming all distances between an node and its corresponding aligned features are distinct positive values, the Fl node should be the unique minimizer of the sum of all the distances serve as the effective surrogate GT. This strategic placement establishes the each Fl as the geometric median, effectively capturing the essence of the associated features' distribution within the unified representation space. Consequently, we can define our nonconformity function as:
NC; = (1 o f)(,)) - "", ll.
(I)
where h represents the feature aligner, and w denotes the z;'s corresponding Fl node. This positioning not only enhances efficiency by eliminating the costs associated with g but also simplifies the overall uncertainty quantification process.

---

Published as a conference paper at ICLR 2025
4.3 UNCERTAINTY QUANTIFICATION WITH
FEATURE SET
Uncertainty quantification entails identifying the most probable FI node at each transformer decoder layer, leveraging insights from the nonconformity function in Eq.1 (see Appendix for further details).
Finding 1: Feature uncertainty significantly affects the NC score of the top-1 1 FI node.
Finding 2: Higher feature uncertainty in transformers with multiple decoder layers leads to significant instability in the top-1 FI node across layers.
Based on these observations, our primary uncertainty value is defined using the metric (p-value) employed in conformal prediction (Vovk et al., 2005):
1x E NC x [Cmed]
H
P mod
|N
where NCealih represents the pool of nonconformity scores computed from all aligned calibration samples, and mcmod denotes the nonconformity score of the current online input data's 's feature. Leveraging this, the final weight for each modality is calculated as:
Smod
Wmod
Qood Smod + QotherSoterr
where Qmod =
Pand-tPutae and Smod denotes the stability score, measured by the frequency of top-1 FI changes across decoder layers. other refers to the modality opposite to mod. The resulting uncertainties used weights (a and B in Fig. 3) in adaptive fusion.
Pond
4.4 TRAINING AND CALIBRATION STAGES
The training process consists of two stages: the first stage involves training the baseline model using a proper training set, ensuring the pre-trained model has no exposure to the calibration samples. The second stage involves joint training of the feature aligner and Fls, while keeping the base model components frozen. Since the first stage follows the same procedure as the original base model's training (Chen et al., 2023; Bai et al., 2022), we focus on the second stage in this section.
Our nonconformity function (Eq. 1) requires each Fl node to act as the geometric median for its associated features. A specifically tailored regularizer is crucial to maintain this positioning.
Joint Training of Fl and Feature Aligner. The Weiszfeld's algorithm stands out as a notable solution method for the geometric median problem (Chandrasckaran & Tamir, 1989). For point to qualify as the geometric median, must meet the following criterion:
172
I ull
We can apply this criterion to problem setting where we have set of 2NN features:
hei = h o fe ( i 1,...,N, hus := h f((mu), i = I.....N
where fo and ft are encoders in the model. and zei and zu represent input data from multiple sensors. With these terms defined, our training objective consists of three terms: Leenter for aggregating different modality features around their corresponding FIs (w"), Lgeomed for ensuring that the Fls serve as the geometric median of the aligned features, and Leeparote for increasing the distance between the positions w", ensuring clear separation between Fls.
L
I|w"" heal l/wo" hall +c
Σ
hei w0* will
11s
Σ luu3 will? 1-1km441
w
Ceenter
Eysomed
Exsperate
Here, C denotes the number of Fl nodes (e.g., the number of classes). This loss function is used to jointly train feature aligner (h) and FIs (w*). ). See Appendix C for the derivation of Eq.
Calibration. This process involves computing nonconfornity scores (Eq. 1) for each calibration instance using the praited FIs and feature aligner h. The pool of calibration scores is then used to assess how unusual the features of the online input data are, facilitating uncertainty quantification. Further details about the offline process are provided in Appendix
7

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy (mAP) Comparison with Baseline Methods Under Different Scenarios. Orange
represents original nuScenes validation set, green indicates natural corruptions collected from real- world driving environments, and blue corresponds to artificial corruptions presented in existing work.
Methods
Type
No Corruption
Rainy Day
Clear Night
Rainy Night
BUS R3D (Chnn 2023)
Statie
68.34
44.51
27.26
AdapNett (Woadd all 20113)
MoF--based
62.33
62.82
41.74
21.58
et (2020s)
Sclf-artention
63.72
64.22
41.17
20.79
Zhang et al. (2023)
Late Fusion
63.93
64.64
42,92
22.56
Cal-DETR (T) (Marrie al, 2023)
Unsertainty-
66.24
44,38
26.29
Cal-DETRR DDETR(OO (Q) (Mastree at, 2031)
Uncertainty-nn aware
66.38
68.42
44,61
27.30
Cocoon (Ours)
Uncertiinty
66.80
68.89
45.68
27.98
Methods
Type
Point Sampling (1.)
Random Noise (C)
Blackout (C)
Sensor Misalign.
Base TOLD aid. 1023)
Statie
65.17
60.39
45.13
53.16
AdapNet (oidee 2017)
MoE-based
61.40
36.43
46.04
Zhu et (2020a)
Self-sttention
62.60
56.07
37.82
45.91
Zhang et al. (2023)
Late Fusion
63.24
56.32
39.39
49.61
Cal-DETR (T)
2023)
Uncertainty-oe
65.29
60.40
44.39
54.87
Cal-DETR (Q) (Mlasires 2021)
Uncertaintly aware
65.40
60.56
45,25
54.54
Cocoon (Ours)
Uncertainty-
65.89
61.83
51.87
55.61
Table 4: Accuracy Breakdown w.r.t. Distance.
70-
60
Near (< 20m)
Mid (20-40m)
Far (> 40m)
Corruption
Static
Ours
Static
Ours
Static
Ours
40-
No Corruption
72.3
72.6
53.9
54,3
7.4
7.7
Confidenc
Point Sampling
72.2
72.5
53.5
53,7
7.0
7.4
30
1(3ur)
Random Noise (C)
69.8
70.9
49.4
50.4
7.4
7.9
20
Ni (oust
Blackout (C)
69.4
37.5
42.8
3,3
4.5
10
Large (Stal))
Misalignment
68.4
69.0
46.3
47.1
4.1
4,4
None
Noise
Figure 5: Confidence Score Change.
EVALUATION
In this section, we evaluate Cocoon's accuracy and robustness across diverse scenarios.
5.1 EVALUATION SETUP
Dataset. Our evaluations utilize the nuScenes dataset (Caesar et al., 2019), a comprehensive au- tonomous driving dataset collected from vehicles equipped with a 32-beam LiDAR and 6 RGB cameras. For conformal prediction, we partition the original training dataset into a proper training set and calibration set with 6:1 ratio. Details on dataset are discussed in Appendix A.
Base Models. Cocoon framework is based on 3D object detectors, FUTR3D (Chen et al., 2023) and TransFusion (Bai et al., 2022), known for their unique architectures and influence on recent advancements in the field (Xie et al., 2023; Yan et al., 2023). This section focuses on FUTR3D, with results for TransFusion detailed Appendix F. FUTR3D encodes 2D and 3D data using ResNet101 He et al. (2016) and VoxelNet Zhou & Tuzel (2018), respectively, fuses the encoded features via concatenation, and then outputs prediction results after passing through six transformer decoder layers. To ensure compatibility with Cocoon, replace the concatenation with element- wise sum following the integration of the feature aligner and Fl nodes.
Corruption Scenarios. simulate natural and artificial corruptions that impact online inference results. Natural corruptions include adverse weather and lighting variations, categorized from the nuScenes validation set. For artificial corruptions, 'blackout' simulate camera malfunctions, while other artificial corruptions, identified challenging conditions (Yu al., 2023; Dong et al., 2023), represent realistic disturbances. Detailed corruption configurations are provided in Appendix B.
Baselines. AdapNet* (Valada et al., 2017) uses an MoE. approach with a trainable gating network; Zhu each modality feature determine weights; Zhang al. (2023) employs late fusion with an enhanced Mon-Maximum Suppression for output-level adaptive fusion; Cal-DETR (Munir al., 2023) measures uncertainty using variance in transformer decoder layers. Cal-DETR (T) determines the weights at the tensor level, while Cal-DETR (Q) determines them at the query level, allowing evaluation of object-level adaptive fusion. See Appendix for details.
al. (2020a) applies self-attention
Training and Inference Details. For computational resources, 4 A40 GPUs were used for training, and 1 RTX 2080 for calibration and inference. The training process consists of two stages: the first stage involves training the baseline model from scratch using proper training set, ensuring the pre- rre-trained model has no knowledge of the calibration samples. The second stage involves training the feature aligner and feature impression nodes. We employ 8-layer MI.P for feature
8

---

Published as a conference paper at ICLR 2025
Static Fusion (2D: 3D =
Adaptive Fusion (2D :00 :B)
Ped 0.32:0.68
Car 0.4:0.6
(a) Occluded pedestrian ('Ped.") detected by adaptive fusion.
Static Fusion :2D:33 :30 1
Adaptive Fusion (2D :3D
CV 0 0.38:62
Car 0:1
TC 0.59:41
(b) Improved detection on the occluded car (left) and the construction vehicle ('CV:; right). "TC": traffic cone. Figure 6: Qualitative Comparison between Static Fusion and Cocoon on Challenging Objects.
Pedestrien (5)
Bicydl M)
(M,
Truck (L)
Pedestriam am (B0)
(BO)
20
optinal region
Figure 7: Alignment of Cocoon's 's resulting weights (a:3, red lines) with optimal regions (grey area). The optimal region is determined the area within a margin of 0.5 from the peak AP value. "S": small, "M": medium, "L": large, "BO": blackout.
aligner align modality-specife features to 228-dimensional representation space. Each FI node indicates per-class feature space within this common representation space. For the first training stage, the baseline models were trained using the same hyperparameters (e.g., learning rate) those used original models. In the second stage, a single batch size was used without changing other hyperparameters. Inference was conducted with a single batch size. In Eq. 3, we set num queries
< num queries and 7x queries
See Appendix D for these coefficient values and further details.
5.2 EVALUATION ON FUTR3D
Quantitative Comparison. Table presents Cocoon's 's accuracy and robustness across various scenarios. Cocoon consistently outperforms other baselines in both normal and unpredictable environments. For further analysis, we compare the static 1:1 fusion (the original fusion method in the base model) and Cocoon from different perspectives. Table 4, we break down accuracy based object locations (distance from the ego agent). Cocoon demonstrates better accuracy in all cases. Notably, the improvements are more sigmificant for mid-distance and far-away objects, which tend to have smaller pixels and sparser points, emphasizing Cocoon's 's robustness for more challenging objects.
We also evaluate Cocoon's 's robustness to varying noise levels to assess its generalizability. Fig. 5 presents results under different severities of random noise interference applied to 2D data. "None" represents normal conditions, "Low" introduces uniform noise in the range of -50 to 50, and "High" adds noise in the range of -150 to 150. For fair comparison, we only evaluate commonly detected objects across both static fusion and Cocoon fusion at all noise levels. Even under extreme noise, Cocoon exhibits less perturbation in the confidence scores of objects, demonstrating its ability to mitigate the impact of corrupted modalities and maintain detection accuracy in diverse conditions.
Alignment between Cocoon and Optimal Region. Our primary hypothesis is that optimally weight- ing different modalities can enhance detection performance, leading to improved APs. To validate this, assess how well the and 3 determined by Cocoon align with the optimal weights. Fig. 7 illustrates this alignment across various object categories in both normal and blackout scenarios. The alignment consistently observed across diverse object sizes in both conditions, supporting our hypothesis.

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of NC Function Effectiveness Across 10 Datasets (A = 0.1). Run 5 times for mean/standard deviation. 'Abs Diff' refers the absolute difference between the mean and the ideal larget of 90%, indicating NC score validity (lower is better). Our method consistently outperforms others, demonstrating the effectiveness of FI, a geometric rredian-based surrogate ground truth.
BasieCP
Feature CP
Dataset
Abs DifT
Abs Diff 4
Ours
Abs Diff :
(Angelopoulos Botes, 2021)
(Teng 2023)
19ps19
90.51 0.25
0.51
90.55 0.21
0.55
90.24 0.65
0.24
Mcps20
89.80 0.75
0.20
89.76 0.74
0.24
90,02 0.7
0.02
Mcps21
89.92 0.66
0.08
89.94 0.66
0.06
90.05 0.63
0.05
Community
89.82 = 1.15
0.18
89.62 = 1.01
0.38
90.18 = 1.30
0.18
Facebook1
90.12 = 0.40
0.12
90.08 0.40
0.08
89,98 = 0.22
0.02
Facebook2
89,95 0.16
0.05
89.96 0.18
0.04
89.98
0.02
Star
90.30 1.42
0.30
90.35 1.21
0.35
90.02 1.73
0.02
Blog
90.11
0.11
90.06 0.39
0.06
89.94
0,06
Bio
90.23 + 0.3
0.23
90,23 0.37
0.23
90.13 0.21
0.13
Bike
89.66 = 1.04
0.34
89.62 = 0.81
0.38
90.25 1.00
0.25
Qualitative Results. As shown in Fig. 6, Cocoon dynamically adjusts the weights between 2D) and 3D data, thereby improving detection performance under complex conditions (e.g., occluded objects).
The primary limitation is the runtime overhead, as discussed in Appendix K. When using FUTR3D, we observe 0.5-second increase in latency. To mitigate this, computing nonconformity scores only in the last three decoder layers reduces latency while maintaining better robustness than static fusion. In the section, we also present further methods to minimize this latency gap (e.g., reducing the number of queries). Another limitation the presence of meaningless queries. Transformer-based architectures, like FUTR3D, often predefine more queries (900) than necessary (up to 200). We found that when a exceeds 0.7, which leads to suboptimal results in all cases in Fig. 2, the meaningless queries predominate, impacting final output due to layer normalization. To address this issue, clipping strategy is applied, whereby meaningless queries undergo static fusion to ensure stable performance. Fig. 6 shows that queries matched to actual objects remain unaffected by clipping.
Limitations.
5.3 VALIDITY OF THE PROPOSED NONCONFORMITY FUNCTION
In this section, we aim to quantify uncertainty at the feature level, similar to Feature CP (Teng et al., 2023), but with a different approach. The effectiveness of the nonconformity (NC) function is quantitatively validated through a coverage check, a method widely used conformal prediction re- search. Our evaluation examines the alignment between the prediction interval at a given significance level (A) and the actual ground truth coverage. For this regression task, our feature aligner projects all features so that they cluster around single Fl node.
Following previous studies, we also use 10 uni-dimensional real-worke datasets for validation. Several datsets (bio, bike, community, facebook1. 1, and facebook2) are sourced from the UCI Machine Learning Repository (Kelly et al., 2023; Singh, 2016). Additionally, we utilize data from the Blog Feedback dataset (Buza, 2014), the STAR dataset (Achilles al., 2008), and the Medical Expenditure Panel Survey datasets (meps19, 19, meps20, and meps21) (Cohen et al., 2009).
In Table 5, our nonconformity function closely aligns with the specified significance level (0.1), consistently including the ground truth in the prediction interval 90% of the time. This demonstrates the effectiveness of our method, leveraging FI (geometric redian-based surrogate ground truth), with empirical coverage highlighting the model's 's proper calibration and effective uncertainty quantification. Qualitative validation on 3D object detection is provided in Appendix ??
6 CONCLUSION & FUTURE WORK
In this work, we introduced Cocoon, an uncertainty-awar adaptive fusion framework designed to ad- dress the limitations of existing modality fusion methods in 3D object detection. By achieving object- and feature-level uncertainty quantification via conformal prediction, Cocoon enables more effective fusion of heterogeneous representations, enhancing accuracy and robustness across various scenarios.
For future work, we plan to extend Cocoon to accommodate additional types and numbers of modalities, such as fusing camera, LiDAR, and radar for autonomous vehicles and robotics, or combining vision and language for VLMs (Yang al., 2023; Liu et al., 2024; Team et al., 2023). This broader applicability will enable systems to integrate diverse modalities more effectively, enhancing their ability interpret complex, challenging environments in ways that mirror human cognition.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDG
We express our gratitude to our area chairs and anonymous reviewers for their invaluable feedback. This work was supported by NSF under the National Al Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant 2112562, in addition to grants CMMI-2038215 and CNS-2321532.
REFERENCES
C.M. Achilles, Helen Pate Bain, Fred Bellott, Jayne Boyd-Zaharias, Jeremy Finn, John Folger, John Johnston, and Elizabeth Word. Tennessee's 's Student Teacher Achievement Ratio (STAR) project, 2008. URL https:///ol. oeeii.org/1.... htt.://.......///////////.
Agency for Healthcare Research and Quality. Data Use Agreement for MEPS. 2017. http: /ttttps::///ss.../ h181doce shtml #Data.
Anastasios Angelopoulos and Stephen Bates. gentle introduction conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021.
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1090-1099, 2022.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Andrew Burnes. Introducing geforce rtx super graphics cards: Best in class performance, plus ray tracing, 2019. URL https: //wtts:::/ww.... nvidia www./nuus/geeeeeeeeeeeeeeeeeeeee ww//gefore//e/w//eeeeeeeeee///se geforce-rtx- wtw.rrt---eeeeesuuueeeeeee
Krisztian Buza. Feedback prediction for blogs. UCI Machine Learning Repository, 2014. DOI: https:/ddooorr///004444//CC55
Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903. 1027, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020a.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020b.
Ramaswamy Chandrasekaran and Arie Tamir. Open questions concerning weiszfeld's algorithm for the fermat-weber location problem. Mathematical Programming, 44:293-295, 1989.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: unified sensor fusion framework for 3d detection. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 172-181, 2023.
Minkyoung Cho, Yulong Cao, Zixiang Zhou, and Morley Mao. Adopt: Lidar spoofing attack detection based on point-level temporal consistency. arXiv preprint 2.10.:2310.15000 2023.
Joel W Cohen, Steven B Cohen, and Jessica S Banthin. The medical expenditure panel survey: national information resource to support healthcare cost research and inform policy and practice. 2009.
Medical care, 47(7_ SSppppppppppplpsssp__11e

---

Published as a conference paper at ICLR 2025
Noah T Curran, Minkyoung Cho, Ryan Feng, Liangkai Liu, Brian Jay Tang, Pedram MohajerAnsari, Alkim Domeke, Mert Pesé, and Kang Shin. Achieving the safety and security of the end-to-end av pipeline. arXiv preprint arXiv:2409.0388 2024.
Yinpeng Dong, Caixin Kang, Jinlai Zhang. Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, and Jun Zhu. Benchmarking robustness of 3d object detection to common corruptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp.
1022-1032, 2023.
Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep mudti-modal object detection and semantic seg- mentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 22(3):1341-136 2020.
Di Feng, Ali Harakeh, Steven L. Waslander, and Klaus Dietmayer. A review and comparative study on probabilissicc object detection in autonomous driving. IEEE Transactions Intelligent Transportation Systems, 23(8):9961-9980, 2021.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp.
IEEE, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues multi-camera 3d object detection. arXiv preprint arXiv:2203.1700 2022.
Junjie Huang. Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du Bevdet: high-perffrmance multi- nulti-cammraaa 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.
Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02700 2022.
Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semanties for 3d object detection. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pp. 35-52. Springer, 2020.
Alireza Javanmardi, David Stutz, and Eyke Hüllermeier. Conformalized credal set predictors. arXiv preprint arXiv:2402.100733 2024.
Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Msmdfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 21643- 21652, 2023.
Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian. Cross-modal center loss for 3d cross- modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10.311 3142-3151, 2021.
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The uci machine learning repository. URL https://archive. ics. uci. edu, 2023.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19994-20006. 2023.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
12

---

Published as a conference paper at ICLR 2025
Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Osear Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
Hyungtae Lee and Heesung Kwon. Dbf: Dynamic belief fusion for combining multiple object detectors. IEEE transactions on pattern analysis and machine intelligence, 43(5):1499-1555 2019.
Xin Li, Tao Ma, Yuenan Hou, Botian Shi, Yuchen Yang. Youquan Liu, Xingjiao Wu, Qin Chen, Yikang Li, Yu Qiao, et al. Logonet: Towards accurate 3d object detection with seal-to-globbl cross-modal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17524-1753 17534, 2023.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural scene flow prior. Advances in Neural Information Processing Systems, 34:7838-7851, 2021.
Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based
representation with transformer for 3d object detection. Advances in Neural Information Processing Systems, 11118444-184555 2022a.
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng. Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V. Le, Alan Yuille, and Mingxing Tan. Deepfusion: Lidar-camera deep fusion for mudti-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17182-17191, June 2022b.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35: 15:10421-10434, 2022.
Shih-Chieh Lin, Yunqi Zhang, Chang-Hong Hsu, Matt Skach, Md E Haque, Lingjia Tang, and Jason Mars. The architectural implications of autonomous driving: Constraints and acceleration. In Proceedings of the twent-ttirrd international conference on architectural support for programming languages and operating systems, pp. 751-766, 2018.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 21 17-2155, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint 2rXiv:2203.05625, 2022.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi- multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3262-3272, 2023a.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's s-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEBE, 2023b.
Yang Lou, Qun Song, Qian Xu, Rui Tan, and Jianping Wang. Uncertainty-encoded multi-modal fusion for robust object detection in autonomous driving. arXiv preprint arXivv2307.161122 2023.
Yan Lu, Xinzhu Ma, Lei Yang, Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. In Proceedings of the JEFF/CVF International Conference on Computer Vision, pp. 3111-3121, 2021.
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone. Sample-efficient safety assurances using conformal prediction. In International Workshop the Algorithmic Foundations of Robotics, pp. 149-169. Springer, 2022a.
13

---

Published as a conference paper at ICLR 2025
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone. Sample-efficienn safety assurances using conformal prediction. In International Workshop the Algorithmic Foundations of Robotics, pp. 149-169. Springer, 2022b.
Arnv Vaibhav Malawade, Trier Mortlock, and Mohammad Abdullah Al Faruque. Hydrafusion: Context-aware selective sensor fusion for robust and efficient autonomous vehicle perception. In 2022 ACM/FEE 13th International Conference on Cyber-Physical Systems (ICCPS), pp. 68-79.
IEEE, 2022a.
Amnv Vaibhav Malawade, Trier Mortlock, and Mohammad Abdullah Al Faruque. Ecofusion: Energy- aware adaptive sensor fusion for efficient autonomous vehicle perception. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 481-486, 2022b.
Oier Mees, Andreas Eitel, and Wolfram Burgard. Choosing smartly: Adaptive multimodal fusion for object detection in changing environments. Intelligent Robots and Systems (IROS), pp. 151-156. IEEE, 2016.
2016 IEEE/RSJ International Conference on
Muhammad Akhtar Munir, Salman Khan, Muhammad Haris Khan, Mohsen Ali, and Fahad Khan. Cal-detr: Calibrated detection transformer. Neural Information Processing Systems (NeurIPS), 2023.
NVIDIA. In-vehicle computing for autonomous vehicles. URL UR..htps::/ww..riidaa...oo en-us/Sa wtw.://wiiiiin-aaaaaaeeeeeeeeeee
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7077-7087, June 2021.
Lisa Putzar, Kirsten Hötting, and Brigitte Röder. Early visual deprivation affects the development of face recognition and of audio-visual speech perception. Restorative neurology and neuroscience, 28(2):251-257) 2010.
Zequn Qin, Jingyu Chen, Chao Chen, Xiaozhi Chen, and Xi Li. Unifusion: Unified multi-view fusion transformer for spatial-temporal representation in bird's s-eeee---vvew.... In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8690 8699, 2023.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 770-779, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Poxet feature set abstraction for 3d object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 00529-10538, 2020.
Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. In 2019 International Conference Robotics and Automation (ICRA), pp. 7276-7282. IEEE, 2019.
Kamaljot Singh. Facebook Comment Volume. UCI Machine Learning Repository, 2016. DOI: http://ddoo.or/11224432...
Sanbao Su, Yiming Li, Sihong He, Songyang Han, Chen Feng, Caiwen Ding, and Fei Miao. Un- certainty quantification collaborative detection for self-driving. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5588-5594, IEEE, 2023.
Sanbao Su, Songyang Han, Yiming Li, Zhili Zhang, Chen Feng, Caiwen Ding, and Fei Miao. Collaborative multi-object tracking with conformal uncertainty propagation. IEEE Robotics and Automation Letters, 2024a.
Sanbao Su, Songyang Han, Yiming Li, Zhili Zhang, Chen Feng, Caiwen Ding, and Fei Miao. Collaborative oojecc tracking with conformal uncertainty propagation. IEEE Robotics and Automation Letters, 2024b.
14

---

Published as a conference paper at ICLR 2025
Jiachen Sun, Yulong Cao, Qi Alfred Chen, and Z. Morley Mao. Towards robust LiDAR- based per- ception in autonomous driving: General blxck-box adversarial sensor attack and countermeasures. In 29th USENIX Security Symposium (USENIX Security 20), pp. 877-894. USENIX Association, August 2020. ISBN 978- 978-1-933133-17555 URL https::///// /ttkps::/www... usen x org/ org/conferencce
Jiachen Sun, Qingzhao Zhang, Bhavya Kailkhura, Zhiding Yu, Chaowei Xiao, and Z Morley Mao. Benchmarking robustness of 3d point cloud recognition against common corruptions. arXiv preprint www.x:222011220000 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Jiaye Teng, Chuan Wen, Dinghuai Zhang, Yoshua Bengio, Yang Gao, and Yang Yuan. Predictive inference with feature conformal prediction. In The Eleventh International Conference on Learning Representations, 2023.
Abhinav Valada, Johan Vertens, Ankit Dhall, and Wolfram Burgard. Adapnet: Adaptive semantic segmentation in adverse environmental conditions. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 4644 4651. IEEE, 2017.
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4614-4612, 2020.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning random world, volume 29, Springer, 2005.
Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmen- tation for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11794 1803, 2021.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. arXiv preprint arXiv:2303. 1926, 2023.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from milti-view images via 3d-to-2d queries. In Conference on Robot Learning, pp. 180-191. PMLR, 2022.
Chao Xiang, Chen Feng, Xiaopo Xie, Botian Shi, Hao Lu, Yisheng Lv, Mingchuan Yang, and Zhendong Niu. Multi-sensor fusion and cooperative perception for autonomous driving: A review. IEEE Intelligent Transportation Systems Magazine, 2023.
Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 17591-17602, 2023.
Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang Cross modal transformer via coordinates encoding for 3d object dectection. arXiv preprint arXiv:2301.01283 2023.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018
Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8947-8958, June 2023.
Hongcheng Yang, Dingkang Liang, Dingyuan Zhang, Zhe Liu, Zhikang Zou, Xingyu Jiang, and Yingying Zhu. Avs-net: Point sampling with adaptive voxel size for scene understanding. arXiv preprint 1xXiv:2402.155/1 2024.
15

---

Published as a conference paper at ICLR 2025
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint 1rXf1:2309.17421, 9(1):1, 2023.
De Jong Yeong, Gustavo Velasco-Hernandezz John Barry, and Joseph Walsh. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors, 21(6):2140, 2021.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11784-11799 2021.
Kaicheng Yu, Tang Tao, Hongwei Xie, Zhiwei Lin, Tingting Liang, Bing Wang. Peng Chen, Dayang Hao, Yongtao Wang, and Xiaodan Liang. Benchmarking the robustness of lidar-camera fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3188-3198, 2023.
Xinyu Zhang, Zhiwei Li, Zhenhong Zou, Xin Gao, Yijin Xiong, Dafeng Jin, Jun Li, and Huaping Liu. Informative data selection with uncertainty for multimodal object detection. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-ent learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499, 2018.
Zixiang Zhou, Donggiangzi Ye, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarformer: A unified transformer-based multi-task network for lidar perception. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14740-14747. IEEE, 2024.
Hu Zhu, Wang, Yu Shi, Yingying Hua, Guoxia Xu, and Lizhen Deng. Multimodal fusion method based on self-attention mechanism. Wireless Communications and Mobile Computing, 2020(1): 8843186, 2020a.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 100.4.001..04159, 2020b.
16

---

Published as a conference paper at ICLR 2025
A DATASET DETAILS
Table 6: Scene Distribution
Our evaluations utilize the nuScenes dataset, comprehen- sive autonomous driving dataset collected from vehicles equipped with a 32-beam LiDAR and 6 RGB cameras (Caesar al., 2019). This dataset includes 700 training scenes and 150 validation scenes, each lasting 20 seconds and annotated a frequency of Hz.
Dataset
Day
Night
Rate (D/N)
Original
616
7.3
Training
528
72
7.3
Calibration
88
12
7.3
In preparation for conformal prediction, we have partitioned the training data into proper training set and calibration set, using 6:1 ratio, as shown in Table 6. The proper training set is used train the parameters of the base detection model, feature aligner, and FI, while the calibration set is applied during the calibration phase.
One crucial assumption of conformal prediction exchangeability, which states that the distribution of sequence of instances remains invariant under any permutation of these instances. Since different scenes are distinct enough to be considered exchangeable (Luo et al., 2022b), we generate a dataset of exchangeable calibration samples by sampling single instance (i.e., frame) at random from each scene belonging to calibration set.
B CONFIGURATIONS FOR ARTIFICIAL CORRUPTIONS
In this paper, we investigate various artificial corruption scenarios, referencing recent benchmarking studies (Yu et al., 2023; Dong al., 2023). We explore two types of 2D data corruptions (applied to six images), one type of 3D data corruption, and sensor misalignment scenarios.
For 2D data corruption, camera blackout conditions simulate potential camera malfunctions by converting images into entirely black frames. Additionally, we introduce random noise interference the six images by adding random noise in the range of -50 to 50 the original RGB values, where each pixel's value ranges from 0 to 255.
For 3D data corruption, randomly delete 30% of the points in frame of LiDAR data. LiDAR density decrease is a common corruption in LiDAR data, shown in various datasets (Dong et al., 2023; Sun al., 2022: Kong al., 2023). A 30% density decrease, shown in Dong al. (2023), significantly impacts accuracy (dropping by 0.39 to 0.5) in 2D-3D fusion models. This corruption arises from adverse weather, surface reflectivity, and sensor resolution issues Dong et al. (2023); Kong et al. (2023), making it representative of long-tail inputs in public datasets (Geiger et al., 2012; Caesar al., 2019). Furthermore, due to the large memory usage of LiDAR point clouds, many works use random sampling at higher rates (Li et al., 2021; Yang et al., 2024; Cho et al., 2023), making density decrease a more frequent use case.
Sensor misalignment occurs due to calibration errors, malfunctioning components, aging sensors, causing spatial or temporal misalignment between sensor data (Dong et al., 2023; Yu et al., 2023). Synchronization issues can arise from external/internal factors, such as spatial misalignment caused by severe vibrations when driving on bumpy roads or temporal misalignment due to clock synchro- nization module malfunctions. Yu al., 2023 provides in-depth investigation into both spatial and temporal alignments. Referencing these works, we add random Gaussian noise to the calibration matrices. In the case of the nuScenes dataset, the calibration matrix, lidar2cam, is normalized, showing a range of (-1463.53, 1516.51) with a mean of -127 112.188 and a standard deviation of 5 516.67. Like the point density drop, introduce 33% noise. (-500, 500), which leads to a meaningful accuracy drop.
C DERIVATION OF TRAINING OBJECTIVE
The challenge of identifying a point that minimizes the sum of unsquared distances to all other points characterizes the geometric median problem. The Weiszfeld's algorithm stands out a solution method for this problem (Chandrasckaran & Tamir, 1989). For a point y to qualify as the geometric median, it must meet the following criterion:
notable
17

---

Published as a conference paper at ICLR 2025
yl
This formula confirms that y, distinct from all other points, accurately represents the geometric median, underpinning the efficacy of our nonconformity function.
Based this algorithm, we can apply this to our problem setting where have set of 2/V features:
hoi := h o (z I,,..,N, hy h fi(au), i
where fo and fi are encoders in the model, and wes and zy represent input data from multiple sensors. With these terms defined, the goal is to ensure that w" is the geometric median of this set of features:
w arg min Σ (1l heall + 1-l--/Wae -1sll)
Assuming w* is distinet from each feature w² hes, w & his then the sufficient and necessary optimality condition for problem equation 4 is
hoi w² |lact 01
his w² |1/4
0
Σ i=1
(5)
wIll
w+1l
Therefore, we can design loss function to penalize the violation of equation 5
N
Bei w' bis
w²
Lgeomed
(6)
h w*|l
w*1|
If Lycomed is zero, confirms that satisfies the geometric median condition as defined in equation 4. As training progresses, the positions of w" tend to converge, which could potentially cause ambiguities in distinguishing between them. address this issue, we introduce a penalty, Leeparatee for increasing distances between the positions of w", aiming to maintain clear separations. Consequently, our total loss function incorporates this additional penalty:
C illw" heal + I|w" hu +C
Σ
hei
w"
hu w²
|105
wtll
(7)
will
| will
Grenter
Cyrament
Exeperste
where C' denotes the number of FI nodes (e.g., the number of classes). The resulting loss function will be used to jointly train feature aligner and feature impression w".
D TRAINING DETAILS
The training process consists of two stages: the first stage involves training the baseline model from scratch using proper training set, ensuring the pre-trained model has knowledge of the calibration instances. The second stage involves training the feature aligner and feature impression nodes.
For the first training stage, the baseline models were trained using the same training objective and hyperparameters (e.g., number of epochs, learning rate, etc.) as those used in existing models. In the second training stage, single batch size was used with the same hyperparameteres the first stage. Here, we use the proposed training loss function (Eq. 7). We set
5
num_queeres' and r) = 7xmmm 7xmum_queries" The coefficient values are determined empirically for stable loss convergence and improved learning of positions. For and Ç, we observed that Leenter and Egeomed values scale similarly, but Leenter converges more slowly. Therefore, assigned a higher priority to d than with a ratio of 5:3.
num_querres'
Regarding n/ if the contribution of Eeeparate is small, the distance between w' of different classes be- comes smaller, causing ambiguities distinguishing between them. Conversely, a large contribution of Cseperate disrupts loss convergence and training stability. Through iterative adjustments, found that ixnriequeries yields optimal results. Fig. 8 shows the loss variation with respect the n value.
18

---

Published as a conference paper at ICLR 2025
the contribution suparate small, the positions of w between classes become closer
x0.2
Lsparate
X0
x0.01
Loss does converge well
x0.3
x0.5
Step
I0k.
Training step
Figure 8: Coefficient Values in Training Objective. This plot shows the effect of the gamma value on Lepparate with seven separate training progressions. Different curves represent different training settings of gamma values. The text located next to each curve indicates the gamma value for each training nuo queries omitted). This figure shows how we can choose the gamma value, which ensures both better loss convergence and stable learning of w*.
E ABLATION STUDY
This section presents an ablation study to assess the contribution of each component. As the feature aligner (FA) and feature impression (FI) are designed for uncertainty quantification based conformal prediction, we conducted the ablation study hierarchically as follows.
Uncertainty quantification (UQ) Object-level dynamic weighting. Table 7 demonstrates that object-level dynamic weighting is ineffective without UQ, and vice versa, highlighting their mutual dependence. When both are disabled, static fusion performed through concatenation. Without conformal prediction, simple MLP model generates a weight value for each object query. When object-level dynamic weighting is disabled, the nonconformity score UQ is computed for the entire input feature.
Table 7: Performance evaluation of different configurations under various conditions. indicates the presence of component.
UQ
Dynamic Weighting
No Corruption
Rainy Day
Clear Night
Rainy Night
Cam. Blackout
Avg. Lateney (sec)
66.
68.34
44.51
27.26
45.13
0.91 + 0.02
63.01
63.51
42.01
21.91
37.01
0.91 + 0.03
66.2
68.37
44.52
27.3
45.21
1.26 - 0.04
66.8
68.89
45.68
27.98
51.87
1.27 1: 0.04
Uncertainty Quantification Analysis. We also conduct ablation study within the scope of uncertainty quantification, examining: 1) The impact of feature aligner (FA) layer count (Table 8), 2) The impact of feature impression (FI) vector dimension (Table 9), 3) The impact of the portion of decoder layers involved in UQ (Table 10). By integrating these findings, we determine the best configuration combination (Table 11). Applying this configuration across all decoder layers reduces the overhead to 0.24 seconds, while improving accuracy by 0.58 % over static fusion.
Table 8: Performance evaluation with varying FA layer counts.
layer dimension
No Corruption
Rainy Day
Night
Rainy Night
Cam Blackout
Avg. Latency (sec)
128
66.01
68.18
44.39
26,99
43,89
1.16 a 0.05
128
66.67
68.56
43.53
27.41
50.99
1.19 = 0.04
128
66.82
68.90
45.55
27.56
51.45
1.24 = 0.05
128
66,80
68.89
45.68
27.98
51.87
1.27 = 0.04
19

---

Published as a conference paper at ICLR 2025
Table 9: Performance evaluation with varying FI dimensions.
layer count
dimension
No Corruption
Rainy Day
Clear Night
Rainy Night
Cam Blackout
Arg. Latency (see)
66.73
68.91
45.62
27.91
51.81
1.16 0.03
66.72
68.96
45.64
27.85
51.79
1.21 0.04
128
66.80
68.89
45.68
27.98
51.87
1.27 0.04
Table 10: Performance evaluation with varying UQ decoder counts.
UQ Decoder Count
FA Layer Count
F1 Dimension
No Corruption
Rainy Day
Clear Night
Rainy Night
Cama Blackout
Aug. Latency (sec)
126
66
68.17
45.11
27,32
$1.3
1,1 113+0003
128
66
45.68
27.98
51.87
1,27 1
F EVALUATION ON TRANSFUSION
To demonstrate Cocoon's applicability other transformer-busee detectors, we evaluate using TransFusion (Bai et al., 2022). TransFusion consists of 2D) (ResNet50 He et al. (2016) and FPN Lin al. (2017)) and 3D (VoxelNet Zhou Tuzel (2018)) encoders, along with a single transformer decoder layer that uses an element-wise sum to combine the 2D and 3D features obtained from the attention mechanisms. See Appendix for further reconfiguration details.
As shown in Table 12, Cocoon achieves better comparable accuracy under various corruption scenarios compared to baseline methods. These results use small set of the nuScenes validation set (120 samples). Since TransFusion has only single transformer decoder layer, cannot apply Cal-DETR (T) and Cal-DETR (Q) (Munir et al., 2023), which require multiple decoder layers for uncertainty quantification.
G BASE MODEL RECONFIGURATION DETAILS
FUTR3D. To use FUTR3D as base model, concatenation (for feature fusion) is replaced with an element-win wiiee sum. This modification maintains accuracy, with both configurations achieving an mAP of 67.39% after training with the proper training set.
Trsnonusion. TransFusion consists of 2D and 3D encoders, along with a single transformer decoder layer that Therefore, unlike FUTR3D, TransFusion satisfies our requirement for an element-wise sum.
element-wise
combine the features obtained from attention mechanism.
Currently, due the need for calibration data separate from the training data, train the model using only the proper training set. However, obtaining an independent calibration set (100 samples) would allow us skip the first training stage, requiring only the feature aligner and FIs be trained with the original nuScenes training set.
H OFFLINE PREPARATION FOR CONFORMAL PREDICTION
During the offline stage of conformal prediction preparation, we first prepare proper training set and calibration set by splitting the original training set, unless separately collected dataset is available for calibration. The pivotal process is the calibration step, which computes nonconformity scores (see Eq. 1) for each instance in the calibration set, establishing benchmarks against which new test instances are evaluated.
Model Reconfiguration. Depending on the initial fusion approach used, specific adjustments are made to the model. For instance, models initially using feature concatenation are reconfigured to adopt an element wise sum, supplemented by simple multi-layer perceptron (MIP) with one two fully connected layers, allowing for a linear combination at a 1:1 fusion ratio. This reconfiguration is confirmed to maintain detection accuracy, as exemplified by the FUTR3D model, which retains its original accuracy of 67.3% post-adaptation. Additional details model reconfiguration discussed in Appendix G.
Training. This phase involves retraining the model with the designated training set, utilizing the specialized loss function detailed in Sec. 4.4. The focus is on aligning and imprinting features to ensure the model's 's readiness for conformal prediction.
20

---

Published as a conference paper at ICLR 2025
Table 11: Best configuration combination for uncertainty quantification.
decoder count
FA
Fl dinsenston
No Corruption
Rainy
Clear Night
Rainy Night
Cam Blackout
Aug. Latency
45.17
50.45
Table 12: Accuracy (mAP) gain of Cocoon across various corruptions and object categories. "Ours": the integration of TransFusion and Cocoon. Categories that do not belong to any scene are omitted. raserrens: (Bai et al., 2022), AdapNet": (Valada et al., 2017)
Corruption
Baseline
Car
Truck
Bus
Pedestrian
Blcycle
T.C.
Total
Type
BaseTrana
88.2
43.7
57.5
65.5
74.0
No Comuuption
AdapNett
86.7 (-1.5)
52.4 (+8.7)
94.3 (+0.9)
94.9 (-0.5)
54.7 (-2.8)
60.4
(-..1-
Ours
(+0)
45.1 (+1.4)
93.6 (+0.2)
95.6 (+0.2)
59.3 (+L8)
66.6 (+1.1)
74.7 (+0.7)
Base (rons
86.7
34.9
92.4
94.1
42.5
61.2
68.6
Misalignment
AdapNet
85.1 (-1.6)
16.9 (-18.0)
93.1 (+0.7)
93.3 (-0.8)
(-1.9)
54.8 (-6.4)
64.0 (-4.6)
Ours
86.4 (-0.3)
33,4 (-1.5)
92.1 (-0.3)
94.3 (40.2)
42,5 (+0)
62.1 (40.9)
68.5 (40.1)
Baservana
84.7
27.0
94.7
93.8
43.3
59.6
61.2
Blackout (C)
Net
83.8 (-0.9)
24.5 (-2.5)
92.5 (-2.2)
91.3 (-2.5)
41.1 (-2.2)
54.9 (-4.7)
64.7 (-2.5)
Ours
84.6 (-0.1)
30.4 (+3.4)
94.7 (+0)
93.9 (+0.1)
40.2 (-3.1)
59.9 (+0.3)
67.3 (+0.1)
Baseтsama
86.6
41.4
91.4
94.7
54.0
66.6
72.5
Sampling (L.
AdapNett
84.6 (-2.0)
50.7 (49.3)
91.9 (40.5)
93.8 (-0.9)
54.1 (40.1)
61.6 (-5.0)
72.8 (40.3)
Ours
86.6 (+0)
44.3 (+2.9)
91.4 (40)
94.7 (40)
55.9 (41.9)
67.9 (+1.3)
73.5 (+1.0)
Calibration. This crucial step involves calculating nonconformity scores for all instances in the calibration set. These scores are derived from the nonconformity function: NC; (h,
Through these methodical subprocesses-modee framework thoroughly prepares the model for effective conformal prediction, ensuring it can generate accurate and reliable predictions consistent with the principles of conformal prediction.
reconfiguration, training, and calibration-
I BASELINE ADAPTIVE METHODS.
Our study investigates various adaptive fusion methods, which serve baseline approaches in our comparative analysis (see Table. 3). Since none of the existing methods were originally designed for 3D object detection, we adapt their fusion mechanisms to work with FUTR3D and train them using FUTR3D's ' original hyperparameters. This ensures that all baseline methods, along with Cocoon, share a common base model, allowing us to focus on comparing the effectiveness of the adaptive fusion mechanisms.
AdapNet (Valada al., 2017) employs mixture-of-experts (MoE) approach with a trainable gating network to enable adaptive fusion, as illustrated in Fig. 1a. The gating network, placed after the 2D and 3D encoders, determines the feature weights for each modality. This mechanism learns scalar weights for both encoded features and applies the weights to the entire feature space of each modality, producing a fused feature. Zhu et al. (2020a) uses self-attention to determine the weights for each encoded feature. In this approach, the encoded features pass through self-attention module, and weighted sum is used to produce the fused feature. Zhang et al. (2023) employs a late fusion approach that introduces adaptiveness between detection outputs generated from separate detection pipelines. For the detection outputs of both modalities, they perform an informative data selection process using Mon-Maximum Suppression (NMS) The decision-level fusion is achieved by combining the box proposals in an enhanced NMS process that incorporates both Intersection over Union (IoU) and confidence scores. our implementation, we use two versions the FUTR3D model-one one for data only and the other for 3D data only-and and apply the NMS process to the box proposals from each perception modality. Cal-DETR (Munir et al., 2023) measures uncertainty in transformer- architectures by calculating variance across the transformer decoder layers. Since this algorithm was originally designed for unimodal perception, we modified FUTR3D's 's architecture by adding two transformer blocks (two sets of decoder layers): for 2D data and another for 3D data. These blocks independently assess the uncertainty for each modality. After applying the weighted sum, the adaptively fused input passes through the original transformer decoder to produce the output.
21

---

Published as a conference paper at ICLR 2025
Cal-DETR (T) determines the weights at the tensor level, while Cal-DETR (Q) determines them the query level, enabling us to evaluate the importance of object-level adaptive fusion.
J INSIGHTS FROM OUR NONCONFORMITY MEASURE ANALYSIS
In Sec. 4.3, we present the findings, also shown below, from applying our nonconformity measures to real-world dataset (Caesar et al., 2019).
Feature uncertainty significantly affects the NC score of the top-1 1 FI node. Higher feature uncertainty in transformers with multiple decoder layers leads to significant instability in the top-1 Fl node across layers.
Finding 1:
Finding 2:
These observations were made using the FUTR3D model, which comprises six transformer decoder layers. Finding is also noted with the TransFusion model. However, due to the TransFusion model having only one decoder layer, Finding Figs. 9 and 10, and Tables 14, 15, and 16.
not observed. We will elaborate on the findings using
When measuring the nonconformity scores for multi-modal features, if the object is clearly visible (Fig. 9), the Fl nodes corresponding to the two modalities are mostly the same. The nonconformity scores for the top-1 1 FI node and the second most promising Fl node show a significant difference (Table 14). For Object 1, the correct label corresponds to the car class (FI node 0), and both sensors predict this accurately, facilitating equal fusion between the camera and LiDAR.
However, when the object is unclear (e.g., overlapping occlusion cases) (Fig. 10), the noncon- formity score for the top-1 Fl node is very high, with no significant difference from the second most promising Fl node (Table 15). For Object the correct label is the construction vehicle class (FI node 2), but the Fl node selection for both modalities does not match this class, Nevertheless, the LiDAR modality predicted a more stable Fl node with lower top-1 nonconformity score, indicating more certain information.
For Object 3 (a person next traffic cone and construction vehicle), recognition with LiDAR is challenging. Here, the camera, excelling at color and small object detection, leads to better perception performance, aligning with our nonconformity measure in terms of top- FI node consistency and nonconformity scores (Table 16). Object 3's correct label is the pedestrian class (FI node 8), and the camera predicts this more stably with a lower nonconformity score compared to the LiDAR. Therefore, the camera is deemed more certain in this case.
Fl Node
2
3
4
6
8
9
Class
Car
Truck
C.V.
Bus
Trailer
Barrier
M.C.
Bicycle
Ped
T.C.
Table 13: FI Node Clarification. this experiment, Fl nodes are designed to indicate classes. 'C.V.:: Construction Vehicle, 'M.C.:: Motorcycle, "T.C.': Traffic Cone.
2
3
1
Figure 9: Clearly Visible Car.
Figure 10: Challenging Case,
22

---

Published as a conference paper at ICLR 2025
Top-3 NC Sorted Ascending
NC Sorted Ascending
Top-3
Ascending Order
Object
Camera
Scores
LIMRR NC Scores
Objject
Camera
LiMAR NC Scores
Object
Comer NC Screes
LiDAR Scores
Layer
3
Layer
Layer
2
0
0.08
9.08
9.75
0.00
9.34
9.88
0
1.31
8.26
9.11
3,83
6.06
6.73
0.07
9.28
10.25
0.24
9.50
10.
0.07
10.14
11.06
0.00
10:25
11:17
1
3.91
10.34
11.68
2.40
6:35
7.40
0.12
12.57
12.90
5.10
11.61
13.22
0.06
10,65
12.22
0.06
10.74
12.31
2
7.09
21.22
21.41
4.79
5.88
6.00
0.10
11.31
12.11
0.22
11.57
12.
0.06
11.00
11.24
0.07
11.11
11.35
3
0.21
6.00
5.69
0.34
6.79
5.60
0.09
10.75
12:72
0.16
10.98
12.92
0.06
12.66
12.81
12:55
12.90
2.15
18:31
18.08
0.20
12.94
13.00
0.07
12.56
12.99
8:21
10:23
12.
0.07
13:56
13.00
13
11.00
6.26
10.44
12.26
4.61
7.00
10.16
0.31
13.00
13.50
1bi Carreponding Nodes For Tap-3 Scores
tb Carsppnding Nodes Por Tap-3 NC Scores
Corregpnding Nodes For Teg-3
Object
Camem Nodes
LIBAR Nedes
Objjct
Camrm Nodes
LIBAR Nodex
Object
Camer
Nodes LaDAR Noxes
Layer
Layer
Layer
Table 14: NC Score Analysis for Object 1. Correct class blue
Table 15: NC Score Analysis for Object Correct class in blue.
Table 16: NC Score Analysis for Object Correct class in blue.
K RUNTIME OVERHEAD
Static: Base model
Adaptive: Base model + Cocoon
FUTR3D
0.91 0.02
1.27 + 0.04
TransFusion
0.7 + 0.1
0.7 + 0.1
Table 17: Comparison of inference latency (in seconds) between static and adaptive fusion. Run 120 times for mean and standard deviation.
Table 17 shows the runtime overhead when Cocoon is integrated. In TransFusion, which uses single transformer decoder layer processing 200 queries, there is no significant increase in latency compared to static fusion. However, in FUTR3D, which has six transformer decoder layers processing 900 queries, the average latency increases by 0.5 seconds, mainly due to computing nonconformity scores in all six layers. To minimize this latency increase, we performed nonconformity computation only on the last three layers. In this case, the latency can be reduced to 1.3 + 0.1 seconds, achieving 66.28% accuracy in normal (no corruption) condition and 51.3% accuracy in 2D data corruption (blackout) condition. To further reduce the latency gap between static fusion and Cocoon's 's adaptive fusion, we can use 200 queries instead of 900, as in TransFusion, or apply model compression (e.g.. quantization and pruning) to the feature aligners.
L FINE-TUNING-BER BASED COMPUTATIONAL OVERHEAD MITIGATION
From the ablation study E, we found that adjusting the MLP layer count and the vector dimension of Feature Impression (FI) can reduce the overhead of 120 ms while maintaining higher accuracy than static fusion.
To further reduce the overhead, devised fine-tuning approach: dynamic fusion is applied only to the last decoder layer, and we fine-tune only the components in the later stages of the fusion process (the earlier parts must remain unchanged to preserve the validity of nonconformity score).
Using this fine-tuning approach, we reduced Cocoon's 's additional overhead (excluding the base model) 50 ms-a significant reduction of 86% from the previous overhead of 360 ms-while achieving the final accuracy shown in Table 18 on an NVIDIA GeForce RTX 2080 (11 TOPS; Burnes (2019)). Given the computational capability of current AV systems -the most common field where Cocoon can be applied (254 TOPS; (VIDIA)-this overhead is negligible, with total latency within the 100 ms threshold for real-time guarantees (Lin et al., 2018).
Table 18: Performance comparison under different conditions.
Fusion
Corruption
Rainy Day
Night
Rainy Night
Point Sampling (L)
Noise (C)
Avg. Latency (sec)
Statie
66.16
68.31
11.51
27.26
65.17
60.39
0.91 1: 0.02
Cocoon (origima)
66.90 6680
68.89
45.68
27.98
65.89
61.83
1:27 1: 0.00
Cocoen (fine -uning)
67.13
69.12
45.70
28.07
66.02
61.99
0.96 0.03
23

---

Published as a conference paper at ICLR 2025
M HANDLING LARGE QUALITY DIFFERENCES ACROSS MODALITIES
camera (black-out)
camera (random nose interfsrence)
Common feature space
surrogate ground truth (FI)
aligned camera feature (calibration set) aligned lidar feature (calibration sat)
10
current test cam input (corrupted)
Number corrupted images (out of 6)
current test lidar input
Figure 12: Nonconformity Scores Under Varying Cam- era Corruption Levels
Figure 11: Feature Alignment Outcomes
Figs. 11 and 12 show the feature alignment outcomes and nonconformity scores under conditions of large quality differences across modalities. In this scenario, low-quality data points positioned farther from the learned surrogate ground truth (i.c., Feature Impression), resulting in lower assigned weights.
N LICENSE OF ASSETS
Code. FUTR3D and TransFusion' are both under the Apache-2.0 license.
Datasets. we utilize the nuScenes dataset (Caesar et al., 2019), which is licensed under BY-NC-S 4.0. Appendix 5.3 shows ten different datasets. Several of these datasets, including Bio, Bike, Com- munity, Facebook 1, and Facebook2, are sourced from the UCI Machine Learning Repository (Kelly al., 2023; Singh, 2016). 6), which is licensed under CC BY 4.0. The Blog dataset (Buza, 201 4) is also licensed under CC BY 4.0, while the Star dataset (Achilles et al., 2008) is under the CCO 1.0 license. For the Medical Expenditure Panel Survey datasets (Meps19, Meps20, and Meps21) (Cohen et al., 2009), we have the necessary permissions for their use, as we utilize these datasets exclusively for statistical reporting and analysis (Agency for Healthcare Research and Quality, 2017).
24


Reviews:
[{"review_id": "y2bRyGvhqI", "reviewer": "Reviewer_XHYu", "summary": "This paper introduces an object- and feature-level uncertainty-aware fusion framework for robust multi-modal sensor fusion in 3D detection scene of autonomous driving. It designs a valid uncertainty estimator for heterogeneous representations, and then dynamically fuses multi-modal sensor features. Experiments indicate the effectiveness of fusing heterogeneous features in several unrobust scene.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) It provides a very intuitive feature fusion analysis to derive the motivation of this paper\n(2) All the designs that underpin the motivation of this article are very valid and reasonable.\n(3) The results are rich and convincing.", "weaknesses": "Although the writing logic and thinking are clear, the explanation of the method is too complicated, which will bring inconvenience to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfnmidSEqV", "reviewer": "Reviewer_9JUv", "summary": "The paper introduces Cocoon, a novel framework that addresses the challenge of multi-modal fusion in 3D object detection. The paper proposes a novel uncertainty-aware fusion framework that operates at both object and feature levels through the introduction of Feature Impression and a feature alignment mechanism. This approach leverages conformal prediction theory to quantify uncertainties in heterogeneous sensor data, enabling dynamic weight adjustment between different modalities such as cameras and LiDAR.", "questions": "1.While you propose using fewer decoder layers or queries to reduce latency: How does this affect the accuracy-speed tradeoff specifically?\n2.Could you provide ablation studies on the importance of different components in your framework? Such as, how much does each component (feature aligner, Feature Impression, uncertainty quantification) contribute to the final performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.Proposes a novel framework (Cocoon) that uniquely combines object-level and feature-level uncertainty awareness in multi-modal fusion, addressing a significant gap in existing approaches.\n2.Thoroughly evaluates the model under diverse corruption scenarios, including both natural corruptions.\n3.Provides solid mathematical foundation for the uncertainty quantification method, with clear derivations and validations across multiple datasets.", "weaknesses": "1.The paper shows a substantial increase in computational latency - from 1.1s to 1.6s when integrating Cocoon with FUTR3D.\n2.While the paper demonstrates consistent improvements over baselines, the actual gains are relatively modest in many scenarios.\n3.Only evaluates camera and LiDAR fusion, missing other important sensors like radar and ultrasonic.\n4. The lack of validation on other datasets raises concerns about the generalizability of the method. Validation can be performed on the Waymo and Argoverse 2 datasets.\n5. The method provides marginal gains and performs worse than previous methods such as CMT, BEVFusion, and DeepInteraction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Quet9a6SH2", "reviewer": "Reviewer_38S7", "summary": "T​this paper propose a multimodal fusion framework called Cocoon for 3D object detection. The core innovation of Cocoon is to achieve object and feature level fusion through uncertainty quantification. The framework introduces a feature aligner and a learnable alternative truth value (called \"Feature Impression\") to fairly compare uncertainties between multimodalities and dynamically adjust the weight of each mode. Experimental results show that Cocoon outperforms existing static and adaptive fusion methods under standard and challenging conditions, and performs more robustly under natural and artificial perturbations.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clearly presented and easy to understand.\n2. The innovation of uncertainty quantification mechanism is feasible from the principles and experiments in Section 4.\n3. What I care about more is that the Cocoon framework has good scalability and can be further applied to more types of modal fusion, such as combining camera, LiDAR and radar data for autonomous driving, or combining visual and language modalities for visual language models (VLM).", "weaknesses": "1. The entire COCOON process, especially the structural comparison shown in Figure 1, and the \"FEATURE IMPRESSION-BASED FEATURE ALIGNER\" and \"UNCERTAINTY QUANTIFICATION\" parts in Sec 4.2 and 4.3, made me feel the huge computational overhead. Whether it is based on RGB, LiDAR, or multi-modal fusion target detection methods, I think they cannot be implemented quickly if the real-time performance cannot be guaranteed. The advantages can only be reflected at the numerical level of the paper.\n2. Does COCOON rely too much on the quality of feature alignment and calibration in the data? Section 4.2 mentioned that since data from different modalities need to be projected into the same feature space for comparison, the accuracy of this part of the alignment directly affects the reliability of uncertainty quantification. If there are large differences in the quality or sampling frequency of the features of each modality, it will be difficult for the feature aligner to effectively align the data distribution, will this in turn affect the overall fusion effect?\n3. At the end of Section 5.1, the value of the hyperparameter $ \\alpha $ is set, but Table 4 directly assigns $ \\alpha $ to 0.1. And Limitations mentions that the change of alpha will bring about a significant change in the algorithm effect. Is COCOON too dependent on hyperparameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)