Predictive Uncertainty Quantification for Bird's Eye View Segmentation: A Benchmark and Novel Loss Function
Authors: Linlin Yu
Bowen Yang
Tianhao Wang
Kangshuo Li
Feng Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Uncertainty Quantification
Evidential Deep Learning
Bird's Eye View (BEV) Segmentation
Primary area: datasets and benchmarks

Abstract: The fusion of raw sensor data to create a Bird's Eye View (BEV) representation is critical for autonomous vehicle planning and control. Despite the growing interest in using deep learning models for BEV semantic segmentation, anticipating segmentation errors and enhancing the explainability of these models remain underexplored. This paper introduces a comprehensive benchmark for predictive uncertainty quantification in BEV segmentation, evaluating multiple uncertainty quantification methods across three popular datasets with three representative network architectures. Our study focuses on the effectiveness of quantified uncertainty in detecting misclassified and out-of-distribution (OOD) pixels while also improving model calibration. Through empirical analysis, we uncover challenges in existing uncertainty quantification methods and demonstrate the potential of evidential deep learning techniques, which capture both aleatoric and epistemic uncertainty. To address these challenges, we propose a novel loss function, Uncertainty-Focal-Cross-Entropy (UFCE), specifically designed for highly imbalanced data, along with a simple uncertainty-scaling regularization term that improves both uncertainty quantification and model calibration for BEV segmentation.

Published as a conference paper at ICLR 2025
PREDICTIVE UNCERTAINTY QUANTIFICATION FOR BIRD'S EYE VIEW SEGMENTATION: A BENCHMARK AND NOVEL LOss FUNCTION
Linlin Yu 1x, Bowen Yang :* Tianhao Wang', Kangshuo Li!, Feng Chen'
i "Department of Computer Science, The University of Texas Dallas, Richardson, TX, USA *Cypress Woods High School, Cypress, TX, USA f (linlin.yu, yu, tianhao.wang, kangshuo.li, feng.chen) @ uutdallasedu,,
8 bowenymail @ gmail.com
ABSTRACT
The fusion of raw sensor data to create Bird's Eye View (BEV) representation is critical for autonomous vehicle planning and control. Despite the growing in- terest in using deep learning models for BEV semantic segmentation, anticipating segmentation errors and enhancing the explainability of these models remain un- derexplored. This paper introduces comprehensive benchmark for predictive uncertainty quantification in BEV segmentation, evaluating multiple uncertainty quantification methods across three popular datasets with three representative net- work architectures. Our study focuses on the effectiveness of quantified uncertainty in detecting misclassified and out-of-distribution (OOD) pixels while also improv- ing model calibration. Through empirical analysis, we uncover challenges in exist- ing uncertainty quantification methods and demonstrate the potential of evidential deep learning techniques, which capture both aleatoric and epistemic uncertainty. To address these challenges, we propose novel loss function, Uncertainty-Fowt Focal- Cross-Entroon (UFCE), specifically designed for highly imbalanced data, along with a simple uncertainty-scalinn regularization term that improves both uncertainty quantification and model calibration for BEV segmentation.
INTRODUCTION
Bird's Eye View (BEV) semantic segmentation is critical component of modern vehicular technology and has received increased attention in recent years. It has been adopted in advanced autonomous vehicle systems, such as Tesla's 's Autopilot. BEV representations offer a top-down perspective of the environment surrounding vehicle, created by fusing data from multiple sensors such as cameras, LiDAR, and radar (Philion & Fidler) 2020). This comprehensive view allows autonomous systems accurately perceive the position and movement of nearby objects, including other vehicles, pedestrians, and obstacles. Therefore, BEV semantic segmentation (BEVSS) plays a vital role in both autonomous driving systems and advanced driver-assistance systems (ADAS) (Liu et al. (203))
Identifying potential errors before they lead to dangerous outcomes is crucial for ensuring the safety and reliability of BEVSS. However, Deep Neural Networks (DNNs) commonly used for BEV representation learning tend to make overconfident predictions on unseen data (Guo et al., (017) and underconfident predictions on noisy data (Wang al. 2021). In 2018, an Uber autonomous vehicle in Arizona failed to correctly identify a pedestrian crossing outside a designated crosswalk at night, resulting a fatal collision (Goodman 2018). In March 2022, Tesla Model S on Autopilot crashed into a stationary vehicle on a Florida highway, injuring five police officers (Shepardson 2022).
Uncertainty prediction in segmentation can enable autonomous systems to return control the driver when necessary. There are two primary types of uncertainty (Kendall & Gal 2017). Aleatoric uncertainty, which arises from inherent randomnesses, such as noisy data and labels. By identifying areas with high aleatoric uncertainty, the vehicle can make better decisions, especially in complex or
Equal contribution.

---

Published as a conference paper at ICLR 2025
ambiguous situations. Epistemic uncertainty, the other hand, stems from a lack of knowledge, such as when the test-time input differs significantly from the training data. Quantifying epistemic uncertainty helps the system handle -ut-o--distributiin scenarios and adapt unexpected conditions, especially in dynamic driving environments.
Uncertainty quantification methods can be broadly categorized based their computational com- plexity and approach. The first category involves multiple forward passes to estimate a model's s predictive uncertainty, including methods such as deep ensembles [Lakshminarayanaal
al.) 2017)
and dropout-based approaches (Gal & Ghahramani (0T6). While effective, these methods are compu- tationally expensive, making them impractical for real-time applications. The second category utilizes deterministic single forward-pat neural networks. Conjugate- ceeuute-proo--ue prior-based methods, like evidential neural networks (Sensoy et af., 2018), predict a conjugate prior distribution of class probabilities, providing multi-dimensional uncertainty estimates. Additionally, post-hoc methods, such as those based on softmax (Hendrycks Gimpel/ 2016) and energy models (Liu et al.) 2020), are notable for their case of adoption without requiring modifications to the underlying network architecture,
This study investigates the uncertainty-awwn BEVSS task, which involves both pixel-level classifica- tion and the associated uncertainty estimations. Our main contributions are
We introduce the first benchmark for evaluating uncertainty quantification methods in BEVSS, analyzing five representative approaches (softmax entropy, energy, deep ensemble, dropout, and evidential) across three popular datasets (CARLA (Dosovitskiy et al.) 2017) 1)), nuScenes (Caesar et al. 2020), and Lyft (Kesten et al. 2019)) using three BEVSS network architectures (Lift-Splat-Shi Sholt--Soottt (Philion Fidler 2020). Cross- Transfornfrr (Zhou & Krähenbühl 2022), and Simple-BEV (Harley et al. (023)).
We propose the UFCE loss, which we theoretically demonstrate can implicitly regularize sample weights, mitigating both mnder-iiting and over- wver-fitting. In addition, we introduce a simple uncertainty scaling regularization term that explicitly adjusts sample weights based on epistemic uncertainty.
Extensive experiments demonstrate that our proposed framework consistently achieves the best epistemic uncertainty estimation, improving the AUPR for OOD detection by an average of 4. 4.758% over the runner-up model. Additionally, delivers top-tier aleatoric uncertainty performance, as evaluated through calibration and misclassification detection, all while maintaining high segmentation accuracy
2 UNCERTAINTY QUANTIFICATION ON BEVSS
Problem Formulation. Suppose we are given n images from RGB camera views surrounding the ego vehicle. Let X ;= (X,, Es, L In km denote the input, where each camera view has feature matrix X R3 (with H and W representing the height and width of the input image), an extrinsic matrix Eg e R3x4 and an intrinsic matrix 1ç E p3x3
The goal of uncertainty-awant aware BEV semantic segmentation is to predict pixel-level classes in the BEV coordinate frame, represented by Y E {0, 1JCXMXN along with aleatoric uncertainty u. uolee
and epistemic uncertainty ui for each pixel indexed by (i.j). Here, C denotes the number of classes, and M and N represent the height and width of the BEV frame, respectively.
Network architecture. We directly apply the network architecture from the well-established BEVSS task. A common BEV semantic segmentation neural network has the general form:
P softmax (f(X; 0))
(I)
where P E [0, MxxxXxN are the pixel-wise class probabilities, and @ refers to the network parameters. We use P.j to denote the class probaabioyyyl vector of the BEV pixel at index (i.j).
Uncertainty quantification methods usually apply post-processing techniques or introduce slight modifications to the network architecture, training, or inference strategy. In our experiments, we consider three network architectures: Sift-Splat-Shoot (LSSS) (Philion & Fidler 2020p, Cross-Vice
Transformer (CVT) (Zhou& & Krähenbühl 2022). and Simple-BEV (Harley et al. 2023). LSS lifts 2D
'hee code is available http://gittuh.ccmm/buufiiihttb
2

---

Published as a conference paper at ICLR 2025
camera images into 3D space, projects them onto BEV plane, and then processes the BEV features for semantic segmentation. CVT, on the other hand, leverages attention mechanisms to transform multi-view image inputs into unified BEV representation, enabling more effective cross-view feature fusion. Simple-BEV defines a 3D coordinate volume over the BEV plane and projects each coordinate into the corresponding camera image. Details are provided in Appendix A.2.
Uncertainty Quantification Benchmark: In this benchmark, we aim to investigate the performance of various uncertainty quantification methods on the uncertainty-aware BEVSS task.
We apply five representative uncertainty quantification methods in the deep learning domain. One widely used metric is entropy (Hendrycks & Gimpel 2016), which captures aleatoric uncertainty for in-distribution samples and is preferred due to its simplicity and low computational overhead. Additionally, Liu et al. (2020) proposed the use of an energy score to distinguish between in- distribution and OOD samples, which is also post-hoc method. We also consider the ensemble- based method (Lakshminarayanaal) et al. (017). which estimates uncertainty by training multiple models and averaging their predictions. Similarly, the dropout-ussed method (Gal & Ghahramani 2016) approximates Bayesian inference by performing dropout during inference. These two methods generate multiple class probability predictions, and the variance is used as epistemic uncertainty metric. These four methods do not require significant architectural changes to the network. In contrast, the evidential neural network (ENN) (Sensoy et al. 2018) replaces the standard softmax activation with ReLU function and produces the multinomial opinions in subjective logic. Although this approach modifies the model architecture, it requires only single training and inference pass to generate both aleatoric and epistemic uncertainty, along with the class probabilities. Detailed descriptions of the models and uncertainty calculations are presented in Appendix AA2.33] [A.2.3
Evidential Neural Networks (ENN): Due to computational efficiency and explainability. we extend ENNs to conduct uncertainty quantification for BEVSS, where pixel-level prediction expected. We replace the last softmax in BEVSS network architecture (LSS, CVT or Simple-Bi (imple-BEV) with the ReLU activation function predict concentration parameters of a non-degenerate Dirichlet distribution.
A TRLLU (S(X;)) + 1, R CxMxN
We assume that the target class label is drawn from a categorical distribution parameterized by probabilities pı.- In this case, we can probabilistically model these probabilities using their conjugate prior, the Dirichlet distribution, which itself is parameterized by the concentration parameters a. = Af; j... For simplicity, we omit the index (i.j) in the subsequent discussion.
~ Cat(y p), p Dir(pla).
The expected class probability can be derived as shown in Equation where an EC aci
p = E Eppia [+] = a/a0
Based subjective logic opinion (Josang 2016 there is bijection between subjective opinions and Dirichlet PDFs, allowing a C-dimensional Dirichlet probability distribution to represent a multinomial opinion. Intuitively, we introduce the concept of "evidence". defined as a metric indicating the volume of supportive observations gathered from training data which suggests a sample belongs to a specific class. Let e, 2 0 represent the evidence for class c, with e, a 1. Higher evidence demonstrates stronger confidence in classifying a sample into the corresponding category, whereas lower overall evidence across all classes suggests a lack of similarity with the training data, indicating higher likelihood of the sample being out out-of-doe distriiiibttion.
Then we discuss the optimization loss for evidential based models. UCE loss associated with an entropy regularizer is commonly used in evidential based models (Sensoy et al. 2018 Charpentier
al. 2020
Li al. 2024), With a training dataset Dirain
[ ENCE (0) = (x DDratrn [CUCE (0, - ENT (0, x)]
(5)
The first component, called UCE loss, aims to minimize the expected cross-entropy loss between the predicted class probabilities and the target categorical distribution. Here, the predicted categorical distribution is sampled from the anticipated Dirichlet distribution, and the target distribution follows one-hot encoding format. This optimization strengthens the evidential support for the true class
3

---

Published as a conference paper at ICLR 2025
while reducing support for other classes. Here, (b)-) is the digamma function.
CUCE (0, x, = (p- [H (p. y)] Σ yo (v((a)) - (v(oo))
The second component, termed the Entropy Regularizer (ER), involves the entropy of the predicted Dirichlet distribution. This can be viewed as the Kullback-Leibler (KL) divergence between the pre- dicted Dirichlet distribution and a uniform Dirichlet prior, promoting a smooth Dirichlet distribution.
The closed-fone form of the Bayesian loss is provided in Appendix A..I
C (0, x) H. (Dir (ax(0, x)) KL (Dir(p)a X)) || Dir(p/1))
METHODOLOGY
In this section, we begin by discussing the limitations of the commonly used UCE loss for ENN models in Section 3.1 Next, we formally introduce our proposed UFCE loss in Section 3.2) highlighting how it partially addresses the limitations of UCE. Finally, we present our proposed uncertainty quantification framework in Section 3.3
3.1 LIMITAION OF UCE
In an uncertainty-aware classification task, there are three levels of ground-truth and prediction pairs:
1. A one-hot encoded ground truth class label y, compared to the predicted label
2. A ground-truth categorical distribution Cat(p) aligned with the predicted expected categori- cal distribution Cat(p) from an ENN.
3. A ground-truth distribution over the categorical distribution Dir(á),, compared to the pre- dicted Dirichlet distribution Dir(a), parameterized by the ENN.
Aleatoric uncertainty (uncertainty of the class prediction) can be calculated with the negative maxi- mum class probability or the entropy of the categorical distribution. Ideally, aleatoric uncertainty should reflect the model's confidence in perfectly calibrated network. However, perrpparameterized deep neural networks, trained with the conventional cross-entropy objective, often exhibit overcon- fidence, leading to significant calibration issues (Guo et al. (017). The overconfidence issue is frequently correlated with overfitting the negative log-likelihood (NLL), since even with a classifica- tion error of zero (indicative of perfect calibration), the NLL can remain positive. The optimization algorithm may continue to reduce this value by increasing the probability of the predicted class. In this section, demonstrate that the UCE loss suffers from similar issue. For example, even with perfect evidence volume prediction, the UCE loss remains positive and increasing the evidence for the predicted class further decreases the UCE loss. To address this, propose the UFCE loss, which aims to improve the calibration performance and the quality of uncertainty estimation.
Epistemic uncertainty (uncertainty on the categorical distribution) can be calculated with the predicted total evidence ag. In the commonly used UCE loss (Sensoy et al.) 2018 Charpentier et al. 2020). minimization continues when total evidence increases. Bengs et al. (2022 highlighted that learners employing UCE loss with first-level ground truth (class label) tend to peak the third-level distribution (Dirichlet distribution). This creates a false impression of complete certainty rather than accurately reflecting uncertainty. To address this, we propose epistemic uncertainty scaling and regularized evidential learning to improve epistemic uncertainty prediction.
Overall, aleatoric and epistemic uncertainty can be estimated based on the Dirichlet parameters o::
alee max Pe: opis C//a..
(6)
Proposition 1. Given predicted distribution p Dir(a),, where a (01, 022,...,00 ac) and C is the number of categories, and a target distribution q ~ Dir(()), assuming a one-hot style target distribution such that à; = 1 for all & where e* is the ground truth label and & we have,
CUCE_ (I(Dir(()) Dir(ā:)) H(Dir((:)) -oog ((()))
(7)
4

---

Published as a conference paper at ICLR 2025
where log(B(á)) is constant with B()) is Beta function.
Proposition I shows that optimizing UCE loss minimizes the sum of the KL divergence between the predicted and ground truth distributions and the entropy of the predicted distribution. Consequently, minimizing UCE loss forces the predicted Dirichlet distribution to align the one-hot target evidence distribution while simultaneously making it more peaked. Since epistemic uncertainty is measured by the spread of the Dirichlet distribution, this results in an overconfident model prone to overfitting.
3.2 UFCE IMPLICIT WEIGHT REGULARIZATION
Motivated by the better calibration capability of Focal loss (Lin et al. 2017) compared to cross- entropy, we propose the Uncertainty Focal Cross Entropy (UFCE) loss, which takes the expectation of focal loss rather than cross-entropy loss.
(0, y. Exxx { DD)-Drrain R (.r.aa((((0.))))
Σ Nel1 - -Po) lo
(8)
B(au. ?) ())xy)-DDrat B(ao act 7) [M(00 +7) +b((a)))
where y is a hyperparameter, e" is the ground truth class index, and B-) is the Beta function. When y 0, the UFCE loss is equivalent to the UCE loss.
Can the UFCE loss improve calibration over UCE loss? We investigate this question by analyzing the relationship between UFCE and UCE losses and their gradient behavior.
The general form of the UFCE loss has the following lower bound (see Appendix Proposition 4
CUPCE > _UCE y ..y.Ep--annn (p--Dr(())) [H(p)]
This bound indicates that minimizing UFCE loss leads to both minimizing UCE loss and maximizing the expected entropy function (H)(()) of the categorical distribution Cat(p). The hyperparameter
balances these two terms. Maximizing the entropy term reduces the gap between ae and the evidence of false classes aee, Vc o c², implying that UFCE loss models are generally less confident in their evidence predictions than UCE loss models, particularly for correctly classified samples.
Proposition 2. Comparing CUFCE and LUCE with numerical analysis the gradient of the parame- ters W in the last linear layer, we have, @CUFCE
LLUCE Ôw.
||
Pf S Ti TT((co 12 [<0
-
iwe-
if > (2(a ?) 1 am
where Ti(Cter 7) and T2( 7) are two thresholds within
respectively.
Proposition 2 shows the relationship between the norms of the gradients of the last linear layer for UFCE and UCE loss under the same network architecture. It is clear that for every y and (i.,, there exists threshold such aCURC
Gradient Norm Difference
0.20
that for all P.. E m)) dw.- 2
This |c Bw.-
0.50
implies that the evidence belonging to the ground truth class predicted by the UFCE mode will initially increase faster than that of the UCE model.
Moreover, there exists T2, and for all Po. E [T2, 1
It implies once Pe- surpasses the threshold r, UFCE will apply regularizing effect to prevent the model from continuing to focus examples it is already confident about, thus avoiding overfitting.
1). Bw..
ag
<
ac dw,-
-000
D--
Figure 1: The y-axis represents the differ- ence between the Ll-norms of the UFCE and UCE gradients with ax, 5, while the axis corresponds p.r. the expected pre- dicted probability of the ground truth class.
Figure I shows an example with 7 1 and a.. 5, T2 s 0.4. This also implicitly acts as a weight regularizer less confident scenarios, which is crucial for highly imbalanced data. Further analysis can be found in Figure and Figure
by pushing the model to focus more
5

---

Published as a conference paper at ICLR 2025
3.3 UNCERTAINTY QUANTIFICATION FRAMEWORK
Epistemic uncertainty scaling (EUS): Owing to the imbalance between in-distribution (ID) and OOD pixels within the BEVSS datasets, we observe a tendency for the model to erroneously classify ID pixels that are proximal to OOD objects. This often leads to the model assigning low confidence to these ID pixels. resulting high epistemic uncertainty. Consequently, this issue significantly increases the occurrence of false positives during the OOD detection task. To address this, we propose using epistemic uncertainty scaling to explicitly apply different weights for samples based on their predicted epistemic uncertainty in the previous step of optimization. The equation is adjusted to:
= (1+ =(1+ -.55 C a0
C CUCCE
(10)
where ç is hyperparameter and is the value predicted by the neural network in the previous optimization step. An example with low evidence in the training set will catch more attention during the training process and may lead to better performance such "difficult" samples.
Regulaie- Learning (ER). We consider the OOD-exposure setting to further enhance the epistemic uncertainty estimation. Assume in is the ID training data and Dérasn is the auxiliary OOD training data. Note that we have true OOD (Dbeal) for the evaluation where the auxiliary OOD and true OOD belong to different distributions. We propose training the model to minimize the KL divergence between predictions and a flat Dirichlet distribution for these auxiliary O0D) samples.
CBR = = ( KL (Dir(a) || Dir(1))
(11)
For fair comparison, we also include setting with OOD exposure for the energy-based based model. With the same strategy as (Liu et al. 2020). we consider
learning for OOD detection.
Details can be found in Appendix |A.2.3]
Proposed framework: We replace the last softmax activation function of the original BEVSS model with a ReLU function in order to produce concentrated parameters for pixel-wise Dirichlet distribution. The model is optimized by the following objective functions:
C UFSE. FRR
C UFC + ACC ER
(12)
Our proposed framework comprises three primary components: the Uncertainty Focal Cross-Entropy (UFCE) loss, epistemic uncertainty scaling (EUS), and Rrguence-Regularized Learning (ER). We consider three hyperparameters. in the UFCE, & in the EUS and À for the ER term.
4 EXPERIMENTS
In this section, introduce a benchmark that evaluates uncertainty- BEVSS. This benchmark includes three datasets, three BEVSS network architectures, five uncertainty quantification models, and two uncertainty estimation tasks. We address the following research questions:
RQI: Are aleatoric and epistemic uncertainties estimated by these models reliable? Specifically, can the aleatoric uncertainty metrics accurately detect misclassified pixels, and can the epistemic uncertainty metrics effectively identify OOD samples?
RQ2: Does our proposed UFCE loss enhance model performance in terms of calibration and uncertainty estimation compared to the traditionally used UCE loss in ENNs?
RQ3:
Does the proposed uncertainty quantification framework improve the prediction of epistemic uncertainty as evaluated by the OOD detection task?
Following these three research questions, we provide extensive experiments and further discussions.
4.1 BENCHMARK SETUPS
Evaluation. Following the task setting from (Phiioon & Fidler 2020) and (Zhou & Krähenbühl 2022). we conduct both vehicle segmentation and drivable area segmentation. We evaluate performance using four metries: (1) Pure segmentation via Intersection-ovvr-Uniioo (loU). (2) Calibration via

---

Published as a conference paper at ICLR 2025
Expected Calibration Error (ECE). (3) Aleatoric Uncertainty via the misclassification detection to identify the misclassified pixels, measured with Area Under ROC Curve (AUROC) and Area Under PR Curve (AUPR). (4) Epistemic Uncertainty via the OOD detection to identify the OOD pixels. measured with AUROC and AUPR.
Datasets. We utilize the CARLA simulator-generated dataset two real-world datasets (nuScenes and Lyft), and a corrupted version, nuScenes-C (Xie et al. 2024). In the default setting, "motorcycle" and "bicycle" serve as the true and pseudo-OOD classes for nuScenes and Lyft, respectively, while CARLA uses "deer" as the true OOD and "bears, horses, cows, elephants" as pseudo-OODs. To evaluate robustness to pssudo-OOD choices, an alternative configuration sets "traffic cones, pushable/pullable objects, motorcycles" as pseudo-OOD OODu and "barriers" as the true OOD for nuScenes, with "kangaroo" " as the true OOD for CARLA, which reduces similarity to pseudo-OO1 OODD compared to the default setting. Detailed dataset descriptions and statistics are in Appendix A.4
BEVSS Network Architecture. We utilize LSS, CVT, and Simple-BEV as model backbones with publicly available implementations. LSS converts raw camera inputs into BEV representations by predicting depth distributions, constructing feature frustums, and rasterizing them onto a BEV grid, while CVT employs transformer-based approach with cross-attention and camera-aware positional embeddings to align features into the BEV space. In contrast, Simple-BEs BEV bypasses depth estimation entirely, projecting 3D coordinate volumes onto camera images to sample features, emphasizing efficiency and robustness to projection errors. Detailed descriptions can be found in Appendix 1
Uncertainty Quantification Methods. We consider widely used uncertainty quantification models in traditional deep learning. The "Entropy" and "Energy" models perform post-hoc processing on predicted logits. with "Energy" model being particularly popular for its adaptability and strong OOD detection performance. We also include the widely-used "Dropout" and "Ensemble" models. The "Ensemble" model consists of three separately trained models with different initialization seeds, while the "Dropout" model employs activated dropout layers during inference with 10 forward passes. Both models use either cross-entropy or focal loss. Lastly, the "ENN" model with UCE-ENT loss serves as a baseline to validate our framework. Notably, only the "Energy" and "ENN" models can be trained with or without pseudo-OO1 OOD information. Details are in Appendix [A.2.3]
Hy Hyperparameters. LSS and CVT are trained with a batch size of 32, while Simple-BEV uses a batch size of 16, all for 20 epochs on NVIDIA A6000 or A100 GPUs Network-specific hyperparameters for LSS, CVT, and Simple-BEV are adopted from their original studies. We use the learning rate scheduler from CVT, setting the learning rate to 4c-3 for focal loss variants (as in CVT) and 1e-3 for cross-entropy variants (as in LSS), with the Adam optimizer and weight decay of 1c-7, consistent with both LSS and CVT. We tune three regularization weights, X, f. and ?/ based on the AUPR metric for pseudo-OOD detection on the validation set. To manage computational complexity, adopt a step-by-step tuning approach, fixing two parameters while adjusting the third, instead of performing a full grid search. Detailed information on hyperparameter tuning strategy, optimal values, and sensitivity analysis can be found in Appendix A.3.
4.2 RESULTS
In the main paper, we present results on nuScenes using LSS and CVT as backbones, covering predicted segmentation and aleatoric uncertainty (Table [.. epistemic uncertainty (Table running time (Table 3), robustness analysis (Table 4 and ablation studies (Table(3).
In the appendix, we provide results CARLA and Lyft using all backbones including LSS, CVT, and Simple-BEV (Tables
Additionally, we include hyperparameter analysis (Tables 7 and robustness experiments (Tables 17-2)), which encompass detailed results corrupted nuScenes- C, diverse town and weather conditions in CARLA, and pseudo-OOD selections. Qualitative comparisons are provided in Figures 46
Benchmark Observations (RQ1): Pure segmentation: (1) Across a comprehensive range of con- figurations (50 in total), focal-based losses demonstrate superior segmentation performance when compared to the standard cross-entropy loss in all but two instances, where the difference is marginal (within 0.7%) This pattern is particularly pronounced in models utilizing the CVT backbone, where focal-based losses yield more significant improvements. (2) The UFCE loss for the ENN consistently
Duue the large size of the dataset, we will provide it upon request.
7

---

Published as a conference paper at ICLR 2025
Table 1: Calibration and misclassification detection performance on the nuScence dataset for vehicle Best and Runner-up results are highlighted in red and blue.
segmentation.
LSS
CVT
Model
Loss
Classification
Misclassification
Pure Classification
Misclassifcattin
tolf f
ECRJ
AUROC 1
AUPPR
FPR95
lol'
PCR.
AUROC f
AUPR 1 FPR95
Without psenda
Entropy
CE
0.332
0,00887
0,909
0.315
0.234
0.277
0.00374
0.949
0.325
0.213
Focal
0.347
0.00301
0.00441
0.949
CB
0.332
0.00887
0.909
0.315
0.234
0.277
0.00374
0.949
0.325
0.213
Energy
Focal
0.347
0.00301
0.941
0.332
0.325
0.00341
0.949
0.321
0.206
Ensemble
CE
0.355
0.00569
0.933
0.317
0.218
0.301
0.00276
0.951
0.315
0.216
Focal
0.370
0.00233
0.946
0.315
0.203
0.344
0.00243
0.953
0.324
0.195
CE
0.332
0.00819
0.905
0.315
0.279
0.00373
0.946
0.332
0.213
Drepout
Focal
0.347
0.00261
0.936
0.325
0.208
0.325
0.00363
0.948
0.327
0,202
0.00429
0.00371
0.305
0.224
ENN
UFCE
0.343
0.00332
0.873
0.310
0.225
0.319
0,0019
0.918
0.319
0.208
pscudo OOD
Energy
CF
0.348
0.00721
0.949
0.331
0.200
0.296
0.00238
0.951
0,316
0.219
Focal
0.346
0.00466
0.00186
ENN
UCR
0.343
0.00342
0.838
0.288
0.296
0.282
0.00336
0.919
0.313
0.227
Ours
0.356
0.00193
0.911
0.317
0.190
0.319
0.00019
0.934
0.321
0.196
OOD in the training phase does not impact pure segmentation, calibration and misclassification detcction. Without pscudo- OOD, the proposed UFCE loss for the ENN model consistently outperforms the commonly used UCE loss across all metrics, showing average improvements of 0.0014 in calibration ECF, 3.6% in misclassification AUROC, 2.5% in AUPR and 6.3% on FPR95 across two backhones. 3. No single model consistently performs best across all metrics, but Focal consistently performs better than CE.
Observations: 1. Involving pscudo-OOD
Table 2: OOD detection performance for vehicle segmentation. Best and Runner-up results are highlighted in red and blue.
nuScenes
CARLA
Model
188
CVT
1.88
CVT
AURUC 1
AUPR
HERUS
AUROCT
AUPR 1
HPR95
AUROC
ALPR
HPRY5
AUROCT
AUPR
EPRY5
Without picudo OOD
Entropy
CE
0.584
0,00052
0.799
0.728
0,00057
0.824
0.692
0.00236
0.782
0.813
0.00007
0.704
0.647
0.00056
0.600
0.00053
0.828
0.693
0.782
0.00221
0.336
Eneegy
0,00029
0,00060
0.801
Fosembla
0.00016
0.963
Focal
0.537
0,00025
0.505
0,00021
0.953
0.962
Dippout
0.411
0.00017
0.975
0.384
0.00017
0.957
0.441
0.80062
0.966
0.36
0.00052
0.971
Focal
0.348
0,00015
0.994
0.402
0,00019
0.931
0.390
0.00086
0.964
0.317
0.00049
0.974
ENN
UCE
0.717
0.00075
0.791
0.661
0.00040
0.857
0.620
0.00172
0.795
0.685
0.00235
0,14
UFCE
0.518
0.00054
0.892
0.683
0,00066
0.816
0.535
0.00163
0.835
0.748
0.00237
0.375
With pecudo 00D
Energy
CE
0.774
0,04760
0.408
0.839
0,02060
0.356
0.897
0.07450
0.259
0.929
0.05140
0.159
0.044400
0.860
0.03370
0.319
0.07500
0.183
0.07880
EXN
0315
0.921
0.305
0.111
0.125
Observations: 1. Without DSeudo-OOD, no uncertainty quantification models can predict satisfying epistemic uncertainty, shown by extremely low OOD detection PR. implying epistemic uncertainty estimation is challenging
BEVSS. Compared all baselines, our proposed model (last line in table) has the best OOD detection performance on all 12 metrics, with an average improvement of 2.7% in AUROC, 6.5% in AUPR and 4.3% in FPR95 over the runner-up model.
8

---

Published as a conference paper at ICLR 2025
outperforms the UCE loss across all 14 configurations. Calibration: In the majority of scenarios, models with focal-base based loss exhibit lower ECE scores, indicating better calibration than cross- entropy based models. Similarly, UFCE consistently outperforms UCE. No single model consistently performs best in terms of calibration. Misclassification detection: (1) Despite most models achieving AUROC scores over 90%, the AUPR values remain below 40% across datasets, indicating significant room for improvement in alcatoric uncertainty estimation. (2) Models using focal-based loss perform better in misclassification detection across most settings compared to those using CE-based loss. Our model performs consistently better than the UCE-based ENN.
Table 3: Running time comparison between uncertainty quantification models
Model
Task
Energy
Ensemble
Dropout
ENN-UCE
ENN-UFCE
Ours
Training Inference
10 hours
30 hours
10 hours
10 hours
10 hours
10 hours
LSS
ms
147 ms
457 ms
55 ms
51 ms
ms
Training
8 hours
24 hours
8 hours 123 ms
hours
8 hours
8 hours
CVT
Inference
ms
32 ms
14 ms
ms
ms
OOD detection: (1) Without pseudo-OOD exposure, all models perform poorly in OOD detection, as evidenced by low AUPR values. (2) Only the energy-baset and evidential-ban -based models can utilize pseudo-OOD data. With pseudo-OOD exposure, both models show significant improvements in AUROC and AUPR, indicating that utilizing pscudo-OODs may be a promising direction. Our proposed framework achieves the highest OOD detection performance across all eight evaluated settings. (3) OOD detection in BEVSS is challenging task, evidenced by the low AUPR.
Complexity analysis: Table presents the estimated duration for training on a pair of A100 GPUs and inference on single A6000 GPU. Ensemble models require significantly more time for training, whereas the Dropout model incurs a longer duration during inference. Conversely, the ENN demon- strates reduced time complexity for both training and inference processes. Our proposed model has similar training and inference time cost with the energy model.
Analysis of UFCE Loss (RQ2): We discuss the effect of UFCE loss from three views. Calibration: (1) Compared to UCE loss without pseudo-OOf exposure, models using UFCE loss achieve slightly higher IoU and lower ECE scores across most configurations for vehicle detection and driveable region detection tasks on three datasets and three network architectures. Additionally, under pseudo- OOD supervision, UFCE--based models consistently show better loU and ECE scores across all experimental setups, highlighting UFCE's superior segmentation accuracy and model calibration.
(2) Compared all models with the same configuration, the ENN using UFCE loss achieves results compara- ble to those designed solely for segmentation. Notably, the ENN with UFCE loss achieves the lowest ECE score on 4 out of 6 configu- rations and another second lowest on configuration, demonstrating its effective- ness in calibration. In addi- tion, with pseudo-OOD ex- posure, the ENN shows im- proved segmentation perfor- mance. This enhancement is due to the model's 's capa-
Table 4: Robustness study on the selection of pseudo-OOD with vehicle segmentation task on nuScenes. Best highlighted in red and blue.
and Runner-up up results are
LSS
CVT
Model
0OD Detection
ODD Detection
AUROC
AUPR
AUROC
AUPR
FPR95
UPCE-EUS-ER
0214
0.40
0.272
UCE-BUS-ER
0.346
UCEER
0.302
0.887
0.354
UFCE
0.895 0.006099
0.919
0.727
0.00118
0,861
We evaluate the alternative pscudo- OOD setting, using traffic cones, pushable/pullable objects, and motorcycles as pscudo-OOD, with barriers as the true OOD. Our model outperforms the runner-up in 5 out of 6 metrics, achieving an average AUPR improvement of 18%, with both UFCE and EUS significantly contributing to these gains.
bility to predict accurate Dirichlet distributions, especially for pixels near OOD instances.
Misclassification detection: UFCE loss surpasses UCE loss with consistently higher AUROC and AUPR scores, generally ranking in the mid-to-upper range among models.

---

Published as a conference paper at ICLR 2025
OOD detection: Compared to UCE loss with pseudo-OOD supervision, our proposed framework with UFCE loss demonstrates significantly better performance, with up to a 12% increase in AUPR and 4% boost in AUROC. These results highlight the significant potential of the UFCE approach to improve the reliability of epistemic uncertainty estimation.
Table 5: Ablation study for vehicle segmentation on nuScenes
Best and
Runner-up results are
highlighted in red and blue.
Pure Classification
Misclassification
OOD Detection
Model
loll
BCE
AUROCT
AUPPR
FPR95
AUROC
AUPR
FPR95
LSS
UPCE-EUS-ER
0.356
0.00193
0.911
0.317
0.190
0,929
0.335
0.219
UCE-BUS-ER
0.339
0.842
0.340
0.329
UCE-ER
0.342
0.00342
0.838
0.289
0.296
0.889
0.315
0.315
ICE
0.341
0.00429
0.819
0.273
0.335
0.217
0,00075
0.791
CVT
UPCE.EUSEER
0.319
0.00019
0.934
0.321
0.196
0.928
00:269
0.244
UCE-BUSSER
0.281
0.00360
0.920
0.314
0.217
0.931
0.210
0.326
UCE-ER
0.282
0.00336
0.919
0.313
0.227
0.921
0.212
0.306
UCE
0.291
0.00371
0.900
0.305
0.224
0.661
0.00049
0.857
The base model is "UCE" and we progressively add the proposed components, First, we introduce ER to obtain "UCE-ER", followed by adding EUS to create "UCE-EUS-ER" Finally, we replace UCE with UFCE, resulting in the madel "UFCE-EUS-ER" Observation: Adding "ER" largely improve the OOD detection performance (average 26%). "EUS" further improve he misclassification detection and OOD detection slightly. The "UFCE" improves the calibration and misclassification detection with a significant gap ECE:0.002555, AUROC:4.15% AUPR:1.7%, FPR95:6.05%)
Ablation Study (RQ3): There are three primary components: UFCE, BUS, ER. We conduct ablation studies using the nuScenes dataset with the LSS and CVT backbones to assess the impact of each component on system performance without compromising generality. The results T Starting with the standard ENN model using the UCE loss as the baseline, we progressively add components to assess their contributions. First, we introduce the ER term, which incorporates pseudo-OOD data during training. This addition leads to a significant improvement in OOD detection performance, with up to 31% improvement in AUPR. Next, we add the EUS regularization, which further enhances both misclassification detection and OOD detection performance. Finally, we replace the UCE loss with our proposed UFCE loss, achieving the best overall results. This change results in up to a 6% improvement in AUPR, particularly benefiting calibration and misclassification detection without sacrificing segmentation accuracy.
summarized in
Discussion on the pseudo-OOD: Intuitively, greater similarity between true and pseudo-OO) -OOD pairs enhances OOD detection performance, while overfitting to pseudo-OODs raises concerns about the model's generalization ability. To further investigate, pseudo-OOD and true OOD pairs compared to the default setting (Table 2). The results for nuScenes are shown in Table 4 while the results for CARLA are provided in Table 17 in Appendix A.5.5 The findings confirm our intuition: higher similarity between true and pseudo-OOD pairs leads to better OOD detection performance. Notably, our proposed model consistently outperformed the best baseline methods across all eight scenarios, achieving improvements of up to 12.6% in AUPR. These results underscore the robustness of our approach, even in settings with less similar OOD pairs.
conducted experiments using dissimilar
CONCLUSION
This paper presents comprehensive evaluation of various uncertainty quantification methods for BEVSS. Our findings reveal that current methods do not achieve satisfactory results in uncertainty quantification, particularly in OOD detection, highlighting the need for advancements in this do- main. Inspired by the robust calibration properties of Focal Loss, we introduce the UFCE loss, which significantly enhances model calibration. Our proposed uncertainty quantification framework, based on evidential deep learning, consistently outperforms baseline models in predicting epistemic uncertainty, well in aleatoric uncertainty and calibration, across wide range of scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDDMEENTS
This work is partially supported by the National Science Foundation (NSF) under Grant No. 2414705. 2220574, 2107449, and 1954376.
REFERENCES
Siddharth Ancha, Philip R Osteen, and Nicholas Roy. Deep evidential uncertainty estimation for semantic segmentation under out-of-distribution obstacles. Proc. IEEE Int. Conf. Robot. Autom, 2024.
J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and Gall. SemantieKITTII. A Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quan- tification through loss minimisation. Advances in Neural Information Processing Systems, 35: 29205-29216,, 2022.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Loick Chambon, Eloi Zablocki, Mickaël Chen, Florent Bartoccioni, Patrick Pérez, and Matthieu Cord. Pointbev: A sparse approach for bev predictions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15195-15204, 2024.
Bertrand Charpentier, Daniel Zügner,, and Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356-1367, 2020.
Vikrant Dewangan, Basant Sharma, Tushar Choudhary, Sarthak Sharma, Aakash Aanegola, Arun Singh, and K Madhava Krishna. Uap-bev: Uncertainty aware planning using bird's s eye view generated from surround monocular images. In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pp. 1-8. IEEE, 2023.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning. pp. 1-16, 2017.
Florian Fervers. Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelha- gen. Encertainty-awan aware vision-based metric cross-vics view geolocalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21621-21631, 2023.
Gianni Franchi, Xuanlong Yu, Andrei Bursuc, Angel Tena, Rémi Kazmierezak, Séverine Dubuisson, Emanuel Aldea, and David Filliat. Muad: Multiple uncertainties for autonomous driving, benchmark for multiple uncertainty types and tasks. arXiv preprint arXiv:2203.01437, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1050-1059. JMg.. 2016. URL http:tttp/rcceeeinngg/ http://ddd....a......rree/////aa
mlm wtt.mm.pprrss/vve.
html
Joshua Goodman. The uber self-driving car fatality: what happened, and what it means. 1EEE Spectrum, 28, 2018. URL hhttts::/eee wik wikiiiiiiaaa... a of Herzberg
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML,, pp. 1321-1330. PMLR, 2017.
11

---

Published as a conference paper at ICLR 2025
Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. Simple-bev: What really matters for multi-sensor bev perception? 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2759-2765, IEEE, 2023.
Russell Alan Hart, Linlin Yu, Yifei Lou, and Feng Chen. Improvements on uncertainty quantification for node classification via distance based regularization. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL. https:/ http::://ppeeevvvii ew net. forum? ? idd=
MJZGCW2hC6
Jianfeng He, Xuchao Zhang, Shuo Lei, Abdulaziz Alhamadani, Fanglan Chen, Bei Xiao, and Chang- Tien Lu. Clur: uncertainty estimation for few-shot text classification with contrastive learning. In Proceedings of the 29th ACM SIGKDI) Conference on Knowledge Discovery and Data Mining, pp- 698-710, 2023.
Jianfeng He, Linlin Yu, Shuo Lei, Chang-Tien Lu, and Feng Chen. Uncertainty estimation on sequential labeling via uncertainty transmission. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2823-2835, Mexico City, Mexico, June 2024, Association for Computational Linguistics. n80cl.180. URL
doi: 10. 10.18665/11/0024.fnnddd 1807
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rXivv1610.002136, 2016.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: future instance prediction in bird's s-eye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15273-15282, 2021.
Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du Bevdet: high-performance multi-camera 3d object detection in bird-eye- ArXiv, abs/21 1790, 2021.
Audun Jøsang. Subjective logic, volume 3. Springer, 2016.
Markus Kängsepp and Meelis Kull. Calibrated perception uncertainty across objects and regions in bird's sird''-eye-view. arXiv preprint arXiv:2211.04344 2022.
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain,
P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang. and V. Shet. Lyft level av dataset 2019. https::/leeeee
lyft wtt.//tttt////////////// 2019.
Balaji Lakshminarayaman, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NewrIPS, 30, 2017.
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. 2019 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 12689-12697,200 2018.
Changbin Li, Kangshuo Li, Yuzhe Ou, Lance M. Kaplan, Audun Jøsang, lin-Hee Cho, DONG HYUN JEONG, and Feng Chen. Hyper evidential deep learning to quantify composite classification uncertainty. In The Twelfth International Conference Learning Representations, 2024. URL
/ettps:/cpperree wtw.eerrwwwee..eee/ooo.m.mm..
net for.mmidd=A7
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuam Li. eneegy-basces out- of out-of distiriibnt detection. Advances in neural information processing systems, 33:21 21475, 2020.
12

---

Published as a conference paper at ICLR 2025
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor sensor fusion with unified bird's s-eye view representation. In 2023 IEEE international conference robotics and automattin (ICRA), pp. 2774-2781. IEEE, 2023.
Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprini arXiv: 181.1281, 2018.
Jishnu Mukhoti, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty for semantic segmentation. arXiv preprint arXiv:2111.00073 2021.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting 3d. In ECCV, 2020.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncertainty. NeurIPS, 31, 2018.
David Shepardson.
Tesla autopilot involved in crash with parked police car in florida. Reuters, 2022. URL https: //ww:/ cnn. http:///////////////aa//////
tes 1 a-crash- ce-car /wt//iiidxx..e//ee html
Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. Advances in Neural Information Processing Systems, 101188333-8043 2021.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ArXiv, abs/1905.1 1946, 2019.
Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. Advances in Neural Information Processing Systems, 34:23768-23779 2021.
Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking and improving bird's eye view perception robustness in autonomous driving. arXiv preprint arXiv:2405. 17426, 2024.
Linlin Yu, Yifei Lou, and Feng Chen. Uncertainty-aware graph- hyperspectral image classifica- tion. Proceeding of the International Conference on Learning Representations (ICLR), 2024.
Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances Neural Information Processing Systems, 13:17727-12836 2020.
Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13760-13769, 2022.
Xiyue Zhu, Vlas Zyrianov, Zhijian Liu, and Shenlong Wang. Mapprior: Bird's-eye view map layout estimation with generative models. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision. pp. 8228 8239, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOFS
Assume the target distribution q Dir(ā) and the predicted distribution p Dir(a),, where a (01, Q2,...,9C)) .,,)) and = (δ1, .d......ue with is the number of categories. We first provide some preliminary results related Dirichlet distribution.
The probability density function for a Dirichlet distribution for a vector p is given by:
C
D(pla) = B(xx
P
(13)
i=1
where ((x) the beta function for the vector a.
B(a) = I T(a:)
(14)
(ao)
We use og = Ei- a₁ and âg Ei= O₂ for simplification.
Then, the Kullback-eibler divergence of from is given by
KL || q) = log B(&) log B(a)= + Σ (an - aa) (v(as) (ao))
(15)
The entropy of p is given by:
H(p) log B(x) + (00 C)
Σ (ai 1))(as)
(16)
Proposition 1. Given predicted distribution the number of categories, and a target distribution q ~ Dir(á),, assuming one-hot style target distribution such that à; 1 for all + c" where c* is the ground truth label and å, have, CUCE KL(Dir(a:) Dir(a)) + H((ir((a)) Log(B(áx)
Dir(a),, where (a1, 0.2,.... ac) and C
(7)
Proof. Combine Equation
16 then we have
(L((ir(aa) || (ir( + H(Dir(a)) log B()) + Σ (ā: 1) (b((n) v(a))] (17)
i=1
Given that & is perfect separable target distribution, i.e. à; = 1 for all i + c², then we have (I.(Dir(a) Dir(á)) H(Dir(a:) (ō 1) ((o) (a ) + log B(x) (18)
The uncertainty cross entropy loss is the expectation of cross entropy loss, i.e.
UCCE E Dir(aa)
ye log Pe = <((a)) - ((c))
(19)
Then we have:
CUCE (L((Dir(a) Dir(ā)) H(Dir((:) log B(ā) â 1 H(Dir(a))
(20)
KL(Dir(a) || Dir(a)) de
log B(k)
(21)
á
áe
Considering the loss for single point, the term e. a. represents the number of events that occurred given the uniform prior, with being the maximum value 0 being the minimum value. Therefore, for each sample observed, only have å ee- +1 while å e 1, & ct. which corresponds to the only event occurred at ground-truth class c.. Consequently, the loss function given by:
UCB KL(Dir(a) Dir(&)) H(Dir(a)) log (()))
(22)
14

---

Published as a conference paper at ICLR 2025
Based on Proposition when minimize the UCE loss, minimizing the KL divergence between a predicted Dinichlet distribution with a target distribution with 'one-hot' evidence, as well as minimize the entropy of the predicted Dirichlet distribution. This will lead the Dirichlet distribution to peak at some point and can not spread to denote the true distribution, leading to overfitting.
Proposition 3. Given random variable following a Dirichlet distribution Di r(o), then the expectation of Focal loss has the following analytical form:
C CUCCE 16 Dir(aa [-(1 -Pe-) Pe-)" log logpe.]
(23)
T(ao Oce + (an) [vb(au + () vb(a)))
(24)
T(ag >)[ Oce.)
where B() denote Beta function and [1, [1,+ is the predicted strength parameter and C is the number of classes.
Proof. Let B()) denote the Beta function and Beta(.) denote the Beta distribution. Define the
Gamma function as I'(.), the Digamma function as (-). and the Trigamma function as (-)).. Let
y be the ground-truth target one-hot vector and be the predicted probability distribution over the simplex A. For simplicity, denote c* as the index of the ground-truth class and define du = Ee dc-
The analytical form of Uncertainty Focal Loss can be derived
C UUCC
Ep-ni ((a)
Σ Ve pe) log Pe
/ (1 (P- Pe-) log(pe. (Dir(pla)dp
/ - 1-Pe->7 log(po. )et : - Oce )dp..
(1 (1-po-)" (o-p->)>lo((p>>> W -1 11 Po- 000000010000000 -dpe-
Blae, 000 - Oce))
/(
da Pe p
(1 Pc- pc: )o )00 dpc
B(ace 010 - ace)
1
d
((--pe))' -11--1111111 - pc- )a0 a. dpc
B(ae, 0ip - ac da
(25)
1
d
/ [10 [pa---(111 (1- pe- ) aa0-a. +7-1 dpc
B(ac, 00 ac
dac d
1
B(a - ac- +7)
B(ac,, ac' da
1
d I(a )[(( - -,- 7)
B(a G0 ag da T(ag + +7)
1
Blac, ar 7)(((((.))) dlao + 7)]
Blae, 00 - ac.
I'aa))I((o ac + + + +2)
[vb +) v(a
T(a, [ a T/(()))
[(oo))((o0 ac +7)
[s/a0 + ((a)))
I(ao + > ac-)
Proposition 4. Given that 1, the UFCE loss has the lower bound involving UCE loss as:
CUFCE > LUCE 7 (E H(p)
(26)
15

---

Published as a conference paper at ICLR 2025
Proof. For any 2 1, by applying Bemoulli's inequality and Hölder's inequality, we have the following relation when having the same predicted a:
C
CURCE pp-Dir(()
(1 (
p-) ye log(pe)
0
2 pi-iii(a)
(1 7 Pe pe)yle log(pc)
-p-Dir((a)
Vc (oge) +
YePe log(Pe)
pprri
Ue log(
Σ YyePe log(pe)
Me log(p) > 1 (p log(p))
Ep-iir(a)
2 pp-
Ve log()) - >/1 ||0r||_o112 log(p)]1
(27)
Σ log(pe)
Bp-Di((a)
Vc log(p.)
max ys
Ve log())) -7 Σ Pe log(pe))
Bp-iir(a)
Ep-Dir(a)
Me log(
- - (.p-oir((a)
Σ
Pe log(pe)
= -E--ii((a)
Σ Ue (po)
- - -pppoir((a)
H(p)
= CUCE 7 Ep-Dir(a)
H(p)
where o denotes dlement-wise product.
Proposition 4 shows that the lower bound of UFCE loss is equivalent to the UCE loss minus the entropy where is a trade-off parameter.
Proposition 5. For uncertainty focal loss LUFCE the partial derivative with respect to the ground truth class e² has the form:
i UPCE doc
[(oo - a. + +7) T(oo + T ac))
vk(a)) vl00 + 7) (ba [w(oo - v(a
(28)
+
Vi + + --1((
(29)
16

---

Published as a conference paper at ICLR 2025
Proof. for UFCE loss, the gradient is
((ao)T(ao- Oc +7)
L
CUCCE_
lvb(ag + rlac.))
dae
dae-
T(ag + ()) ac+)
0
B(au, 7)
(s'(a0 +7) - (bac.)]]
dac
Blao act
B(
1b(00 + 7) rb(c))))
B(ay act,
da.
B(ao,
[1b(an) 2/00000000000000 + +)]l lib(av + + -
(30)
B(ao act,
B((n,
1/1 110 +7) - ((c)))
B(ap act
((oo) Oc +7)
{ (b(ao) +0au + +))
vb(a + + ((ac))
T(av + r)[((a) act)
(i 7) 21 (act)
In the following proposition, we consider the influence of gradients on the prediction of ag- Let We denote the vector of weight parameters in the last linear layer that influences the prediction of the true-class evidence c². Let denote the logits and be the input the last linear layer. The prediction of a,- has the following form:
2
= RRLLU
w
where the last weight in w. is related to the intercept.
Proposition 2. Comparing LUFCE and LUCE with numerical analysis on the gradient of the parame- ters we in the last linear layer, we have,
OLUPCE
CCUCE owe
(2 Pf S Ti (ac <<
Ow.-
Pe 72 (2
where Ti(ar,,?)) 7) and T2(a 7) are two thresholds within ( 1- oqg
respectively.
Proof. Define w. as the model parameter of the last linear layer. Using the chain rule, we can easily derive the gradient for the last linear layer's 's parameter:
OCURCE
Üs
Oace
@CCURCE
Owe
Owe
Øs
dac
(31)
acuCE
Øs
dac
(DLUCE
Ow.-
Ow.-
Ōs
Dag.
This establishes connection between the gradient of the last layer's weight and the gradient of the loss functions with respect o,,,. Thus, Bac
ag!ir
implies
Do.
ow.-
ow.-
The uncertainty cross entropy loss is given by:
C UCE ()) De log
Σ Me (v(()) - = (/(0)) (b((c))
(32)
c=1
Its gradient is given by the difference of trigamma functions:
aCUcE
1/1 (ao) (1(.).).
(33)
da.
17

---

Published as a conference paper at ICLR 2025
According to Proposition 5 we have:
aCURCR
I(ao)I ae + ?) (1v(a0) (000 + 7) (v(p +7) (( [ [01 +7) th 10 }1} T( + +)) ac))
do.-
The difference between the 11-morm of gradients is:
OC
('ao) ae +7) T(ao + 7)) - (c-)
[bb(a)) - (aa0 + >)] + + - ((a=))]
doo
dao
[1'1 +7) - 15i }1 + (1'((0)) + (ac )), (34)
aCURCE Bag 8mg
DC
where and
can be shown be negative for all og > Oge and 7 E [0,5]. 5].
Let Pe- aet and Oc Po- We now analyze the relation between the difference and the projected class probability term Pe. Rewriting the difference based on Pe-, we denote the resulting
do
form as (Pe- ao 7):
((a)) - 00 + 7) [s/(cro) - (0 + 7) (1 + +7) a0) T(oo + ()0 Pe an)
f'Pe, 00.
[au + 7) - 1/2 (p.. an)l } + (#1((0)) - the (o)-
Next, we derive the gradient of this difference function with respect to pe- Let:
[((o))I(ag p-- + 7)
A(p.-) T(ao + + (x)) B( = (b((o)) - -0(000 + +)] [ub(a0 + 7) - (pp an)] + [wi + 7) - 1h 00))
Then,
f ?) = A(pe. -(p (t) (ao) + ap))
First, we find 0A(0.-) Recall that the derivative of the Gamma function with respect to its argument
Op.- [z)))(z).
is: 4 I(()
Using the product rule and the chain rule:
OA(F.-)
[(ao)
d T'a00 - -e- ao +22
Ope-
T(ao +
dp..
T(a0 - Pr. an)
[(ao)
d I(av P0. + 7) 1-agb(ao Pe. ao + +7) + autiaa Po av)
T(ao + 7)
dp.- I(ao- P.- +7) (-o0/( 00 + 7) + 000 pe- T(au ac)
N(ao Po- (kg)
=A(pe) [- a0 + +7) + anvi pc-
(35)
Next, we find 0B(p),
0p.-
0B ( [((a) - vhan + +)] (p. + a0-
(36)
Op..
Then, we find the total derivative of [fer 7): (pe:: Op.-
AA(pe.)
B(pa)) - A(p.)) B
+ (-aoti2(pe-ao))
Op.-
Op.-
= A(pe.) [-aot [
p. ag + 7) + anv(an Po ap)) B(...)
A(pe-) [[v)(a)) v0 + 7) a0 (p.- + (F
(Fe
-A(pe )am (d/(0 an + - v(an D. ap) B(F..) B(F..)
A(pe- )au [[v((a)) v(an + 7)) 1/ (p.- an) + +2 an)] - (p (37)
18

---

Published as a conference paper at ICLR 2025
Noting that this derivative involves polynomial terms of polygamma functions, which are differen- tiable on (0, +00), this verifies f-) is also differentiable on (0; +oo). Given that dg 2 ag- + and
0e- 2 1, the feature range of Pe- oig 1 - ((-) remains differentiable this interval. We
demonstrate that Of(e (00.1)) is positive and negative at the bounding points of this range, respectively. According the intermediate--vllee theorem, there exists least configuration of Pe- in this range such that af(p.+:00.1) within
8p,.
0. Therefore, there exist two thresholds Ti ; 7) and T2 (2(tttt a when Pe- TI TT(e (), and agtiK Da.-
8p..
[ 1 -e 0o
such that acurce da.-
actic
coo
Da.-
da.
when Pe- T2( 7).
First, substitute Pe-
f ( 1 (0.)))) - I(ao + +a) 1) 00
M(a)) -1+)) [xb(ao) viav + >b) -1((1
(38)
+ W (a0 + +7) xh) + (1'1 (ao) 1 (1)).
Using the properties of the Gamma function, I'(ao - 1) [(an] where TE is the Euler-Mascheroni constant, the expression simplifies to:
cup and (1) = -YE and 1 (1)
6
f 1 00. [_.0.)))
06 + 7 - { [bb(ao) - vsa0 + 7)] (bb + + - xb(1]]
Op -
+ |//2 (ap + 7) - -3/ (1) + (eh (0)1 - (1))
(39)
an -1
ao + 7 [wb(an) vian 7)] [bb( + + + 7ml
n² } + 4/1 (0s) x²
1/ + +7)
6
is evident from numerical analysis that f - a0:7) > 0 for all y E [0, 5] and ag 2.
Substituting Pe
we have
0g
5(1 1
1 00.07
I(ao)[(1+ +7) [v(ao) (aa [3] 7) v0(00 - 1)]
000
T(ao +I]
(40)
[Mi(a0 + 7) v1 - 1)1 (wi(a)) - in (a0 1)).
is evident from numerical analysis that f (1 :0 0 for all 7 E 5] and ap >>
A.2
RELATED WORK
A.2.1 VISION-RASED BIRD'S ' EYE VIEW SEMANTIC SEGMENTATION
Bird's-eye view (BEV) serves as an effective representation for fusing information from multiple cameras, making central component in autonomous driving systems. However, transforming camera images into BEV maps presents significant challenges, primarily due to the complexity of
depth estimation and 3D geometric transformations. The key component BEVSS is the 2D-to-3D
lifting strategy, which employs techniques such depth- weighted splitting (Philion & Fidler 2020
Hu et al. (222). attention mechanisms (Zhou & Krähenbühl 2022). and bilinear sampling (Har ley al. 2023).
In this paper, we use three representative methods, Lift splat shoot (LSS) (Philion & Fidler 2020).
Cross-Viee Transformer (CVT) (Zhou & Krähenbühl 2022), and Simple-BEV (Harley et al. 2023
19

---

Published as a conference paper at ICLR 2025
Figure 2: Numerical analysis
for different composition of ag and
210
0.000
34-
-0415
0005
.429
5.505
-0009
2-3
6.134
-0.0
1.0
0.000
110
201
-1410
.007
20

---

Published as a conference paper at ICLR 2025
Figure 3: Implicit weight regularization impact by UFCE with de 10
v 0.00
Y = 0.00
0.010
V = 0.10
V == 0.10
V = 0.20
2.-
Y = 0.20
0.50
Y 0.50
y=
Y
== 1.00
y = 1.00
Ufice
0,000
Tow."
-0.005
62
0.6
0.2
0.6
10
Ps
Pe
LSS leverages raw pixel inputs from multiple surrounding cameras and "lifts" each image individually into frustum of features. Initially, it predicts categorical distribution over predefined set of possible depths. Subsequently, the frustum of features is generated by multiplying the features with their predicted depth probability. By utilizing known camera calibration matrices for each camera, point cloud of features the ego coordinate space can be obtained. LSS then "splats" the frustums into a rasterized bird --ew grid using PointPillar (Lang al.) 2018) model. These splatted features are then fed into a decoder module to predict BEVSS. The concept of transforming from camera pixels to 3D point clouds and subsequently to BEV pixels has inspired several subsequent models, such as BEVDet (Huang et al., 2021 and FIERY (Hu et al. 2021).
CVT takes distinct approach by leveraging transformer architecture and cross-attention mechanism. CVT begins by extracting features from multiple surrounding camera images using pre-trained EfficientNNet-44TTa5 & Le 2019) model. These extracted features serve the attention values in the subsequent cross-attention step. To create the attention keys, the features are concatenated with the camera-awan positional embedding. This positional embedding constructed using known camera pose and intrinsic information, enabling the model to account for the specific characteristics of each camera. The positional encoding of the BEV space serves as the queries during the cross attention process.
Simple-BEV simplifies the process of generating BEV representations by avoiding the use of estimated depth maps. Instead, it defines a 3D coordinate volume over the BEV plane and projects each coordinate into the corresponding camera images. Image features are then sampled from the surrounding regions of the projected locations. While the resulting features are not precisely aligned in the BEV space and are distributed across potential locations. this approach significantly enhances efficiency and robustness to projection errors. PoinBEV (Chambon et al. 2024) further improves the 'feature pulling' strategy with the sparse representations.
A.2.2 UNCERTAINTY QUANTIFICATION ON BIRD'S S EYE VIEW SEMANTIC SEGMENTATION
To the best of our knowledge, there no pioneer work on the uncertainty quantification on the BEVSS task. We introduce the most relative literature in the section.
Uncertainty quantification on input-dependent data: The uncertainty quantification methods used in this work are primarily designed for input-independent data, such images for classification. However, developing uncertainty quantification techniques tailored for input-dependent data remains underexplored but necessary. Zhao et al. (2020) applied evidential deep learning to node-level graph tasks, while Hart et al.) (202)); Stadler et al. (2021) incorporated graph topology relationships. He et al. (2023) explored uncertainty in text classification through conservative learning, and He et (2024) investigated evidential deep learning for sequential data. We consider this an important direction for future research.
21

---

Published as a conference paper at ICLR 2025
Uncertainty quantification in camera view's semantic segmentation. Uncertainty quantification in pixel-level camera semantic segmentation is the most relevant research task. Similarly, uncertainty in this task also arises from two primary sources: alcatoric uncertainty, which reflects inherent ambiguity in the data (e.g., sensor noise or occlusions), and epistemic uncertainty, which represents the model's lack of knowledge due to limited or biased training data. Mukhoti & Gal (2018) evaluated Bayesian deep learning methods, including MC Dropout and Concrete Dropout, applied to semantic segmentation and introduced new patch-based evaluation metrics. However, these metries focused on the relationship between accuracy and uncertainty. limited to aleatoric uncertainty. In addition, Bayesian- based approaches require multiple forward passes, making it impractical for real- time applications such as autonomous driving. Mukhoti et al. (222)) proposed single-pass uncertainty estimation by extending deep deterministic uncertainty to semantic segmentation tasks. They used Gaussian Discriminant Analysis (GDA) to model feature space means and covariance matrices per class, enabling the quantification of epistemic uncertainty through feature densitics, while alcatoric uncertainty was estimated via the softmax distribution. However, this work did not provide comprehensive quantitative evaluation of the quality of the estimated epistemic uncertainty. Anchaa et al. (2024) advanced uncertainty estimation with model based the evidential deep learning framework. Their approach is built on natural posterior network that estimates a pixel-level Dirichlet distribution with density estimator based on a GMM-enhanced normalizing flow and linear classifier. To mitigate feature collapsing issues, they introduced decoder that reconstructs image patches from pixel-Ievel latent features. The proposed model evaluated from both calibration view and OOD detection- like tasks. However, the heavy modification on the network architecture makes it hard to apply to the BEVSS task. Yu et al. (2024) constructed graph over image pixels and incorporated graph topology into uncertainty estimation. However, this method relies on high-dimensional pixel features from hyperspectral images, making it unsuitable for RGB images.
Fusing camera view uncertainties to BEV space: Challenges and Opportunities. Uncertainty quantification BEV semantic segmentation (BEVSS) lacks established literature, and extending methods from the camera space to BEV is non-trivial. To illustrate this challenge, we conducted a simple experiment. Using the CARLA dataset, we trained an Evidential Neural Network (ENN) model based on Ve quantify pixel-level uncertainty in camera images. The predicted uncertainty was then mapped from the camera space to the BEV space using a ground truth mapping function, with animals included the OOD category. Experimental results showed average AUPR of 47% in the camera space, but only 7% in the BEV space.
This discrepancy arises possibly because the predicted OOD regions are often larger than the actual objects. When mapped to BEV, these over-predicted regions expand further due to perspective distor- tion. Conversely, correctly predicted regions are restricted to the visible parts of the object, resulting in significant inaccuracies and poor performance in BEV. Finally, we highlight that incorporating camera- view uncertainties can enhance the quality of uncertainty quantification in BEV, offering a promising direction for future research.
Robutness in bird's eye view semantic segmentation. OOD robustness in BEV semantic segmen- tation focuses on maintaining model performance under novel and unexpected conditions, such as varying weather, lighting changes, or sensor malfunctions. RoboBEV Xie et al. (2024) provides an extensive benchmark for evaluating the robustness of BEV perception models in autonomous driving, testing their performance under various natural corruptions. For instance, they demonstrate significant performance drops in segmentation tasks when evaluating the CVT model trained clean datasets against corrupted datasets.
We further clarify the distinction between OOD robustness and OOD detection within exper- imental context. OOD robustness (or generalization) aims to ensure that models maintain high performance on OOD samples with domain shifts. In contrast, OOD detection emphasizes model reliability by identifying samples with semantic shifts-cases cases where the model cannot or should not generalize. Importantly, these concepts can be complementary: an uncertainty-awarg model should also be robust to domain shifts, meaning it must maintain effective OOD detection even under conditions of domain shift.
Calibration in bird's eye view semantic segmentation. There indeed several works focus primarily on calibration while neglecting challenges such as out-of- diistiib-ttiiiiiinnnn data and dataset shifts, which require epistemic uncertainty estimation. Kängsepp & Kull (2022) focus on overconfident probability estimates semantic segmentation issue by applying isotonic regression for pixel-wise calibration
22

---

Published as a conference paper at ICLR 2025
and beta calibration for object-wise calibration. This method improves confidence reliability but does not explicitly address epistemic uncertainty. MapPrior (Zhu et al. 2023) integrates a generative prior with traditional discriminative BEV models to enhance BEV semantic segmentation performance while focusing on calibration, particularly aleatoric uncertainty. It combines an initial noisy BEV layout estimate from predictive model with generative refinement stage that samples diverse latent space. However, MapPrior's ' approach is limited to alcatoric
outputs from a transformer-base uncertainty and relies on an additional model architecture, making it less adaptable to other BEV semantic segmentation models.
Uncertainty quantification on downstream tasks. Several works have explored uncertainty in downstream tasks that use bird's s-eye view (BEV) maps as intermediate representations. However, these work are also limited calibration. UAP (Dewangan et al. 2023) targets motion planning and trajectory optimization by building uncertainty-awart occupancy grid map. It estimates collision probabilities based on sampled distances to nearby occupied cells and optimizes trajectories approach, incorporating uncertainty into the planning process. Fervers et al. (2023) focus on metric cross-view geolocalization (CVGL), which aims to localize ground vehicles relative to aerial map images. Their approach matches BEV representations aerial images by predicting soft probability distributions over possible vehicle poses. The uncertainty derived from these probability distributions is integrated into Kalman filter, improving the accuracy and robustness of trajectory tracking over time.
using a sampling-base
A.2.3 UNCERTAINTY QUANTIFICATION BASELINES
Softmax- -base (Hendryeks & Gimpel, 2016) Softmax entropy one of the most commonly used
metrics for uncertainty (Hendrycks & Gimpel 2016). is the entropy (HH(p(Y,,) (X);)()))) of softmax distribution p( xX:
H Y i.j x)) = Σ
c=1 Peji.j log log.....--
(41)
This metric is known to capture aleatoric uncertainty, but can not capture epistemic uncertainty reliably.
Energy-based (Liu et al. 2020) The anegyy-based model is designed distinguish OOD data from ID data. Energy scores, which are theoretically aligned with the input's probability density, exhibit reduced susceptibility to overconfidence issues and can be considered as epistemic uncertainty. As a post-hoc method applied to predicted logits, energy scores can be flexibly utilized as a scoring function for any pre-trained neural classifier.
epis
energy Uij
- log Σ
expl /T
(42)
where Le,i.j represents the predicted logits for pixel (i,j) associated with class where logits are defined as the output of the final layer prior to the softmax activation function standard classification model, i.e. L f(x; 0). T is the temperature scaling. The energy score will be employed to quantify epistemic uncertainty. There is no aleatoric uncertainty explicitly defined for energy-based models, use the softmax confidence score to measure aleatoric uncertainty.
alea Uij
conf Uij
maxxPe,...
(43)
Ensembles-based [Lakshminarayanall et al., 2017). Deep Ensembles-based method learn M different versions of network weights ()( ... M(M) and aggregates the predictions of these versions. The aleatoric uncertainty is measured by the softmax entropy of the mean of the predictions from different network weights. The epistemic uncertainty is measured by the variance between the model predictions.
alea i.j
(
1
Σ
M
(m)
log
1
Σ
M
(m)
m=1
.M
m=1
(44)
epis Vi
var((pe..))
(m) 1M (Pe.)))==1))
c argmax
M
Pis
(m)
M
m=1
23

---

Published as a conference paper at ICLR 2025
(m) where Pi.j [0, 1Jc refers to the predictions of BEV network based on the network weights O(m). We use M = 3 for experiments.
Dropout-based (Gal & Ghahramani 2016). Dropout-bart baseu methods approximate Bayesian inference based on activated dropout layers. conducts multiple stochastic forward passes with active dropout layers test time. Similar to Deep ensembles, the entropy of expected softmax probability and variance of multiple predictions are used as aleatoric and epistemic uncertainty scores, respectively. We use M 10 for experiments.
Energy-bounded Learning (EB). For fair comparison, we also include OOD-exposure for an energy-based model. With the same strategy as Liu al. (2020), we consider energy-bounded learning for OOD detection. This approach entails fine-tuning the neural network by assigning lower energy levels ID data and higher energy levels to OOD data. Specifically, the classification model is trained using the following objective function:
[CoEd = [x DDD H(p, y) + 7 C
(45)
where p is the predicted class probabilities and Lb is the regularization loss defined in terms of energy:
L E X X DDraan mmax (0
X....
energy min)) ))
(46)
L (X (Xoue Dirain (max (0 Mout u (nergy))
(47)
where the Min; Mout are two bounded hyperparamters, Details be found in Liu et al. (2020) We
can also replace cross entropy loss with focal loss for better calibration:
FooEB = E(x, y)D Focal y) + n L
(48)
where Focal y) = Ee Ve(1 Pc)7 log
A.3 IMPLEMENTATION DETAILS
For the model training, we will use equation 49 for the original ENN model optimized by Bayesian loss and equation for our proposed UFCE-based models.
CUCE-ENT-ER CUCE-ENT + ALER
(49)
CUPCE-ER = CUCCE + XCER
(50)
It is important to highlight that _R and FNTT represent distinct concepts within our framework. _R identified a targer loss, assumes that the level-2 ground truth for pseudo OOD samples corresponds to flat Dirichlet distribution, reflecting the actual discrepancy between predictions and ground truth. Conversely, CENT is characterized as a surrogate loss, fulfilling an auxiliary role. Through the employment of the ENT regularizer, the model is implicitly encouraged to predict smoother distribution for all training samples.
A.3.1 HYPERPARAMETERS
Fixed hyperparamters: We set the weight of the entropy regularization term in the UCE loss (3 in Equation 5) to 0.001. The energy-bounded regularization weight in the energy model (n in Equations 46 and 48 is set 0.0001. For all models where "vehicle" is considered the positive class, we assign a positive class weight of 2. In scenarios where "drivable region" is a positive class, the class weight set to We employ learning rates of 4 10-3 for focal loss variants and 1 x 10-3 for cross-entropy variants, using the Adam optimizer with a weight decay of 1 x 10-7 The batch size is set to 32 across all experimental scenarios.
We tune three regularization weights related to proposed model: X, ç. and y. Due to computational constraints, we did not perform full grid search over these hyperparameters. Instead, task, specifically optimizing for the AUPR metric evaluated on the validation set. First, we performed coarse grid search over À E {0.001, 0.005, 0.01, 0.05, 0.1} with & = and 7 0, finding that
Hyperparameters tuning strategy:
conducted step-by-step search based on the pseudo-OOD detection
24

---

Published as a conference paper at ICLR 2025
À = 0.01 generally provided good performance across most scenarios. Next, with À fixed at 0.01, we tuned Ç E {8, 16, 32, 64, ] with 7 = O, and selected Ç = 64 for the remaining experiments. Finally, we fine-tuned y {0.05, 0.5, 1.5) while keeping À 0.01 and 64 fixed. The detailed hyperparameters used in our reported results are listed in Table 6
Table 6: Hyperparameter Settings for Different Datasets and Methods
Dataset
Method
LSS
0.5
0.01
64
CARLA
CVT
0.05
0.01
64
Simple-BEV
LSS
0.01
nuSoemes
CVT
0.05
0.01
64
Simplo-BEV
0.05
0.01
64
LSS
0.01
64
Lyft
CVT
0.01
64
Simple-BEV
0.05
0.01
64
Hyperparameters Sensitivity Analysis:
To evaluate the robustness of our model with respect to its hyperparameters, we conducted comprehensive sensitivity analysis on the nuScenes dataset with vehicle segmentation task using the LSS model the backbone.
We first analyze the impact of the hyperparameter À on model performance, with results presented in Table 7 Intuitively, larger À corresponds to higher pseudo-OOD regularization weight, encouraging the model to predict a uniform Dirichlet distribution for pseudo-OOD pixels. However, this may also negatively affect the original task performance. Experimentally, observe that larger À values lead to decline in pure segmentation and calibration performance. Due to the observed significant variations in segmentation metrics and misclassification results are closely tied to segmentation performance, we ignore the misclassification detection performance here. For OOD detection, we emphasize the AUPR metric key indicator. With increasing X. we initially observe an improvement in AUPR (accompanied by higher AUROC and lower FPR95), followed by a decline as > becomes too large. This indicates that while moderate pseudo- regularization enhances OOD detection, overly strong regularization can degrade overall performance.
Table 7: Hyperpameter sensitivity analysis for varying A. We conduct the vehicle segmentation with LSS as the backbone on nuScenes with 64, 7 0.5.
Pure Classilication
Misclassification
OOD Delection
toU 1
ECE 4
AUROC 1
AUPR
FPR95
AUROC
AUPR
FPR95 |
0.001
0.361
0.00335
0.895
0.320
0.192
0,801
0,224
0.410
0.005
0.357
0.00327
0.890
0.318
0.199
0.833
0.225
0.340
0.01
0.343
0.00035
0.925
0.324
0.191
0,743
0.165
0.383
0.05
0.304
0.01090
0.936
0,300
0.233
0,840
0.115
0.276
0.259
0.919
0.860
0.230
We then analyze the impact of the hyperparameter 7 on model performance, with results presented in T8] We observe that varying " does not significantly affect segmentation performance. However, selecting an appropriate value for y leads better calibration performance, as evidenced by the lower Expected Calibration Error (ECE) score, and improved aleatoric uncertainty prediction, indicated by better misclassification detection performance. This demonstrates that the UFCE loss can enhance both calibration and misclassification detection performance. Besides, we observe that OOD detection performance quite sensitive to the selection of o.
Finally, we investigate the effect of hyperparameter & on model performance, with results presented in Table 9 First, We observe that increasing & led to a slightly higher IoU), indicating improved segmentation accuracy (then omit the misclassification detection performance considering the correlation between these two tasks). We noted that higher values of & resulted in increased AUROC on OOD detection performance in general.
25

---

Published as a conference paper at ICLR 2025
Table 8: Hyperpameter sensitivity analysis for varying y. We conduct the vehicle segmentation with LSS as the backbone on nuScenes with 0, X = 0.1.
7
Classification
Misclassification
0OD Detection
Toll f
ECEL
AUROC1
ALPPR
FPR95J
AUROCT
AUPR
FPR95
0.05
0.349
0.00378
0.913
0.321
0.195
0.876
0.184
0.278
0.348
0.00166
0.901
0.328
0.187
0.774
0.341
0.358
1.5
0.349
0,01170
0.924
0.331
0.183
0.634
0.334
0.473
Table 9: Hyperpameter sensitivity analysis for varying &. We conduct the vehicle segmentation with LSS as the backbone on nuScenes with 0.5, = 0.01.
Pure Classification
Misclastification
0DD Detection
loU
ECE 4
AUROC1
AUPR
4
AUROC
AUPR 1
0 0.348
0.001666
0.901
0.328
0.187
0.774
0.341
U.358
8
0.349
0.00079
0.913
0.327
0.186
0.785
0.273
0.348
16
0.351
0.00089
0.910
0.321
0.186
0.747
0.328
0.390
32
0,350
0.00154
0.900
0.324
0.192
0.822
0.321
0,315
64
0.351
0.00133
0.904
0.321
0,187
0.797
0.315
0.364
128
0.353
0.001337
0.893
0.818
A.4 DATASET DETAILS
Datasets. We consider both synthetic and real-world datasets for our experiments and details are presented in Table 10 For synthetic data, utilize the widely recognized CARLA simulator (Doso vitskiy et al. 2017) to collect our dataset, which will be made available upon request due to its large size. Our simulated CARLA dataset features five towns with varied layouts and diverse weather conditions to enhance dataset diversity. In terms of real-world data, we employ the nuScenes (Caesar et al. 2020) and Lyft (Kesten et al. 2019) dataset, which is also used in the evaluation of the two segmentation backbones used in our paper: LSS and CVT, as well as an updated leaderboard. We choose not to use the KITTI (Behley et al. 2019) dataset for several reasons. KITTY primarily features suburban streets with low traffic density and simpler traffic scenarios, with annotations limited the front camera view instead of full 360-degree perspective. It also lacks radar data and is designated for non-commercial use only. In contrast, nuScenes aims to enhance these features by providing dense data from both urban and suburban environments in Singapore and Boston.
Below, we provide detailed descriptions of each dataset: (1) nuScenes. This dataset comprises of 35661 samples. It offers 360 view around the ego-vehicle through six camera perspectives, with each view providing both intrinsic and extrinsic details. We resize the camera images to 224x480 pixels, and produce Bird's-Eye--o View (BEV) labels of 200x200 pixels for analysis. (2) Lyft. This dataset comprises of 22,888 samples. It offers 360 view around the ego-vehicle through multiple camera perspectives, providing both intrinsic and extrinsic details for each view. We resize the camera images to 224x480 pixels and generate --ydes-Eyye-- View (BEV) labels at resolution of 200x200 pixels for analysis. (3) CARLA. In this simulated environment, six cameras are installed at 60-degree intervals around the ego vehicle, emulating the setup found in nuScenes. The simulation includes various weather conditions (e.g., Clear Noon, Cloudy Noon, Wet Noon) and urban layouts (e.g., Town10, Town03) to enrich the dataset's diversity. For each camera, we capture and record intrinsic and extrinsic information. The dataset consists of 224x480 pixel camera images and 200x200 pixel BEV labels, similar nuScenes. total, 40,000 frames are collected for training, with an additional 10,000 frames designated for validation.
True OOD and pseudo OOD setting. We evaluate the quality of the predicted epistemic uncertainty through an nut-of-distribute (OOD) detection task. In-Distribution (ID) pixels are those that belong to the segmentation task with clear labels, such as "vehicle" and "background". True-OOD pixels represent a semantic shift from the training data; they only appear in the test dataset and were not seen during training. The OOD detection performance reported in this paper identifies pixels as either ID or true OOD. Pseudo-OOD pixels are artificially designated as OOD during training to regularize the model. helping it learn to distinguish between ID and OOD pixels. These pscudo OOD pixels exist in the training and validation sets, and hyperparameter tuning is based the pseudo-OOD)
26

---

Published as a conference paper at ICLR 2025
Table 10: Dataset details with diversity descriptions.
Camera Space
BEV Space
Datuset
Diversity Description
Camerns
Positions
Resolution
FOV
Scale
Resolution
CARLA
60 degree intervals around the ego vehicle
224 480
90
100 m
200
10 weather conditions (e-g.. Clear Noon. Cloudy Noon, Wet Noon) and urban layouts (e.g.. Town10, Town03)
Diverse
100 m around vehiele
200
muSeenes
Front, Front-Leff, Fromt Right, Rack--Teft
224 480
70,
100 m
200
1,000 diverse scenes collected over various weather, time day, and traffic conditions
100 m around vehicle, resolution
200
Back-Right, Back
Lyft
Lefl, Front Right,
224
driving scenes with varying traffic densities environments Alto, Californi
480
150 m around vehicle
Side-Left, Side-Right, Back
detection performance on the validation set. The detailed setting we used in this paper is presented in Table II
Criteria used to select the pseudo-OOD data. We initially adhered the criteria outlined in the benchmark study by Franchi et al. (2022), seminal work in the field of autonomous driving, to identify candidate OOD classes. In this context, OOD data typically encompasses less frequently encountered dynamic objects (e.g., motorcycles, bicycles, bears, horses, cows, elephants) and static objects (e.g., food stands, barriers) that are distinct from primary segmentation categories like vehicles, road regions, and pedestrians. These objects were designated candidate OOD classes. Subsequently, in our experiments, we randomly partitioned these candidate classes into pseudo-OO1 and true OOD categories.
is worth noting that the OOD benchmark dataset MUAD httpss//muad-datasee cantaset.github.ioo/ provided by Franchi al. (2022) collected using simulator based on front-camera imagery for image segmentation. However, it does not include a collection of images from multiple cameras or labels for BEV segmentation. To address this, we adopted similar procedure and generated a BEV segmentation dataset with OOD objects using the well-established CARLA simulator.
Table 11: Dataset configurations for different settings.
Setting
1 (Default)
Setting
Setting
Dalaset
ID
Pscudo-OOD
True-
ID
Pscude-OOD
True-
ID
Pscudo- 0OD
Truc- OOD
OOD
CARLA
vehiele
bears, horses,
doer
vehicle
bears, horses, cows, elephants
kangaroo
n.a.
n.a.
n.a.
cows, elephants bicycle
nuScenes
vehicle
motorcycle vehicle
traffic pushable/pullable objects, motorcycles
barriers
drivable region
hicycle
matorcycle
Lyft
vehicle
bicycle
motorcycle n.a.
n.A.
Splits for evaluation. (1) nuScenes: Considering that only public training and validation sets are available, we report results on the validation set, following standard practice in the literature. Notably, we remove all frames that contain true OOD pixels from both the training and validation sets used for training and evaluation, respectively. For models that are not exposed to pseudo-OOD pixels during training, we also remove frames containing pseudo-OOD objects from the training set. (2) Lyft. We use the same split strategy nuScenes. (3) CARLA. To introduce pscudo and true OOD objects into our synthetically generated dataset, we employ 3D models of specific objects, integrating these models into the CARLA simulator scenes to generate custom objects. We first gather clean datasets for training, validation, and testing separately, where each frame exclusively contains in-distribution objects. Then, gather separate train-pseude and val- val-pseudo datasets, incorporating pseudo-OOD objects into each scene. Finally, we collect a dataset featuring true OOD objects, specifically deer.
In Table 12 provide the number of frames and OOD pixel ratio for splitter datasets.
27

---

Published as a conference paper at ICLR 2025
Table 12: Dataset split information
train
train-aug
val
val-ong
test
Dataset
No. frames
No. frames
Pscudo- OOD
No. frames
No. frames
Pseudo-
No.
True-
OOD
frames
OOD
ratio
rabio
ratio
CARLA
40,000
80,000
0.11%
40,000
80,000
0.14%
2,000
0.07%
nuSoenes
19,208
23,831
0.017%
3,878
5,082
0.012%
1,204
0.02%
Lyft
11,487
16,184
0.18%
4,431
6,094
0.016%
0.009%
Dataset split principle: "train" and "val" datasets only contain ID pixels. "train-aug" and "val-aug" contain ID and pscado-OOD pixels. and "test" contains both ID and true OOD pixels. For models without pseul-OOD expusure during training, datascts. For models with pscudo-OOD exposure, we utilize the "train-aug" and "val-aug" datasets. All models are evaluated on the "teat" dataset. calculating pure segmentation misclussficattinn performance, mask the pixels solely on the pixels.
the "Irin" and "val"
CARLA Data generation process We first introduce key concepts for dataset generation. These are listed from the bottom up terms of scale to make everything easy to understand. Then we introduce the main control loop.
Tick: A tick is single unit of simulation in the simulator. Every tick, the position of vehicles, states of sensors, weather conditions, and traffic lights are updated. A tick the smallest denomination that
Frame: A frame is single sample of data. Each frame consists of 6 RGB images and single BEV semantic segmentation image. The RGB images have a resolution of 224x480, and the BEV image has a resolution of 200x200. The BEV image covers an area of 100 meters x 100 meters.
We define a scene as a time-frame of N ticks on a specific map. In the beginning of the scene, the map is loaded. Then, 5 vehicles with sensors attached (ego vehicles) spawned at random locations. These vehicles have 6 RGB cameras attached to them at 60 degree intervals These intervals represent front, front left, front right, back, back let, and back right. We also gather the depth ground truth maps at the same intervals. We place a birds-eye-view segmentation camera above the vehicle that captures an area of 100x 100m a resolution of 200x200 pixels. Then, 50 non-ego vehicles are spawned in random locations. Then, if needed, 40 OOD objects are spawned random positions. The simulator will run through each of the N ticks one by one in sequential manner, and will save one frame for each ego vehicle every five ticks.
Scene:
Main control loop: The main control loop runs through M scenes, systematically varying weather conditions and urban layouts to generate diverse dataset.
Weathers Condition: ClearNoon, CloudyNoon, WetNoon, MidRainyNoon, Soft- RoinyNoon, ClearSunset, CloudySunset, WetSunset, WerCloudSunset, SoftRainySun- set
Urban Layouts: Town10, Town07, Town05, Town03, Town02. Specifically, we have
Town 02: A small simple town with mixture of residential and commercial buildings.
*
Town 05: Squared-grid town with cross junctions and a bridge. It has multiple lanes per direction. Useful perform lane changes.
Town 07: A rural enviromment with narrow roads, corn, barns and hardly any traffic lights.
* Town 10: downtown urban environment with skyscrapers, residential buildings and an ocean promenade.
Within each scene:
Weather Variation: Every N/10 ticks, the weather changes to the next condition in the set. This approach ensures that each weather condition is represented equally throughout the scene.
Urban Layout Selection:
The town for each scene is selected in a cyclic manner using the following logic: town = towns [i]; = (i + 1) mod 5. where townsis the list of urban layouts, and is the index that cycles through the towns. The modulo
28

---

Published as a conference paper at ICLR 2025
operation ensures that after reaching the last town, the index wraps around to the first, providing a continuous loop through the available urban layouts.
After each scene: All objects are destroyed, and the town environment is cleared. A new scene begins by loading the next town as determined by the selection process. The number of frames can be calculated with (N/5) M * (No. of ego vehicles).
A.5 ADDITIONAL EXPERIMENTS
In this section, we first provide clear definitions and calculations for the evaluation metrics used in study, ensuring transparency and reproducibility. Then we introduce the additional experimental results.
Pure segmentation via IoU: Intersection over Union (loU) is used to evaluate the segmentation
performance of the models. It is calculated as the IOU for positive class, which is defined as:
True Positives
IoU True Positives + False Positives False Negatives
Higher IoU values indicate better segmentation performance, as they reflect accurate predictions for both the object and background regions.
Calibration via ECE: Expected Calibration Error (ECE) measures how well the predicted probabili- ties align with the true likelihood of correctness. It is computed by dividing the confidence scores into M bins and calculating the weighted average of the difference between accuracy and confidence for each bin:
M
Bml
ECE
Jacc (B.) - conf Bmm)
where Bm represents the set of predictions in bin m, [Bm] is the number of samples in the bin, the total number of samples, acc(B,, the aceuracy, and conf( B.,) is the average confidence. Lower ECE values indicate better calibration. We use M = 10 for experiments.
Misclassification detection:
To evaluate misclassification detection, we treat misclassified pixels as the positive class and do the binary classification task with the aleatoric uncertainty as the score. Metrics such as Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision-Recall curve (AUPR) are used.
For out-of-distribution (OOD) detection, we assess the model's ability to differenti- ate OOD pixels from in-distribution (ID) pixels. Similar to misclassification detection, AUROC and AUPR are used as evaluation metrics, and OOD pixels are positive classes with epistemic uncertainty the score.
OOD detection:
A.5.1 FULL EVALUATION ON LYFT
Results on Lyft dataset. Table 13 presents the comprehensive results on the Lyft (Kesten et al. 2019) dataset using LSS and CVT model backbones. Our findings on Lyft align with those from our studies on CARLA and nuScenes. Overall, our proposed model employing the UFCE loss consistently outperforms the UCE loss in terms semantic segmentation accuracy, calibration, and misclassificattin detection. Furthermore, incorporating epistemic uncertainty scaling and pseudo- OOD exposure significantly enhances OOD detection performance.
A.5.2 QUANTITATIVE EVALUATION ON CALRA (FULL)
Result on calibratton/miscllasffcaaaai detection on CARLA: Table 14 presents the segmentation, calibration, and misclassification detection results on the CARLA dataset using LSS and CVT as model backbones. We observe that our proposed model demonstrates the best OOD detection performance, the second-best calibration performance, and comparable results segmentation and misclassification detection compared to other models.
A.5.3 QUANTITATIVE EVALUUTION ON NUSCENES (ROAD SEGMENTATION)
Result on OOD detection on nuScenes dataset for road segmentation: Table 15 presents the segmentation and calibration performance the nuScenes dataset, with "road" designated as the
29

---

Published as a conference paper at ICLR 2025
Table 13: Evaluation on Lyft dataset for vehicle segmentation. highlighted in red and blue.
Best and
Runner-uy up
results are
LSS
CVT
Baseline
Loss
Pure Classification
Misclssiiiatiion
OOD Detrction
Pure Classifiction
Misclssifiation
OOD Detoction
ToU
BCE.
AUROC1
AUPR
AUROC
AUPR
IoU
ECE1
AUROCT
AUPR
AUROC
AUPR
Without pscudo ODD
Entropy
CF
0.353
0.01530
0.876
0.316
0.555
0.00124
0.322
0:00519
0.934
0.311
0.795
0.00259
Focal
0.422
0000831
0.928
0.341
0.638
0.00147
0.329
0.00630
0.937
0.317
0,752
0.00216
Energy
CE
0.393
0.01530
0.876
0.316
0.594
0.00127
0.322
0,00619
0.934
0,311
0,789
0.00266
Focal
0.422
0100631
0.928
0.342
0.629
0.00133
0.329
0.00300
0.937
0.317
0.729
0.00207
Ensembll
CE
0.411
0,00993
0.883
0.317
0,408
0.00087
0.351
0.00402
0.948
0.354
0.399
0.0008
Focal
0.446
0.00406
0.917
0.00071
0.306
0.00135
0.535
0,00093
Dipport
0301700
0.570
0.317
0.00047
0.322
0,362
0.925
0.318
0.00057
ENN
0.00855
0.762
0.00086
0.541
0.00235
UFCE
0.427
0.00766
0.547
0.00069
0.383
0300344
0.665
0U00213
OUD
Esergy
0.442
0002100
0.500
0.341
0.724
0.133
0.385
01957
0,335
0,802
0.019
Focal
00:45
0.001:9
0.922
0.355
0,757
0,042
0.429
0,00516
0.964
0.349
0.761
0.024
ENN
UCE
0.499
0.00650
0.830
0.286
0.877
0.149
0.384
0.00588
0.905
0, 0,306
0.913
0.060
Ous
0.467
0.00720
0.520
00.79
00.36
0.184
0.419
6:00085
01934
0.340
0.386
0.145
Observations:
The proposed UFCE loss the ENN model consistently outperforms the commonly used UCE loss across 20/24
metrics, 2. Our proposed model (last rew) shows the best OOD detection performance across LSS and CVT backbone, the specifically on AUPR metric.
Table 14: Calibration and Misclassification detection performance on the CARLA dataset for vehicle segmentation Best and Runner-up results are highlighted in red and blue.
CVT
Pure Classificntion
Misclassification
Pure Classification
Misclassification
IoU'
BCRJ
AUROC
AUPR
FPR95
loU
ECE,
AUROC +
AUPR 1
Without pseudo OOD
Entropy
CE
0.403
0.00309
0.928
0.272
0.222
0.361
0.00183
0.952
0.246
0.215
Focal
0.403
0.00309
0.928
0.272
0.221
0.435
0.00146
0.970
0.265
0.148
Energy
0.403
0.00309
0,952
0.246
Focal
0.403
0.00309
0.928
0.272
0.221
0.435
0.001466
0.970
0,265
0.148
Fasemble
0.433
0.00202
0.941
0.266
0.00160
0.961
0.222
0.227
Focal
0.462
0.00140
0.971
0.270
0.157
0.471
0.00135
0.973
0.260
0.154
CE
0.409
0.00277
0,922
0.269
0.243
0.366
0.00183
0.944
0.238
0.235
Drepout
Focal
0.422
0.00135
0.956
0.295
0.167
0.442
0.00142
0.966
0.264
0.167
ENN
UCB
0.407
0.00212
0.817
0.252
0.345
0.398
0.00098
0.917
0,253
0.192
0.424
0.00033
0.933
0.263
With pseudo OOD
Energy
CE
0.441
0.00249
0.314
0.00124
0.977
0.264
0.119
Focal
0.469
0.00588
0.979
0.277
0.094
0.465
0.00096
0.980
0.266
0.090
0,428
0.00131
0.00457
0.961
0.263
Ours
0.459
0.00040
0.950
0.303
0.099
0.468
0.00180
0.967
0.294
0.097
Involving pseudo-OOD the training phase does impact pure segmentation, calibration and misclassification detection. 2. Without pscudo-OOD, the proposed UFCE loss for the ENN model consisteurly ourperforms the commonly used UCE loss accoss 10 mettics. 3. No single model consistently performs across metrics, Focal consistently performs better than CE.
Observations:
positive class for segmentation. To verify the calibration improvements brought by the proposed UFocal loss, all evaluations are conducted on the clean validation dataset. We observe that our proposed UFocal loss consistently achieves lower ECE and higher loU scores, indicating better calibration performance and segmentation accuracy compared to the UCE loss. Additionally, the misclassification detection performance is also superior to that of the UCE loss. Compared to other baselines, our model demonstrates top-tier performance.
A.5.4 QUANTITATIVE EVALUATION ON SIMPLE- BEV ON NUSCENES
Full Results with SiM backbone on nuScenes dataset for vehicle segmentation. Table [6 presents the full results with the SimpleBEV (Harley et al. 2023) the backbone. While the original
30

---

Published as a conference paper at ICLR 2025
Table 15: Segmentation, Calibration and Misclassification detection on road segmentation for Runner-up results are highlighted red and blue.
nuScenes. Best and
LSS
CVT
Model
Loss
Pure Classification
Misclasifiication
Pure Classification
Misclassification
Road loUt
ECEμ
AUROCT
AUPRI
Road IoU*
BCEJ
AUROC
AUPR
CE
0.756
0.0448
0.870
0.330
0.637
0.0495
0.835
0.354
Entropy
Focal
0.763
0.0266
0.886
0.341
0.678
0.0458
0.848
0.345
Ensemble
CE
0.776
0.0156
0.883
0.309
0.656
0.0269
0.840
0.324
Focal
0.769
0.0308
0.882
0,314
0.698
0.0263
0.853
0.324
CE
0.744
0.0455
0.867
0.329
0.624
0,0573
0.831
0.351
Dropout
Focal
0.752
0.0300
0.883
0.337
0.670
0.0511
0.844
0.343
UCE
0.752
0.0417
0.783
0.291
0.639
0.0455
0.806
0.335
ENN
UFCE
0.760
0.0250
0.839
0.319
0.679
0,0438
0.828
0.336
Observations: 1. The propesed UFCE loss for the ENN model consistently outperforms the commonly used UCB Joss across all metrics. showing improvements of 2.6% segmentation loUl calibration ECE, 3.9% in misclassiiiction ficationn AUROC, and 1.5% in AUPR LSS and CVT backboone. 2. For road detection, there no clear evidence showing that Focal loss performs better than classic CE loss in of segmentation aleatoric uncertainty prediction.
paper reported vehicle segmentation loU of 44.7 nuScenes, report 38.2. This discrepancy arises because we adopt a consistent experimental setup across all backbones, including LSS and exclude frames containing
OOD objects from the training and use camera images with a resolution of 224 x 480 without any image augmentations. Additionally, we employ bilinear sampling as the lifting strategy, EfficientNet-BA as the network backbone, and batch size of 16.
CVT. Specifically,
We observe that (1). The proposed model achieves the second-best performance in pure segmentation, with gap of only 0.1 % from the best baseline, while outperforming all models in calibration. (2). All models demonstrate comparable performance misclassification detection. (3). For OOD detection, the proposed model achieves the best performance in AUROC and AUPR, with 12% improvement in AUPR over the second-best model and FPR95 is only 0.008 higher than the best baseline.
Table 16: Simple-BEV backbone: Segmentation, Calibration, Misclassification detection, OOD
detection performance for vehicle segmentation on nuScenes.. Best and Runner-up results are highlighted in red and blue.
Pure Classification
Misclassification
OOD
Detection
model
loss
IoU
ECE!
AUROCT
AUPRT
FPR951
AUROC1
AUPR
FPR95
Without pscudo OOD
0.352
0.00721
0.936
0.326
0.196
0.684
0.001
0.816
Entropy
Focal
0.366
0.00697
0.942
0.322
0.193
0.661
0.001
0.841
CE
0.352
0,00721
0.936
0.326
0.196
0.638
0.000
0.819
Energy
Focal
0.366
0.00698
0.942
0.322
0.193
0.614
0.000
0.835
0.373
0.00456
0.946
0.322
0.197
0.534
0.920
Ensemble
Focal
0.388
0.00453
0.946
0.322
0.185
0.504
0.000
0.937
CE
0.350
0.00617
0.934
0.325
0.207
0.458
0.000
0.964
Dropout
Focal
0.364
0.00623
0.940
0.323
0.200
0.523
0.000
0.931
UCE
0.361
0.00647
0.844
0.285
0.286
0.657
0.000
0.832
ENN
UFCR
0.370
0.00296
0.848
0.294
0.279
0.604
0.000
0.831
With pscudo OOD
CE
0.360
0,00586
0.946
0.332
0.183
0.820
0.071
0.326
Energy
Focul
0.372
0.00606
0.947
0.332
0.186
0.838
0.058
0.325
UCE
0.364
0.00455
0.882
0.314
0.207
0.914
0.215
0.272
ENN
Ours
0.372
0.00315
0.896
0.322
0.187
0.845
0.319
0.356
31

---

Published as a conference paper at ICLR 2025
A.5.5 ROBUTNESS ON SELECTION OF PSEUDO-OOD
Robustness the selection of pseudo-OOD (cont.). We investigate how the similarity between pseudo-OOD and true OOD affects epistemic uncertainty predictions. For this, we use two pseudo- OOD and true-OOD pairs for nuScenes and CARLA, with detailed settings presented in Table Generalization results for nuScenes are shown in Table 4 with the main discussion provided in Section 4.2 Generalization results for CARLA are presented Table where we evaluate OOD detection performance using "kangaroo" as the true OOD and "bears, horses, cows, elephants" as pseudo-OO1 OODs, For this true/pseudo-OOD pair, our proposed framework consistently outperforms others across all six metrics, achieving average AUPR improvement of 3.2% over the runner-up. sharing the same observation on nuScenes.
Table 17: Robustness Analysis (selection of pseudo-OOD): OOD detection performance for vehicle segmentation on CARLA ("kangaroo" as true OOD). Best and Runner-up results are highlighted in red and blue.
LSS
CVT
Model
Loss
AUROCT
AUPR
EPR95
AUROCT 1
AUPR
FPR95 J
Without pscudo OOD
CE
0.665
0.005
0.768
0.767
0.007
0.691
Entropy
Focal
0.725
0.006
0.783
0.007
0.675
CE
0.672
0.005
0.742
0.749
0.006
0.69
Energy
Focal
0.717
0.006
0.726
0.755
0.007
0.675
Ensemble
CE
0.485
0.001
0.963
0.487
0.001
0.962
0.449
Dropour
CE
0.432
0.001
0.966
0.388
0.001
0.962
Focal
0.401
ENN
UCE
0.623
0.004
0.779
0.68
0.005
0.784
LFCE
0.593
0.712
With pseudo OOD
CE
0.746
0.045
0.556
0.746
0.034
0.442
Energy
Focal
0.727
0,049
0,489
0.786
0.051
0.425
UCE
0.818
0.097
0.417
0.911
0.077
0.353
ENN
Ours
0.882
0,111
0.368
0.914
0.127
0.322
Table 18 presents the OOD detection performance across various pscudo/true OOD pairs. The results support our intuition that higher similarity between true and pseudo-OOD pairs leads to improved OOD detection performance, suggesting that identifying representative pseudo-OODs is a promising direction for advancing OOD detection tasks. Notably, our proposed model consistently outperforms the best baseline methods across all eight scenarios, achieving improvements of up to 12.6% in AUPR. This highlights the robustness of our approach, even in settings with less similar OOD pairs.
Table 18: OOD detection performance (AUPR 1) for similar and dissimilar OOD pairs across datasets and backbones, comparing our model to the best baselines and its ablated variants.
Similarity between
LSS
CVT
Dataset
True/Pseudo OOD
Best Baseline
Ours
Best Baseline
Ours
Similar
31.5
33.5
21.2
26.9
nuScenes
Dissimilar
20.8
21.5
11.7
15.3
Similar
14.7
20.4
11.1
23.7
CARLA
Dissimilar
9.7
11.0
7.7
12.7
A.5.6 ROBUTNESS ON CORRUPTED DATASET
Robustness nuScenes-C. C. nuScenes- Xie et al. 2024 introduces various corruptions to the validation set of the nuScenes dataset, comprising eight types of corruption, each with three levels of severity: easy, mid, and hard. Brightness, Dark, Fog, and Snow represent external environmental
32

---

Published as a conference paper at ICLR 2025
dynamics, such as illumination changes extreme weather conditions. Motion Blur and Color Quant simulate effects caused by high-speed motion and image quantization, respectively. Camera Crash and Frame Lost model camera malfunctions.
Using a model trained on the clean nuScenes training dataset (excluding frames containing true OOD pixels), evaluate its performance on the corrupted validation set to assess its robustness to domain shifts. We compare our model against the most relevant baseline, "ENN-UCE," and the model that generally performs best in OOD detection tasks, "Energy-Focal". The results are presented in Table 19
For misclassification detection, our model achieves the highest AUROC in 14 out of 21 scenarios and the second highest in the remaining 7 scenarios, Our model archives the highest AUPR in 13 out of 21 scenarios and the second highest the remaining 8 scenarios. Our model consistently outperforms the ENN-UCE baseline across all 21 scenarios on both AUROC and AUPR. For OOD detection, our model achieves the best AUPR in 16 out of 21 scenarios and the second- best AUPR for the other 5 scenarios. Our model archives the best AUROC in 8 out of 21 scenarios and second-best in the remaining 13 scenarios. We note that AUROC and AUPR offer different perspectives to measure the quality of ranking on data points (BEV pixels in our context) for separating positives and negatives. relatively lower AUROC but higher AUPR for our method in some scenarios suggests our model identifies more true positives among top-ranked pixels than the ENN-UCE model, while ENN-UCE model better separates true positives and negatives among lower-ranked BEV pixels. It indicates that data corruptions in these scenarios may affect the quality of epistemic uncertainty quantification for some lowly ranked pixels, but not for the top ranked ones. However, high AUPR is particularly important in applications where human experts manually verify top-ranked misclassified instances or anomalies. Given the high cost of manual verification, ensuring a high rate of true positives among the top-ranked data points makes AUPR a more suitable metric in such cases.
We observe greater performance drop under domain shifts with higher corruption severity, consistent with the findings Xie al. (2024), where significant performance degradation was noted in the segmentation task using CVT on nuScenes compared to the clean dataset. Additionally, sensor-driven distortions, such as color quantization and motion blur, have the least impact, while snow and camera crashes cause the most severe performance degradation. These experiments highlight the need for robust models or optimization strategies, which we plan to explore as a future direction.
A.5.7 ROBUTNESS ON DIVERSE CONDITIONS (CARLA)
We evaluate our models on diverse subsets of the test sets, considering variations in urban layouts and weather conditions. In the CARLA dataset, we include four towns across 10 weather conditions, with details on dataset generation provided in Appendix A.44
Evaluation results on different towns are shown in Table 20 Towns 2 and 5 represent denser ur- ban layouts compared to Towns 7 and 10. We observe consistent performance across each subset, aligning with the evaluations conducted on the full CARLA dataset. Specifically, our model con- sistently outperforms all baselines in OOD detection tasks across all towns, while also achieving better segmentation, calibration, and misclassification detection compared to the standard UCE loss. Furthermore, we observe performance gaps across urban layouts, highlighting the need to improve model robustness in diverse environments.
While the results align with the conclusions drawn from evaluations the full dataset when comparing our proposed model to the baselines, we also observe performance variations across different weather conditions. Interest- ingly, the results challenge the intuition that the best performance would occur under "ClearNoon" conditions. Notably, OOD detection performance is positively correlated with segmentation per- formance, which may be attributed to clearer environments improving segmentation quality and, consequently, OOD detection.
Evaluation results on varying weather conditions are shown in table 21
22 23
A.5.8 ROBUTNESS ON MODEL INITIALIZATIONS
We report the model variance for our proposed model in Table 24 and the evidential UCE baseline. We randomly initialize the model three times and the variance is within 3%.
33

---

Published as a conference paper at ICLR 2025
Table 19: Robustness Analysis (data corruption): evaluation on estimated uncertainty with CVT as the backbone on nuScenes-C.
Misclassification
OOD Detection
Coruptiin Type
Severity
Model
AUROCT
AUPR
FPR95 1
AUROC
AUPR f
FPR95
Energy-Pocal
0.955
0.321
0.196
0.860
0.024
0,319
None
Clean
ENN-UCE
0.919
0.313
0.227
0.921
0.212
0.306
Ours
0.934
0.321
0.196
0.928
0.269
0.244
Energy-Foca
0.863
0.226
0,436
0.710
0.016
0.581
Easy
ENN-UCE
0.823
0.195
0.478
0.859
0.052
0.459
Ours
0.878
0.318
0.365
0.787
0.109
0.525
Energy-Foca
0.785
0.140
0.644
0.615
0.011
0.742
CameraCrash
Mid
ENN-UCE
0.727
0.117
0.659
0.723
0.008
0.734
0.867
0.351
0.538
0.651
0.067
0.700
Energy-Foca
0.796
0.151
0.605
0,623
0.004
0,809
Hard
ENN-UCE
0.751
0.140
0.611
0.791
0.003
0.756
Ours
0.813
0.251
0,440
0,663
0.027
0.773
Energy-Focal
0.880
0.235
0.411
0.712
0.021
0.536
Easy
ENN-UCE
0.846
0.227
0.423
0.862
0.118
0.483
Ours
0.880
0.291
0.364
0.805
0.148
0.533
Baeryy-Foca
0.783
0.132
0.654
0.316
0.003
0.792
Foamel.ost
Mid
ENN-UCR
0.727
0.117
0.673
0.743
0.044
0.790
Ours
0.848
0.290
0.503
0.605
0.038
0,843
Energy-Foue Focal
0.728
0.112
0.723
0.002
0,871
Hard
ENN-UCE
0.657
0.074
0.785
0.674
0.032
0.799
Ours
0.826
0,280
0.572
0.55y
0.024
0.898
Energy-Foue Focal
0.949
0.315
0.209
0.850
0.025
0.354
Easy
ENN-UCE
0.905
0.307
0.246
0.926
0.174
0.301
Ours
0.925
0.310
0.219
0.923
0.243
0.275
Energy-Focal Focal
Energy-
0.929
0.297
0,263
0.825
0.028
0.407
ColorQuant
Mid
ENN-UCE
0.866
0.273
0,302
0.915
0.100
0.322
Ours
0.902
0.287
0.289
0.889
0.144
0.367
Energy-Fock Focal
0.866
0,223
0.436
0.777
0.025
0,579
Hard
ENN-UCE
0.775
0.197
0.462
0,843
0.008
0.507
Ours
0.831
0.227
0.445
0,866
0.008
0.431
Energy-Focal
0.947
0.309
0.229
0.831
0.024
0.357
Fasy
ENN-UCE
0.899
0.284
0.279
0.921
0.194
0.384
Ours
0.934
0.307
0.228
0.931
0.244
0.266
Eneryy-Focal
0.900
0.249
0.374
0.748
0.020
0.373
MotionBlur
Mid
ENN-UCE
0.820
0.225
0,387
0.853
0.149
0.443
Ours
0.898
0.260
0.342
0.857
0.155
0.417
Energy-Focal
0.864
0.214
0.443
0.739
0.020
0.616
Hard
ENN-UCE
0.772
0.195
0.451
0.836
0.138
0.494
Ours
0.867
0.235
0.402
0.833
0.131
0.463
Energy-Focal
0.939
0.300
0.253
0.875
0.049
0.314
Easy
ENN-UCE
0.881
0.278
0.278
0.915
0.176
0.346
Ours
0.911
0.288
0.254
0.898
0.172
0.329
Emergy- Emergy-Focal Focal
0.909
0.261
0.342
0.852
0.03K
0.375
Brightness
Mid
ENN-UCE
0.833
0,227
0,379
0.860
0.057
0.434
Ours
0.881
0.248
0.360
0.883
0.075
0.324
Energy-Focal
0.887
0.239
0.400
0.803
0.008
0.461
Hard
ENN-UCE
0.792
0.202
0.448
0.792
0.010
0.494
Ours
0.862
0.221
0.442
0,870
0.037
0.420
Energy-Focal
0.890
0.233
0.387
0.809
0.017
0.619
Fasy
ENN-UCE
0.741
083
0.482
0.750
0.057
0.600
Ours
0.841
0.237
0.384
0.830
0.104
0.480
Energy- Focal
0.839
0.169
0.523
0.785
0.006
0.636
Snow
Mid
ENN-UCE
0.613
0.080
0.732
0.679
0.004
0.762
Ours
0.798
0.176
0.512
0.724
0.053
0.626
Energy-Foca
0.826
0.158
0.547
0.792
0.005
0.685
Hard
ENN-UCE
0.580
0.063
0.797
0,604
0.001
0.809
0.754
0.137
0.604
0.676
0.067
0.709
Energy-Poon Pocay
0.920
0.280
0.310
0.873
0.029
0.341
Easy
ENN-UCE
0.857
0.249
0.352
0.840
0.134
0.485
Ours
0,906
0.271
0.318
0.880
0.173
0.353
Energy Focal
Mocal
0.899
0.250
0.376
0.849
0.019
0.416
Fog
Mid
ENN-UCE
0.858
0,248
0,372
0.829
0.114
0.498
Ours
0.892
0.251
0.368
0.856
0.160
0.426
Energy-Foce
0.878
0.227
0.426
0.803
0.014
0.427
Hard
ENN-UCE
0.851
0.239
0.395
0.061
0,500
Ours
0.890
0.241
0.388
0.823
0.090
0.432
34

---

Published as a conference paper at ICLR 2025
Table 20: Robustness Analysis (urban layouts): evaluation with LSS backbone CARLA in diverse
towns. Best results highlighted in red.
Pure Classification
Misclassification
OOD Detection
pseudo 0OD
model
loss
BCEJ
AUROCT
AUPR1
FPR95L
AUROCT
AUPRI
FPR95]
Town 10
Baseline
CE
0.377
0.00331
0.931
0.270
0.228
0.688
0.003
0.805
Focal
0.392
0.00140
0.956
0.291
0.177
0.755
0.003
0.749
CR
0.377
0.00331
0.931
0.270
0.228
0.667
0.002
0.784
Energy
Focal
0.392
0.00140
0.956
0.291
0.177
0.748
0.003
0.754
No
Ensemble
CR
0.409
0.00218
0.942
0.257
0.228
0.491
0.001
0.963
Focal
0.431
0.00165
0.964
0.262
0.194
0.453
0.001
0.961
CB
0.381
0.00299
0.921
0.264
0.252
0.462
0.001
0,960
Dropout
Rocal
0.398
0.00151
0.941
0.276
0.213
0.391
0.001
0.967
UCE
0.384
0.00231
0.813
0.245
0.352
0.604
0.002
0,816
Evidential
UFCE
0,390
0.00031
0.905
0.274
0.167
0.578
0,003
0.820
CR
0.413
0.00263
0.969
0.297
0.141
0.857
0.058
0.358
Energy
Yes
Focul
0.427
0.00586
0.976
0.278
0.111
0.861
0.061
0.284
Evidential
UCE
0,403
0.00141
0.876
0.276
0.228
0.866
0.106
0.338
Ours
0.427
0.00064
0.939
0.286
0.133
0.946
0.180
0.235
Town
Baseline
CR
0.407
0.00329
0.930
0.289
0.208
0.677
0.003
0.784
Focal
0.414
0.00158
0.960
0.303
0.158
0.765
0.003
0,729
CE
0.407
0.00329
0.930
0.289
0.208
0.678
0.003
0,763
Energy
Focal
0.414
0.00158
0.960
0.303
0.158
0.766
0.003
0.719
No
Ensemble
CB
0.438
0.00207
0.942
0.278
0.201
0.493
0.001
0.961
Focal
0.457
0.00139
0.969
0.283
0.158
0.442
0.001
0.960
CE
0.413
0.00294
0.919
0.276
0.235
0.437
0.001
0.963
Dropout
Focal
0.420
0.00170
0.948
0.291
0.186
0.401
0.001
0.964
Evidential
UCE
0.413
0,00229
0.822
0.256
0.334
0.630
0.002
0.798
UFCB
0.416
0.00007
0.916
0.291
0.147
0.520
0.002
0.832
Energy
CB
0.443
0.00267
0.975
0.315
0.117
0.878
0.298
Focal
0.456
0.00609
0.979
0.106
0.214
Yes
Evidential
UCE
0.441
0.00136
0.893
0.294
0.196
0.256
Ours
0.455
0.00049
0.945
0.306
0.111
0.957
0.213
0.192
Town 7
CE
0.376
0.00348
0.935
0.283
0.206
0.599
0.001
0.877
Baseline
Focal
0.393
0.00182 82
0.967
0.311
0.126
0.725
0.002
0.821
CE
0.376
0.00348
0.935
0.283
0,206
0.604
0.001
0.849
Energy
Focal
0.393
0.00182
0.967
0.311
0.126
0.706
0.002
0.820
No
Ensemble
CE
0.400
0.00224
0.951
0.276
0.181
0.482
0.001
0.953
Focal
0.425
0.00132
0.975
0.295
0.134
0.460
0.001
0.957
CE
0.374
0.00322
0.929
0.279
0.225
0.509
0.001
0.951
Dropout
Focal
0.387
0.00182
0.960
0.308
0.149
0.461
0.001
0.952
Evidential
UCE
0.384
0.00244
0.822
0.259
0.333
0,532
0.001
0.868
UFCB
0.394
0.00017
0.922
0.304
0.136
0.550
0.001
0.839
Energy
CR
0.418
0.00282
0.975
0.325
0.097
0.349
Yes
Focal
0.420
0.00478
0.982
0.311
0.078
0.888
0.076
0.236
Evidential
UCE
0.410
0.00156
0.899
0.308
0.184
0.858
0.126
0.326
Ours
0.438
0.00035
0.957
0.320
0.079
0.949
0.230
0.227
Town 2
Baseline
CE
0.407
0.00320
0.943
0.294
0.185
0.696
0.002
0.788
Focal
0.437
0.00115
0.971
0.310
0.114
0.759
0.002
0.754
CE
0.407
0.00320
0.943
0.294
0.185
0.697
0.002
0.753
Energy
Focal
0.437
0.00115
0.971
0.310
0.114
0.757
0.002
0.750
No
Ensemble
CE
0.445
0.00195
0.954
0.287
0.174
0.490
0.001
0.964
Focal
0.478
0.00125
0.978
0.292
0.117
0.461
0.001
0.954
CB
0.411
0.00291
0.935
0.287
0.204
0.436
0.001
0.967
Dmpout
Focal
0.440
0.00133
0.964
0.299
0.136
0.397
0.001
0.962
Evidential
UCE
0.422
0.00203
0.842
0.275
0.297
0.632
0.002
0.800
UFCE
0.435
0.00029
0.925
0.306
0.130
0.540
0.001
0.870
Energy
CB
0.458
0.00246
0.979
0.321
0.101
0.894
0.057
0.261
Yes
Focal
0.473
0.00571
0.985
0.308
0.067
0,893
0.062
0.225
Evidential
UCE
0.444
0.00114
0.907
0.309
0.168
0.896
0.093
0.271
Ours
0.479
0.00022
0.963
0.319
0.074
0.964
0.156
0.161
35

---

Published as a conference paper at ICLR 2025
Table 21: Robustness Analysis (weather conditions part 1): evaluation with LSS backbone on CARLA in diverse weather conditions.
Best results are highlighted in red.
Pure Classification
Misclassifiattin
OOD Detection
pseudo OOD
model
loss
ECEJ
AUROCT
AUPR1
FPR95,
AUROCI
AUPR
FPR95
ClearNoon
Baseline
CB
0.401
0.00318
0.927
0.275
0.217
0.654
0.002
0.839
Focul
0.420
0.00159
0.959
0.295
0.156
0.737
0.002
0.761
CE
0.401
0.00318
0.927
0.275
0.217
0.648
0.002
0.809
Energy
Focal
0.420
0.00159
0.959
0.296
0.156
0.740
0.002
0.757
No
Ensemble
CE
0.433
0.00207
0.941
0.261
0.200
0.493
0.001
0.958
Focal
0.460
0.00146
0.969
0.271
0.160
0.454
0.001
0.953
CE
0.398
0.00292
0.920
0.270
0.235
0,484
0.001
0.947
Dropout
Focal
0.419
0.00154
0.952
0.289
0.178
0.411
0.001
0.957
Evidential
UCE
0.411
0.00220
0.815
0.244
0.348
0,573
0.002
0.831
UFCR
0.416
0.00015
0.912
0.287
0.157
0,543
0.002
0.834
CB
0.449
0.00255
0.975
0.317
0.111
0,852
0,051
0.347
Energy
Focal
0.449
0.00493
0.980
0.299
0.087
0.862
0.064
0.278
Yes
UCE
0.443
0.00133
0.889
0.292
0.203
0.866
0.299
Evidential
Ours
0.455
0.00038
0.950
0.310
0.945
0.159
0.255
CloudyNoon
CB
0.434
0.00300
0.926
0.278
0.213
0.679
0.002
0,801
Baseline
Focal
0.428
0.00136
0.969
0.307
0.130
0.731
0.002
0,760
CE
0.387
0.00343
0.940
0.284
0,206
0.664
0.002
0,783
Energy
Focal
0.413
0.00133
0.963
0.300
0.145
0.731
0.002
0.764
No
Ensemble
CE
0.419
0.00219
0.947
0.277
0,202
0,507
0.001
0.948
Focal
0.460
0.00118
0.975
0.304
0.132
0.463
0.001
0.956
CE
0.415
0.00290
0.933
0.286
0.220
0.485
0.001
0.959
Dropout
Focal
0.417
0.00163
0.955
0.303
0.164
0.428
0.001
0.951
Evidential
UCE
0.418
0.00222
0.833
0.273
0.313
0.587
0.002
0.836
UPCE
0.423
0,00016
0.923
0.307
0.136
0.560
0.002
0.833
CE
0.454
0.00256
0.981
0.331
0.082
0,913
0.068
0.231
Energy
Yes
Focal
0.487
0.00415
0.985
0.070
0.158
Evidential
UCE
0.451
0.00120
0.913
0.158
0.236
Ours
0.471
0.00010
0.965
0.059
0.104
WetNoon
Baseline
CE
0.415
0.00316
0.940
0.294
0.189
0.643
0.002
0,832
Focal
0.418
0.00159
0.967
0.319
0.128
0.754
0.003
0.739
CB
0.415
0.00316
0.940
0.294
0.189
0.654
0.002
0.799
Energyy
Focal
0.418
0.00159
0.967
0.319
0.128
0.755
0.003
0.739
No
Ensemble
CR
0.439
0.00199
0.956
0.300
0.162
0,501
0.001
0.954
Focal
0.460
0.001 18
0.975
0.303
0.132
0.463
0.001
0.957
CB
0.415
0.00290
0.933
0.286
0.220
0.485
0.001
0.959
Dropout
Focal
0.417
0.00163
0.955
0.303
0.164
0.428
0.001
0.951
UCE
0.418
0.00222
0.833
0.272
0.313
0,587
0.002
0.835
Evidential
UFCE
0.423
0,00015
0,923
0,307
0.136
0,560
0.002
0.833
CR
0.454
0.00256
0.981
0.330
0.082
0.913
0.068
0.231
Energy
Yes
Focal
0.467
0.00549
0.987
0.308
0.052
0.911
0.078
0.177
Evidential
UCE
0.451
0.00120
0.913
0.317
0.158
0.897
0.124
0.236
Ours
0.471
0.00010
0.965
0.333
0.059
0.975
0.229
0.104
MidRainyNoon
Baseline
CB
0.410
0.00319
0.938
0.302
0.192
0.657
0.002
0.838
Focal
0.428
0.00136
0.969
0.306
0.130
0.731
0.002
0,760
CE
0.410
0.00319
0.938
0.302
0.192
0.672
0.002
0.798
Energy
Focal
0.428
0.00135
0.969
0.306
0.130
0.735
0.002
0.765
CB
0.445
0.00196 99
0.953
0.283
0.173
0.474
0.001
0.969
No
Ensemble
Focal
0.466
0.00129
0.978
0.297
0.121
0.457
0.001
0.963
CE
0.418
0.00283
0.927
0.286
0.220
0.475
0.001
0.957
Dropout
Focal
0.432
0.00142
0.956
0.296
0.167
0.442
0.001
0.955
UCE
0.420
0.00211
0.834
0.264
0.311
0.595
0.002
0.817
Evidential
UFCE
0.429
0.00014
0.926
0.297
0.128
0.515
0.001
0.871
CB
0.453
0.00255
0.981
0.327
0.083
0.904
0.085
0.249
Energy
Focal
0.462
0.00564
0.986
0.058
0.189
Yes
Evidential
UCE
0.447
0.00118
0.910
0.164
0.259
Ours
0.469
0.00034
0.963
0.312
0.068
0.965
0.268
0.148
36

---

Published as a conference paper at ICLR 2025
Table 22: Robustness Analysis (weather conditions part 2): evaluation with LSS backbone Best results are highlighted in red.
CARLA in diverse weather conditions.
Pure Classification
Misclassification
OOD Detection
pseudo OOD
model
loss
ECEJ
AUROCT
AUPRT
FPR95,
AUROCT
AUPR
FPR95
SoftRainNNon
Baseline
CE
0.395
0.00333
0.942
0.293
0.197
0.655
0.002
0.842
Focal
0.413
0.00133
0.963
0.299
0.145
0.734
0.002
0.758
CR
0.387
0.00343
0.940
0.284
0.206
0.664
0.002
0.783
Energy
Focal
0.413
0.00133
0.963
0.300
0.145
0.731
0.002
0.764
CR
0.439
0.00199
0.956
0.300
0.162
0,501
0.001
0.954
No
Ensemble
Focal
0.460
0.00118
0.975
0.303
0.132
0.462
0.001
0.957
CB
0.415
0.00290
0.933
0.286
0.220
0.485
0.001
0.959
Dropout
Rocal
0.417
0,00163
0.955
0.303
0.164
0.428
0.001
0.951
UCE
0.418
0,00222
0.833
0.272
0.313
0,587
0.002
0.836
Evidential
UFCR
0.423
0.00016
0.923
0.308
0.136
0,560
0,002
0.833
CE
0.454
0.00256
0,981
0.330
0.082
0,913
0.068
0.231
Energy
Rocal
0.441
0.00603
0.983
0.297
0.076
0,902
0.071
0.209
Yes
UCE
0.451
0.00120
0.913
0.317
0.158
0.897
0.124
0.236
Evidential
Ours
0.471
0.00010
0.965
0.332
0.059
0.975
0.229
0.104
ClearSunset
Baseline
CR
0.387
0.00343
0.940
0.283
0.2066
0.668
0.002
0.804
Focal
0.413
0.00134
0.963
0.300
0.145
0.734
0.002
0.758
CB
0.387
0.00343
0.940
0.283
0.2066
0.664
0.002
0.783
Energy
Focul
0.418
0.00159
0.967
0.318
0.128
0.755
0.003
0.739
No
Ensemble
CB
0.439
0.00199
0.956
0.300
0.162
0,501
0.001
0.954
Focal
0.460
0.00118
0.975
0.303
0.132
0.463
0.001
0.956
CE
0.415
0.00290
0.933
0.286
0.220
0.485
0.001
0.959
Dropout
Focal
0.417
0.00163
0.955
0.303
0.164
0.428
0.001
0.951
Evidential
UCE
0.418
0.00222
0.833
0.272
0.313
0.587
0.002
0,835
UFCB
0.423
0,00015
0.923
0.307
0.136
0,560
0.002
0.833
CB
0.454
0.00256
0.981
0.330
0.082
0,913
0.068
0.231
Energy
Focal
0.445
0.00588
0.977
0.278
0.106
0.868
0.073
0.261
Evidential
UCF
0.451
0.00120
0.913
0.316
0.158
0.897
0.124
0.236
Ours
0.471
0.00010
0.965
0.333
0.059
0.975
0.229
0.104
CloudySunset
Baseline
CE 0.374
0.00349
0.933
0.281
0.218
0.678
0.002
0.790
Focal
0.380
0.00145
0.966
0.307
0.143
0.771
0.003
0.771
CR
0.373
0.00342
0.938
0.279
0.209
0.658
0.002
0.799
Energy
Focal
0.391
0.00133
0.966
0.300
0.141
0.766
0.003
0.717
No
Ensemble
CR
0.401
0.00215
0.950
0.277
0.190
0.489
0.001
0.957
Focul
0.427
0.00143
0.973
0.282
0.157
0.425
0.001
0.964
CB
0.376
0.00317
0.930
0.267
0.235
0.448
0.001
0.959
Dropout
Focal
0.398
0.00162
0.953
0.289
0.176
0,383
0.001
0.952
UCE
0.370
0.00246
0.834
0.265
0.310
0.612
0.002
0.791
Evidential
UFCB
0.387
0.00030
0.919
0.298
0.148
0.538
0.002
0.803
CR
0.404
0.00277
0.974
0.319
0.121
0.865
0.059
0.318
Energy
Yes
Focal
0.427
0.00613
0.979
0.289
0.113
0.889
0.065
0.213
Evidential
UCE
0.401
0.00147
0.885
0.288
0.209
0.871
0.104
0.328
Ours
0.431
0.00062
0.947
0.297
0.117
0.959
0.182
0.217
WetSunset
Baseline
CB
0.371
0.00344
0.933
0.283
0.213
0.684
0.002
0,780
Focal
0.391
0.00133
0.966
0.299
0.141
0.772
0.003
0.717
CE
0.373
0.00342
0.938
0.278
0,209
0.658
0.002
0,799
Energy
Focal
0.391
0.00133
0.966
0.299
0.141
0.766
0.003
0.717
CB
0.402
0.00214
0.950
0.277
0.190
0.489
0.001
0.957
No
Ensemble
Focal
0.427
0.00143
0.973
0.282
0.157
0.425
0.001
0.964
CR
0.376
0.00317
0.930
0.267
0.235
0.448
0.001
0.960
Drpout
Focal
0.398
0.00162
0.953
0.289
0.176
0,383
0.001
0.953
UCE
0.370
0.00246
0.834
0.265
0.310
0.612
0.002
0,791
Evidential
UFCE
0.387
0.00030
0.919
0.299
0.148
0.538
0.002
0.803
CB
0.404
0.00277
0.974
0.319
0.121
0.865
0.059
0.318
Energy
Focal
0.427
0.00613
0.979
0.113
0.213
Yes
Evidential
UCE
0.401
0.00147
0.885
0.209
0.328
Ours
0.431
0.00062
0.947
0.296
0.117
0.959
0.182
0.217
37

---

Published as a conference paper at ICLR 2025
Table 23: Robustness Analysis (weather conditions part 3): evaluation with LSS backbone Best results highlighted in red.
CARLA in diverse weather conditions.
Pure Classification
Misclassification AUPR
OOD Detection
pscudo OOD
model
loss
I
BCEJ
AUROCT
FPR95
AUROCT
AUPR
FPR9S
WodCloudySunset
Baseline
CR
0.361
0.00353
0.932
0.279
0.214
0.680
0.002
0.788
Focal
0.391
0.00133
0.966
0.299
0.141
0.772
0.003
0.718
CB
0.373
0.00342
0.938
0.279
0.209
0.658
0.002
0.799
Energy
Focul
0.391
0.00133
0.966
0.299
0.141
0.766
0.003
0.717
No
Ensemble
CB
0.401
0.00215
0.950
0.277
0.190
0.489
0.001
0.957
Focal
0.427
0.00143
0.973
0.281
0.157
0.425
0.001
0.964
CE
0.376
0.00317
0.930
0.267
0.235
0.448
0.001
0,960
Dropout
Focal
0.398
0.00162
0.953
0.289
0.176
0.383
0.001
0.953
UCE
0.370
0.00246
0.834
0.266
0,310
0.612
0.002
0.791
Evidential
UFCB
0.387
0.00030
0.920
0.299
0.148
0.538
0.002
0.803
CB
0.404
0.00277
0.974
0.319
0.121
0.865
0.059
0.318
Energy
Yes
Focal
0.427
0.00613
0.979
0.113
0.213
UCE
0.401
0.00147
0.209
0.328
Evidential
0.431
0.00062
0.217
Baseline
CE
0.373
0.00342
0.938
0.279
0.209
0.681
0.002
0,810
Focal
0.448
0.00170
0.958
0.313
0.141
0.737
0.003
0.711
CB
0.434
0.00300
0.926
0.277
0,213
0.674
0.002
0.370
Energy
Focal
0.448
0.00170
0.958
0.313
0.141
0.736
0.003
0.708
No
Ensemble
CR
0.463
0.001 94
0.944
0.276
0.187
0,504
0.001
0.958
Focal
0.490
0.00112 27
0.970
0.292
0.144
0.463
0.001
0.960
CB
0.431
0.00272
0.924
0.277
0.219
0.478
0.001
0.955
Dropout
Focal
0.442
0.00140
0.952
0.311
0.154
0.428
0.001
0.953
UCE
0.443
0.00206
0.816
0.258
0.347
0.595
0.002
0.806
Evidential
UFCR
0.441
0.00025
0.914
0.301
0.153
0.553
0.002
0.826
CB
0.490
0.00238
0.979
0.324
0.084
0.906
0.072
0.250
Energy
Yes
0.00415
0.985
0.158
Evidential
UCE
0.482
0.197
0.195
Ours
0.494
0.00029
0.956
0.325
0.072
0.975
0.276
0.105
38

---

Published as a conference paper at ICLR 2025
Table 24: Variance for CVT on nuScenes.
Pure Classification
Misclasiffictiie
OOD
Model
IoU
ECE
AUROCT
AUPR
AUROCT
AUPR
URCE-US-ER
924 0.001
UCE-EN-EUS-ER
000005511
0,000123
A.5,9 QUALITATIVE EVALUATIONS FOR MODEL COMPARISON
We qualitatively analyze the model performance with pixel-level prediction when viewed from bird's s eye perspective, and show the effectiveness our proposed model.
We first present the semantic segmentation predictions in Figure Our model demonstrates com- parable segmentation performance to energy-based dropout, and ensemble models. Compared to ENN-UCE, model produces tighter object boundary predictions.
Figure 4: Comparison of Semantic Segmentation Performance: Each row represents example, with the first column showing the ground truth labels, where the yellow regions indicate the positive class ("vehicle" in these examples). We visualize the predicted probabilities for the positive class generated by our model and four baselines. Brighter regions correspond to higher probability values.
Ground BEV
Ours
ENN-UGE
Energy
Dropout
Ensemble
We present the predicted aleatoric uncertainty in Figure We anticipate that correctly classified pixels will exhibit low aleatoric uncertainty, while misclassified pixels will display high aleatoric uncertainty. Analyzing misclassification detection is complex because the ground truth varies across different model predictions. Based these three frames, we cannot see a large performance difference between the various model variants.
the predicted epistemic uncertainty. Ideally, in-distribution (ID) pixels should exhibit low epistemic uncertainty, while out-of-distribution (OOD) pixels should show high uncertainty. Our focus is on the relative uncertainty levels between ID and OOD pixels rather than the absolute uncertainty scale. The results demonstrate that our proposed model achieves the most accurate identification of OOD pixels, indicating the best-predicted epistemic uncertainties. In contrast, the ENN-UCE baseline shows a significant increase in false positives. Even with the assistance of pseudo-OOD, the energy model fails to accurately locate OOD pixels. Dropout and ensemble models, unable to leverage Aseudo-OOD, exhibit the worst performance.
39

---

Published as a conference paper at ICLR 2025
Figure 5: Comparison of Predicted Aleatoric Uncertainty for Misclassification Detection: Each row represents example, with each pair of columns corresponding shows the misclassified labels, where yellow indicates misclassified pixels, while the right column visualizes the predicted aleatoric uncertainty for the same model, with brighter regions representing higher uncertainty values.
model. The left column
Misc Doetts
Abaatono Une.
Visc Docals
Alaatori Line
Misc Pools
A Aestoric Unn..
Micc Pasts
Alaatori Une.
Misc Plzats
Alhator Une
Ous
ENN-UGE
Energy
Dropout
Ensembe
Figure 6: Comparison of Predicted Epistemic Uncertainty for OOD Detection: Each row represents example, with the first column displaying the ground truth labels, where yellow regions indicate OOD pixels (" ("motorcycle" in these examples). The predicted epistemic uncertainty visualized, with brighter regions indicating higher uncertainty values.
Ground Truth OOD
Ours
ENN-UCE
Energy
Dropout
Ensemble
40


Reviews:
[{"review_id": "z3qkS6voqd", "reviewer": "Reviewer_QCfp", "summary": "This paper presents a thorough study on predictive uncertainty quantification of BEVSS and introduces the Uncertainty-Focal-Cross-Entropy (UFCE) loss function to enhance model calibration.  Experiments demonstrate that the uncertainty quantification framework can outperforms baseline methods  in aleatoric uncertainty and calibration in various situations.", "questions": "- It would be better to clarify the meaning and difference between OOD and pseudo-OOD.\n- It is also better to provide the clear definition on the evaluation metrics. \n- Is there anything missing in Appendix A.5? There are two empty subsections.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper proposes a novel loss function, Uncertainty-Focal-Cross-Entropy (UFCE), specifically designed for highly imbalanced data, which significantly enhances model calibration and segmentation accuracy.\n+ The introduction of an uncertainty-scaling regularization term improves both uncertainty quantification and model calibration for BEV segmentation.\n+ The comprehensive  experiments are conducted across three popular datasets, two representative backbones and five uncertainty quantification methods.\n+ The theoretical proof is solid.", "weaknesses": "- Without pseudo OOD, the model performs not well. It is crucial to elucidate the criteria for selecting pseudo OOD data and to discuss its impact on the model's generalization capabilities. Furthermore, an explanation of how varying choices of pseudo OOD data might influence model training would be beneficial. What criteria were used to select the pseudo-OOD data? How sensitive is the model performance to different choices of pseudo-OOD data? Did the authors conduct any ablation studies with different types of pseudo-OOD data?\n- As indicated in A.3.1, the selection of parameters like λ and γ varies across different datasets and backbone networks. The experiments should include a comprehensive hyperparameter analysis to illustrate the optimal setting of these parameters for diverse datasets and models. Please discuss any patterns observed in optimal hyperparameter settings across datasets and backbones. If possible, propose guidelines for selecting these hyperparameters on new datasets or architectures.\n- While the proposed framework demonstrates notable enhancements over the baseline models, it would be strengthened by a comparison with the latest methods in uncertainty quantification. Is there any specific reason of not choosing more recent baselines instead of LSS and CVT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JN2QA9yjFd", "reviewer": "Reviewer_DVo9", "summary": "Summary:\n\nThis paper is motivated to explored the uncertainty estimation in bird’s-eye-view segmentation task, as uncertainty estimation can be utilized to handle out-of-distribution scenarios and adapt to unexpected conditions, especially in dynamic driving environments.. The contribution is two-folds: \n\n1. Testing of Uncertainty Quantification: The paper evaluates predictive uncertainty quantification methods in Bird's Eye View (BEV) semantic segmentation with existing datasets, highlighting the model’s ability to detect misclassified and out-of-distribution (OOD) pixels. This incorporates five existing approaches across multiple backbones.\n2. New Loss Function – UFCE: The authors claim a novel Uncertainty-Focal-Cross-Entropy (UFCE) loss function. This loss function is designed to handle the challenges of highly imbalanced data, improving both model calibration and the reliability of uncertainty predictions for BEV segmentation tasks. The UFCE loss, inspired by Focal Loss, enhances the model’s focus on uncertain predictions, leading to better aleatoric (data-related) and epistemic (model-related) uncertainty estimation.\n3. Improvement in Model Calibration and OOD Detection: The proposed UFCE loss function, coupled with an uncertainty-scaling regularization term, shows superior results over traditional methods, especially in improving the AUPR (Area Under Precision-Recall curve) for OOD detection by approximately 4.758%. The framework achieves high accuracy in segmentation while also performing well in detecting misclassifications and OOD pixels, thus bolstering both model reliability and safety for autonomous vehicle applications.", "questions": "Question\n\n1. What is the reason of choosing such pseudo-OOD classes in those dataset? Any statistical supporting evidence here? Or any quantitative justification for the choice of pseudo-OOD classes, such as similarity metrics between the chosen classes and true OOD examples?\n2. Also the reason of choosing LSS and CVT as baselines. These two methods are back in 2022 and there are many more advanced BEV segmentation models such as BEVSegFormer[1], PETR-V2[2], and so on. It is suggested that authors either include more recent baselines in their comparison or provide a clear justification for why LSS and CVT were chosen despite the availability of newer methods.\n\n[1] Peng, Lang, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. \"Bevsegformer: Bird's eye view semantic segmentation from arbitrary camera rigs.\" In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5935-5943. 2023.\n\n[2] Liu, Yingfei, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. \"Petrv2: A unified framework for 3d perception from multi-camera images.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3262-3272. 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strength\n\n1. Effectiveness Across Datasets and Models: Through extensive experiments, the paper demonstrates that the proposed uncertainty quantification framework performs robustly across various datasets (CARLA, nuScenes, Lyft) and model architectures (Lift-Splat-Shoot, Cross-View Transformer), outperforming existing methods on key metrics like AUROC (Area Under ROC Curve) and Expected Calibration Error (ECE).\n2. Computational Efficiency: The proposed Evidential Neural Networks (ENN) with UFCE loss achieves comparable or improved results with lower computational complexity compared to methods requiring multiple forward passes, making it practical for real-time applications.\n3. Methodical Analysis: The study rigorously evaluates aleatoric and epistemic uncertainties, using metrics like Area Under the ROC Curve (AUROC) and Area Under Precision-Recall Curve (AUPR). Also a runtime efficiency comparison is included as well. This multi-faceted approach allows for a clearer understanding of each method's effectiveness. \n4. Real-world Relevance: By focusing on predictive uncertainty in the context of autonomous vehicles, the research addresses a critical area of real-world application where model reliability and safety are paramount.", "weaknesses": "Weakness:\n\n1. This paper looks more than a study than a question-motivated paper. The benchmark part utilizes existing dataset as well as existing methods and no clear conclusion is drawn from the study. The proposed new loss and the regularization terms are not quite related to the study in terms of motivation. The author is encouraged to explain more on the logical motivation of the proposed new loss function part. \n2. Missing details in experiments part. For different datasets, they have different sensor setting (camera position on the vehicle, FOV, number of camera, number of beams for LiDAR) as well as the scale of bird’s-eye-view grid (0.25m or 0.5m or 1m?). It would be great if the authors could provide a table summarizing the sensor configurations and BEV grid scales for each dataset used in the experiments.\n3. Some typos. For example, L49: Aleatoric uncertainty, which arises from inherent randomnesses, such as noisy data and labels. **not sure if it's missing Subject–verb–object or Subject-Link verb-Predicative Structure** L075: We propose the UFCE loss, which we theoretically **demonstrate can** implicitly regularize sample weights, mitigating both under-fitting and over-fitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjpgngRH4h", "reviewer": "Reviewer_6JWb", "summary": "The paper addresses the challenge of uncertainty quantification in BEV segmentation, a critical component of autonomous vehicle perception. The authors propose a loss function, called uncertainty-focal-cross-entropy (UFCE), in order to improve uncertainty estimation and calibration of BEV segmentation. The authors conduct a comprehensive benchmark, evaluating various uncertainty quantification methods on popular datasets and backbones.", "questions": "Why is the evidential neural network selected to conduct uncertainty quantification for BEV semantic segmentation? Is only the last softmax function replaced by using the relu function so as to have the Dirichelt distribution of the pixel-wise classification prediction? How about this method compared to other uncertainty estimation methods? \n\nIs there any comparison between the ENN and other uncertainty measure methods? Or the combination of these methods? For example, the methods mentioned in Sec.2, like Bayesian Neural Networks, Monte Carlo Dropout, and Deep Ensembles? \n\nHow do the authors prove that the UCE loss has also the issue that the optimization reduces the NLL value by increasing the probability of the predicted class? Is there any case study about the limitation of UCE loss?\n\nIn Sec. 3.2, the authors claim that the UFCE loss improves the calibration over UCE loss. Is there any detail or training results for comparison? \n\nFor the comparison of calibration between UCE and UFCE, the authors claim that models with focal-based loss exhibit lower ECE scores, indicating better calibration than cross-entropy based models. But in Table 1, the performance of UFCE is not consistently better than UCE. \n\n\nThe presentation and organization of the experimental results/tables make it difficult to follow. For example, Sec. 4.2 presents the results from Table 1 to 10.  Not all tables need to show comparison with all methods. For instance, some tables can focus on comparing UCE and UFCE.\n\nAre there any visualisation results to show the calibration between UCE and UFCE? \n\nHow about the comparison with the most recent BEV methods? Also the comparison with other robust BEV segmentation models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper presents an evaluation benchmark for uncertainty quantification in BEV semantic segmentation, which consists of three datasets like CARLA, nuScenes, and Lyft. \n\n\nThe benchmark includes two backbones that are proposed for general BEV semantic segmentation. The benchmark provides insights into potential improvements of previous BEV semantic segmentation models.\n\nThe paper introduces a novel loss function, UFCE, which is specifically designed for imbalanced data in BEV segmentation.", "weaknesses": "While the paper focuses on uncertainty quantification of the BEV segmentation task. As mentioned in the introduction, it would be beneficial to widely explore the impact of uncertainty on other downstream tasks such as motion planning and decision-making of the advanced driver-assistance systems (ADAS). \n\nA more detailed comparison with other state-of-the-art uncertainty quantification methods would strengthen the finding and prove the effectiveness of the proposed loss function. For example, there is only a comparison between UCE and UCFE. \n\nWhile the paper focuses on evidential deep learning, a more comprehensive comparison with other uncertainty quantification methods, such as Bayesian neural networks, Monte Carlo dropout, and deep ensembles, would be valuable.\n\nThe organization of the results is not easy to follow. For example, Sec. 4.2 presents results from ten tables, but the tables are not well structured. Therefore, the presentation and organization of the experimental results could be improved. \n\nThe comparison with other robust BEV methods is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3lFq9jR904", "reviewer": "Reviewer_Uxrf", "summary": "The task tackled in this paper is Bird’s Eye View (BEV) semantic segmentation. The authors present  a comprehensive benchmark for predictive uncertainty quantification and introduce a novel loss function, i.e., Uncertainty-Focal-Cross-Entropy\n(UFCE). The studies focus on model calibration, uncertainty quantification (detecting misclassified pixels) and out-of-distribution (OOD) detection. The methods are evaluated across three popular datasets and two representative backbones.", "questions": "- The two selected networks seem to be representative, but not SOTA, which would be expected for benchmark paper. What were the reasons for selecting these two networks and how does this choice affect the generalizability of the benchmark results?\n- The in-distribution results between the uncertainty methods are very similar and provide little information. What is this supposed to show?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problems addressed in the paper are very important in the area of safety in neural networks.\n- The new loss function presented is well thought out and mathematically motivated.\n- The very well-known uncertainty methods are compared with each other. Different datasets are used, both synthetic and real-world.\n- The methods are briefly explained in the main part, and details are provided in the appendix. It is a good balance between information and details in the method section.\n- The implementation details are very well documented, also the dataset details in terms of creation and splitting etc.\n- The complexity analysis and the ablation studies show a good comparison.", "weaknesses": "- A figure for BEVSS and uncertainty estimation at the beginning of the paper would have an explanatory effect.\n- The experiments section is very confusing, constantly going back and forth between results in the main section and in the appendix. Differences between the datasets are not emphasized enough. For example, you could focus on one dataset and one network in the main part and have all the other experiments in the appendix. If the results for the datasets/networks differ, a paragraph with the most important findings could be written.\n- The appendix on page 24 is also a bit mixed up with the headings printed, but the tables are in different places.\n- I am missing a related work section on the papers that have already dealt with uncertainty estimation and OOD detection for the BEVSS task. This makes it difficult for me to assess the added value of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qk8mBVfl4o", "reviewer": "Reviewer_CuPe", "summary": "This manuscript introduces a new benchmark and a new approach for predictive uncertainty quantification in Bird’s Eye View Semantic Segmentation (BEVSS) tasks. \n\nBEV segmentation plays a key role in understanding the vehicle’s surrounding environment by creating a top-down representation using sensor data (e.g., cameras, LiDAR, radar). However, neural networks used for these tasks often produce overconfident predictions on unseen data and underconfident predictions in noisy data. The paper focuses on the two types of uncertainty—aleatoric (caused by noise in data) and epistemic (caused by model uncertainty due to lack of knowledge)—and their impact on safety-critical tasks like autonomous driving.\n\nThe authors propose a new loss function called Uncertainty-Focal-Cross-Entropy (UFCE), which mitigates under- and overfitting in BEV segmentation models by integrating regularization on uncertainty. \n\nAdditionally, the paper introduces a benchmark for evaluating five uncertainty quantification methods across three large autonomous driving datasets (CARLA, nuScenes, and Lyft) using two popular BEV segmentation backbones: Lift-Splat-Shoot and Cross-View Transformer. The primary goal is to quantify how well these methods detect misclassified pixels, improve model calibration, and identify out-of-distribution (OOD) data.\n\nExperiments demonstrate that the UFCE loss improves both aleatoric and epistemic uncertainty prediction compared to the existing softmax and ensemble-based approaches. The paper reports a 4.758% improvement in OOD detection performance and gains in misclassification detection. The benchmark serves as a good resource for future work in predictive uncertainty quantification for BEVSS.", "questions": "- **Q1:** Could the authors provide more detailed insights into how the UFCE loss function handles both aleatoric and epistemic uncertainty in BEVSS? Specifically, what mechanisms within the loss function enable it to balance both types of uncertainty effectively?\n\n---\n\n- **Q2:** How does the proposed UFCE loss function compare to other more recent uncertainty quantification techniques? Are there any specific examples where the UFCE loss provides a distinct advantage over methods such as deep ensembles or MC-Dropout?\n\n---\n\n- **Q3:** The paper discusses its benchmark evaluation on the CARLA, nuScenes, and Lyft datasets. Could the authors provide more information on how well the model performs in more complex or dynamic environments, such as dense urban settings or environments with adverse weather conditions?\n\n---\n\n- **Q4:** Could the authors explain the potential computational overhead of the UFCE loss function and other uncertainty quantification techniques? Specifically, how feasible is the proposed method for real-time applications in autonomous driving, considering resource-constrained environments?\n\n---\n\n- **Q5:** The manuscript lacks comparisons with recent benchmarks such as RoboBEV, MapBench, and Robo3D for out-of-distribution detection. Could the authors conduct additional experiments on these datasets to strengthen the claims about the robustness of their model?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(+) The focus on uncertainty quantification for BEV segmentation is highly relevant to real-world applications in autonomous vehicles, where handling uncertain and dynamic environments is crucial for decision-making.\n\n(+) The manuscript provides some good evaluations of uncertainty quantification techniques, analyzing their performance across various autonomous driving datasets and network backbones.\n\n(+) The introduction of the UFCE loss function is a good contribution, effectively enhancing the calibration and uncertainty prediction of BEVSS models. It mitigates the shortcomings of traditional loss functions in handling aleatoric and epistemic uncertainties.", "weaknesses": "(-) While the manuscript provides a thorough benchmark for predictive uncertainty quantification, the individual uncertainty quantification techniques themselves are not particularly novel. Many of these approaches, such as deep ensembles and MC-Dropout, are well-established in the literature, and the work lacks an in-depth exploration of more recent or advanced uncertainty quantification methods.\n\n(-) Although the paper addresses both types of uncertainty (aleatoric and epistemic), the explanations regarding how the proposed method specifically handles these uncertainties are not well developed. More technical insights on how the UFCE loss function balances both types of uncertainty and why it is better than existing methods are needed.\n\n(-) The benchmark, while extensive, is focused primarily on the CARLA, nuScenes, and Lyft datasets. These datasets are useful but do not encompass the full range of scenarios an autonomous vehicle might encounter, particularly when it comes to highly dynamic environments like dense urban settings or environments with poor weather conditions. Evaluating on a wider variety of datasets, including those that feature more diverse sensor modalities or environments, would strengthen the paper’s claims. It would also be interesting to see how the benchmarked models perform on out-of-distribution data, such as those corruption sets from RoboBEV [R1], MapBench [R2], and Robo3D [R3].\n\n(-) The manuscript lacks clarity in some sections, particularly in the technical description of the UFCE loss function and the details of the ablation studies. Some important concepts, such as the handling of noise and sparsity in the LiDAR point clouds, are not adequately explained, which might make it difficult for readers to fully grasp the method’s contributions.\n\n\n**References:**\n- [R1] S. Xie, et al. “Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving,” arXiv, 2024.\n\n- [R2] X. Hao, et al. “Is Your HD Map Constructor Reliable under Sensor Corruptions?” arXiv, 2024.\n\n- [R3] L. Kong, et al. “Robo3D: Towards Robust and Reliable 3D Perception against Corruptions,” ICCV, 2023.\n\n---\n\n### Justification of Rating\nThe manuscript makes a contribution by focusing on uncertainty quantification in Bird’s Eye View Semantic Segmentation (BEVSS) for autonomous driving tasks. \n\nThe introduction of the Uncertainty-Focal-Cross-Entropy (UFCE) loss function is an interesting attempt to improve model calibration and uncertainty prediction, addressing a relevant real-world problem. The inclusion of a benchmark to evaluate different uncertainty quantification techniques across three datasets adds practical value.\n\nHowever, I have limited experience in the area of OOD detection and am not fully familiar with the theoretical derivations behind the bounds of the UFCE loss function. As such, I leave more room for the other reviewers to assess the technical novelty and rigor of this contribution. \n\nWhile the results are compelling, the work could have benefitted from more in-depth explanations of key components, such as handling aleatoric and epistemic uncertainties and how the proposed loss function truly differentiates itself from previous works.\n\nAdditionally, although the benchmark evaluation is thorough, it could be further expanded to include more diverse datasets and dynamic environments. The manuscript also lacks clarity in some technical sections, which slightly hampers the overall readability. For these reasons, I believe that while the manuscript has merits, it leaves some gaps that need to be addressed before it reaches its full potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)