UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model
Authors: Timo Kaiser
Thomas Norrenbrock
Bodo Rosenhahn
Venue: ICML 2025 poster (2025)
Keywords: Uncertainty Quantification
Segment Anything
Primary area: applications->computer_vision
TL;DR: A novel lightweight uncertainty quantification method that estimates multiple sources of uncertainty of the Segment Anything Model.

Abstract: The introduction of the Segment Anything Model (SAM) has paved the way for numerous semantic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of particular interest. However, the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification (UQ) approaches. This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation jointly respecting aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, a lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under-parameterised models, insufficient prompts or image ambiguities. Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accuracy and cost efficiency.

UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model
Timo Kaiser'
Thomas Norrenbrock
Bodo Rosenhahn
Abstract
Input
The introduction the Segment Anything Model (SAM) has paved the way for numerous seman- tic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of partic- ular interest. However, the ambiguous nature of the class-agnostic foundation model SAM chal- lenges current uncertainty quantification (UQ) ap- proaches. This paper presents a theoretically mo- tivated uncertainty quantification model based on Bayesian entropy formulation jointly respect- ing aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under- parameterised models, insufficient prompts or image ambiguities. Our proposed determinis- tic USAM demonstrates superior predictive ca- pabilities on the SA-V, MOSE, ADE20K, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accu- racy and cost efficiency.
Tiny Model
IoU:: 80.2%
Ag:
7.5%
Large Model
loU: 89.9%
Figure I. Large and Tiny SAM with our epistemic UQ on the DAVIS dataset. The Large model predicts an accurate mask while the Tiny model fails in the area of the tail. Our USAM MLP A estimates potential epistemic gap of 7.5% intersection over union (IoU) directly from the Tiny model, helping the user to balance the trade-off between efficiency and accuracy.
model training. Similarly, the adoption of deep learning in critical domains such as medical diagnosis (Wang et al., 2019) or autonomous driving (Yan et al., 2024a; Gottwald et al., 2024) intensified research in uncertainty quantification (UQ) to make systems more reliable and safe. Moreover, UQ helps improve model training (Kaiser et al., 2022), supervisability (Vujasinović et al., 2024), and in other ap- plications (Paul et al., 2024; Kaiser et al., 2024; Wehrbein et al., 2025; Kruse Rosenhahn, 2025). UQ is therefore also moving into focus with regard to SAM (Zhang et al., 2023b; Jiang et al., 2024). Like other segmentation methods, SAM exhibits uncertainty caused by various factors. For example, Figure 1 illustrates the prediction of SAM using tiny and large backbones. While the tiny model is more cost-efficient, it fails to segment the details on the fishtail. Knowing about potential accuracy gaps (i.e. uncertainty) based on under-paramettrriztion can help determine if a switch to a larger model is worthwhile. Existing applica- tions rely on SAM's inherent confidence score or apply stan- dard approaches like test-time augmentation (Deng et al., 2023) which are resource-intensive and poorly suited be- cause SAM is subject to a unique kind of uncertainty. In ad- dition to model (epistemic) and data uncertainty (aleatoric), the task is undefined, i.e. the object to segment is unknown priori, and the user defines the prior with a prompt. Thus,
1. Introduction
In the last years, significant effort and resources have been spent into the development of foundation models. One well-established model is the Segment Anything Model (SAM) (Kirillov et al., 2023) that segments arbitrary objects in images based on multi-modal prompts. SAM enables researchers and engineers rapidly develop new appli- cations (Zhang al., 2023a) such interactive segmen- tation (Wu & Xu, 2024) or satellite imagery (Ren et al., 2024) without the need for expensive data annotation or
'Institute for Information Processing / L3S Leibniz Uni- versity Hannover, Germany, Correspondence to: Timo Kaiser <kaiser@ int.un www.uiihannoverde::
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

UncertainSAM Fast and Efficient Uncertainty Quantification
Entropy Hy. Ha. HA. Hap
Caurdinate Prompis
Prompt Encoder
Output Variations
Mask Decoder
User
U Upsample
A
with Task
Random Embbeddings
MLPs
Masis
Token
MLP
Image Encoder
MI.P
SamScore
CeEaRaaen Estimator USAM
Augmented Image ((x))
Aj. DG. A' Al.-
Finhediing
Concal
USAM,, USAMSAM. USAM
Figure 2. The SAM framework with our USAM extension and Bayesian entropy approximation to quantify uncertainty. Starting from the the bottom left, image 21 is the input. user defines one or more coordinate prompts zy to specify his desired segmentation task a. The image and prompt are encoded into embeddings. concatenated together with random embeddings. and fed into the mask decoder which applies attention, MLPs and upsampling layers. The SAM framework estimates three potential masks that address different tasks A. Additionally, SAM estimates the IoU between the ground truth of and the masks corresponding to A, denoted as SamScore. We extract and concatenate the mask and confidence tokens (Kirillov al., 2023) to train MLPs that estimate the expected predictive, epistemic, task, and prompt uncertainty. Furthermore, the process calculate the entropy of uncertainty is visualized in blue. Multiple prompts are chosen by the user, augmentations are applicd to the input, and models of diffcrent size θ are used to apply variational inference.
SAM also underlies task uncertainty because the prompt is potentially describing multiple valid tasks or SAM fails to decode
usage. Notably, the simple MLPs of USAM perform on par or surpass the Bayesian approximation and existing UQ methods, establishing a new state-of-the-art in UQ for SAM. Easy-to-use code is available here'.
To advance UQ research for SAM and better classify the sources of uncertainty, this paper:
Section 2 recapitulates SAM, UQ and UQ in SAM. Then, Section 3 introduces the Bayesian approximation and our novel USAM to estimate uncertainty. Section 4 shows in- depth experimental evaluation concluded in Section 5.
Introduces a complete UQ formulation for SAM,
Presents its Bayesian UQ approximation, and
Introduces USAM, an estimator that outperforms ex- isting methods in estimating sources of uncertainty.
2. Segment Anything and Uncertainty
This section offers preliminaries and related work SAM and UQ. Then, we present new Bayesian formulation of UQ in SAM that is used later in our method.
Our formulation of UQ in SAM divides the predic- tive uncertainty into uncertainties stemming from under- parameterized models (epistemic uncertainty), insufficient image prompts (aleatoric prompt uncertainty). or ambiguous images (alcatoric task uncertainty). The Bayesian approxi- mation employs Monte Carlo sampling (Mooney, 1997) to quantify those. Since the Bayesian formulation an impor- tant basis for our investigations, but unfortunately computa- tionally expensive, we introduce UncertainSAM (USAM), simple training strategy to train a lightweight estimator that accounts for the same uncertainties. USAM uses multi-layer perceptrons (MLPs) to directly estimate uncertainties from SAM's pretrained latent representations, enabling practical use without bells and whistles in arbitrary applications. Our experimental results demonstrate that our UQ methods ac- curately estimate if prompt should be refined, supervision is required, or larger models improve the prediction. The last application, combined with the efficiency of our MLPs, can reduce the energy requirements of applications. While the computational expensive Bayesian approximation is the- oretically grounded, USAM is also relevant for practical
2.1. Segment Anything Model (SAM)
SAM (Kirillov et al., 2023) is foundation model for class- agnostic image segmentation. It predicts image masks for arbitrary objects based on an input image øj and one or more multi modal input prompts Tp. An overview, extended with our contributions, is presented in Figure 2. Formally, SAM estimates the pixel-wise probability of being foreground y for an unknown ground truth J. The framework com- prises three basic blocks: The image and prompt encoders as well as the mask decoder. A Vision Transformer (Doso- vitskiy et al., 2021) is used to encode the input image, while multi-modal prompt encoders encode prompts. Prompts can be point or box coordinates encoded as positional encod- ing (Tancik al., 2020), text encoded with CLIP (Radford et al., 2021), or dense masks added the image embedding with convolutions. The resulting embeddings are combined
hhtgs//giilbbboomGGrennuuALLLLL AAAA
2

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
with a randomly chosen embedding and fed into the mask decoder which updates and fuses all prompts. The resulting new embedding/ / is split into mask and IoU token. The mask token is projected with multi-layer perceptrons (MLP) and multiplied with the upsampled image embedding to cre- ate pixel-wise foreground probabilities, i.e. the object mask. The IoU token fed into another MLP that estimates the intersection over union (IoU) (Jaceard, 1901) to the ground truth, denoted SamScore in this paper.
2020) sample from an approximation of O. Efficient BNN approaches (Gal & Ghahramani, 2016) assume θ to be Bermoulli-distributee modeled with Dropout (Srivastava et al., 2014). The aleatoric uncertainty p(y|z,, ()) is some- times undefined or biased. Variational inference can ap- proximate applying test-time augmentations (Wang et al., 2019), where the input measurement æ is slightly perturbed (e.g. with flips and rotations) with static 6. This approach assumes that target values y of ambiguous samples are over- fitted on sample-specific biases during training and only memorized by spurious patterns. Perturbations activate dif- ferent biases, approximating the distribution of y.
Since the ground truth mask may be unknown even to the user encoding it into the prompt, we omit the notion of ground truth and instead define the user-encoded propo- sition as the task a. Depending on the image 21, multiple prompts (
For convenience, some methods (Lahlou et al., 2023; Quil- lent et al., 2024) quantify uncertainty with the variance or entropy of the sampled distribution. This allows determinis- tic approximations of the epistemic, aleatoric, or predictive variance/entrooyy For example, DEUP (Lahlou et al., 2023)
can represent the same task a and multiple tasks {a1, ... .) can be represented by the same prompt Zp, e.g. different granularities in hierarchical struc- tures. To tackle this task ambiguity, SAM adds the random embedding to the latent. With multiple random embeddings, multiple masks based on different task assumptions à e Á are generated. Selecting the closest to ground truth mask and using the minimum loss (Guzzán-riveen et al., 2012; Charpiat et al., 2008) during training induces variability in the network that only depends on the random embedding. During test time, the mask with highest SamScore is kept.
trains NN to estimate the epistemic variance of MC sam- pling, Compensation learning (Kaiser et al., 2023) predicts pixel-wise aleatoric uncertainty, or Quillent et al. (2024) spatial aleatoric uncertainty. Deterministic UQ methods are lightweight, applicable post-hoc, and efficient.
While the presented methods work well in practice, the the- oretical foundations for separating aleatoric and epistemic uncertainty remain debatable and under active discussion. The task-agnostic nature of SAM further amplifies this chal- lenge, as discussed next. More insights into uncertainty can be found in (Gawlikowski et al., 2023; Gruber et al., 2023).
2.2. Uncertainty Quantification (UQ)
According to (Gawlikowski et al., 2023), a neural network (NN) parameterized with weights 0 from the function space θ maps a set of inputs to set of targets. The finite training set D is used to condition 0 during optimization, enabling the NN to predict targets for new samples. UQ aims to quantify the confidence that the prediction is correct.
2.3. UQ in SAM
SAM inherently faces challenges related to uncertainty, as ambiguities are present in images or prompts. Quantifying the reliability of SAM's prediction is critical, especially in sensitive domains. For instance, in the medical domain, SAM exhibits excessive uncertainty, necessitating Yan et al. (2024b) to retrain SAM or Jiang et al. (2024) to fine-tune the ability to predict different masks. Existing works have intro- duced or utilized UQ methods in SAM: Deng et al. (2023) or Zhang et al. (2023b) approximate aleatoric uncertainty with the entropy of MC simulations by augmenting prompts, (Vujasinović et al., 2024) caleulate the mean pixel-wise en- tropy between different object masks to detect errors, or (Xu et al., 2024) quantify uncertainty with subjective logic (Jsang, 2018) and use the uncertainty to refine prompts. (Li et al., 2024) adapt UQ in SAM to sample more diverse predictions and (Liu et al., 2024) use entropy to improve finetuning.
From a Bayesian perspective (Gal & Ghahramani, 2015), the predictive uncertainty distribution is defined as
f plulo, P(( do
p(ulor D)
Predictive Uncertainty
Aleatoric Egistemic
over all parameter configurations 0 e Q conditioned on the unseen input sample x and the training data D. The formulation allows to decompose uncertainty into more fine- grained aspects, namely the Aleatoric uncertainty inherent in x and the Epistemic uncertainty that originates from im- perfect A. The posterior distribution p(0 (D) over all 0 is infeasible, and ply|r, 0) is often undefined. Thus, many methods approximate both terms using Monte Carlo simula- tion (MC) (Mooney, 1997).
The epistemic posterior distribution p(O can be approx- imated with MC samples 0 E θ For example, model ensembles (Lakshminarayanan et al., 2017) reduce Equa- tion (1) to a tractable sum of a few models, while cost in- tensive Bayesian neural networks (BNN) (Goan & Fookes,
Despite these efforts, the implications of SAM's class- and task-agnostic framework on UQ remain underexplored. Un- like other frameworks, the task is undefined a priori and SAM is trained to cover a large set of tasks A. During inference, the abstract task a e A is defined by the user
3

---

UncertainSAM Fast and Efficient Uncertainty Quantification
and represented by prompt, e.g. persons that should be segmented is encoded by a single point coordinate. Decom- posing SAM's input into the image 21 and the prompt æp, and explicitly including the subtask of estimating the task, the Bayesian formulation from Equation (1) extends to
ground mask estimation y is correct, i.e. is equal to the ground truth y a. The uncertainty can be quantified for single pixel y or entire masks y. This paper quantifies the latter to address applications that utilize per image evalua- tions like supervision management (Vujasinovié et al., 2024) or adaptive model scaling (Aggarwal et al., 2024).
p(ular 21; D,a)= S S p(y (0.
3.1. A Bayesian Entropy Approximation
Predictive Une.
Aleatoric
p( [1/21, Zp. 0) p ( a) (( dodrpdå (2)
The predictive uncertainty from Equation (2) is intractable. To address this, we approximate it using ideas from ex- isting sampling-based methods (Wang et al., 2019; Deng et al., 2023; Lakshminarayanan et al., 2017). Specifically, we predict set of two-dimensional foreground probability maps J {y> ..... y } using SAM, where the cardinality is defined as K = |T| |xp] | |0]. It combines the sets of image augmentations t T (aleatoric), randomly sam- pled point-coordinate prompts 2p E (a) drawn equally distributed from the ground-truth mask (prompt), the tasks A encoded SAMs mask proposals (task), and all avail- able pre-trained models @ of different size (epistemic). The models are denoted as Q {L. B+, S, T} and correspond to the names Large, Base+, Small, and Tiny. We compute the task probability p(a)t(x1), zp, 0) by normalizing over all SamScores of a prediction. Using the weighted entropy
Prompt Decoder Prompt Encoder Epistemic
In this formulation, SAM estimates the task probability p(alar, xp, 0) given the human encoded prompt Tp. This introduces new potential sources of uncertainty: Ambiguity in prompt en- and decoding. The first stems from the user inspecting the image content, the latter from SAM's interpre- tation of 2 and zp. SAM inherently tackles task ambiguities with multi-mask approach that follows Bayesian concepts to estimate epistemic uncertainty. SAM generates three masks representing predictions for different à e Á using dif- ferent random input embeddings (see Figure 2). However, the true task probability p (()) Xp, B) is intractable.
Reviewing the former mentioned methods, MC variations of O like ensembles or BNNs to estimate epistemic uncertainty not applicable post-hoc and are highly cost-intensive for foundation models. Aleatoric approximations that augment the prompt Tp (Deng et al., 2023; Zhang et al., 2023b) only address aleatoric uncertainty that stems from memorized bi- ases in the task decoder. Image augmentations like in (Wang et al., 2019; Lahlou et al., 2023) may only address the image encoder, neglecting prompt-induced uncertainties. shows that the task-agnostic nature of SAM challenges current UQ strategies. To address these challenges, we propose a more fine- fine-grained decomposition of uncertainty in SAM. Beyond epistemic uncertainty, we subdivide alcatoric uncertainty into prompt and task uncertainty."
H(y) -- vey Tyll
1/
(ylog2()) (v) + (1-3) loga (1- -y)), (3)
the predictive uncertainty is quantified for an entire image by calculating the weighted sum of y
Hy( a, Toa,,,T,0 T,D,A.))
p(a)t(=)), zp, [T I [x (x)) (0) )(onale)))
H
Σ (L.z0.0,)) TxXo(a)=OxA
(4)
where the indices yf uration. Higher entropy Hy indicates greater uncertainty, though the specific source of uncertainty remains unknown. Thus, we compute entropies for distinct uncertainty sources. The epistemic model uncertainty is quantified by
y. < ) indicate the prediction config-
3. Method
As shown, SAM necessitates more fine-gras fine-grained evaluation of uncertainty. First, we formulate Bayesian approximations of the predictive, epistemic, prompt, and task uncertainty to evaluate known ideas and to baseline our main method. To make UQ usable for applications, we secondly present USAM that directly estimates the sources of uncertainty from SAM's s latent without sampling or retraining.
Ho xp. a, O.A.Y) v
Σ 600
y x (a,
O
(5) calculated for a given image Ti, prompt Tp, and the known ground truth task a. The operator Best(*) selects the best fitting mask proposal according to the ground truth and thus follows the standard SAM evaluation protocol. Similar, the prompt uncertainty is quantified by
The task is to quantify the probability that SAM's fore-
During the conference, Kirchhof et al. initiated discussion on new perspectives regarding uncertainty in large language models, highlighting limitations in the commonly made distinction between aleatoric and epistemic uncertainty (Kirchhof et al., 2025). These concepts are also applicable this paper, particularly their term "Underspecification " that highly relates to Equation (2).
1 (x6]
Hxg (x1, (x1,0,0, a, A. ). = H
Σ (ppAp(()
V mn (e,
66)

---

UncertainSAM Fast and Efficient Uncertainty Quantification
Thny vs. Large Model So, Ab (S) (1)
ing the respective input image ii and the refined prompt zp-- Furthermore, USAMr(1), USAMs(!), USAMB+ (1), and (1 estimate the expected IoU that the Tiny, Small, Baset, and Large SAM model achieve, respec- tively, and are in summary denoted as USAMa((). The last MLP USAMsAM(() estimates the accuracy achieved with the mask selected by the SamScore, i.e. without apply- ing Best( and comparing to the ground truth as done in SAMs evaluation.
Training Sample 21.0
Single vs. Dense Prompt Dug Di,, USAM (1) US)Ms()
Un- vs. Supervised 53 USAMc(1) - (SAMs
Figure 3. Training objectives of our MI.Ps. They estimate the gap between simple and cheap (left) and refined (right) predictions.
During test-time with a given SAM model 0 that cre- ates the tokens I, we quantify the predictive uncertainty by predicting and inverting the expected IoU with 1 USAMa (1). A low expected IoU indicates a higher un- certainty without indicating the cause. To quantify the prompt uncertainty, we calculate the expected gap 4x; (1) USAMa (1) USAM&(I) between the single-coordinate and refined prompt. Similar, we quantify the task un- certainty by estimating the gap (1 = (S)
and requires pre-selected model weights. Finally, we quan- tify the task uncertainty with all three mask proposals:
0., s H Σ p( 0), aed
(7)
This Bayesian framework allows the quantification of pre- dictive uncertainty Hy. epistemic model uncertainty He, aleatoric prompt uncertainty Xxz, and aleatoric task uncer- tainty HA through MC simulations. They allow evaluating known concepts for the task-agnostic SAM setting.
USAMsAM(!) between the supervised and unsupervised mask selection. The model uncertainty is quantified with the expected gap (!) = USAM, (1) - US)Mr(() be- tween Large and Tiny SAM. Since all models are trained equally and only differ in size, we expect high epistemic uncertainties causing the largest drop in the tiniest model.
More details to T, Xp, A, and Q are given in Appendix A. Note that A depends on the respective 24, Tp, and which we omit in the equations due to space limitations.
Furthermore, additional MLPs estimate (1) directly from the latent and are denoted as (!)... They aim to recog- nize patterns that indicate specific uncertainties. The direct MLPs estimate the expected gap without the expected IoU, thus focusing on the relevant aspects during optimization while further reducing computation. USAM is lightweight alternative to the Bayesian approximation.
3.2. USAM: A Deterministic Estimator
While the Bayesian entropy approximation closely reflects Equation (2), the computational effort is huge, ground truth required, and a user needs to draw multiple prompts. To address these limitations, we introduce USAM and ex- tend SAM with lightweight estimators that directly estimate uncertainty proxies. We do this by concatenating SAMs 256 -i--immnnio mask and IoU tokens, further denoted as 1, and feeding them into MIPs that are trained to predict uncertainty as indicated in Figure 2. The simple design of the MLPs aligns with the architecture of SAM's inherent MLP for predicting the SamScore, as described in the Ap- pendix of (Kirillov et al., 2023). They have three layers and sigmoid activation, but we use 512 input dimensions and hidden states due to the concatenated input tokens. All MLPs are trained to minimize the mean squared error to the objectives that are discussed next and visualized in Figure 3.
4. Experiments
This paper presents a theoretical close approximation and the efficient USAM to quantify SAM's uncertainty. The fol- lowing section presents experimental results that investigate the methods and highlight the effectiveness for practical tasks. First, we describe the experimental settings. Then, we analyse the ability to estimate the uncertainty that lies in insufficient models, inaccurate prompts, or ambiguous tasks. Finally, we analyze the overall segmentation uncertainty and present qualitative results of our MLPs.
4.1. Setting
For each dataset image 21 é X, we predict the set y and extract the tokens I. Additionally, we predict 3i using refined prompt oه U Xp that consists of all single-coordinate prompts and represents user that spends high-effort to create a prompt with high certainty. We calculate the IoU values from Y' to the ground truth to train USAM. Given an arbitrary I that can stem from any model in θ, our first MLP USAM (!) estimates the IoU that is achieved us-
In our experiments, we use SAM models pretrained by (Ravi et al., 2024) to implement the Bayesian approximation and USAM. The MLPs of USAM are trained with the SGD opti- mizer (weight decay 0.001). Hyperparameters are optimized using the Bayesian optimization framework SMAC3 (Lin- dauer et al., 2022). The number of epochs is limited to between 5 and 80, the batch size between 16 and 256, the
5

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Certain Prompt
Certain Task
Certain Tiny Model
0% 40 %
A 9%
Nr, 0%
X/
A A 0%
A 0%
A 5%
A 4%
A -3%
Uncertain Prompt
Uncertain Task
Uncertain Tiny Model
A S 22%
A Xo %
A X; -2
A A 16 %
A 40% %
A A 1 %
Ag 7%
A 4%
A 6%
Figure 4. Samples with low (certain) and high (uncertain) uncertainty scores. The uncertainty is quantified with USAM, tee expected difference between simple and refined prompts (,,))), supervised and SamScore selection (A7)), and the Tiny and Large model (Ag).
learning rate between 0.0001 and 0.1, and SGDs momen- tum between 0.1 and 0.9. During SMAC3 optimization, we split the training set into 80% training and 20% validation subsets. Our best trained models on the large-scale dataset SA-V are publicly available in our code repository.
Table I. Model uncertainty quantification. The table shows the area under curve (AUC) when predicting variable fraction of the most uncertain samples with the Large model, others with the tiny one. The uncertainty determined by the respective method.
rel. AUC in %]+
DAVIS
ADF20k
MOSE
COCO
SA-V
Combining the dataset selection of (Kirillov et al., 2023) and (Ravi et al., 2024), we run experiments on different scaled instance segmentation datasets DAVIS (Perazzi et al., 2016), ADE20k (Zhou et al., 2019), MOSE (Ding et al., 2023), COCO (Lin al., 2015), and SA-V (Ravi et al.,
SAM
SamScore
64.25
$2.85
68.44
57.97
58.83
Entropy
71.78
53.48
71.56
61.61
63.49
USAM Boyes
Hy
51.48
52.06
61.36
54.27
55.39
He
73.46
$7.65
75.60
64.43
65.66
Ag
66.85
60.32
71.78
63.66
61. 61,23
2024). The training is performed only using the training data of the respective dataset without access to validation data. Due to missing public test data for DAVIS, COCO, and ADE20K, we use the validation set for extensive evaluation. For similar reasons, we use the SA-V trained models to evaluate on the MOSE train data.
Ag
59.08
61.55
73.05
66.60
63.71
We compare our MLPs to the introduced Bayesian baseline and further standard UQ approaches derived from SAM. indica-
On one hand, we use the inverse SamScore tor for uncertainty. A larger SamScore indicates higher expected IoU and is proxy for UQ. Furthermore, we use the mean entropy of the pixel-wise foreground probabilities that correspond to the predicted mask and denote it as H Mask entropy is often used as UQ method in applications, e.g. in (Vujasinović et al., 2024).
To rank competing methods, we employ the mean inter- section over union (a.k.a mloU) of predicted and ground truth masks. The uncertainty estimation capability is evalu- ated with the area under curve (AUC) of the mloU, when a variable ratio between 0% and 100% of the most uncertain samples are corrected or refined with a better estimate. De- pending on the evaluated task, we correct uncertain samples with the ground truth or refine the prediction using a larger model, better prompts, or task supervison. Good uncertainty estimation methods assign large scores to samples that are likely wrong. Therefore, a correction of those samples leads to better mloUs and the AUC increases, such that a better UQ estimation is indicated. For readability, we normalize the AUC between the best and worst possible results (rel. AUC). All corresponding absolute values can be found in the Appendix. We follow (Ravi al., 2024; Kirillov al., 2023) and choose the best mask proposal during evaluation.
4.2. Model Uncertainty
As commonly known, larger models have larger predictive ability compared to smaller ones. This also true for SAM. However, even if large models are often superior, they go along with larger energy consumption and hardware require- ments (Glandorf et al., 2023). Ideally, large model should only be used when the small model is significantly more uncertain and is expected to have a lower accuracy. Thus, we evaluate the ability to estimate the uncertainty stemming from the model. We predict the test set with the Tiny model
6

---

UncertainSAM Fast and Efficient Uncertainty Quantification
Model Selection
Prompt Refinement
Task Supervision
Segmentation Uncertainty
0.95
0.76
0.76
0.7
0.9
mlo
0.85
0.74
0.74
0.65
0.8
0.75
0.72
0.72
0.6
0.7
0.25
0.25
0.25
0.25
Ratio of Corrected Samples
Ratio of Corrected Samples
Ratio of Corrected Samples
Ratio of Corrected Samples
Oracle
Hust
SamSSore
Oracle
Hust
SomScore
Oracle
Huse
SamSScor
Oracle
Must SamSScore
Ab
Myy Mx; A Ap
My
A
Hx,
Ho MLP.
Figure 5. Performance gain while improving predictions selected with UQ on the COCO dataset. We evaluate the SamScore, mask entropy Hsus. the Bayesian entropy approximations Hy. H.A. x,,, He and our USAMg. A' Axy, Ag. The dashed line denotes an oracle estimation. Beginning from left right, the first plot shows the improvement when replacing ratio of the most uncertain predictions of the Tiny model with the Large models. The second plot shows the improvement, when using refined prompts to the most uncertain samples. The third and foruth, when the best mask in A selected ignoring the SamScore and if ratio of the most uncertain predictions is replaced by the ground truth. It shows that our MLPs are consistently superior to all other methods or on-par with the Bayesian approximation.
Table Prompt uncertainty quantification. The table shows the area under curve (AUC) when predicting a variable fraction of the most uncertain samples with refined prompt containing multiple point coordinates, others with a single-coordinate prompt. The
Table 3. Task uncertainty quantification. The table shows the area under curve (AUC) when predicting a variable fraction of the most uncertain samples with the correct task, otherwise with the one selected by the SamScore. The most uncertain samples are determined by the respective method.
uncertainty is determined by the respective method.
rrll, AUC in %)
DAVIS
ADE20k
MOSE
COCO
SA-V
rell AUC in %]+
DAVIS
ADE20k
MOSE
COCO
SA-V
SAN
SamScore Entropy
71.82
69.12
66.85
60.55
54.13
SAM
SamScore
68.36
66.31
70.58
64.09
58.87
77.13
70.15
69.24
68.05
63.56
Entropy
68.77
76.49
73.55
74.56
70.88
Baycs
Hy
54.11
60.82
63.74
61.25
55.78
Bayes
Hy
74.88
64.53
67.32
68.12
70.50
Hx.
80.75
79.41
74.33
74.20
67.69
HA
43.86
$2.79
66.04
57.55
78.13
USAM
Ax
75.04
83.23
74.21
78.17
10.
USAM
A
94.05
93.08
94,21
94.85
94.17
A
75.53
83.41
74.50
78.89
71.27
A
94.31
92.38
94.01
94.87
94.61
and re-predict ratio of the most uncertain samples with the Large model. The ratio parameter reflects the trade-off between accuracy and energy consumption.
ble shows that SAM with the largest backbone is superior in most situations on all datasets. Furthermore, Table 10 shows that the large model is notably more robust image corruptions and noise.
Table shows the AUC of the mloU with a corresponding plot given in Figure 5 (Model Selection). The Bayesian entropy He and our direct USAM MLP A are on-par and the most precise UQ methods. Moreover, the default UQ method SamScore is consistently outperformed by all other methods, except from Hy. The weak performance of Hy is reasonable, because it includes all uncertainty aspects including the task and prompt uncertainty that are irrelevant and potentially misleading for this evaluation. However, USAM performs often best and requires negligible com- putational efforts compared the Bayesian entropy and is therefore the preferred method to preserve energy. The only setting in which USAM is not competitive is on the small-scale DAVIS dataset, indicating that more data vari- ance is required for training which also observable in the next experiments. Additional insights about the impact of the model size are briefly analyzed in the Appendix: Ta-
4.3. Prompt Uncertainty
Similar the former experiment, evaluate the ability to quantify uncertainty stemming from prompts. We do this by predicting the dataset with a simple coordinate prompt in the centroid of the mask and replace a ratio of the most uncertain prompts with refined ones consisting of 8 equally distributed coordinate prompts from the ground truth mask.
The AUC of the mloU is reported in Table with a cor- responding plot in Figure 5 (Prompt Refinement). Similar to the former experiment, the Bayesian entropy Hx and our direct USAM MLP A'x perform superior with Ax being slightly better. The experiment shows that our method can be used to notify users when prompts lead to uncertain results.
7

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Table Uncertainty quantification. The table shows the area un- der curve (AUC) when correcting variable fraction of the most uncertain samples. The most uncertain samples are determined by the respective method. The non-corrected samples are predicted with the Large SAM model. Similar numbers are given for the Tiny, Small, and Base+ models in the Appendix, Table 14,
Table 6. Token ablation. The UQ performance of USAM when removing mask or IoU tokens from the MLP input on the COCO dataset, measured in relative AUC as in the main experiments.
Mask Token
IoU
Model
Prompt
Task
Token
Uncertainty
Uncertainty
Uncertainty
X
/'
61.19%
72.08%
86.89%
[rel. AUC in %t]
DAVIS
ADE20k
MOSE
COCO
SA-V
/
x
62.63%
76.42%
91.96%
SAM
SamScore
83.36
75.40
89.48
79.60
80.80
/
/
63.66 %
78.30 %
94.82 %
Hsua
82.00
70.23
82,09
74.27
79.60
Hy
38.79
46.87
56,58
45.36
57.32
Bayss
He
84.24
74.39
84,02
78.26
85.90
Table 7. Runtime of SAM with and without UQ methods on a
Hx
87.16
83.79
86.23
79.13
83.23
regular image performed NVIDIA RTX3050 Ti. Entropy is calculated on SAMs logit map. [7] and [ipp denote the number of applied image and prompt augmentatiins used for MC sampling. and USAM contains the calculation of all our proposed MLPs.
HA
52.24
52.69
66.07
56.96
60.51
USAM,
91.59
92.60
92,78
90.01
89.37
Compared to the runtime of all other UQ methods, our USAM is faster and casier to implement.
Table 5. Pearson correlation between different UQ measures on the COCO validation dataset using by the Large SAM model. loUcr denotes the real intersection over union between SAMs prediction and the ground truth. Equivalent tables obtained by the other SAM models available in the Appendix, Tables 15 to 17.
Seconds Benaiio
SAM
+Entropy
+/7|= 5
+/Xp|= =
+USAM
Large
0.437
0.452
2.187
0,500
0.441
Base+
0.205
0.233
1.028
0.289
0.210
Small
0.134
0.157
0.688
0.232
0.142
loUer
SamS
Hsu
My
Ha
HA
Hx,
USAM,
AG
A
Tiny
0.122
0.149
0.584
0.198
0.139
ToUer
1.00
0.49
0.10
0..09
0.42
0.71
0.26
0.17
.0.15
SamS
0.49
1.00
-0.44
-0.09
-0.63
-0.14
-0.4
0.58
-0.38
-0.53
-0.24
Hau
-0.30
0..4
1.00
0.41
0.81
0.36
0.74
.0.88
0.69
0.49
0.66
Hy
0.10
-0.09
0.41
1.00
0,26
0.42
0.39
0.17
0.23
0.35
0.27
quantification with the mIoU when correcting a ratio of the most uncertain samples with the ground truth. Table presents the AUC of the mloU scores. Without exception, our MLP USAM, that predicts the expected IoU clearly outperforms all other methods. Interestingly, the prompt and model uncertainty Hx, and He lead to acceptable results, too, indicating that those are the main causes of predictive uncertainty. It is important to note that the SamScore is not on-par even if it is optimized to solve exactly this task during SAM's training. A corresponding plot is given in Figure 5 (Segmentation Uncertainty).
fiee
-0.54
.0.63
0.81
0.26
1000
0.33
0.74
-0.60
0.62
0.45
0.51
НA
-0,09
-0.14
0,36
0.42
0.33
1.00
0.36
-0.11
0.31
0.17
0.17
Hx,
-0.42
0.41
0.74
0.39
0.74
0.36
1.00
-0.42
0.61
0.39
0.53
USAM,
0.71
0.17
00.11
0.42
0.37
00.21
Ab
.0.26
0.69
0.23
0.31
.0.37
0.35
A
-0.17
-0.53
0.49
0.33
0.45
0.17
0.39
-0.23
0.35
0.29
A5
-0.15
-0.24
0.66
0.27
0.51
0.17
0.53
-0.21
0.72
0.29
1.00
4.4. Task Uncertainty
During evaluation of the class-agnostic open set segmenta- tion problem, the best mask proposal per sample is used for evaluation. This does not reflect the workflow in real-world pipelines, in which the mask with the highest SamScore is chosen. Thus, uncertainty stemming from ambiguous tasks with multiple valid results can be used to ask for supervision.
4.6. Qualitative results and Discussion
Visual examples reveal insights about SAM's capabilities as shown in Figure 4 and in the Appendix. On the left side, samples with high and low prompt uncertainty Ax,
The UQ from ambiguous tasks is evaluated in Table 3. Dif- ferent to the former experiments, we choose SAMs mask proposal with the highest SamScore. Then, for ratio of the most uncertain samples, we replace it by the mask proposal that is best fitting to the ground truth. USAM is performing best with a large gap as visualized in Figure 5 (Task Super- vision). Interestingly, the Bayesian entropy is not a suitable approximation in this setting. However, the quality of our USAM superior and the favorable method.
[rel. %]
Predictive Uncertainty
SumsScove (0:45) (Kirilloe al., 2023)
Msa (0.49)
(Vujisinovid 3924)
Tesk Uncertainty
Model Uneartinity
0.2
Our Bayesian M, (0.36)
Our USAM A (0.68)
4.5. Segmentation Uncertainty
Prompi Uncertintty
Uncertainty correlates to the expected segmentation error. Thus, we measure the capability of predictive uncertainty
Figure 6. Uncertainty quantification capabilities on the COCO Dataset extracted from Tables I 4 with their enclosed areas.
8

---

UncertainSAM Fast and Efficient Uncertainty Quantification
are shown. With a single coordinate prompt in the centroid of the blue highlighted mask, SAM is able to segment the person, but struggles with the partially occluded bike. Better prompts could help to indicate the segments. In the middle column, the upper image has high task certainty A,, be- cause a coordinate prompt on the mobile is not pointing to other objects (e.g. the girl). Contrary, the centroid of the blue area in the lower image could belong to the person, the kite, or both together. With respect the model uncertainty Ae in the right column, visual inspection shows that the Tiny model is consistently accurate on clearly visible dogs, but drops on homogeneous structures as the zebras com- pared to the Large model. This indicates different strengths of the pre-trained SAM models.
to quantify uncertainty and can be used for benchmarking.
5. Conclusion
This paper formalizes UQ in the context of SAM by split- ting it into epistemic model as well as aleatoric prompt and task uncertainty. To model those uncertainties, we evaluate a theoretically grounded Bayesian entropy approximation. To overcome the computational costs, we introduce USAM that consists of deterministic MLPs to estimate the same uncertainties efficiently. Our experiments demonstrate com- pelling results, comparing the Bayesian and deterministic approaches with standard SAM UQ methods. We reliably identify the source of uncertainty in real-world applications, like prompt refinement, supervision of SAMs proposals, and adaptive model scaling, Remarkably, our lightweight USAM perform better or on-par with the Bayesian approach which are therefore the new -fateoof-the-arr Despite their simplicity, the improvements delivered by USAM, even for tasks similar to those handled by the SamScore, underscores the potential for future iterations of SAM to address UQ more effectively. Our contributions help to improve uncer- tainty handling mandatory in safety critical domains.
A view on the correlations between our calculated values in Table 5 helps to understand the behaviour of uncertainty and UQ. Compared to our MLP that estimates the IoU, the SamScore has a lower correlation to the overall uncertainty IOUGT- This is interesting, because both are trained on the same objective and ours only differs in the usage of the mask token as input, the optimization with SMAC3, and trained post-hoc. Furthermore, the IoUGr is highly correlated to the Bayesian model and prompt entropy (He, Hx₂) that explicitly modelled in this paper. The large correlation to He supports the findings that a lot of uncertainty stems from the model (Appendix, Tables 9 and 10).
Acknowwedgmentss
This work was supported partially by the German Federal Ministry of the Environment, Nature Conservation, Nu- clear Safety and Consumer Protection (GreenAutoMLATAAS project no. 67K132007A), the Federal Ministry of Educa- tion and Research (BMBF), Germany, under the Al service center KISSKI (grant no. 01IS22093C), the MWK of Lower Sachsony within Hybrint (VWZN4219), the Deutsche Forschungsgemeinschattt (DFG) under Germany's Excel- lence Strategy within the Cluster of Excellence PhoenixD (EXC2122), the European Union under grant agreement no. 136006 XTREME
The runtime comparison presented in Table 7 shows that the MLPs of our USAM perform UQ efficiently with the lowest computational overhead. Bayesian methods based on MC sampling heavily increase the computation time depending the number of samples. Also the calculation of the logit entropy is slower than the lightweight execution of our proposed MIPs.
To further investigate our method, we set either the IoU to- ken or the mask token to to remove potentially informative patterns. We evaluate using the tiny model across the three tasks described in the main experiments performed on the COCO dataset. The results that are shown in Table 6 indi- cate that both tokens contribute to the predictive capability, yet they remain accurate as standalone features. Both in combination lead to the best results.
Impact Statement
Uncertainty quantification leads to more reliable and inter- pretable predictions of machine learning algorithms. Es- pecially prior works in critical domains such as medical diagnosis show that the integration of uncertainty is impor- tant (Deng et al., 2023; Wang et al., 2019; Zhang al., 2023b). This paper contributes better knowledge about uncertainty in the context of the popular Segment Anything Model that is publicly available and employed in myriad applications. Additionally, our results have the potential to decrease the carbon footprint of those applications.
All in all, the lightweight and straight forward USAM leads to unexpected superior results which outperforms all other methods. A graphical comparison with corresponding closing area values is visualized in Figure 6. Since all results we present are applied post-hoc, an integration into the cost- intensive training of SAM may lead to improved results and should be considered in the future. We already provide USAM here. The Bayesian predictive entropy Hy does not lead to competitive results in the presented practical prob- lems. However, the well-performing He, Hx,, and HA show that Equation (4) includes all aspects that are relevant
To the best of our knowledge, uncertainty quantification in general is not a topic of interest for unethical applications or dual-use research of concerns.

---

UncertainSAM Fast and Efficient Uncertainty Quantification
References
Gottwald, T., Schier, M., and Rosenhahn, B. Safe resetless reinforcement learning: Enhancing training autonomy with risk-averse agents. In Proceedings of the European Conference on Computer Vision Workshops (ECCVW), 2024.
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P.,, Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappa- ganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and M. Automix: Automatically mixing language mod- els. Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS), 2024.
Gruber, C.,, Schenk, P. O., Schierholz, M., Kreuter, F, and Kauermann, G. Sources of uncertainty in machine learning a statisticians' view. arXiv preprint, 2023.
Charpiat, G., Hofmann, M., and Scholkopf, B. Automatic image colorization via multimodal predictions. Pro- ceedings of the European Conference on Computer Vision (ECCV), 2008
Buuzáán-rivera, A., Batra, D., and Kohli, Multiple choice learning: Learning produce multiple structured outputs. In Advances in Neural Information Processing Systems, 2012.
Deng, G., Zou, K., Ren, K., Wang, M., Yuan, X., Ying, S., and Fu, H. SAM-U: Multi-box Prompts Triggered Un- certainty Estimation for Reliable SAM in Medical Image. In In Proceedings of the Medical Image Computing and Computer Assisted Intervention Workshops (MICCAI), 2023.
Jaccard, P. Étude comparative de distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 1901.
Jiang, M., Zhou, J., Wu, J., Wang, T., Jin, Y., and Xu, M. Uncertainty-aware adapter: Adapting segment anything model (sam) for ambiguous medical image segmentation. arXiv preprint, 2024.
Ding, H., Liu, C., He, S., Jiang, X., Torr, P. H., and Bai, S. MOSE: A new dataset for video object segmentation in complex scenes. In Proceedings of the International Concference on Computer Vision (ICCV), 2023.
Jsang, A.
Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incor- porated, 2018.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
Kaiser, T., Ehmann, L., Reinders, C., and Rosenhahn, B. Blind knowledge distillation for robust image classifica- tion. arXiv preprint, 2022.
Kaiser, T., Reinders, C., and Rosenhahn, B. Compensation learning in semantic segmentation. In Proceedings of Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023.
Gal, Y. and Ghahramani, Z. Bayesian convolutional neural networks with bernoulli approximate variational infer- ence. arXiv preprint, 2015.
Kaiser, T., Schier, M., and Rosenhahn, B. Cell track- ing according to biological needs-strong mitosis-aware multi-hypothesis tracker with aleatoric uncertainty. arXiv preprint arXiv:2403.150111 2024.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian ap- proximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference Machine Learning (ICML), 2016.
Kirchhof, M., Kasneci, G., and Kasneci, E. Position: Uncertainty quantification needs reassessment for large language model agents. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL https::/p///.....t... htt.tttssss/////neeeeeett /ppenevieneeeeeeeeeeeeeeeee net./ forum? id=Lrv20S5R2v.
Gawlikowski, J., Tassi, C., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., Shahzad, M. Yang, W., Bamler, R., and Zhu, X. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 2023.
Glandorf, P., Kaiser, T., and Rosenhahn, B. Hypersparse neural networks: Shifting exploration to exploitation through adaptive regularization. In Proceedings of the International Conference on Computer Vision Workshop (ICCVW), 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-.., al. Segment anything. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Goan, E. and Fookes, C. Bayesian neural networks: An in- troduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean- Morlet Chair, Fall 2018, 2020.
Kruse, M. and Rosenhahn, B. Multi-flow: Multi-view- enriched normalizing flows for industrial anomaly detec- tion. In CVPR Workshops 2025, June 2025.
10

---

UncertainSAM Fast and Efficient Uncertainty Quantification
Lahlou, S., Jain, M., Nekoei, H., Butoi, V. I., Bertin, P., Rector-Brooks, J., Korablyov, M., and Bengio, Y. DEUP:
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.., Mintun, B., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. Sam 2: Segment anything in images and videos. arXiv preprint, 2024.
Direct epistemic uncertainty prediction. Transactions Machine Learning Research, 2023.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Process- ing Systems, 2017.
Ren, S., Luzi, F., Lahrichi, S., Kassaw, K., Collins, L. M., Bradbury, K., and Malof, J. M. Segment anything, from space? In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2024.
Li, C., Li, W., Liu, H., Liu, X., Xu, Q. Chen, Z., Huang, Y., and Yuan, Y. Flaws can be applause: Unleashing potential of segmenting ambiguous objects in sam. Ad- vances in Neural Information Processing Systems, 37: 45578-45599 2024.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhurdinov, R. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 2014.
Lin, T.-Y., Maire, M., Belongie. S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L.,, and Dollár, P. Microsoft coco: Common objects in context. arXiv preprins, 2015. 2015.
Tancik, M., Srinivasan, P., Mildenhall, B., -ridovich-Keil,, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Bar- ron, J., and Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 2020.
Lindauer, M., Eggensperger. K., Feurer, M., Biedenkapp. A., Deng. D., Benjamins, C., Ruhkopf, T., Sass, R., and Hutter, F. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 2022.
Vujasinović, S., Becker, S., Bullinger, S., Scherer- Negenborn, N., Arens, M., and Stiefelhagen, R. Strike the balance: On-the-fly uncertainty based user interactions for long-term video object segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2024.
Liu, K., Price, B., Kuen, J., Fan, Y., Wei, Z.,, Figueroa, L., Geras, K., and Fernandez- Granda, C. Uncertainty- aware fine-tuning of segmentation foundation models. Advances in Neural Information Processing Systems, 37: 53317-53389, 2024.
Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., and Vercauteren, T. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing, 2019.
Mooney, C. Z. Monte carlo simulation. Sage, 1997.
Paul, R. D., Quercia, A., Fortuin, V., Nöh, K., and Scharr, H. Parameter-efficient bayesian neural networks for uncertainty- depth estimation. arXiv preprint arXiv:2409. 17085, 2024.
Wehrbein, T., Rudolph, M., Rosenhahn, B., and Wandt, B. Utilizing uncertainty in 2d pose detectors for proba- bilistic 3d human mesh recovery. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), February 2025.
Perazzi, F., Pont- Tustt, J., McWilliams, B., Van Gool, A. benchmark
L., Gross, M., and dataset and evaluation methodology for video object seg- mentation. In Computer Vision and Pattern Recognition, 2016.
Wu, and Xu, M. One-prompt to segment all medical images. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Quillent, A., Bismuth, V., Bloch, I., Kervazo, C., and Lad- jal, S. Deep-learning uncertainty estimation for data- consistent breast tomosynthesis reconstruction. Pro- ceedings of the International Symposium on Biomedical Imaging (ISBI), 2024.
Xu, Y., Tang. J., Men, A., and Chen, Q. Eviprompt: A training-free evidential prompt generation method for adapting segment anything model in medical images. Transactions on Image Processing, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICMI.), 2021.
Yan, J., Wang, P., Wang, D., Huang, W., Watzenig, D., and Yin, H. Segment-anything models achieve zero-shot robustness in autonomous driving. In Proceedings of the International Automated Vehicle Validation Conference (IAVVC). IEFE, 2024a.

---

UncertainSAM- - Fast and Efficient Uncertainty Quantification
Yan, Z.,, Sun, W., Zhou, R., Yuan, Z., Zhang, K., Li, Y., Kim, S., Song, S., Ren, H., Liu, T., Li, Q., Li, X., He, L., and Sun, L. Biomedical SAM-2: Segment anything in biomedical images and videos. In Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond, 2024b.
Zhang, C.,, Liu, L., Cui, Y., Huang, G., Lin, W., Yang, Y., and Hu, Y. A comprehensive survey segment anything model for vision and beyond. arXiv preprint, 2023a.
Zhang, Y., Hu, S., Jiang, C., Cheng, Y., and Qi. Segment anything model with uncertainty rectification for auto- prompting medical image segmentation. arXiv preprint, 2023b.
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 2019.
12

---

UncertainSAM Fast and Efficient Uncertainty Quantification
A. Details to A Bayesian Entropy Approximation
Section introduces Bayesian entropy approximation that formalizes a UQ baseline for our evaluation based on known Monte Carlo sampling concepts. For better reproducability, we recapitulate some main aspects with more detailed notations.
The predictive uncertainty from Equation (2) is intractable. Thus, approximate it by combining ideas of former sampling methods (Wang et al., 2019; Deng et al., 2023; Lakshninarayanan et al., 2017). We predict set of predictions )' ... The cardinality is defined by multiple sampling sets K |7] [Pp] x |A| | We use sets of 5 image augmentations E T, the identity (i.e. no pertubation), vertical flip. JPEG compression (10% and 30% image quality). Gaussian blur with a kernel size of 5 pixel, and Gaussian noise. The prompts Zp Xp are randomly sampled point-coordinate drawn equally distributed from the ground-truth mask that corresponds a. The 3 task estimations A are defined by SAMs mask predictions and encode the three most likely tasks. Finally, the 4 publicly available pre-trained models of different size define the weights O {L, B+, S, (Large, Base+, Small, and Tiny). It important to note that the prompts Xp are depending on Ii and a, and the estimated mask proposals A depend on the image 21, prompt Tp, and model 0. Correctly, they need to be denoted as (x1 a) and A(z1, Fp, 0). We omit the correct notation due to space limitations. We assume that the relation is evident and self-explaining in the equations.
The probabilities in Equation (2) underlined with Prompt Decoder, Prompt Encoder, and Epistemic are not defined. Thus, we set the prompt probability and the model probability to
1 Xpp
(x)))
and
(8)
1
p(0ID)
(9)
O
Moreover, we approximate the task probability p ((()((()) ap, 0) using the SamScore. Since SAM was trained on large scale dataset with large set of tasks, we assume SAM to inherently estimate a better IoU to more frequent, i.e. more likely, tasks. We normalize over all three SamScores of prediction to get the task probability:
SamScore
p(a)t(m)). xp,)) Σ EacAuce). SamScore(a)
(10)
Applying the sampling sets and the probability definitions, the predictive uncertainty is defined as
((a(((m)) xp,
pyy 1y= x
Σ
K7l- p(()((=)))
(11)
(t.)),,) TxXpxOxA
All further details can be found in the main paper.
13

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Table Hyperparameter optimization results using (Lindauer et al., 2022) in descending order. The results are obtained during training of
the USAM, (1) model trained on ADE20K. Variations are performed on the batch size, epochs, learning rate, and momentum.
(MSE)
Batch
Epochs
Learning Rate
Momentum
Rank
Loss (MSE)
Size
Espochs
Learning Rate
Momentum
0.02436
106
79
0.00131
0.83033
36
0.02505
164
0.00186
0.82966
0.02436
79
0.83276
37
0.02518
0,01451
0.51234
0.02437
79
0.82986
38
194
0.0013
0.80417
0.02437
79
0.00116
0.83177
39
0.02521
193
0.00126
0.815
0.024388
79
0,00112
0.832
40
0,02532
224
0.00148
0.79977
0.02439
193
70
0,00242
0.81485
41
0,02536
208
0,04111
0.24591
0.02439
176
79
0.00148
0,82823
42
0.02555
27
0.00241
0.86083
0.02439
98
80
0.00134
0,83175
43
0.0256
129
72
0.00064
0.8245
0.0244
106
76
0.00131
0,83136
44
0.02561
19
75
0.00211
0.83676
10
0,02441
98
74
0.00128
0,83464
45
0.02564
93
68
0.0275
0,58042
11
0.02442
98
75
0.00128
0,83464
46
0.02566
199
30
0.03576
0.64499
12
0.02443
102
80
0.00108
0.82989
47
0.02569
80
0.01084
0.78907
13
0.02444
102
79
0.00118
0.83004
48
0.0257
215
40
0.057
0.64422
0.02446 14
0.02445
153
68
0.00248
0.82004
49
0.02575
46
31
0.02255
0.25254
167
79
0.00132
0.82778
50
0.02576
2.66
0.08722
0,52532
0.02446
95
74
0.00143
0.82605
51
0.02577
158
0.02941
0.70937
0.02447
68
0,00235
0.81844
52
0.02579
0.0467
0.024447
53
0.83178
0.02448
70
0.82183
54
128
0.06786
0,48176
0.02451
62
0,00183
0,83002
55
0.02624
220
0.03376
0.86587
0.02452
67
0,00128
0.78672
56
0.02636
170
0.04239
0.62245
0.02453
107
80
0.00215
0,83304
$7
0.0265
103
0.0888
0.38777
0.02457
106
79
0.00208
0,83412
58
0.02667
223
0.03961
0.88511
24
0.02457
94
75
0.00214
0. 0.82338
59
0.02675
151
63
0,07027
0.30754
25
0,02457
95
69
0.00227
0,81132
60
0.02707
80
53
0,06813
0,56828
26
0.12465
182
70
0.00144
0.83128
61
0.0274
100
77
0.00023
0,84102
27
0.0247
164
51
0.00198
0.82948
62
0.02757
119
35
0.03685
0,85836
28
0.02472
97
G7
0.00245
0.8231
63
0.02767
121
39
0.03343
0,88359
29
0.02474
91
75
0.00301
0.83894
64
0.02852
58
49
0.05615
0.57001
0.02481
233
47
0.01569
0.12675
65
0.02874
228
0.00175
0.27064
0.02489
67
0.74238
66
0.02907
170
0,09892
0.41356
0.02494
27
0.00131
0,83092
67
0.0292
0.00125
0.02496
35
0.01568
0.48597
68
0.02988
0.0018
0.34614
0.125
36
0.02999
224
0.00143
0.33011
0.025
80
0.00131
0.83033
70
0.03083
23
0.07553
0.65109
14

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Table ratio samples per dataset where models different size perform best. Larger models consistently perform better average, but not on every individual sample. The performance gap between large and small models depends the dataset.
Table 11. Model uncertainty quantification. The table shows the area under curve (AUC) when predicting variable fraction of the most uncertain samples with the largest model, others with the tiny one. The uncertainty is determined by the respective method. Supplementary data Table 1. Compared Table 1, this table shows absolute AUC, To help interpreting the values, the best (Oracle) and worst (Worst) possible correction order based on the uncertainty score added.
Dataset MOSE
Ratio
DAVIS
ADE20K
SA-V
Model Size
Tiny
6.38%
18.24%
7.85%
15.64%
15.72%
Small
10.40%
20.55%
11.06%
17.65%
18.39%
Base+
27.76%
27.69%
25.63%
27.46%
28.01%
Large
55.44%
33.50%
55.44%
39.24%
37.86%
Standard Measures
[UCC in %]
DAVIS
ADF20k
MOSE
COCO
SA-V
Oracle Worst
82,518
64.603
82.131
76,032
81.805
77.778
59.761
77.685
71.169
77.235
SAM
SamScore
80.823
62.320
80.728
73,988
79.924
Hsu
81.180
62,351
80.867
74.165
80.
Hy
80.218
62.282
80.414
73.808
79.767
Bayes
He
81.260
62.553
81.047
74.302
80.236
HA
80.475
62,233
80.478
73.794
79.805
H
81,236
62.424
81.016
74.320
80.224
Classifiers Trained on DAVIS
[UUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USAM
Ag
80.946
62,396
80.513
73.852
79.528
A9
80.578
62.397
80.427
73.815
79.708
Classifiers Trained on ADE20K
|UC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USKM
A
80.996
62.682
80.690
73,983
79.917
Table The impact image augmentations and noise (Averaged over all datasets) model performances evaluated with models with different size. We flipped and scaled the image, removed color (Gray) and high-frequen patterns (JPEG), and added Gaussian blur or noise. All models are scale and flip invariant but decrease the performance when adding noise. Interestingly, the smaller model with fewer weights is more prone for most image noise.
A9
80.417
62.741
80.326
73.849
79.834
Classifiers Trained on COCO
AUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USAM
Ae
81.044
62.451 62.482
80.877
74.265
80.105
A
81.105
80.933
74.408
80.145
Classifiers Trained on SA-V
[mloU in %
Normal
Flip
Scale
Augmentation Gma
GBlur
GNoise
JPEG
[AUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
Model Size
Tiny
76.47
-0.01
-0.07
-2.06
-8.08
-19.36
-16.80
Ae
81.038
62.457
80.856
74.130
80.034
Small
77.42
-0.03
-0.09
-1.11
-8.54
11.55
-15.97
A4
81.040
62.417
80.898
74,239
80. 147
Base+
78.64
.0.05
0.06
11588
8.77
17.94
16.62
Large
79.58
.0.05
-0.10
-1.45
-8.67
-14.8
-1299
15

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
4.96%
A A A 44.44%
12.09%
A 28.20% A
A Xp 11.44%
6.53%
A 7.35%
A 45.79%
A 6.17%
A X 14.54%
A X 1.92%
A 42.86%
8.22 % A A A
A 8.99%
A A 5.01%
4.84%
A A 39.30% A 7.03%
A N 6.96%
A Ni
8.20%
A 42.46 %
A 28.78 %
A 5.30%
A X2 10.50%
A 6.60%
A A 40.21 %
A 6.06 %
A 8.37%
A 3.94%
A A 36.09% %
A A 45.67%
A 4.92%
A XP 12.97%
A 3.69%
A 20.16%
A 7.81%
A 10.83% %
A X 9.02%
A 5.42%
A 44.65%
A 39.17% %
A' = 33.69%
AS 6.18%
7.02% A
A 5.48%
A 7.52%
A N/P
8.53% %
A 11.40% %
X
A A
46.16% %
46.50 1 A
A A
33.61% %
A 5.32%
AS
4.77%
A 6.61
Figure 7. Samples with uncertainty estimations. Complementary samples to Figure 4. The ground truth mask is visualized by the blue mask SAM's prediction of Tiny model in green.
16

---

UncertainSAM- - Fast and Efficient Uncertainty Quantification
A 1.19% A
A X 8.52%
X 30.59%
A A 36.07% %
A X 8.38%
A 2.42%
A 5.57%
A 37.26% %
A 4.71%
À Ai 1.71%
A X 6.84% %
A 44.45%
30.04% A A
4 2/F 1.4%%
A 2.44%
9.29% A e
A 21.84%
A 4.15%
A' 0.01% % A 0.07% % A 1.71%
A X A A 7.13% A 2.92%
0.36 %
A 2.12 %
A X 0.80 %
A 3.28%
A X 2.47%
A A A A A 8 3.16%
0.26 %
A 14.92%
0.96 %
A 3.17%
A 5.54%
A A 42.63%
A 4.16%
A Xp 2.31%
A N 1.82%
A Ni. 9.95%
A 0.39 %
A A 10.12%
A 37.46%
A 4.73%
4.17%
A 5.94%
A X 2 %
A
X = 0.79%
A = 2.63%
A 1 %
A A 4.12%
A 29.90% %
A 6%
A 3.03%
A 3.17%
Figure 8. Samples with uncertainty estimations. Complementary samples to Figure 4. The ground truth mask is visualized by the blue mask SAM's prediction of Tiny model in green.
17

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Table 12. Prompt uncertainty quantification. The table shows the under curve (AUC) when predicting variable fraction of the most uncertain samples with refined prompt containing multiple point coordinates, others with a single-coordinate prompt. The uncertainty is determined by the respective method. Supplementary data Table Compared to Table AUC. To help interpreting the values, the best (Oracle) and worst (Wors)) possible correction order based on the uncertainty score is added.
Table 13. Task uncertainty quantification. The table shows the area under curve (AUC) when predicting a variable fraction of most uncertain samples with correct task, otherwise with the one selected by the SamScore. The most uncertain samples are determined by respective method. Supplementary data Table 3. Compared to Table 3, this table shows absolute AUC. To help interpreting the values, the best (Oracle) and worst (Worst) possible correction order based the uncertainty
table shows absolute
added.
Standard Measures
Standard Measures
[AUC
DAVIS
ADE20k
MOSE
COCO
SA-V
[AUC in %]
DAVIS
ADF20k
MOSE
COCO
SA-V
Oracle Worst
74.583
59.938
76.687
70.635
76.688
Oracle
82.483
70.917
80.912
75.159
81,177
60.567
51,860
70.400
61.273
66.643
Worst
77.500
61.976
76.523
70.854
76.341
SamScore
70.149
57.216
74.837
67.273
72.556
SAM
SamScore
81.079
68.156
79.457
74.066
78.959
Hid
70,205
58.039
75.024
68.253
73.762
Hsu
81.343
68.248
79.562
74.464
79.415
Hy
71.063
57.073
74.633
67.650
73.724
Hy
80.197
67.414
79.321
74.103
79.039
Bigg
He
81,618
68.616
79.790
74.616
79.532
Bayes
He
57.827
74.956
67.916
73,517
HA
66.715
56.125
74.552
66.661
74.490
AA
80.428
66.723
79.248
73.794
78.960
Hx
81.524
69.077
79.786
74.790
79.615
H
69.112
56.870
74.810
67.678
73.524
Classifiers Trained on DAVIS
Classifiers Trained on DAVIS
[UCC %b]
DAVIS
ADE20k
MOSE
COCO
SA-V
[AUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USAM
A %
81.240
67.801
79.362
73.931
79.047
USAM
A
73.749
58.506
75.625
69.338
73.131
A
73.785
58.289
75.307
69.096
72.860
A 1
81.264
67.671
79.374
74.084
78.989
Classifiers Trained on ADE20K
Classifiers Trained ADE20k
[AUC %
DAVIS
ADE20k
MOSE
COCO
SA-V
[UC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USAM
x A
81.386 81.227
69.418
79.555
74.563
79.276
USAM
A
73.682
59.379
76.223
69.916
75.767
4
73.581
59.323
75.968
69.712
75.458
69.434
79.449
74.480
79.213
Classifiers Trained on COCO
Classifiers Trained COCO
[AUC in %
DAVIS
ADE20K
MOSE
COCO
SA-V
[AUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
USAM
Aus
81,594
68.903
79.780
75.001
79.626
USAM
A
73.958
59.274
76.323
70.152 70.155
75.946
A
74.135
59.268
76.311
75.970
A
81.610
69.081
79.793
75.039
79.656
Classifiers Trained on SA-V
Classifiers Trained on SA-V
[AUC in %
DAVIS
ADE20k
MOSE
COCO
SA-V
[UCC in %}
DAVIS
ADE20K
MOSE
COCO
SA-V
USAM
Axs
81,507
68.769
79.741
74.817
79.734
USAM
A A
73.680
59.165
76.304
70.009
76.102
A'
73.842
59.104
76.266
69.984
76.146
A x,
81.488
68.832
79.723
74.824
79.788
18

---

UncertainSAM - Fast and Efficient Uncertainty Quantification
Table 14. Uncertainty quantification. The table shows the area under curve (AUC) when correcting a variable fraction of the most uncertain samples. The most uncertain samples are determined by the respective method. Oracle denotes an optimal uncertainty quantification. Supplementary material to Table 4 in the main paper. Compared to Table 4, this table shows absolute AUC. To help interpreting the values, the best (Oracle) and worst ( (Worst) possible correction order based the uncertainty score is added.
JoUGr
SamS
Hag
Hy
He
HA
Hx.
USAM,
A2
43
4%
loUar
1.00
0.46
-0.39
0.06
+0.63
-0.12
40.53
0.76
-0.37
-0.20
-0.25
SamS
0.46
1.00
-0.63
-0.22
-0.68
-0.31
-0.54
0.63
-0.56
+0.82
-0.36
Hsa
0.39
-0.63
1.00
0.41
0.81
0.36
0.74
-0.48
0.69
0.49
0.66
Hy
0.06
-0.22
0.26
0.39
0.11
0.23
0.27
He
-0.68
0.26
1.00
0.74
-0.66
0.62
0.51
HA
-0.12
-0.31
0.36
0.42
0.33
0.36
-0.15
0.31
0.17
0.17
Hx.
0.53
-0.54
0.74
0.39
0.74
0.36
1.00
-0.51
0.61
0.39
0.53
USAM,
0.76
0.63
-0.48
0.11
-0.60
-0.15
-0.51
1.00
-0.52
-0.27
-0.35
Model Tiny
A3
0.37
-0.56
0.69
0.23
0.62
0.31
0.61
-0.52
1.00
0.35
0.72
(AUC in %]]
DAVIS
ADE20k
MOSE
COCO
SA-V
A7
-0.20
40.82
0.49
0.33
0.45
0.17
0.39
-0.27
0.35
1.00
0.29
A3
.025
-0.36
0.66
0.27
0.51
0.17
0.53
-0.35
0.72
0.29
1.00
Oracle
94.621
88.322
93.696
91.501
94.473
Worst
83.036
72.780
83,647
80.491
83,564
Table 15. Correlation between different UQ measures on the COCO validation dataset using by the Thyy SAM model. IoUcr denotes the real intersection over union between SAMs prediction and the ground truth. Supplcmentary data to Table 5.
Bayes SAM
SamScore
92.274
84.695
92.366
88,897
92.482
Hsua
92.353
83.647
91.614
88,309
92.355
Hy
87.783
80.166
89.724
85.727
89.983
He
93.349
84.693
92.638
89,597
93.256
H
92.773
85.433
92.218
88,975
92.740
HA
89.219
80.688
90.258
86.697
90.441
USAMr
93.665
87.074
92.852
90.322
93,222
Model Small
loU
SamS
Hug
Hy
He
HA
Hy,
USAM;
Ag
A%
[AUC in %6]
DAVIS
ADE20k
MOSE
COCO
SA-V
loUlT
1.00
0.45
-0,38
0.07
-0.62
-0.12
-0.50
0.74
-0.34
-0,20
-0.23
SamS
0.45
1.00
-0.56
-0.22
-0.68
-0.27
-0.50
0.57
-0.49
40.69
-0.33
Oracle
95.087
88.684
94.182
91.878
94.718
Hsa
-0.38
-0.56
1.00
0.41
0.81
0.36
0.74
-0.48
0.69
0.49
0.66
Worst
83.816
73.101
84.491
81.022
84.064
Hy
0.07
-0.22
0.41
1.00
0.26
0.42
0.39
0.12
0.23
0.33
0.27
Bayes SAM
SamScore
92.935
84,967
92.900
89.277
92.885
Hi
.0.68
0.26
0.74
0.62
0.51
Hsua
93.071
84.068
92.305
88.889
92.772
HA
-0.12
-0.27
0.36
0.42
0.33
1.00
0.36
-0.15
0.31
0.17
0.17
Hx.
-0.50
-0.50
0.74
0.39
0.74
0.36
1.00
-0.50
0.61
0.39
0.53
Hy
88.348
80.462
90.299
86,147
90.320
USAMg
0.74
0.57
-0.48
0.12
0.66
10.15
-0.50
1.00
-0.48
-0.28
-0.32
He
93.768
84,949
93.064
89.915
93.489
A4
-0.34
-0.49
0.69
0.23
0.62
0.31
0.61
-0.48
1.00
0.35
0.72
Hx,
93.326
85,857
92.764
89,436
93.066
A
-0.20
-0.69
0.49
0.33
0.45
0.17
0.39
-0.28
0.35
1.00
0.29
AA
89.808
81.145
90.803
87.156
90.761
A
-0.23
-0.33
0.66
0.27
0.51
0.17
0.53
-0.32
0/22
0.29
1.00
USAMs
94.143
87.489
93.431
90.793
93.608
Table 16. Correlation between different UQ measures on the COCO validation dataset using by the Small SAM model. loUGr denotes the real intersection over union between SAMs prediction and the ground truth. Supplementary data to Table 5.
Model Base+
[AUC in %]
DAVIS
ADE20K
MOSE
COCO
SA-V
Oracle
95.839
89.173
94.896
92.401
95.
Worst
85.224
73.586
85,881
81.766
84.941
Bayes SAM
SamScore
93.502
85.189
93.786
89.372
93.247
Hssa
93.743
84,395
93.225
89.311
93.231
Hy
89.380
80.819
91.125
86.757
90.863
He
94.361
85.225
93.650
90.313
93.813
loUar
SamS
Hya
Hy
Hg
HA
Hag
USAMs,
Ag
AG.
94.313
86.634
93.680
89.938
93.584
loUar
1.00
0.35
.0.35
0.07
10.59
_0.11
10.46
0.72
.0.31
.0.20
.0.20
HA
90.407
81.868
91.736
87.642
91.282
Sam'
0.35
1.00
-0.52
-0.29
40.61
-0.27
-0.45
0.42
-0.42
-0.66
-0.32
USAMg
94.994
87.998
94,200
91.328
93.981
Fgu
-0.35
-0.52
1.00
0.41
0.81
D.
36
0.74
-0.45
0.69
0.49
0.66
Hy
0.07
.0.9
0.41
1.00
0.26
0.42
0.39
0.12
0.23
0.33
0.27
Model Large
le
0..9
00.61
0.81
0.26
1.00
D.33
0.74
-0.64
0.62
0.45
0.51
[ACC in
DAVIS
ADE20k
MOSE
COCO
SA-V
-0.11
.0.7
0.36
0.42
0.33
1.00
0.36
-0.14
0.31
0.17
0.17
Hx
-0.46
.0.55
0.74
0.39
0.74
0.36
1.00
-0.48
0.61
0.39
0.53
Oracle
96,291
89,361
95,394
92.735
95.407
USAMs.
0.72
0.42
-0.45
0.12
40.64
-0.14
-0.48
1.00
-0.44
-0.28
.0.29
Worst
86.336
73.896
87.080
82.473
85.595
Ab
-0.31
-0.42
0.69
0.23
0.62
0.31
0.61
-0.44
1.000
0.35
0.72
Bayes SAM
SamScore
94.635
85,556
94.519
90.641
93.523
AA
-0.20
0.66
0.49
0.33
0.45
0.17
0.39
-0.28
0.35
1.00
0.29
A5.
.0.20
.0.32
0.66
0.27
0.51
0.17
0.53
-0.29
0.72
0.29
1.00
Haa
94,500
84,757
93.905
90,095
93.405
Hy
90.198
81.144
91.784
87.128
91.219
Table 17. Correlation between different UQ measures on the COCO validation dataset using by the Base+ SAM model. IoUar denotes the real intersection over union between SAMs prediction and the ground truth. Supplementary data to Table 5.
Ho
94.722
85.400
94.065
90.504
94.023
Hx,
95.013
86.854
94.249
90,593
93.762
HA
91.536
82,045
92.572
88.318
91.533
USAMr.
95.454
88.217
94.793
91.710
94.365
19


Reviews:
[{"review_id": "Lpt18soJZQ", "reviewer": "Reviewer_nnaG", "summary": "The paper introduces UncertainSAM (USAM), a method for uncertainty quantification (UQ) in the Segment Anything Model (SAM). By decomposing uncertainty into epistemic (model), aleatoric (prompt/task), and task ambiguity components, USAM employs a Bayesian entropy framework and lightweight MLPs to estimate uncertainty efficiently. Experimental results across multiple datasets demonstrate USAM's superiority over SAM's inherent confidence score and standard UQ baselines, offering a practical solution for applications requiring reliability and efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the detailed feedback and comprehensible remarks that will help enhance the paper. Our responses to the proposed improvements are as follows:\n\n---\n\n## Weakness 1 \n\nWe only partially agree with the statement that we do not compare to SAM-specific UQ. For example, our Prompt UQ is closely related to Deng et al. (2023, see main paper for reference). Furthermore, SAM-specific UQ is usually application-specific and not readily translatable to our holistic view of UQ. However, this weakness will be addressed with a modification of the paper, which is further elaborated in our response to reviewer 3EoJ, who identified a similar concern. Please see response to reviewer 3EoJ for the details.\n\n---\n\n## Weakness 2\n\nWe agree that the **architecture** is underdescribed. It is motivated by SAM's original MLP design for predicting the SamScore. We differ only in the input dimension and hidden state (from 256 to 512) because we employ and concatenate both tokens. We suggest the following modifications to clarify this in the paper:\n\n- Line 237: \"The simple design of the MLPs aligns with the architecture of SAM’s inherent MLP for predicting the SamScore, as described in the Appendix of Kirillov et al. (2023). They have three layers and a sigmoid activation, but we use 512 input dimensions and hidden states due to the concatenated input tokens. The architecture is visualized in Figure XX.\"\n- In addition, we will include a figure presenting an MLP with the input tokens.\n\nRegarding **SMAC3:** Due to space limitations, we presented the upper and lower hyperparameter bounds optimized using SMAC3 in ll. 247–252 (right column). We suggest adding a more detailed description and presenting the per-iteration results in tabular form. Since the optimization is not the primary focus of the paper, we will include this in the Appendix and refer to it in line 252. The SMAC3 results will provide greater transparency regarding the robustness of the proposed method.\n\n---\n\n## Weakness 3\n\nOur proposed MLPs are very small compared to the entire SAM model and require only a single forward pass, unlike MC-based methods. However, as mentioned by the reviewer, an explicit performance comparison should be included in the final version of the paper to demonstrate the impact of our method on efficiency. To address this, we designed hardware-dependent runtime experiments and added them to our anonymous code repository at https://anonymous.4open.science/r/UncertainSAM-C5BF/scripts/speed_test.ipynb for reproducibility.\n\nTo illustrate the practical efficiency impact, we conducted tests on a small consumer GPU (GeForce RTX 3050) using setups described in the paper. We compare us to vanilla SAM, entropy, and MC sampling (image and prompt) as employed in our paper. Notably, we performed the test with all nine (!) MLPs presented in our paper. Thus, the efficiency of our method could be increased by up to a factor of nine if only specific MLPs are necessary for a given application. It shows that our method is superior to all other extended UQ methods. We thank the reviewer for this suggestion and propose adding the following results to the camera ready:\n\n\n**Large Model**\n| Config                   | Time (s/iter) | Params          |\n|--------------------------|--------------|----------------|\n| SAM2                     | 0.43796      | 224.4M         |\n| SAM2 + Entropy           | 0.45278      | 224.4M         |\n| SAM2 + 5 MC img aug      | 2.18681      | 224.4M         |\n| SAM2 + 8 MC prompt aug   | 0.50030      | 224.4M         |\n| SAM2 + 9 MLPs **(USAM)**     | **0.44144**      | 224.4M + 9×526K |\n\n**Base+ Model**\n| Config                   | Time (s/iter) | Params          |\n|--------------------------|--------------|----------------|\n| SAM2                     | 0.20538      | 80.8M          |\n| SAM2 + Entropy           | 0.23294      | 80.8M          |\n| SAM2 + 5 MC img aug      | 1.02753      | 80.8M          |\n| SAM2 + 8 MC prompt aug   | 0.28902      | 80.8M          |\n| SAM2 + 9 MLPs **(USAM)**     | **0.20966**      | 80.8M + 9×526K |\n\n**Small Model**\n| Config                   | Time (s/iter) | Params          |\n|--------------------------|--------------|----------------|\n| SAM2                     | 0.13444      | 46.0M          |\n| SAM2 + Entropy           | 0.15716      | 46.0M          |\n| SAM2 + 5 MC img aug      | 0.68734      | 46.0M          |\n| SAM2 + 8 MC prompt aug   | 0.23210      | 46.0M          |\n| SAM2 + 9 MLPs **(USAM)**     | **0.14160**      | 46.0M + 9×526K |\n\n**Tiny Model**\n| Config                   | Time (s/iter) | Params          |\n|--------------------------|--------------|----------------|\n| SAM2                     | 0.12166      | 38.9M          |\n| SAM2 + Entropy           | 0.14876      | 38.9M          |\n| SAM2 + 5 MC img aug      | 0.58362      | 38.9M          |\n| SAM2 + 8 MC prompt aug   | 0.19736      | 38.9M          |\n| SAM2 + 9 MLPs **(USAM)**     | **0.13899**      | 38.9M + 9×526K |", "claims_and_evidence": "good", "methods_and_evaluation_criteria": "good", "theoretical_claims": "good", "experimental_designs_or_analyses": "good", "relation_to_broader_scientific_literature": "Please see weakness", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. It introduces a novel decomposition of uncertainty into epistemic, prompt, and task sources, addressing SAM's unique class-agnostic nature. This provides a nuanced framework for understanding SAM's limitations. WDSI effectively addresses the challenge of progressive occlusion, where parts of objects become invisible due to overlapping or changes in camera viewpoint. By leveraging hypergraph representations and the WSW metric, the framework ensures reliable tracking and segmentation even in occlusion-heavy scenes.\n\n2. USAM’s lightweight MLPs achieve state-of-the-art performance while avoiding the computational overhead of Bayesian methods. Extensive experiments on five datasets (DAVIS, ADE20k, MOSE, COCO, SA-V) validate USAM’s effectiveness in tasks like model selection, prompt refinement, and task supervision.\n\n\nWeaknesses\n1. While USAM outperforms SAM’s confidence score and entropy-based methods, it does not compare against recent SAM-specific UQ approaches (e.g., prompt-augmented methods), leaving its relative novelty unclear. And, though USAM is claimed to be efficient, no runtime or memory benchmarks are provided to quantify gains over Bayesian sampling or other UQ methods. This information would help researchers understand the practical feasibility of implementing USAM.\n\n2. The MLP architecture and SMAC3 optimization setup are underdescribed. The author should provide more structural diagrams to clarify.\n\n3. As a lightweight method, the authors should add parameter and speed comparison.", "other_comments_or_suggestions": "none", "questions_for_authors": "please see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kESyGc6rKG", "reviewer": "Reviewer_6x8E", "summary": "This paper makes a series of efforts to enable uncertainty quantification for SAM. To achieve this, the authors first adopt Monte Carlo sampling to estimate the predictive, epistemic model, aleatoric prompt, and aleatoric task uncertainty. Then, to release the computation burden of the sampling process during testing, the authors concatenate SAMs 256-dimensional mask and IoU tokens and feed them into MLPs that are trained to predict uncertainty.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for highlighting our strenghts and asking questions that are valuable for eliminating weaknesses and enhancing our paper in terms of readability and evaluation. \n\n---\n\nBefore addressing the questions, we need to clarify an issue that affects this review as well as reviewer **3EoJ**.\nAfter carefully re-reading the paper with temporal distance, we acknowledge that our description of the usage of SAM2 is misleading and needs to be improved:\n\n- First, line 254 (right column), \"We combine the evaluation of SAM and SAM2,\" is misleading because we do not combine the models during evaluation. Rather, we combine the datasets evaluated in the corresponding papers. We do this to provide a broader overview that also includes the latest data publication from SAM2.\n- Second, line 242 (right column), \"We use the pretrained SAM2,\" suggests that we use the video segmenation mechanism of SAM2, which is designed for segmenting temporally consistent masks in videos. In reality, while we use the pretrained SAM2 model, we only use its single-image implementation, which is equivalent to SAM and does not include the video extension. We do this because the authors of SAM2 provide more stable code, better interfaces, and slightly better accuracy.\n\nPotentially, our MLPs could be used ad hoc to estimate uncertainty from the refined tokens of SAM2 applied to videos, but that is not evaluated and is beyond the scope of this paper. To prevent misunderstandings for future readers, we will avoid the term SAM2 and suggest the following modifications in the camera-ready version:\n\n\n**Replace:** \"In our experiments, we use the pretrained SAM2 (Ravi et al., 2024)\" **with** \"In our experiments, we use SAM models pretrained by Ravi et al. (2024).\"\n\n**Replace:** \"We combine the evaluation of SAM and SAM2 ...\" **with** \"Combining the dataset selection of Kirillov et al. and Ravi et al., we ...\"\n\n--- \n\n## Weakness and Question 1: Why the mask and IoU tokens can reflect uncertainty\n\n(Un)certainty refers to the likelihood that the prediction is correct. In our case, correctness is measured by the IoU metric, making the expected IoU an ideal proxy for uncertainty. In general, estimating uncertainty does not require ground truth. For example, predictive uncertainty is often described as the variance of the predictive distribution. As seen in Equations 1 and 2, this distribution does not depend on the ground truth. Therefore, the absence of ground truth during inference does not invalidate the expressiveness of our method that uses tokens for uncertainty estimation.\n\nAdditionally, the SamScore in SAM (Kirillov et al.) is based on the same principle and also does not have access to ground truth during inference. However, the question of whether and how the mask and IoU tokens reflect uncertainty is an interesting one:\n\nEssentially, the MLPs attempt to map patterns from the object’s latent space (i.e., the tokens) to the expected IoU gap (i.e., uncertainty) based on statistical evidence. The already given effectiveness of the SamScore suggests that such patterns exist in general. \n\nTo further investigate and provide more details for future readers, we conducted an additional ablation study. In this ablation, we set either the IoU token or the mask token to 0 to remove potentially informative patterns.\nWe evaluated using the tiny model across the three tasks described in the submitted paper on the COCO dataset. The results indicate that both tokens contribute to the predictive capability, yet they remain accurate as standalone features. Both in combination lead to the best results. The results are:\n\n\n**Token to Uncertainty Ablation** **in relative AUC in %** \n\n| **Mask Token** | **IoU Token** | **Model UQ** | **Prompt UQ** | **Task UQ** |\n|:---:|:---:|:---:|:---:|:---:|\n| No | Yes | 61.19 | 72.08 | 86.89 |\n| Yes | No | 62.63 | 76.42 | 91.96 |\n| Yes | Yes | **63.66** | **78.30** | **94.82** |\n\nWe will add this interesting findings to the camera ready version.\n\n## Weakness 2: The MLPs' design choice\n\nThe reviewer’s remark about our MLP design choice is similar to the comment from reviewer **nnaG**, indicating that our current explanation lacks clarity. There was no dedicated architecture design process for the UQ estimator. Instead, the architecture is motivated by SAM’s original MLP design for predicting the SamScore. We differ only in the input dimension and hidden states (increasing from 256 to 512) because we use and concatenate both tokens instead of using just one.\n\nThe sigmoid activation function is a reasonable choice since the expected IoU gap ranges between 0 and 1. To improve clarity, we suggest modifying the method description and adding a figure to illustrate the MLP architecture. Please see our response to reviewer **nnaG** for details on the suggested changes.", "claims_and_evidence": "The claims are well-supported by the experiments.", "methods_and_evaluation_criteria": "The proposed method is reasonable.", "theoretical_claims": "There are no new theoretical proofs.", "experimental_designs_or_analyses": "The experimental designs are reasonable with no major issues. One minor issue is the lack of details on how to \ncombine SAM and SAM2 for the evaluation.", "relation_to_broader_scientific_literature": "This work extends the existing Uncertainty Quantification into SAMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. Uncertainty quantification is important for segmentation tasks. \n2. The decomposition of aleatoric uncertainty into prompt and task uncertainty is reasonable.\n3. The proposed method is easy to follow.\n\nWeaknesses\n1. Lacks the analysis of why the mask and IoU tokens can reflect the uncertainty. Since the uncertainty depends on the ground truth mask, the mask and IoU tokens can not reflect uncertainty entirely.\n\n2. Why choose MLPs and how to confirm their specific parameters e.g., 3 Layers with 512 hidden states and a Sigmoid activation.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Since the uncertainty depends on the ground truth mask, the mask and IoU tokens can not reflect uncertainty entirely. Thus, it remains questionable that the usage of only the mask and IoU tokens can reflect the uncertainty.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbIf4m3Ekl", "reviewer": "Reviewer_3EoJ", "summary": "This paper introduces an interesting method to measure the uncertainty of SAM in image segmentation tasks. To achieve this, this paper proposes USAM, an efficient post-hoc method that can quantify the uncertainty of SAM and help users determine whether the model results are reliable, which tunes a lightweight MLP-based estimator. Specifically, the authors design a formula based on Bayesian entropy that can simultaneously consider three levels of uncertainty: model uncertainty (for example, the model is too small, resulting in inaccurate predictions), prompt uncertainty like ambiguous user prompts, and task uncertainty like unclear user prompts. The paper demonstrates the effectiveness of USAM on several datasets, showing its ability to identify sources of uncertainty and improve model predictions. Overall, this paper is well-organized and theoretically grounded.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge the in-depth review that recognizes our strengths and provides valuable directions for addressing flaws, particularly regarding related methods. Below, we address questions and concerns. Issues already covered in other responses are only referenced.\n\n---\n\n## Claims \n\nThank you for pointing out that this claim in the abstract is not sufficiently elaborated in the introduction. A more detailed discussion of SAM and its implications for UQ is provided in Section 2.3, \"UQ in SAM\".\nMost UQ methods are based on MC strategies and motivated by Bayesian principles (see Equation 1 ff.). While these methods work well in practice, their theoretical justifications rely on assumptions that are often unrealistic and cannot be guaranteed. For example, test-time augmentation (Wang et al., 2019), used to estimate aleatoric uncertainty, assumes that the sample is in-distribution. Similarly, epistemic UQ estimators, such as model ensembles, assume that uncertain samples are out-of-distribution. These issues, which already exist in well-established UQ approaches, are further exacerbated by the task-agnostic nature of SAM.\nTo the best of our knowledge, there is no general discussion in the literature on SAM's implications on UQ. However, we acknowledge that our formulation of this claim is unclear. To address this, we suggest adding the following to line 126 (right column):\n\n\"While the presented methods work well in practice, the theoretical foundations for separating aleatoric and epistemic uncertainty remain debatable and under active discussion (Gruber et al., 2023). The task-agnostic nature of SAM further amplifies this challenge, as discussed next. More...\"\n\n## Weakness 1\n \nAs the reviewer mentioned, several papers propose UQ methods, such as [1] and [2]. The first encodes variance into the latent to sample ambiguous segmentations, while [2] estimates pixel-wise epistemic uncertainty to improve finetuning. They are designed for specific tasks and not directly comparable to us. USAM neither samples mask nor estimates pixel-wise uncertainty, making it unsuitable for both. \n\nHowever, our UQ comparison already aligns with the comparisons performed in [1] and [2].\nIn [1], UQ performance is evaluated by applying prompt perturbations (similar to our H_XP)  to SegGPT and image augmentations (similar to our H_Y) to SEEM. Note that by changing the frameworks to SegGPT and SEEM, the ambiguous segmentation task is evaluated but UQ is not evaluated in isolation. In [2], a pixel-wise equivalent to our \"Entropy\" is used, termed SUM Confidence-Uncer, alongside a pixel-wise equivalent to our H_A, termed SUM Discrepancy-Uncer. These study designs underline that standard UQ methods are suitable for evaluating new UQ approaches.\nNevertheless, we acknowledge that our paper does not sufficiently address the variety of UQ approaches used in conjunction with SAM. We propose to expand Section 2.3 (\"UQ in SAM\") and include a discussion of more related works that propose or employ UQ with SAM, such as [1] and [2].\n\n---\n\n## Q1\n \"For several tasks, quantifying the uncertainty of SAM is of particular interest\". We mention some tasks in the proposed paper: For example, Vujasinovic et al. (2024) use UQ to estimate if supervision is required to correct tracking errors, or in the healthcare domain to gain trust and improve prediction (Deng et al., 2023; Zhang et al., 2023). The papers [1] and [2] mentioned by the reviewer employ UQ in SAM to sample prediction or to improve finetuning. We will add these additional references to the camera ready.\n\n--- \n\n## Q2\n\nA similar concern was raised by reviewer **nnaG**. We agree that the simplicity of our MLP does not sufficiently support claims of superior efficiency. To address this, we conducted runtime experiments demonstrating that USAM incurs minimal computational overhead and is significantly faster than MC-based methods.\nFor details, see our response to reviewer **nnaG**, Weakness 3, or run the experiment in our anonymous repository https://anonymous.4open.science/r/UncertainSAM-C5BF/scripts/speed_test.ipynb. We propose adding these findings to the camera-ready version.\n\n--- \n\n## Q3 and Q4\n\nOur submitted paper contains ambiguities regarding our use of SAM vs. SAM2. For a detailed clarification, please see our introduction of the response to reviewer **6x8E**.\n\n--- \n\n## Q5\n\nWe are uncertain how our method could be applied to the ambiguous segmentation benchmark. As stated by [X], the goal of this benchmark is to achieve \"better agreement between prediction and the ground truth\".\nIn contrast, our method estimates the variance of the distribution (i.e., uncertainty). Since we do not modify SAM’s mask prediction, USAM’s segmentations would be identical to vanilla SAM’s.\n\n[X] Rahman, Aimon, et al. \"Ambiguous medical image segmentation using diffusion models.\" CVPR. 2023.", "claims_and_evidence": "The main claims about uncertainty quantification in SAM are supported by clear mathematical formulation, method, proofs, and experiments. The minor concern is the claim that \"the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification approaches.\" is not well justified. As SAM can generate ambiguous predictions (Top-K), it inherently has uncertainty awareness, making the uncertainty analysis easier. Why do the authors claim that this will challenge UQ?", "methods_and_evaluation_criteria": "The proposed method introduces Bayesian Entropy Approximation in SAM, which makes sense and is technically sound. It considers uncertainty from different levels (Eq.2), which is well justified by both theoretical proofs (Sec. 3) and experiments (Sec.4.2 to Sec. 4.4).", "theoretical_claims": "I have checked the theoretical claims about the uncertainty quantification formulation (Sec. 2.2), the grounding in SAM (Sec.2.3), and Bayesian Entropy Approximation (Sec.3 and Appendix A). I did not find major theoretical errors and think that this paper is well organized.", "experimental_designs_or_analyses": "Most of the experiments were well conducted, especially for the detailed justification of claimed uncertainty terms from the model, prompt, and task aspects. The experimental analysis is also comprehensive and convincing. However, there are two major concerns. The first concern is insufficiently comparing existing UQ methods in SAM model, such as [1-3]. The second concern is the lack of comparison of specific settings focusing on uncertain predictions, like ambiguous segmentation, which is critical to highlight the strength of the proposed UQ methods [1,2].\n\n[1] Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM (NeurIPS 24)  \n[2] Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 24)", "relation_to_broader_scientific_literature": "The proposed method has a potentially broader impact on understanding the existing SAM model from the uncertainty perspective, thereby inspiring the algorithmic designs of the next-generation SAM.", "essential_references_not_discussed": "The proposed method explores the uncertainty quantification of the SAM model. However, this is not a new direction, and there are many related works. Even though the proposed method is comprehensive, technically sound, and theoretically correct, it is suggested to make necessary discussion with related UQ methods using SAM [1-2]\n\n[1] Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM (NeurIPS 24)   \n[2] Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 24)", "other_strengths_and_weaknesses": "Strength:\n- The paper is well organized and gives a comprehensive and theoretically sound analysis of the uncertainty in SAM. \n- The proposed Bayesian approximation is reasonable.\n- The perception of the model, prompt, and task uncertainty is interesting and comprehensive.\n- The experimental justification is solid.\n\nWeakness:\n- Lack some comparison with related works about modelling uncertainty in SAM\n- Lack some clarification (see Questions For Authors)", "other_comments_or_suggestions": "N/", "questions_for_authors": "- The authors claim that \"For several tasks, quantifying the uncertainty of SAM is of particular interest.\". It is suggested to clarify what tasks require measuring uncertainty in SAM.\n- The authors are suggested to give a justification for the efficiency of the proposed method.\n- In Sec. 4, the authors implement the method in SAM2, but Fig.1 illustrates SAM, which is inconsistent.\n- Considering that the authors use SAM 2 as the base model, the theoretical analysis lacks a formulation for time-series signals. Will it introduce extra uncertainty when considering the object motions in videos?\n- Lack of a comparison on the ambiguous segmentation benchmark.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9iY4C64kiF", "reviewer": "Reviewer_pXDa", "summary": "This work introduces a method for uncertainty quantification of SAM, based on Bayesian entropy formulation. A lightweight post-hoc UQ method is trained based on the formulation. Results on multiple public benchmarks demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\nI am glad to keep my original positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and positive assessment.\n\nTo further support the reviewer’s inclination toward acceptance, we would like to highlight the linked code repository in the main paper, which demonstrates the ease of use of our proposed framework. We remain open to any additional questions, remarks, or suggestions to refine our proposed paper during the author-reviewer discussion.", "claims_and_evidence": "Claims are clear and convincing.", "methods_and_evaluation_criteria": "Evaluation criteria make sense.", "theoretical_claims": "Theoretical claims seem correct to me.", "experimental_designs_or_analyses": "The experimental designs and analyses are reasonable to me.", "relation_to_broader_scientific_literature": "This uncertainty quantification method of SAM can benefit a wide range of computer vision tasks. Considering the wide application of SAM, this method will be influential.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength: The experiments on multiple public benchmarks validate the effectiveness of the proposed method comprehensively.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received four weak accept ratings. Initially, the reviewers had a few concerns, including some technical clarifications, runtime and parameter analysis, more experiments and analysis of uncertainty. After the rebuttal, the reviewers are satisfied with the answers and thus all support the paper's acceptance. The AC took a close look at the paper/rebuttal/reviews, and agrees with the reviewers' assessment. Thus, the AC recommends the acceptance decision and encourages the authors to revise the paper following the reviewers' suggestions, as well as releasing the code/models for reproducibility.
Decision: Accept (poster)