Uncertainty-aware Fine-tuning of Segmentation Foundation Models
Authors: Kangning Liu
Brian L. Price
Jason Kuen
Yifei Fan
Zijun Wei
Luis Figueroa
Krzysztof J. Geras
Carlos Fernandez-Granda
Venue: NeurIPS 2024 poster (2024)
Keywords: Segmentation foundation model
Primary area: machine_vision
TL;DR: We introduce the Segmentation with Uncertainty Model (SUM), which enhances the accuracy of segmentation foundation models by incorporating an uncertainty-aware training loss and prompt sampling based on the estimated uncertainty of pseudo-labels.

Abstract: The Segment Anything Model (SAM) is a large-scale foundation model that has revolutionized segmentation methodology. Despite its impressive generalization ability, the segmentation accuracy of SAM on images with intricate structures is often unsatisfactory. Recent works have proposed lightweight fine-tuning using high-quality annotated data to improve accuracy on such images. However, here we provide extensive empirical evidence that this strategy leads to forgetting how to "segment anything": these models lose the original generalization abilities of SAM, in the sense that they perform worse for segmentation tasks not represented in the annotated fine-tuning set. To improve performance without forgetting, we introduce a novel framework that combines high-quality annotated data with a large unlabeled dataset. The framework relies on two methodological innovations. First, we quantify the uncertainty in the SAM pseudo labels associated with the unlabeled data and leverage it to perform uncertainty-aware fine-tuning. Second, we encode the type of segmentation task associated with each training example using a $\textit{task prompt}$ to reduce ambiguity. We evaluated the proposed Segmentation with Uncertainty Model (SUM) on a diverse test set consisting of 14 public benchmarks, where it achieves state-of-the-art results. Notably, our method consistently surpasses SAM by 3-6 points in mean IoU and 4-7 in mean boundary IoU across point-prompt interactive segmentation rounds. Code is available at https://github.com/Kangningthu/SUM

Uncertainty-aware Fine-tuning of Segmentation
Foundation Models
Kangning Liu 1
Brian Price*
Jason Kuen²
Yifei Fan
Zijun Wei?
Luis Figueroa²
Krzysztof J. Geras
Carlos Gernandez-Granda'
New York University
Adobe
Abstract
The Segment Anything Model (SAM) is a large-scale foundation model that has revolutionized segmentation methodology. Despite its impressive generalization ability, the segmentation accuracy of SAM on images with intricate structures is often unsatisfactory. Recent works have proposed lightweight fine-tuning using high-quality annotated data to improve accuracy on such images. However, here we provide extensive empirical evidence that this strategy leads to forgetting how to "segment anything": these models lose the original generalization abilities of SAM in the sense that they perform worse for segmentation tasks not represented in the annotated fine-tuning set. To improve performance without forgetting, we introduce novel framework that combines high-quality annotated data with a large unlabeled dataset. The framework relies on two methodological innova- tions. First, we quantify the uncertainty in the SAM pseudo labels associated with the unlabeled data and leverage it to perform uncertainty-ave Second, we encode the type of segmentation task associated with each training example using a task prompt to reduce ambiguity. We evaluated the proposed Segmentation with Uncertainty Model (SUM) on a diverse test set consisting public benchmarks, where it achieves state-of-the-art results. Notably, consistently surpasses SAM by 3-6 points mean IoU and 4-7 in mean bound- ary IoU across point-prompt interactive segmentation rounds. Code is available at htttts://github com/ Komm/Kagggingtu/SUSS
fine-tuning.
method
Introduction
The Segment Anything Model (SAM) [1] is ground vision foundation model to perform interactive binary segmentation, where the goal to identify pixels belonging to target region in an input image. The task is interactive, because the model accepts prompts from the user, which specify the desired target region. These prompts are typically the form of a set of points that should be included or excluded from the target region, or a box mask that encompasses it.
SAM is trained via a boot-strapping training procedure called a data engine on massive dataset of unlabeled images. The key idea is to leverage model-generall pseudo-labels to annotate the unlabeled data and use the pseudo-labels to further train the model. SAM is notable for its impressive general-purg purpose capabilities, allowing it to segment wide range of objects, parts, and visual structures. Yet, the model encounters difficulties achieving high-quality segmentation of intricate structures and often produces imprecise segmentation boundaries. This is problematic for applications such as image editing and automatic image annotation 2 where accurate fine- meeggraieed predictions are essential.
Our analysis in Appendix Dreveals that inaccuracies within pseudo-labels are substantial, especially in boundary regions. It has been reported in [ that simply scaling the amount of pseudo-labeled data by 10x only achieves marginal improvement (around 0.5 mloU). To improve segmentation
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SegTask Input Image
Ground Truth
SAM
HO-SAM
SUM
Mean Boundary IoU
Salient
Object
Entity
Part
2 7
Number of point promots
User-input prompes
Positive Prompts
Negative Prompts
SAM
SUM (proposod)
Figure 1: SUM improves SAM without forgetting to "segment anything." Performance compari- son between the Segment Anything Model (SAM), HQ-SAM model fine-tuned on the HQ-Seg--4XX dataset, and the proposed Segment with Uncertainty Model (SUM) fine-tuned using a high-quality annotated dataset and unlabeled data. Each row corresponds to held-out datasets focusing on different segmentation tasks (SegTask). Left: Both HQ-SAM and SUM show qualitative improvements over SAM, particularly in salient-object segmentation of complex structures (top row). HQ-SAM, however, struggles with background entities (middle row) and part segmentation (bottom row), often erro- neously prioritizing objects in the foreground entire objects. Right: SUM consistently outperforms SAM and HQ-SAM in quantitative comparisons, achieving the highest mean boundary IoU across diverse evaluation sets and interactive segmentation rounds. We adhere the same point prompt sampling evaluation strategy as SAM (Appendix B.3). More examples are in Appendix H.2
quality, recent works 4 [ suggest fine-tuning SAM exclusively with data paired with high-qualiy, human annotation. A key challenge is that the size of those datasets is usually relatively small, and the diversity of images and annotations they contain is limited. Consequently, the model may overfit the characteristics of the fine-tuning dataset, which degrades its performance other situations (e.g., segmentation tasks different from the fine-tuning dataset). Currently, the most common strategy to mitigate this is lightweight finetuning, which only updates a small subset of model parameters 5
7 g However, our experiments find that the overfitting problem still persists, as demonstrated Fig. and Section 4.2
In this paper, our goal is to investigate how to better learn from pscudo-labels, with the help of high-quality numan-annotaale data. We then apply these insights improve the accuracy of SAM without compromising its generalization. Our proposed framework, the Segmentation with Uncertainty Model (SUM), performs fine-tuning in manner that is aware of the regions where model-generated labulslabels have questionable accuracy. This is achieved by explicitly estimating pseudo-labe uncertainty in the unlabeled training data, in the form of an uncertainty map generated by specialized module. The uncertainty map is then incorporated into a novel uncertainly-awee training loss, which reduces the influence of pseudo-label regions that are expected to be inaccurate. The framework has additional advantage when training seerrative-segmentaaee training, user prompts are typically generated randomly based on the available labels. Our framework additionally enables uncertainty-avet aware prompt sampling, which avoids producing misleading prompt locations.
models. During
Our framework also addresses an additional challenge in fine-tuning segmentation foundation models. Human annotations usually correspond to specific segmentation tasks. For instance, the HQ-Seg- 44K A used by previous tine-tuning approaches [4] focuses on salient-object segmentation, where the target regions may include multiple entities
complex arrangement. In contrast, current general-purpose interactive segmentation models, such SAM, are mainly trained for entity or part segmentation, where target regions are associated with individual entities or object parts, respectively. To take this into account, we propose novel task prompt, which informs the model of the desired

---

segmentation task during training. This resolves ambiguities between the human and generrtee labels, and can also be leveraged specify the desired segmentation task during inference.
To evaluate SUM, we performed systematic experiments with diverse segmentation datasets, resulting in the following insights. (1) Existing fine-tuning approaches forget segment anything- After fine-tuning on HQ-Seg-44K, they outperform SAM on salient-object segmentation but underperform on held-out datasets where annotations are associated with entities or object parts (Fig. || and Sec-
tion 4.2 (2) Including unlabeled data associated with SAM pseudo-labels in the fine-tuning dataset prevents forgetting, but also yields a limited gain in performance if noise inherent pseudo labels is not effectively managed (see Section 4.3) (3) SUM is able to effectively leverage the labeled and unlabeled data achieve superior performance (see Section 442 and Appendix [..]). This is demonstrated in controlled experiments using publicly available fine-tuning datasets, as well as on a large-scale internal dataset. An ablation study (see Section
4.3) shows that the different method- ological innovations (uncertainty-baser loss, snerrtainty-assee prompting, and task prompting) all contribute this gain in performance.
2 Related Work
Segmentation Foundation Models Segmentation a fundamental task in computer vision, which includes semantic segmentation [0 12] instance segmentation [13[4) panoptic segmenta-
19 20). Inspired by the breakthroughs of language foundation models [21 22 23 24], the field has recently witnessed the emergence of large-scale
tion [6), and interactive segmentation 17
segmentation models pre-trained on massive datasets, which demonstrate remarkable generalization
capabilities. Painter (25] and SPg (66 proposed robust in-context learning frameworks enabling
segmentation of images given image-mask prompts. Recently, SEEM (27) introduced a multi-modal reference-pin reference-prompeed general segmentation model. Our work builds on the Segment Anything Model (SAM) M1 a ground-breaking foundational model for interactive segmentation. SAM is trained leveraging a boot-strapping training procedure called a data engine, which encompasses multiple phases with different degrees of human supervision, culminating in aully-automated stage where the pseudo labels are generated automatically from the model.
Fine-tuning Segmentation Foundation Models Different methods have been proposed to extend
SAM, and other foundation models, in different ways, such as increasing inference speed [28] 29]
performing all- all-purpose matching (30] personalized segmentation B1], increasing granularity [2
enabling fanguage-informed segmentation guidance (33), and adapting to specific domains (e.g..
medical imaging [34]]]]]] 35
6 7
36 37 38
39 40 41. agriculture [8) remote sensing 442. shadow
detection 43
45 or water leakage (46)).
Our goal is to enhance SAM's accuracy while maintaining its generalizability. HQ-SAM 4 pursues
a similar goal, but primarily focuses on minimal architectural adaptations for weight fine-tuning on high-quality human-annotaa data. Here, we focus an effective framework for leveraging limited human-annotated data and diverse unlabeled data.
Semi-supervised Semantic Segmentation Supi-super semantic segmentation also aims to leverage unlabeled data when labeled data is limited. However, the purpose is usually to enhance the performance of a model on a specific task. this work, we aim to instead leverage the unlabeled data to preserve the generalization ability of foundation model for diverse segmentation tasks (e.g. part, object, background), which may not be represented in the labeled data. The majority of semi-supervised approaches enforce consistency between predictions corresponding augmented versions of an unlabeled image. This is achieved through various specialized architectures such as seacher-student networks [5T] 52 (58), parallel networks [54] 1551. or incorporating additional
representation heads [56]. In contrast, our proposed method seeks to improve the pseudo-labels based on the data engine of SAM, requiring minimal modifications to the existing architecture. Combining these two types of approaches is an interesting direction for future investigation.
Uncertainty-awae aware Segmentation Uncertainty-awar segmentation has been explored for semi- supervised settings (e.g. U2PI. (51)), and
segmentation 57
58 59 These methods
typically employ prediction confidence [51 (7) or discrepancy among multiple prediction heads [59] 58 to quantify uncertainty. Our approach differs fundamentally from previous approaches in both
3

---

the generation of uncertainty maps and the utilization of uncertainty maps. SAM undergoes multiple rounds of self- self-training. which leads to the accumulation and overfitting of errors in pseudo-labels [60]. During our evaluation, the model frequently predicts incorrect regions with high confidence, and different heads agree on these erroneous areas. Experiments detailed in Section 4.2]show that previous approaches are ineffective in our context. Our main insight is to use external supervision to identify and correct systematic biases that accumulate during training. Further discussion can be found in Appendix E.2
3 Method
3.1 Preliminary Background: Segment Anything Model (SAM)
The Segment Anything Model (SAM) [ consists of three modules: an image encoder based on a large vision transformer, lightweight prompt encoder for geometric inputs (e.g.. points, boxes, mask indicating the target in interactive segmentation), and a lightweight mask decoder. The decoder combines the outputs of both encoders to produce a predicted segmentation mask.
During training, the model is provided interactive prompts iteratively. The first prompt is obtained by randomly selecting a point or bounding box from the target mask. Then, subsequent point prompts are randomly sampled within the error region, which is the discrepancies between the target mask and prior model prediction. At each training step, the model receives the preceding mask prediction as an additional input.
3.2 Segment with Uncertainty Model (SUM)
The Segment with Uncertainty Model (SUM) is a fine-tuning framework to improve the segmentation quality of existing foundational segmentation model, which builds upon the Segment Anything Model (SAM). A diagram illustrating the framework is provided in Fig. 2 The framework assumes the availability of human-labeled set with high-quality labels (e.g. HQ-Seg-44K3 and a large, diverse unlabeled set (c.g. SA-1B (... These datasets are used to fine-tune SAM via interactive segmentation training. Training performed with human annotations for the labeled data and with SAM-generated pseudo-labels for unlabeled data. These pseudo-labels are produced in accordance with SAM's automatic mask generation pipeline (for more details, see Appendix (B).
SUM contains several novel elements designed to improve the fine-tuning process. The pseudo- labels generated by SAM are inaccurate, which is problematic for training. SUM addresses this by exploiting the insight that this inaccuracy is systematic and prediciable. Uncertainty maps are leveraged modify the fine-tuning cost function and the generation of training prompts for the unlabeled examples to make them uncertainty aware, explained in Sections 3.2. uncertainty maps are produced by an
[and and 3.2.2 The module described in Section 3.2.3 Additionally, SUM incorporates task prompt, described in Section 3.2.4. which allows us to specily the desired segmentation task.
3.2.1
Uneertainty-aware Loss
SUM combines human-annotated and unlabeled data, utilizing different training losses for each. For the human-annotated data, SUM minimizes the same training loss as SAM: a linear combination of focal loss [61] and Dice loss [62]. For the unlabeled data, we propose to utilize uncertainty- versions of both losses instead, which are then combined in the same way as SAM. The goal is to down-weight the influence of uncertain areas in the pscudo-labels and increase the influence of regions with high confidence.
The umcertainty-awart focal loss TuAtocal is obtained by incorporating pixel-specific weights to the standard focal loss [61]. Let 9: denote the binary pseudo-label corresponding to the ith pixel in an input image, P: the corresponding probability predicted by the model, and u, the corresponding entry of the uncertainty map obtained from the SUM incertainty-quantffccal The uncertainty is between 0 (highly certain) and (highly uncertain). The contribution of each pixel to the focal loss (ps, ( is weighted by c, 1 - u4 which can be interpreted as the certainty
module (see Section 3.2.3)

---

Task Prompt
Inpul Image
Image Encoder
Image Embedding
Mask Decoder
Mask
Loss
Human
Prediction
Annotation
Conv
Prompt Encoder
Prompi Sampling
Human Labeled Image
Prediction
Precomputed
Filtering Sampling
SAM
Pseudo Anmolation
32x32 regular grid point prompes
Uncertainty
Uncertainly Map
Quantiflcation
Task Prompt
Image Encoder
Image Embedding
Mask Prediction
Uncertainty-awar
Input Image
Mask Decoder
Loss
Prosppt Encoder
Uncertalnty-aware Prompt Sampling
Unlabeled Image
Previous Mask Prediction
Components represent contribution
Figure Segment with Uncertainty Model (SUM) framework. The diagram describes the proposed SUM framework, which utilizes both aunan-annotated and unlabeled data to tinettne foundation model for interactive segmentation. Top: When processing numan-annotated examples, interactive prompts are sampled based on the binary-mask labels and fed iteratively into the model along with the image. Since this binary mask depends on the type of segmentation task desired by the user, SUM incorporates task prompt that specifies the task relevant to each annotation for salient-object segmentation and 2 for entity segmentation). Bottom: For unlabeled images, the iterative prompts are sampled based on model-generated binary pseudo-labels, which may be inaccurate. SUM includes an -reertaitt-quantificatti module that processes the pseudo-labels, generating an uncertainty map. This map is leveraged within uncertainty-aware loss function used for training, and also informs how the interactive prompts are sampled. For all unlabeled data, the task prompt is set to O. Components highlighted in red represent novel contributions, whereas the remaining components adhere to the SAM framework.
associated with that pixel:
1 Σ c Tocal (Pi. It)
ITAoocal (p, -=
(I)
i=1
The uncertainly-aware Dice loss IUADice is an adaptation of the standard Dice loss, which only preserves pixels where the certainty above certain threshold (see Appendix for discussion on the choice of this hyperparameter):
JuA (p...... (.u) +E
lUADice (p. ()
SuA (p. 0, u) +
m
(pu G. u)
Σ 0: p. I 1c > 0) Sua(p. 8,
- (Us pi) [[c 0].
where is small positive constant to ensure numerical stability. JuA and SUTA denote the filtered intersection and sum, respectively, of the predicted probabilities p and pseudo-labels ÿ.
3.2.2 Uncertainty-aware Prompt Sampling
SUM applies the same prompt-sampling strategy SAM for the human-annotated data during interactive training. The first prompt obtained by selecting point bounding box at random from the target mask containing the positive segmentation labels. Then, subsequent positive point prompts
5

---

are selected from regions in the target mask not present in the predicted mask, and negative point prompts are selected from regions the predicted mask not present in the target mask. When the target mask is inaccurate, which is often the case when it relies on pseudo-labels, this is problematic because the point prompts may be completely incorrect. To address this, SUM leverages the uncertainty maps generated by the intertainty-uuantficeee prompt sampling. Let R denote the region used by SAM for prompt sampling Uncertainty-awarr sampling selects each pixel i in R with probability P: c:/ 2 Cj. where o; is defined above as the certainty associated with the pixel. This strategy ensures that prompts are sampled mostly from regions with high confidence.
module (see Section 3.2.3) to perform uncertainty-awant
3.2.3 Generation of Uncertainty Maps
In this section, we describe how uncertainty quantification is performed within the SUM framework, detailing the generation of uncertainty maps associated with pseudo-labels. For this purpose, SUM uses specialized mask-refinement module. The module receives the logits produced by the SAM mode?" ? as an input and outputs a refined prediction of the segmentation mask. The uncertainty map is calculated by taking the absolute difference between the refined prediction and the sigmoid- transformed probabilities of the SAM logits. The uncertainty value for the i-th pixel in the uncertainty map u; is given by:
== (a()))
(3)
where a is the sigmoid function, S; denotes the SAM logits for the i-th pixel, and r; denotes the refined prediction for the i-th pixel. This yields values between 0 (no difference, indicating low uncertainty) and (large difference, indicating high uncertainty). This is illustrated in Fig. 3 The mask-refinemen module also receives as input box prompt encompassing the pseudo-label and the point prompt used to generate the pseudo-labels.
Mask-refinement Module
Decoder
Refined
Imput Image
Prediction
Conv
Prompt Fncoder
Image
Image
Mask Decoder
SAM
Prompt Sampling
Facoder
Embedding
Prediction
Uncertainty Map
SAM
Point Prompt
Prompt Encoder
Figure 3: Generation of uncertainty maps. (1) The mask-refinement module receives as input the segmentation prediction produced by SAM. (2) The module produces a refined segmentation mask.
(3) The uncertainty map equals the absolute difference between the SAM and refined predictions. Additional visual examples of the uncertainty map are presented in Appendix Fig. 10
The mask-refinement module is implemented by fine-tuning SAM's decoder and prompt encoder. To be clear, these fine-tuned components are exclusively used to generate uncertainty maps before training SUM. We train the mask-refinement module refine the SAM pseudo-label masks using the corresponding high-quality labels in the high-quality human-annotated dataset. During training, we feed the SAM pseudo- logits to the mask-refinement module in the same way that SAM incorporates previous mask predictions in its interactive segmentation training protocol.
Appendix D provides an in-depth description of the design of the mask-refinement module, including quantitative and human-subject evaluations of the resulting segmentation maps, the relationship between uncertainty and pseudo-label accuracy, alongside comparative analysis of model variants.
Although it is possible to replace SAM pseudo-labels with refined masks in a naive manner, we empirically find that it less effective than uncertainty-awar training, as reported in Table 2 This suggests that potential errors mask refinement (see Appendix Fig. may impact performance.
'The regions the target mask not present in predicted mask are taken as positive prompts. while region the predicted mask not present in the target mask are taken as negative prompts. "These are the logits fed through sigmoid and binarized to produce the SAM pseudo-labels.

---

3.2.4 Task Prompt
Segmentation datasets often focus on different segmentations tasks, resulting in dramatic and system- atic differences between their labels. For example, the HQ-Seg-44KK 4 dataset targets salient-object segmentation, where segmentation mask may include multiple entities in complex arrangement. In contrast, the pseudo-labels produced by SAM for the SA-IB dataset mostly correspond to entity segmentation (where masks are associated with individual entities) or parl segmeniation (where masks indicate parts of objects). This is problematic for joint training, as introduces a high degree of ambiguity regarding the segmentation mask that the model should predict after the initial prompt.
To resolve this ambiguity, SUM includes a task prompt to accompany the first point or box prompt, indicating the segmentation task associated with each training example. In practice, we identify the task based on the dataset associated with each example. We assign 0 examples from the unlabeled dataset, 1 to examples from numan-annotated datasets focused on salient-object segmentation, and 2 to examples from human -nnotttte datasets focused on entity segmentation. See Appendix [H.] for an illustration. More categories can be added to account for additional segmentation tasks (e.g.. 3 for part segmentation). The task prompt is implemented via an embedding layer, which acts as a lookup table mapping each discrete task identifier to a unique high-dimensional vector embedding learned during training. This task embedding is combined via element-wise addition with the embedding produced by the SAM image encoder, illustrated by Fig. 2
4 Experiments
4.1 Experiment Settings
Training Sets We utilize multiple training sets built using the following datasets:
SA-250K: An unlabeled subset extracted from SA-1B dataset [ containing 250,000 images. A human-ame annotated set of 44,320 images with high-quality salient-object masks. set of 31,913 images, each with average of 20 entity masks, detailing both foreground and background.
HQSeg-44K 4
EntitySeg Training Set [63] A
human-anman annotated set containing 252,798 images with salient-object masks, 60,79 70,798 images with entity masks, and 153,046 images focused on human parsing. More details are provided in Appendix C
Internal Dataset:
Based these datasets, define three fine-tuning (FT) sets with varying budgets of human- annotated data: (1) FT-Small: SA-250K and HQSeg-44K, enabling a fair comparison with existing fine-tuning methods such as HQ-SAM [4]; (2) FT-Medium: SA-250K, HQSeg-44K and EntitySeg; (3) SA-250K and the internal dataset.
Evaluation Sets All the models are evaluated on the same collection of test sets, designed to assess performance on three different segmentation tasks:
Salient Object: COIFT 64
validation sets of DIS 65 and ThinObject5K 65,
HR-SOD (6).
VOCEVAL [67], and BIG 67 These datasets contain object-centric images with salient objects and binary masks. The training sets in DIS-VD and ThinObject5K are included in the HQSeg-44 dataset. COIFT, HRSOD, VOCEVAL, and BIG are used for zero-shot evaluation.
Entity: Three EntitySeg 663 validation sets with 454, 459 and 401 images. These sets feature complex scene images with multiple entities ranging from foreground instances to background regions (e.g., sky, road).
Part: Three human/clothing parsing datasets: the validation set of Fashionpedia [68] containing 1,148 images (861 of which have subpart masks), the first 1,000 images of the validation sets of Multi-Human Parsing [69 and Easyportrait [70]. We also include the first 1,000 images of the validation set of Paco
which contains a wider variety of object types.
Evaluation 1) Metries: We use standard metrics to evaluate segmentation quality: mean Intersection over Union (mloU) and mean boundary loU (mBloU). We aggregate these metrics via averaging for the test datasets in each category. Also, an overall average derived by aggregating results from all test datasets. 2) Test prompts: Unless specified otherwise, generate test prompts using the
7

---

center point sampling technique described in SAM [.. We also report box box-prompt segmentation results in Appendix G 3) Comparison to single-output model: SAM and SUM generate multiple masks when a single prompt is provided. For evaluation, we generally select the output mask that most closely aligns with the provided ground truth. However, when comparing it to HQ-SAM, which produces single output, we use a single output fixed in advance.
Baselines We compare the proposed method SUM against the following baselines: (1) Human anmotation only, a method where fine-tuning is exclusively performed with human-labeled data; and (2) Vanilla, where fine-tuning is performed on unlabeled and numan-annotaan data following the standard SAM training protocol, without discriminating between human and SAM pseudo labels.
Implementation Details: SUM is implemented by fine-tuning a SAM model based on ViT-H backbone. The experiments were done adhering to SAM's s configuration for loss weighting. Key training parameters, including learning rate, duration, hyperparameters, and schedules for both human-annotated and unlabeled data, are detailed in Appendix B Although the SUM framework introduces additional steps to generate uncertainty maps, these steps remain manageable within practical constraints. A detailed discussion on training complexity and computational overhead is provided in Appendix B.2
4.2 Comparing SUM with SoTA Methods
Comparison with HQ-SAM Fig. 4 evaluates and compares different fine-tuning strategies, which are all constrained to utilize the same lightweight fine-tuning scheme and architecture as HQ-SAM [ (NeurIPS 2023) for a fair comparison. We compare (1) the proposed SUM method;
(2) the original baseline approach (i.e., fine-tuning exclusively on HQ-Seg-44k); and (3) the vanilla baseline, which utilizes all of FT-Small. Evaluation is carried out over several rounds of interactive segmentation. HQ-SAM excels in salient-object segmentation unsurprisingly, as is fine-tuned on a dataset associated with this segmentation task, but its performance is worse than the other models for entity and part segmentation (and in fact also worse than SAM, as shown in Figure DT. Conversely, the vanilla approach, which uses unlabeled data and SAM pseudo labels, performs worse than HQ-SAM on salient-object segmentation. SUM, utilizing the same HQ-SAM lightweight structure, achieves similar performance to HQ-SAM on aalient-object segmentation while clearly outperforming all other models on entity and part segmentation. Detailed wise comparisons are provided in Appendix G.3
HQ-SAM, fine-tuned following the Nluman-Annotation-.
Salient Object
Entity
Part
80
HO-SAM
Vanilla
60
(HQ SAM arch.)
SUM (HQ-SAM arch.)
40
6
1 2
3
2
6
Number point prompts
Figure 4: Comparison of HQ-SAM [4] with Vanilla and SUM fine-tuned using the same lightweight scheme as HQ-SAM. SUM matches HQ-SAM and outperforms Vanilla in sabient-object segmentation and is superior in entity and part segmentation.
Comparison with Other Light-weight Fine-tuning Strategies The left subfigure of Fig. 5 il-
lustrates the comparison of SUM fine-tuned on the FT-Small against various strategies applied to fine-tune SAM on the human-annotated HQSeg-44k dataset: fine-tuning only the decoder and prompt encoder, Adapter FT: fine-tuning only the Adapter [72) plugged into the image encoder and LoRA FT: fine-tuning SAM with LoRA [73 in the same way as specific human-annotn
Preerr+trrmpteeeeee -nccoder FT:
SAMed [35). Our results suggest that fine-tuning SAM exclusively
dataset can easily result in overfitting, regardless of the fine-tuning strategy. Conversely, SUM enhances the overall performance and mostly retains SAM's generalization capabilities.
Comparison with Semi-supervised Methods To assess SUM's performance relative to other meth- ods utilizing unlabeled data, we adapt AugSeg [49]. a SoTA semi-supervised semantic segmentation method, to the interactive segmentation pipeline. AugSeg employs teacher-student framework. For
8

---

70
70
80
Seliet Object
Ently
Overall Aveinge
Swleet dbject
Eneiyy
Part
Overall Avenngg
5AM
LoRA
SUM
SAM
AugScop Mixco Prompt
U2PL
SUM Confidence uncer
Adapter
Frompe Encoder
Prompe
Discnepancy Uncer
Figure 5: Left: Comparison of single point-prompt segmentation mloU for SUM versus models fine-tuned using various strategies on the HQSeg-44K dataset. All competing models improve on the salient-object segmentation task associated with this dataset but deteriorate on other segmentation tasks. Right: mloU for single point-prompt segmentation: comparisons models fine-tuned on FT- Small dataset with various strategies. SUM clearly outperforms all other strategies salient-object and part segmentation, while preserving competitive performance on entity segmentation.
Table I: 3-point-prompt segmentation. This table reports 3-point-prompt interactive segmentation mean loU and mean Boundary IoU of different models fine-tuned on FT-Small.
Metries
SAM (without fine tuning)
Vanilla
AugSog Mixed Prompt
SUM Confidence-
SUM Discrepancy-
Uncer
Uncer
Salient object
86.1
89.2
90.6
91.2
89.1
89.1
mloU
Emtity
Part
82.7
63.2
81.2
61.8
82.9
59.9
64.2
83.4
67.2
83.4
82.1
59.3
62.4
83.4
average
Salient object
77.0
81.0
82.5
81.0
83.2
81.0
81.0
Emilty
76.5
mBloU
Purt
59.4
57.5
55.8
60.2
63.3
55.2
58.1
Overall average
71.3
71.9
the human-annotatee data, we employ SAM's training procedure to train the student network, with both point and box prompts. For the unlabeled data, apply consistency loss based on predictions from two augmented versions of the same image, processed by the student and teacher networks. Interactive segmentation requires prompt sampled from target labels. We utilize model-generated pseudo-labels to generate the prompts for unlabeled data in two different ways: 1) AugSeg Mixed- Prompt employs a combination of point and box prompts, and 2) AugSeg Box-Prome uses only box prompts. We provide more implementation details and discussion in Appendix [ As depicted in the right subfigure of Fig. 5 SUM clearly outperforms both versions, achieving a mloU of 72.8, compared to 68.8 for AugSeg Mixed- Preddprropp, 67.0 for AugSeg Box Box-Prompt.
The choice of uncertainty quantification is critical for the effectiveness of pseudo-label utilization. To understand how ours compares with existing uncertainty quantification approaches, we further evaluate two variants: (1) SUM Confidence- Uncer, which uses logit confidence of SAM prediction quantify pseudo-label uncertainty, similar to U2PL (51) (CVPR 2022); (2) SUM Discrepancy-Uneee which quantifies uncertainty using the discrepancy between multiple predictions for target mask, similar to Zheng et al. [58) 's work (IJCV 2021). Results in the right sub-figure of Fig. 5 indicate that these variants underperform ours significantly. We also tested an improved U2PL incorporating the same uncertainty map as SUM and task prompt training, but it still underperforms SUM. Further details are in Appendix We provide ppoint-prompp interactive segmentation comparison in Table The performance of SUM on part segmentation improves, which is not explicitly represented in the human annotations. This demonstrates SUM's capability enhance accuracy across general tasks.
Performance of SUM under Various Annotation Budgets In Appendix G. we evaluate our proposed model (SUM) when fine-tuned under various annotation budgets, utilizing the FT-Medium and FT-Large datasets. SUM achieves consistent improvements regardless of the fine-tuning budgets.
Performance of SUM on Additional Evaluation Sets To further validate the generalization of our proposed methods, we have expanded our evaluation by testing SUM and SAM on 8 additional evaluation sets that include a variety of image types. For reproducibiiltty SUM fine-tuned on the public dataset FT-Medium. As shown in Appendix G.6 SUM consistently outperforms SAM.

---

4.3
Ablation Experiments
Table 2: Ablation study. This table reports interactive segmentation mean IoU of different ablated versions of SUM, showing individual gains provided by uncertainty- fine-tuning and task prompts. Mean Boundary IoU and detailed dataset-wise comparisons are provided in Appendix F4
Metries
Number
Point
Task
SAM (without fine-tuning)
Vanilla
Mask Refinement
SUM wlo Task
Prompt
SUM
Continuous TP
SUM
Salient object
78.7
80.4
81.5
81.1
85.2
84.7
Eutity
77.4
78.8
78.1
78.6
79.8
79.2
Part
52.2
50.2
52.3
54.4
53.3
54.6
mloU
Overall average
69.0
69.3
70.3
71.0
73.4
72.8
Salient object
86.1
90.4
91.3
91.4
V1.6
89.6
Eatity
82.7
86.5
86.2
86.7
86.6
85.7
Part
63.2
66.7
67.8
67.9
67.7
Overall lverrage
77.2
80.3
81.5
82.0
82.1
81.0
Ablation on Uncertainty-aware Training and Task Prompt Table the Vanilla baseline and the following ablated versions of SUM on public dataset FT-Medium to ensure reproducibility: 1) Mask Refinement: This variant uses the refined masks produced by the mask-refinemee module of SUM as pseudo labels for the unlabeled images in SA-250K, and is otherwise the same as the Vanilla baseline. 2) SUM w/o Task Prompt: SUM without task prompts but with uncertainty-an aarear loss and sampling. 3) SUM Continuous TP: This is variant of SUM where the task prompt is utilized continuously in all rounds of interactive segmentation during training and inference (as opposed to just in the first round, as in SUM).
The results demonstrate the advantage of uncertainty-aware training over direct mask refinement. SUM w/o Task Prompt outperforms Mask Refinement, especially for part segmentation, which is the segmentation task that is not included the human-annotated labels. Both models outperform Vanilla, indicating that mask refinement provides an advantage over a naive combination of unlabeled and labeled data. Additional experiments analyzing mask refinement are reported in Appendix D
SUM is superior to SUM without Task Prompt across all segmentation tasks for a single- prompt, but both models essentially equivalent for 3 prompts. This suggests that task prompting is very useful when there is more ambiguity as to which mask is desired by the user. In fact, SUM Continuous TP underperforms SUM for 3-point prompts, suggesting that persistent task prompting may be counterproductive as semantic ambiguity decreases.
Additional Ablation In Appendix F.2 compare SUM to version of SUM without uncertainty- aware sampling. In Appendix Table SUM implemented with different backbones, including ViT-B, VIT-L, VIT-H. Our model consistently outperforms SAM across different model backbones for all mask types and interactive rounds. The improvement is most significant for single-point prompt segmentation of salient objects. In Appendix F.I we compare SUM and different variants of SUM via different lightweight fine-tuning methods.
6 we provide a comprehensive comparison between SAM and
5 Discussion and Future Work
Fine-tuning foundation models effectively is an important challenge. Our work suggests that key consideration is how to account for varying degrees of diversity and label quality in the available data. Specifically, we show that in the case of segmentation, incorporating a diverse unlabeled dataset can be effective in preserving the generalization ability of the original foundation model, as long as they are utilized in an uncertainty-awar manner.
As detailed in Appendix D our methods have limitations: the quality of mask refinement in SUM dependent on the initial performance of the SAM model. This suggests that leveraging SUM to improve pseudo-labed quality in the training data could lead to further improvement. Our work suggests some possible avenues for future research to improve the training of foundation models. The uncertainty maps generated by SUM can potentially be used to improve pseudo-label filtering in foundation- modell data engines. In fact, a human study reported in Appendix Somddix[[ouu found that filtering approaches utilized in SAM are unable to accurately discern pseudo labels with inaccurate boundaries. Finally, the efficacy of the proposed task prompts suggests that it may be possible develop automatic tubel-generatie schemes to produce pscudo-labels associated with more diverse tasks.
10

---

Acknowledgments and Disclosure of Funding
The authors acknowledge Markus Woodson for valuable discussions and feedback. They also thank Weicheng Zhu and Yunzhen Feng for their assistance with manuscript proofreading. Kangning Liu was supported by NSF grant NRT-1922658. Krzysztof Geras was supported by The Shifrin Myers Breast Cancer Discovery Fund (SMBCDF) and NYU Langone Health, Department of Surgery.
References
[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Segment anything. ICCV, 2023.
[2] Dengsheng Zhang, Md Monirul Islam, and Guojun Lu. A review on automatic image annotation techniques. Puttern Recognition, 45(1):346-362, 2012.
[3] Venkatesh N Murthy, Subhransu Maji, and R Manmatha. Automatic image annotation using deep learning representations. In Proceedings of the 5th ACM International Conference Multimedia Retrieval, 2015.
[4] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. NeurlPS, 2024.
[5] Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian
Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails to segment anything?-sann saannng-------apptet Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint arXiv:2304.09145 2023.
[6] Jay N Paranjape, Nithin Gopalakrishnan Nair, Shameema Sikder, Swaroop Vedula, and Vishal M Patel. Adaptivesam: Towards efficient tuning of sam for surgical scene segmentation. arXiv preprint arXiv:2308.08: 03336088.376,..... 2023.
[7] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306. 13465, 2023.
[8] Yaqin Li, Dandan Wang. Cao Yuan, Hao Li, and Jing Hu. Enhancing agricultural image segmentation with agricultural segment anything model adapter. Sensors, 23(18):7884, 2023.
[9] Vijay Badrinarayanan. Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 39(12):4481-2499 2017.
[10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 4044):834-848, 2017.
[11] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021.
[12] Bowen Cheng, Ishan Misra. Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[14] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Soloy2: Dynamic and fast instance segmentation. NeurlPS, 2020.
[15] Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention-guided unified network for panoptic segmentation. In CVPR, 2019.
[16] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In CVPR, 2019.

---

[17] Konstantin Sofiiuk, Ilia Petrov. Olga Barinova, and Anton Konushin. f-brs: Rethinking back- propagating refinement for interactive segmentation. In CVPR, 2020.
[18] Yuying Hao, Yi Liu, Zewu Wu, Lin Han, Yizhou Chen, Guowei Chen, Lutao Chu, Shiyu Tang, Zhiliang Yu, Zeyu Chen, et al. Edgeflow: Achieving practical interactive segmentation with edge-guided flow. In ICCV, 2021.
[19] Xi Chen, Zhiyan Zhao, Feiwu Yu, Yilei Zhang, and Manni Duan. Conditional diffusion for interactive segmentation. In ICCV, 2021.
[20] Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng. and Yefeng Zheng. Interactive segmentation as gaussion process classification. In CVPR, 2023.
[21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xXiv:8110004500 2018.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[23] OpenAl. Gpt-4 technical report, 2023.
[24] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[25] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023.
[26] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint 2rXiv:2304.03284, 2023.
[27] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint wrXiv:2304.067/1 2023.
[28] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint 2rXiv:2306.12156 2023.
[29] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung- Ho Bae, Seungkyu Lee, and Choong Scon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023.
[30] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305. 13310, 2023.
[31] Renrui Zhang, Zhengkai Jiang. Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048. 2023.
[32] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything any granularity. ECCV, 2024.
[33] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:23100 15308 2023.
[34] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12622 2023.
12

---

[35] Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation, 2023.
[36] Xinyu Xiong, Churan Wang, Wenxue Li, and Guanbin Li. Mammo-sam: Adapting foundation segment anything model for automatic breast mass segmentation in whole mammograms. In International Workshop on Machine Learning in Medical Imaging, 2023.
[37] Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W Remedios, Yucheng Tang, and Yuankai Huo. All-in-sam: from weak annotation to pixel-wise nuclei segmentation with prompt-based finetuning. arXiv preprint arXiv:2307.00290 2023.
[38] Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam on medical images: comprehensive study on three prompt modes. arXiv preprint 1aXiv:2305.00033 2023.
[39] Deepa Anand, Gurunath Reddy M, Vanika Singhal, Dattesh D. Shanbhag, Shriram KS, Uday Patil, Chitresh Bhushan, Kavitha Manickam, Dawei Gui, Rakesh Mullick, Avinash Gopal, Parminder Bhatia, and Taha Kass-Hout. One-shot localization and segmentation of medical images with foundation models, 2023.
[40] Bertrand Chauveau and Pierre Merville. Segment anything by meta as a foundation model for image segmentation: new era for histopathological images. Pathology, 2023.
[41] SeungKyu Kim, Hyun-Jic Oh, Soonghui Min, and Won-Ki Jeong. Evaluation and improvement of segment anything model for interactive histopathology image segmentation, 2023.
[42] Lucas Prado Osco, Qiusheng Wu, Eduardo Lopes de Lemos, Wesley Nunes Gonçalves, Ana Paula Marques Ramos, Jonathan Li, and José Marcato Junior. The segment anything model (sam) for remote sensing applications: From zero to one shot. International Journal of Applied Earth Observation and Geoinformation, 124:103540, 2023.
[43] Yonghui Wang. Wengang Zhou, Yunyao Mao, and Houqiang Li. Detect any shadow: Segment anything for video shadow detection, 2023.
[44] Xiao Feng Zhang, Tian Yi Song, and Jia Wei Yao. Deshadow-anything: When segment anything model meets zero-shot shadow removal, 2023.
[45] Leiping Jie and Hui Zhang. Adaptershadow: Adapting segment anything model for shadow detection, 2023.
[46] Shichang Liu, Junxin Chen, Ben-Guo He, Tao Chen, Gwanggil Jeon, and Wei Wang. Adapting segment anything model for shield tunnel water leakage segmentation. In Workshop on Advanced Multimedia Computing for Smart Manufacturing and Engineering, pages 13-18, 2023.
[47] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17.009911, 2017.
[48] Jaemin Na, Jung-Wer Woo Ha. Hyung Jin Chang, Dongyoon Han, and Wonjun Hwang. Switching temporary teachers for semi-supervised semantic segmentation. NeurlPS, 2024.
[49] Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation matters: A simple-yeteeffective approach to semi-supervised semantic segmentation. In CVPR, 2023.
[50] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. NeurlPS, 2021.
[51] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In CVPR, 2022.
[52] Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. Perturbed and striet mean teachers for semi-supervised semantic segmentation. In CVPR, 2022.
13

---

[53] Ven-Cheng Liu, Chih- Yao Ma, Zijian He, Chia- Ween-Wen&&r Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09488, 2021.
[54] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise semi-supervised learning. ECCV, 2020.
[55] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR, 2021.
[56] Ying Jin, Jiaqi Wang. and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. NeurIPS, 2022.
[57] Yuxi Wang, Junran Peng, and ZhaoXiang Zhang. Uncertainty-aware pseudo label refinery for domain adaptive semantic segmentation. In ICCV, 2021.
[58] Zhedong Zheng and Yi Yang. Rectifying pscudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV, 129(4):1 1220, 2021.
[59] Nikita Araslanov and Stefan Roth. sulf-supe augmentation consistency for adapting semantic segmentation. In CVPR, 2021.
[60] Sheng Liu, Kangning Liu, Weicheng Zhu, Yiqiu Shen, and Carlos Fernandez-Granda. Adaptive carly-learning correction for segmentation from noisy annotations. In CVPR, 2022.
[61] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[62] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
[63] Lu Qi, Jason Kuen, Weidong Guo. Tiancheng Shen, Jiuxiang Gu, Wenbo Li, Jiaya Jia, Zhe Lin, and Ming-Hsuan Hssan Yang. Fine-grained entity segmentation. arXiv preprint arXiv:2211.05777 2022.
[64] Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In WACV, 2021.
[65] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. ECCV, 2022.
[66] Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. In ICCV, 2019.
[67] Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR, 2020.
[68] Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, and Serge Belongie. Fashionpedia: Ontology, segmentation, and an attribute localization dataset. ECCV, 2020.
[69] Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiple-human parsing the wild. arXiv preprint 170.:2170, 2017.
[70] Alexander Kapitanov. EasyPortrait Dataset.
kapitaao//easppotraiit 2023. Accessed: 2023-09-01.
[71] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In CVPR, 2023.
[72] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXiv preprint
2023.
14

---

[73] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2021.
[74] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[75] Yanfei Song, Bangzheng Pu, Peng Wang. Hongxu Jiang, Dong Dong, Yongxiang Cao, and Yiqing Shen. Sam-lightening: A lightweight segment anything model with dilated flash attention to achieve 30 times acceleration, 2024.
[76] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In CVPR, 2016.
[77] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- Yu-Wing Tai, and Chi-Keung Tang, Fss- 1ss-:000: A 1000-class dataset for few-shot segmentation. CVPR, 2020.
[78] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection on extended essd. TPAMI, 38(4)7177772, 2015.
[79] Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. TPAMI, 37(3):569-555 0886699------- 2014.
[80] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuar Yang. Saliency detection via graph-based manifold ranking. In CVPR, 2013.
[81] Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, and Jiaya Jia. Open world entity segmentation. TPAMI, 2022.
[82] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
[83] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[84] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand- object segmentation: Dataset, model, and applications. In ECCV, 2022.
[85] Nadav Cohen, Yael Newman, and Ariel Shamir. Semantic Segmentation in Art Paintings. Computer Graphics Forum, 2022.
[86] Lei Yang, Yan Zi Wei, Yisheng He, Wei Sun, Zhenhang Huang, Haibin Huang, and Haoqiang Fan. ishape: A first step towards irregular shape instance segmentation. arXiv preprint arXiv:2109.15068, 2021.
[87] Cameron Trotter, Georgia Atkinson, Matt Sharpe, Kirsten Richardson, A Stephen McGough, Nick Wright, Ben Burville, and Per Berggren. Ndd20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation. arXiv preprint arXiv:2005. 13359, 2020.
[88] Jungseok Hong, Michael Fulton, and Junaed Sattar. Trashcan: semantically-segmeene dataset towards visual detection of marine debris. arXiv preprint arXiv:2007.08097, 2020.
[89] Jiazhou Chen, Yanghui Xu, Shufang Lu, Ronghua Liang, and Liangliang Nan. 3-d instance segmentation of mvs buildings. IEEE Transactions on Geoscience and Remote Sensing, 60:1-14, 2022.
15

---

Supplementary Material: Uncertainty-awarc Fine-tuning of Segmentation Foundation Models
Contents
Checklist Details
18
License of the Assets
18
Broader Impacts
18
Implementation Details
18
B.1
Training
18
B.2
Training Complexity and Computational Overhead
19
3 Evaluation
20
Additional Dataset Information
21
C.1
Training Sets
Evaluation Sets
Mask-refinement Module
21
Design
D.2 Training
22
D.3 Evaluation
22
Semi-supervised Baseline
26
E. Implementation Details
26
Discussion
28
Additional Ablation Study
30
Ablation on Light-weight Fine-tuning
30
F2
Uncertainty-aware Sampling
30
Different Backbones
32
F4
Additional Details Main Ablation Experiments
32
F5
Threshold for the Uncertainty-aware Dice Loss
36
Additional Quantitative Results
37
Performance of SUM under Various Annotation Budgets
37
G.2
Box Prompt Evaluation and Complementary Metrics
37
16

---

G.3
Dataset-wise Comparison with HQ-SAM
42
G.4
Dataset-wise Comparison with SAM
46
G.5
SUM vs SAM under the Single Output Evaluation Setting
G.6
Results on Additional Evaluation Sets
51
H
Additional Qualitative Results
53
Task Prompt
53
H.2 SUM versus SAM
55
H.3 SUM versus HQ-SAM
60
Human Evaluation
63
1.1
SAM Pseudo Label Quality
63
Mask-relinement Module
67
17

---

A Checklist Details
A.1 License of the Assets
Licence for the Codes We build upon SAM
1 which is under Apache-2.0 license https
www.wappche.... wwppache.orgliceeeeeeeeeeeeeeeee
We reproduce the code for AugSeg 49). U2PL 51 Adapter 722 LoRA 73 35
all of which
are under MIT License under Apache-2.0 license
to http://ppwsoourrroooroooeeeaeees
HQ-SAM is
2.0
Licence for the Dataset HQ-Seg-44k org//icenses/LLCCNSEE-22
under Apache-2.0 license https: //w:://wwwww.. wwt.p::/www.apchee
COIFT 64
is under deed.
4.0 /
DIS 655 is under htwp://waia.iiiiiioioo/////////n
license http:///www.....aaee/eeoo///a//a
ThinObject5K
is under Apache-2.0
license https://ww..ppccde.or////ieaaee
LICENSE-2.0 2.0
HR-SOD 666] under MIT License according to httpss//ppaoourr..og/////ooooo//
VOCEVAL 67
and BIG are under MIT License according htttpss//opnsssureeeoorgg
licenses/MIT
EntitySeg 63
under CC BY-NC 4.0 license httper//rreaivvemmmoss..///Loooe
by-nc/4.0/
Paco 71 is under MIT License according to ottpps//pppaauurc.c//////oooo//
Fashioupedia 68 is under Creative Commons Attribution 4.0 40t.e//nhhttt//////ttt/////ommo org/licensos by 4. 0/legalcode
Easyportrait 70 is under Creative Commons Attribation-SharrAlike 4.0 International License http: //creativecommons org/ ftrg/liz htt////eeeeebba/////
The Multi-Human Parsing 669 dataset is made freely available to academic and non-academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation. The license terms is provided at hhttss://lv-mhh httpvv--hpp.iihttbb. io/
A.2 Broader Impacts
We want comment on potential societal outcomes associated with this work. The training of large- scale foundation models incurs carbon emissions, which contributes to global warming and raises concerns about environmental sustainability within the Al research community. Additionally, the black-box characteristics of these deep-learning models may result in unintended biases in practical applications. Nonetheless, we hope these effects may be offset to some extent when the models are applied in domains such as medical image segmentation. We believe that advancements in accurate segmentation foundation models have positive societal implications across a variety of fields, including medicine, autonomous driving, and creative content generation.
B Implementation Details
B.1
Training
In this section, we present comprehensive training details for our proposed Segmentation with Uncertainty Model (SUM), including loss functions, optimization strategy, interactive-ssemennttiio training pipeline, and the selection and use of hamen-labeled and unlabeled data.
Training Resources The original SAM I model was trained using 256 100 GPUs. For fine- tuning, we default to 40 Nvidia A000 80GB GPUs for fine-tun fine-tuning, contingent the availability of
18

---

these GPUs. Regarding the SUM model deployed in the HQ-SAM framework 4 we adhere to the HQ-SAM configuration, utilizing 8 GPUs ensure fair comparison.
For the human-annotated data, SUM minimizes the same training loss as SAM: a linear combination of focal loss [6]] and Dice loss 62 with the same ratio of 20:1. For the unlabeled data, we propose to instead utilize uncertaint--awar versions of both losses, which are then combined in the same way. The hyperparameter / in the uncertainty-ave aware Dice loss is set to 0.6. We study the effect of varying this hyperparameter in Appendix F.3
Losses
Following SAM, we train the IoU prediction head by optimizing the mean- mean-square-error between the predicted IoU and the IoU of the predicted mask compared to the target mask. This then incorporated into the mask loss, applying a scaling factor of 1.0.
Training Procedure Following SAM (. we use the AdamW 74 optimizer. The learning rate set at x 10-3 Our fine-tuning process takes 6,000 iterations, which is equivalent to a single SA-250K epoch, with batch size of 40 images. To account for GPU memory constraints, we train with a single image and up to 30 randomly sampled masks per GPU. Consistent with the SAM framework [1]. we apply layer-wise learning rate decay of 0.8. We do not apply futa-nuumentttion techniques. SUM is initialized from an SAM ViT-H model. Training is distributed across 40 GPUs.
Interactive Segmentation Training We conduct interactive training for 7 rounds. The initial round randomly selects between bounding box a point prompt with equal probability. This is followed by sequence of five rounds, where point prompts are sampled as follows. 'Positive' prompts are selected from regions within the target mask that are absent in the predicted mask, while 'negative' prompts are selected from areas included in the predicted mask but not in the target mask. In line with the SAM framework, we incorporate an additional round without additional prompts after the final stage.
We adjust the data sampling strategy to form the training batches, according to the relative frequency of human-annotated data and unlabeled data. For FT-Small and FT-Medium include one human-annotated image for every unlabeled image. In the case of FT-Large, where human-labeled data surpass unlabeled data, we include one unlabeled image for every two human-labeled images.
Utilization of Human-labeled and Unlabeled Data
Pseudo-label Generation In the fully automatic stage of SAM's data engine [1 the model is provided with prompts of 32x32 regular grid of points. The model then predicts a set of masks for each point. The masks then filtered discard low-quality ones. Specifically, there three filters in SAM:
The masks are selected based on the model's predicted IoU score, setting minimum threshold of 88.0.
For maintaining mask stability, SAM evaluated pairs of binary masks derived from the same soft mask but thresholded at slightly different values. mask was retained only if the loU between the binary masks, created by thresholding the logits at 0 and then at 1 and +1, was 95.0 or higher.
Lastly, to climinate non- inniinformatie masks, SAM excluded any mask that occupied 95% or more of the total image area.
Although the filtering mechanisms succeed in eliminating a significant portion low-quality masks, their effectiveness is constrained when comes further distinguishing the accuracy of mask boundaries among the remaining masks. This limitation is evidenced by our human study detailed in Appendix [.T. which assesses the boundary precision of the filtered masks.
B.2 Training Complexity and Computational Overhead
While the SUM framework adds steps to obtain uncertainty maps, they remain manageable within practical constraints, as detailed below.
19

---

Training the uncertainty map genera- tion module is relatively efficient on the human-labeled samples. It involves tuning a small number of parameters and can be completed within 4 hours using 8 A100 GPUs.
Training the Uncertainty Map Generation Module:
The computational overhead introduced by the SUM framework minimal. The process of generating pseudo labels and uncertainty maps for unlabeled data utilizes the existing SAM encoder, thereby sharing the primary computational workload. Beyond pseudo-label generation, the additional time required to produce refined masks using lightweight decoder is negligible compared to the time taken by the image encoder. ViT-H image encoder in SAM has 632M parameters while the prompt- decoder only has 3.87M parameters. As per the reference [75], on an RTX4090, the decoder adds only approximately 2% more computational time compared to the encoder.
Generating Uncertainty Map for the Unlabeled Image:
Fine- MuninT the SAM Model:
Once the uncertainty map is generated, the fine-tuming SAM model using the uncertainty map is similar to the standard training used in SAM.
Iterative Point Prompt Sampling: This method is used in standard SAM training as well, and replacing the original uniform sampling with weighted (uncertainty-awarc sampling results in a negligible training burden. Sampling a point from a 1024x 1024 candidate pool using both methods can be completed within 0.006-0.009 seconds (average of 1000 runs) on the CPU of a MacBook.
Task Prompt: The task prompt, a learnable single vector, is combined via glement-wise addition with the embeddings from the SAM image encoder and is used only in the first round of interactive segmentation. The element-wise addition of two tensors is relatively fast.
This involves thresholding and a weighted loss computation, which requires a similar running time to the original loss.
Uncertainty-awe Loss Computation:
Inference Phase: Uncertainty maps are utilized only during training. Once our model is trained, inference operates similarly to the SAM model, without additional computational burden. This results in negligible computational overhead.
B.3 Evaluation
Interactive Segmentation Evaluation Evaluation is carried out using six interactive segmentation rounds. The initial prompt may be either a bounding box or point. Given that point-based prompts are standard for interactive segmentation evaluations (as utilized in both SAM and HQ-SAM) our analysis predominantly concentrates on them.
Point-prompt Sampling Method during Evaluation Our point-prompp sampling during inference follows the one in SAM (1. The initial point is deterministically chosen the one farthest from the object's boundary. Subsequent points are selected similarly, as the farthest from the boundary of the error region between the ground-truth segmentation mask and the model prediction. was first introduced by Xu et al. (76) and has become the norm in literature for evaluating interactive segmentation. It ensures consistency by deterministically selecting the initial point be farthest from the object boundary. Subsequent points are placed farthest from the error region's 's boundary, which varies depending on the discrepancies between the ground truth and prior model predictions. While the initial point is consistent across all models, subsequent points are model-specifs in light of distinct error patterns.
In order to generate more challenging prompts, in certain experiments we perform boundary sampling instead. We utilize morphological operations for the precise identification of boundary points within specified proximity to the mask's 's edge. In our specific configuration, this distance is set to 10 pixels.
Comparison to Single-output Model Given the possibility of semantic ambiguity when only a limited number of prompts is provided, we generally opt for a fair comparison strategy: in cases where several output masks are generated, we select the one that most closely aligns with the provided ground truth. However, this approach is not applicable in comparisons involving HQ-SAM, which, by design, produces a deterministic single output. To maintain fairness and consistency when comparing with HQ-SAM, we only utilize single output from all other methods (which is fixed beforehand),
20

---

even if they have the capability to produce multiple outputs. The single output is set equal to the estimate corresponding to the fourth output token, following HQ-SAM. We include additional comparisons between models using deterministic single outputs in Appendix G.5
C
Additional Dataset Information
C.1 Training Sets
HQ-Seg-44k HQ-Seg-44k [4] comprises 44,320 highly precise image mask annotations for the salient object segmentation task. It was built by merging 6 pre-existing datasets: DIS [65] (train set), ThinObject-5K 64) (train set), FSS 777, ECSSD [78), MSRA-1OK (7),, and DUT-OMRON [8).
Train Set This dataset is large-scale and high-quality dataset for entity seg- mentation 81. It encompasses a diverse array of image domains and entities, enriched with vast collection of high-resolution images and superior-quality mask annotations. Additionally, is designed to be open-world and not restricted to predefined classes.
EntitySeg 63
Internal Dataset We collected a high-quality human annotated internal dataset containing 252,798 images with salient-object masks, 798 images with entity masks, and 153,046 images focused on part segmentation, primarily for human parsing and portrait. Each annotation in this dataset has been meticulously carried out by human annotators. The salient-object masks encompass wide range of categories, such as people, vehicles, street signs, animals, household items, kitchenware, food, furniture, electronics, and personal accessories. The entity masks include both foreground elements (like people, vehicles, animals) and background components (including sky, road, grass, water, clouds, and mountains). The part segmentation masks predominantly focus body parts, various clothing items, and accessories.
C.2 Evaluation Sets
the validation sets of DIS and ThinOb- ject5K 665 HR-SOD (66). VOCEVAL 67 and BIG (67). These datasets contain object- centric images with salient objects and binary masks. The training sets in DIS- VD and ThinObject5K are included in the HQSeg-44e dataset. COIFT, HRSOD, VOCEVAL, and BIG are used for zero-shot evaluation.
Salient Object: We utilize COIFT
64
Entity: Three EntitySeg (63) validation sets with 454, 459 and 401 images. These sets feature complex scene images with multiple entities ranging from foreground instances to background regions (e.g. sky, road).
Part: Fashionpedia 68 is a large-scale dataset that contains a comprehensive ontology of fashion elements. We include the main evaluation set (containing 1 1148 images) with part segmentation annotation and subset with even more fine-gr tine-grained subpart masks (containing 861 images). We also include 1000 images from each of the Paco, LVIS, Multi-Human Parsing, and Easyportrait datasets. Paco 71 is an extensive detection dataset that enriches the traditional scope of object detection by incorporating detailed annotations for object parts and attributes. The Multi-Human Parsing (69 dataset aims to partition a crowd scene image into semantically consistent regions belonging to the body parts or clothes items while differentiating different identities. Easyportrait [70] is a large-scale image dataset for portrait segmentation and face parsing.
D Mask-refinement Module
D.1 Design
We built the mask refinement module to identify systematic biases in the SAM pseudo labels before SUM training. The module trained by finetuning separate SAM model with mask-prompt segmentation on the human-annotated dataset. We limit fine-tuning to the decoder and prompt encoder, keeping the image encoder fixed, to prevent overfitting. In the final architecture, we only require the fine-tuned decoder and the prompt encoder, as the encoder is the same as the one in SAM.
21

---

D.2 Training
The training data for the mask-refinement module is obtained by generating pseudo masks by applying SAM numan-annotated datasets. This provides the prompt-target pairings necessary for our fine- tuning process. These pseudo masks are generated using a single point prompt from each ground truth mask, mirroring the conditions under which SAM pseudo labels produced. To address semantic ambiguity, the mask prompt is set to be the SAM mask output that is most aligned with the human annotation. We exclude pairs where the IoU between the SAM-generated pseudo label and the ground truth is below a threshold (set to 0.7). Additionally, we apply the same sample filtering technique as SAM during mask generation to ensure mask quality (see Appendix B in SAM (I). Fine-tuning is conducted on the prompt-target pairs, as shown in Fig. 6 The mask-refinement module also receives as input a box prompt encompassing the pseudo label mask, and the point prompt used to generate the pseudo labels. We empirically found that including isolated box or point prompts is counterproductive, so they were excluded (see Fig. prompt.
Precompated
Inpua firage
Eriiddilig
Refrel Prediction
Poongi Escoder
Pronept Sampling
SAM penada
Figure 6: Training of the mask-refinement module: The mask module is implemented by fine-tuning the decoder and prompt encoder of SAM. By leveraging SAM's automatic mask generation, we acquire initial pseudo labels and corresponding image embeddings. Subsequently, these pseudo labels undergo a refinement process to produce the refined prediction. During training the prediction is fit using high-quality, human-annotated.
D.3 Evaluation
The mask-refinemmnt module refines SAM predictions via mask mask- -rompt segmentation. Fig. 7 shows a comparison between SAM, our proposed mask-refinenn refineeeen module and three versions of HQ-SAM-
HQ-SAM - Original HQ-SAM model that uses distorted ground truth as the mask prompt during training.
HQ-SAM during training.
as HQ-SAM but without incorporating separate box or point prompts
HQ-SAM MP SAM pseudo label Same HQ-SAM MP but using SAM generated pseudo labels as the mask prompt in place of the distorted ground truth, the approach taken by the original HQ-SAM.
For fair comparison, all model variants were fine-tuned using only the HQSeg-44K 4 dataset, which contains human-annotater masks associated to the salient object segmentation task. The SAM pscudo labels are generated using using single point prompt sampled from the ground truth segmentation mask. The proposed mask-refinement module elearly outperforms the alternatives, which overfit the salient-object segmentation task.
Fig. g presents an ablation analysis of the mask- mask-refinement module, comparing the following ablated models fine-tuned using only the HQSeg-44K [4]
Without Filtering Proposed refneeetnn module without pseudo-label filtering, which disregards pairs with an loU below a set threshold.
With Synthetic Label Utilizes synthetic labels mask prompt in training.
22

---

(a) Mean IoU
Mean loU
Sal lency Object
Entity
Part
(b) Mean Boundary IoU
Mean Boundary tou
70
60
50
40
Sallency Object
Entity
Part
Configuration
SAM Input
HQ-SAM MP
Mask-vefinement module
HQ-SAM
HQ-SAN w SAM pseudo label
Figure 7: Comparative evaluation of the mask-refinement module. Comparison between SAM, our proposed mask-refinement module and three versions HQ-SAM. The original HQ-SAM suffers from overfitting specific tasks. HQ-SAM MP, which limits HQ-SAM to mask-prompt segmentation tasks while omitting individual box or point prompts during training achieves better generalization. Additional incorporation of SAM- -eM-eeeerteed pscudo labels yields improved results. Our mask-refinement module achieves the best performance.
With MAE Encoder -Emplyys encoder pretrained with Masked Autoencoder (MAE) weights.
With Encoder Adapter Incorporates lightweight adapter into the image encoder.
The ablation results show that the proposed design clearly outperforms the ablated alternatives, particularly in terms of generalization to other segmentation tasks.
Table 3 presents the outcomes of utilizing the same mask-refinement module featured in Fig 7 and to refine SAM predictions that were generated using one-box prompt. The results demonstrate that our approach is also capable of refining pseudo labels generated by box prompts.
In Fig ? we present scatter plot that compares the boundary IoU of SAM predictions (produced using a singlc-poin prompt) before and after refinement masks using the HRSOD dataset. For the majority of examples, mask-refinement module enhances the quality of the SAM prediction. However, in some cases the refined outputs are worse than the original prediction. Motivated in part by this, we propose leveraging the refined prediction via uncertainty-and -wwar aware training instead. To evaluate the correlation between certainty (1 (1-uncertainty) and SAM pseudo-labed accuracy: we used the HRSOD dataset and set a threshold of on certainty, identify regions high and low certainty areas. Results show that high-certainty regions have superior accuracy (ace: 98%) over the uncertain ones (ace: 28.55 28.5%). When the threshold 0.8, the accuracy in high-certainty regions increases 99.2%. indicating positive correlation between certainty and pseudo-labe accuracy.
23

---

(a) Mean IoU
70
Mean loU
60
50
40
Sall ency Object
Entity
Part
(b) Mean Boundary IoU
Mean Boundary lou
70
60
50
40
Sall ency Object
Entity
Part
Model Configuration
Without Filtering
asaa
Mask-reelinmment Module
Encoder
Adapter
With Synthetic Label
Figure 8: Ablation analysis of the mask-refinement module. Performance evaluation of different mask-refinement variants trained the HQSeg-44K dataset. We one-point prompt generate the SAM predictions, which are then taken as the mask prompt input to all the model variants.
Table Mask refinement with box prompts. Comparison SAM and the result of applying the proposed mask-refinement module in Fig. 7 and g to SAM predictions generated by a one-box prompt. The refinement process yields enhanced results, with notable improvement in boundary loU metrics.
Salient Object
Entity
Purt
Metrics
SAM
Mask- refinement Module
SAM
Mask- refinement Module
SAM
Mask- refinement Module
mloU
84.5
86.0
82.6
82.7
66.6
67.0
mBIoU
75.4
77.8
76.4
77.4
63.2
63.6
We provide some visualization examples of the refinement results via the mask-refinement module in Fig. T0
We further test whether our method improves upon the quality of SAM pseudo label generated via the automatic mask generation stage. Since there no ground truth mask provided corresponding to the automatic mask generation, we conduct the human evaluation. The results are provided in Appendix I.2
24

---

Boundary IoU Comparison
100
Mask-refinemen module output
95
80
75
70
90
SAM input
Figure 9: Comparison of ToU scores between one-point prompt SAM predictions and the correspond- ing refined prediction produced by mask-refinement module. Refinement improves segmentation quality for most examples.
Irout Image
Zoomedinn image
S4M Sceudo Label
Befined Psoudo Label
Uncortainty map
Input Image
Zoomed Image
SAM Pocudo Label
Refined Pseuda Label
Uncertainty map
Inout
Input Image
Image
Uncertainty map
Figure 10: Visual examples of the refinement results via the mask-refinement module.
25

---

E Semi-supervised Baseline
E.1 Implementation Details
AugSeg AdamW
49 employs teacher-student framework. Following SAM we use the (74 optimizer. The learning rate is set at 1 x 10-5.
AugSeg
For the human-annotace data, employ SAM's training procedure to train the student network, with both point and box prompts. We conduct interactive segmentation for 7 rounds. We minimize the same training loss as SAM: linear combination of focal loss [61] and Dice loss 62 with the same ratio of 20:1.
For the unlabeled data, we apply consistency loss based on predictions from two augmented versions of the same image, processed by the student and teacher networks. We binarize the prediction from the teacher network as the target, focal loss and Dice loss added between the prediction from the teacher network and student network with a weight ratio of 20:1. We found that this works better than directly computing the KL divergence or mean square errors between two predictions. We follow the augmentation process in [49]. Interactive segmentation requires prompts sampled from target labels. We utilize model-generatee pseudo-labels generate the prompts for unlabeled data in two different ways:
AugSeg Mixed-Prompp which employs combination of the point and box prompts. In scenarios where only single prompt employed, may be either a point or a box prompt. Conversely, when multiple prompts are utilized, they may either be a mix of point and box prompts or exclusively point prompts;
AugSeg Box-Prompt, which exclusively employs box prompts, applicable solely in single prompt scenarios.
In scenarios where a single prompt (either point or box) is provided, SAM designed to yield three outputs to accommodate ambiguity associated with the target mask. To incorporate consistency loss between two sets of predictions, we explored two approaches:
AugSeg Mixed- This is the default setting, where we select the most similar pre- diction to the pseudo-labe from each set, and add consistency loss between these two predictions. This approach aligns with the SAM training protocol, where only the prediction most closely matches the target mask is considered for loss calculation.
This variant computes the consistency loss between corresponding predictions from both sets, as determined by the same output token. Given that there three output tokens (each for one output) for single-prong prcpee-rromptteec scenarios, this results in three separate consistency losses.
AugSeg
In addition to sampling prompts from the pseudo-label, we employ a uniform prompt sampling strategy, denoted AugSeg Uniform Point-Prompt. This approach is akin to the automatic mask generation process utilized in SAM. Specifically, we provide the model with a regular grid of points as prompts. For each point, the model predicts a set of masks potentially representing valid objects. We then compute the consistency loss between the corresponding predictions from both sets, as determined by the same output token. Such non-discriminatory approach is feasible exclusively in single prompt scenarios, given the absence of target masks for interactive prompt sampling. However, employing this strategy has resulted in significantly inferior outcomes in our experiments. One plausible explanation could be the absence of filtering mechanisms similar to those in SAM's pseudo-label generation process, resulting in the retention of plenty low-quality predictions serving as regularization targets.
In the Table 4 we report the single-point prompt mean IoU comparison of different model variants fine-tuned on FT-Small. This shows that adding multiple consistency losses actually hurts the model performance.
For multi-prone scenarios, we sample interactive prompts for both teacher and student networks based on SAM pseudo-labels.
26

---

Table Single-poing prompt mean IoU comparison of AugSeg Mixed -eokdppromptt with version that adds multiple consistency losses
AugSeg Mixed-Prompt (multi-consistency)
AugSeg MixeddPrrepp
AugSeg Bex-Prompt
AngSeg Uniform Point-Prompt
Task
Salient object
82.3
81.4
83.2
82.5
Entity
74.6
Part Overall
48,4
44.6
43,5
37.8
68.8
66.8
67.0
64.2
70
700
60
60
Sallent Object
Entity
Part
Overall Average
Sallent Object
Entity
Part
Overall Average
Confidence-Uncer
Vanilla
SUM
SUM Discrepancy-Uncer
AugSeg Mixed-Prompt
Mean IoU and mean boundary IoU of point-prompt segmentation evaluation. Com- parisons of models fine-tuned FT-Small dataset with various strategies. SUM clearly outperforms all other strategies. Note that AugSeg Box-Prompt is not applicable as explained in Appendix
Figure 11:
U2PL Besides the standard loss function for both labeled and unlabeled examples, U2PI. [51] also employs contrastive loss to regularize the encoder's image embeddings based on the uncertainty of pseudo labels.
The original U2PL method relies on the confidence of model prediction logits for uncertainty quantification. However, our experiments show that this approach does not perform well with the segmentation foundation model. A possible explanation is that the foundation model, trained on substantial volume of labeled and unlabeled data, begins with relatively high initial quality. This contrasts sharply with semi-supervised settings, where both data and labels are limited. To ensure fair comparison, we adopt the same uncertainty map as used in SUM for the U2PI.. Additionally, we incorporate task prompts into the U2PL. framework. Following SAM [1, our training loss incorporates both pseudo and human labels, with added contrastive loss as specified in the original paper.
Following SAM M.
the AdamW
7 optimizer. The learning rate at 1 10
For the camman-annotated data, we employ SAM's training procedure to train the student network, with both point and box prompts. We conduct interactive segmentation for 7 rounds. We minimize the same training loss as SAM: a linear combination of focal loss [61] and Dice loss [62] with the same ratio of 20:1. In addition the standard SAM loss, we also incorporate contrastive loss in the embedding space for the pseudo-labeled data, we search the loss weight and find setting the contrastive loss weight to be gets the best performance.
However, U2PL performance falls short of that achieved by SUM. This suggests that merely adding uncertainty to the image embedding space, while continuing to train the prediction space with noisy pseudo labels, is less effective. We additionally provide the three-point prompt comparison results plot corresponding to Table in Fig I1 From this, observe that AugSeg's 's performance is comparable to that of Vanilla, while SUM surpasses both in terms these metrics.
this variant of the SUM model (SUM we adopt methodologies previously established for domain adaptive segmentation by Araslanov et al. [59] and Zheng et (58). This involves generating two distinct predictions for specified target mask and using the discrepancy between these predictions to quantify uncertainty.
SUM Discrepancy-Unccraaitty
27

---

We evaluated two different architectures: conventional teacher-student framework, as described by Araslanov and Roth [59], and a dual- -uulde structure based on the work of Zheng et [58]. In the latter, a single encoder feeds into two separate decoders that each produce prediction. To clarify the target mask for prediction, we employed a bounding box derived from the pseudo label as a prompt. Our experimental results indicate that the teacher-student framework generally outperforms the dual- decoder setup. Nonetheless, neither approach matches the efficacy of our proposed method, which involves training specialized mask refinement module utilizing high-quality human annotations.
E.2 Discussion
Semi-supervised Segmentation Directly integrating AugSeg within the interactive segmentation framework did not outperform the performance of directly utilizing SAM pseudo-labels as targets (i.e. the Vanilla bascline). We identify several potential avenues for enhancing semi-supervised methods in interactive segmentation scenarios:
Designing better supi-super methods for interactive segmentation necessitates a better framework for prompt sampling. Generating prompts from for unlabeled data is crucial. Here, the concept of uncertainty- aware prompt sampling, as introduced in SUM, shows potentials to improve model performance by avoiding producing misleading prompt locations.
pseudo-labels
Moreover, the occasional inaccuracies in the output of the teacher network, which provides a guiding target mask for the student network's training, suggest that incorporating an uncertainty-aware component into the loss function could further refine performance.
Our approach follows SAM, where consistency is enforced between the model's predictions and the pseudo-labels. The intuition is that the model should yield consistent outcomes in line with the pseudo-labels, despite variations in the sampling locations of the prompts from these pseudo-labels. This is orthogonal to augmentation-base Exploring the potential combination between the augmentation-baase semi-supervised strategies and our proposed approach presents a compelling avenue for future research.
semi-supervised strategies.
Uncertainty-aware Segmentation Our approach differs fundamentally from previous approaches of uncertainty-awan segmentation in both 1) the generation of uncertainty maps and 2) the utilization of these uncertainty maps, detailed below. We will clarify this in the revised manuscript.
The uncertainty- (57. or prediction differences from different heads to generate uncertainty [59]58). They are designed for Nomain-adapiin semantic segmentation or sumi-supervised semantic segmentation, where labels or data scarce, and segmentation quality is typically low. These methods are also specific to the limited task. In contrast, our proposed strategy is designed for the interactive binary segmentation foundation model, which has not been tackled by the previous method.
fine-tune fine-tuning strategies generate uncertainty via model prediction logit values [1
Uncertainty Map Generation: Our main insight is to utilize external supervision to detect and correct systematic biases that accumulate during training. This design logic is fundamentally different from relying the model self-training itself, as most previous models [51 17.. SAM undergoes several rounds of self-training, leading to the accumulation and overfitting of errors in pseudo-labels. In our evaluation, the model frequently predicts erroneous regions with high confidence, and different heads concur on these incorrect areas due to the memorization of label noise [60]. Therefore, previous methods [5] (5 59 58) to generate uncertainty maps, do not capture this uncertainty effectively. To further improve segmentation quality, it is logical to integrate form of scalable external supervision. This serves to identify and address the systematic biases that have accumulated during the training of foundation models. Our investigation into the design of mask- mensk-rffiement models is an initial endeavor in this regard, and we are optimistic about future advancements in this field.
Utilization of the Uncertainty Map: Our method the first to introduce uncertainty- aware prompt sampling for training interactive segmentation models, representing novel contribution to the field. In contrast, the semantic segmentation tasks addressed in [51] 57 do not involve prompt sampling. Additionally, we have tailored the uncertainty-aware focal and Dice loss to train the binary segmentation foundation model. They are not proposed in [7]57]
28

---

Quantitative comparison with previous methods: As reported in Table the proposed SUM method outperforms existing methods based on uncertainty quantification [57] 58 "SUM Confidence- Uncer" corresponds to [57, and "SUM Discrepancy- "ereppancy-Uneer corresponds to 58 Table
| also provides comparison with UP2L. [51. an uncertainty- -ware semi-supervised segmentation method that uses uncertainty maps in the feature space via contrastive learning, which also performs worse than SUM (see also Figure 5).
Continual Learning for Segmentation We also want to comment on how our setting differs from the typical frameworks of continual learning segmentation. Most of the papers on continual designed for inasmmnnremental segmentation. However, in case, it not clear how to adopt such an idea for the binary mask segmentation case, where the aim is to not forget different segmentation tasks. Notably, several seminal studies within this domain focus model weight distillation, creating a loss function that minimizes the discrepancy between predictions from previous and current models. This concept parallels the pseudo-labeling approach utilized in the SAM paper, which aims to maintain the prediction to be similar to aseudo-labels associated with different tasks.
learning for segmentation
Furthermore, some techniques strive to preserve previously learned knowledge by distilling inter- mediate feature maps. However, our experiments indicate that despite employing strategies like freezing the image encoder or using minimally light-weight fine-tuning methods such as HQ-SAM, task-specific knowledge degradation, forgetting, still occurs in segmentation tasks.
29

---

Additional Ablation Study
F1 Ablation on Light Fine-tuning
Salient Object
Entity
Part
SAM
SUM (Encoder
mlau
Adapter arch.)
SUM (HO-SAM arch.)
SUM (Full
Hine tuning arch.)
Number of point promots
Figure 12: Comparison of SAM and different versions of SUM fine-tuned on the FT-Small
dataset. Full parameter tuning shows the most significant improvement.
Fig. 12 compares SAM and different variants SUM fine-tuned on FT-Small: (1) fine-tuning
the decoder, the prompt encoder and the Adapter 722 plugged in the image encoder, (2) using
the same lightweight fine-tuning scheme and architecture as HQ-SAM, (3) fine-tuning whole SAM (the default setting). We observe that all variants of SUM outperform SAM salient-object segmentation and retain the performance on entity and part segmentation. Besides, full fine-tuning clearly outperforms the other models.
All the ablation experiments carried on the public dataset FT-Medium ensure reproducibility,
F.2 Uncertainty-awwe -waaee Sampling
Fig. 13 compares SUM to version of SUM without uncertainty-aware sampling on two evaluation datasets (DIS-VD and ThinkOhject5K val datasets), which contain objects with intricate structures. When evaluating, we provide prompts located close to the object boundaries, which are more challenging for the models. We observe that SUM with uncertainty-avee sampling has much better performance.
We compare SUM to version of SUM without uncertainty-awarc sampling during training for one-point segmentation Table 5 When evaluating, we provide prompts located close to the object boundaries, which are more challenging for the models (see Appendix [B.3]. We observe that SUM fine-tuned with incertainty-awart sampling has much better performance, especially for datasets with intricate structures like DIS-VD and ThinObject5k.
Input Image
No
Yes
Ground Truth
mloU Comparison
90
YES
NO
70
ThinObject5K val uncertainty-aware sampling on
DIS-VD
Figure 13: Comparison of SUM and version of SUM without intricate structures when the prompts are close to object boundaries. In such cases, uncertainty-aware sampling provides visually superior results (left) and a higher single-prompt mean loU comparison on two challenging test sets (right).
30

---

Table 5: Comparison of SUM and a version of SUM trained without uncertainty-awan sampling. Full table of 1-point-prompt mean IoU comparison corresponding to Fig. [3 Here, we adopt more challenging boundary-sensitive point selection tactics. Specifically, we select the "median" point from set boundary points, guaranteeing that the sample remains within the mask yet distanced from the geometric center.
w/o uncertainty sampling
COIFT DIS-VD HRSOD
91.7
91.0
66.1
Salient object
hinOhject5K
86.9
86.8
Big
79.3
79,5
Average
79.8
81.2
Entityseg 0
66.2
67.2
Entity
Enrityseg
68.6
69.0
Eniitysgg 2
69.9
71.0
mloU
Average
68.2
69.1
Bodypart
50.9
54.4
Easyportrai
45.2
Part
Rashionpedia
45.9
48.4
Fashionpedia subpart
31.6
Paco
43.3
43.4
44.6
Average
COIFT
86.6
86.3
DIS-VD
Salkent object
HRSOD
79.5
80.0
ThinObjeet5K
70.6
75.5
Big
68.1
68.4
Voceval
63.7
64.8
Average
72.4
Entityseg
62.1
Entity
Enrityseg
63.7
64,0
Earityseg
Average
63.0
63.88
Bodypart
46.6
49,5
Easyportrait
Part
Fashioopodia
42.9
45.6
Fashionpedia subpart
29.7
Paco
41.8
Average
40.4
41.0
All
Average
58.2
59.3
31

---

F.3 Different Backbones
In Table 6 we provide a comprehensive comparison between SAM and SUM implemented with different backbones, including ViT-B, ViT-L, VIT-H. Our model consistently outperforms SAM across different model backbones for all segmentation tasks and interactive rounds. The improvement is most significant for one-point prompt segmentation of salient objects.
Table 6: We present comparative evaluation of the mean Intersection Union (loU) and boundary IoU metrics for interactive segmentation, assessing the performance of SAM and SUM models across range of Vision Transformer (ViT) backbones.
Backbone
Metries
Points
Salient Object
Entity
Part
SAM
SUM
SAM
SUM
SAM
SUM
70.9
78.4
73.2
74.7
52.3
55.0
VTT-B
86.8
57.7
66.6
64.4
66.7
48.0
50.3
mBloU
mlnU
78.0
84.5
77.0
77.9
52.7
52.9
VIT-
mBloU
67.4
74.9
69.5
71.1
48.8
49.0
86.8
mioU
78.7
85.2
77.4
79.8
52.2
55.3
VITH
mBIoU
68.3
75.6
70.1
73.3
48.3
51.2
87.3
70.9
F.4
Additional Details of Main Ablation Experiments
In Fig. 14 we provide mloU comparison of the Vanilla baseline and the ablated versions of SUM. We provide an mean Boundary IoU comparison of the Vanilla baseline and the ablated versions of SUM in Fig. 15 Detailed dataset-wiss comparisons for interactive segmentation performance across
different ablated versions are provided in Tables and 9
Number of point prompts: 1
Number of point prompts:
90
80
mloU
60
Salient Object
Entity
Part
Salient Object
Entity
Part
Vanilla
SUM w/o Task Prompt SUM
SUM Continuous TP
Mask Refinement
Figure 14: Mean IoU for the ablation analysis. This figure reports the interactive segmentation
mean loU of different ablated versions of SUM, demonstrating the individual gain provided by uncertainty fine-tum fine-tuning and task prompts.
32

---

Number of point prompts: 1
Number of point prompts:
mBloU
60
50
Salient Object
Entity
Part
Salient Object
Entity
Part
Vanilla
SUM w/o Task Prompt SUM
SUM Continuous TP
Mask Refinement
Figure Mean boundary IoU for Ablation study. This figure reports the interactive segmentation mean boundary IoU of different ablated versions of SUM, demonstrating the individual gain provided by uncertainty-awan fine-tuning and task prompts.
Table 7: Ablation study: one-point-prompt segmentation evaluation. This table reports the results of different ablated versions of SUM, demonstrating the individual gain provided by uncertainty-aware fine-tuning and task prompts.
SUM w/o
SUM
Metries
Task
Dataset
Vanilla
Mask- relinement
Task Prompt
SUM
Contiruous
TP
DIS-VID>
54.7
59.22
58.1
69.1
68.
Salient object
HRSOD
86.5
87.7
87.1
90.1
89.6
ThinObeet5K
Rig
Voceval
83.1
82.7
82.9
84.8
84.5
Average
80.4
81.5
81.1
85.2
84.7
Entilyseg D
77.1
76.3
77.1
78.1
77.4
Emiiy
Entitysgg
78.7
78.1
78.1
79.3
79.0
Entilyseg
mioU
Average
78.8
78.1
78.5
79.8
79.2
Bodypart
Easyportrait
45.7
52.3
53.2
$5.3
53.8
Part
Fishionpedia
53.3
55.3
58.4
58.4
57.2
Pashioepedia suhport
38.4
40.2
45.3
Paco
49.0
48.6
49.0
51.8
$1.6
Average
50.2
$2.3
54.4
55.3
$4.6
All
Average
70.3
71.0
73.4
72.8
COIFT
90.2
89.7
S3.5
88.4
90.1
DIS-VD
60.8
661.2
Salient object
ThinObjat5K
71.2
71.9
71.3
76.1
76.7
Rig
67.6
69.2
70.0
75.2
71.3
Voceval
70.1
69.1
69.55
70.0
Average
70.7
71.9
75.6
75.3
Entitysog
Emitt
Entityseg
72.3
71.6
71.7
73.2
72.9
Entityog 2
73.0
72.4
73.1
75.0
74.4
mBloU
Averapo
72.0
71.2
71.8
73.3
72.9
Bodypart
$8.4
58.6
61.00
19.4
601.1
Easyportrait
38.0
44.2
45.1
47.4
45.7
Part
Fushionpedia
Fashioepedia suhpart
36.8
38.6
42.2
43.3
41.9
Paco
47.1
46.7
47.1
49.9
49.7
All
Average
62.3
63.3
64.0
66.4
65.9
33

---

Table 8: Ablation study: three- point-prompt interactive segmentation evaluation. This table reports the results of different ablated versions of SUM, demonstrating the individual gain provided by uncertainty-awar fine-tuning and task prompts.
SUM wlo Task Prompt
SUM
Metries
Task
Dataset
Vanilla
Mask- refinement
SUM
Concinuous
COIFT
95.7
96.0
95.9
95.9
95.7
DIS-VD
75.9
79.0
79.3
80.1
75.2
Salient object
ThjnCieee
HRSOD
92.1
92.1
Big
93.1
95.2
94.5
94.8
91.8
Voceval
92.9
92.9
93.3
93.2
91.7
91.6
Entitysgg
84.5
84.1
$4.9
$4.9
83.7
Emity
Entilyseg
Entityseg
88.1
S8.
88
$7.7
Average
86.5
86.2
$6.7
86.6
85.7
Bodypart
74.3
76.9
74.9
76.6
Easyportaitt
59.9
66.1
67.5
67.6
67.0
Part
Fashionpedia
68.8
70.8
Fishioepodia subpart
58.8
59.5
62.7
61.8
Phcn
62.0
62.2
62.9
63.5
63.2
Average
64.4
66.7
67.8
67.99
67.7
All
Average
80.3
81.5
$2.0
82.1
$1.0
COIFT
8
92.3
92.2
Salient object
Thin(Oteeat5K
HRSOD
82.1
87.9
83.4
88.5
X3.3
88.6
83.9
88.7
80.9
$7.0
Big
83.9
86.0
Viceval
81.3
79.6
Average
X2.7
83.9
84.1
84.4
81.5
Entitysgg
79.9
79.1
80.3
78.4
Emity
Entityseg
82.0
81.6
82.2
82.0
$0.9
Eutilyseg
83.4
82.7
83.7
83.5
82.3
mBBoUU
Average
Bodypart
68.9
71.1
70.7
69.4
70.8
Eacyponrait
56.4
Part
Fashionpedia
66.8
68.8
68.8
68.0
Fashionpedia subpart
$7.1
58.0
59.9
61.0
60.1
Paco
62.0
Average
Averago
34

---

Table 9: Ablation study: six-point-promy interactive segmentation evaluation. This table reports the results of different ablated versions of SUM, demonstrating the individual gain provided by uncertaint--aware fine-tuning and task prompts.
SUM wlo
SUM
Metries
Task
Dataset
Vanilla
Mask- refinement
Task Prompt
SUM
Concinuous
COIFT
960
96.2
96.2
96.1
95.9
DIS-VD
83.1
85.3
88.5
85.4
81.8
Salient objoct
HRSOD
ThjnCieee
Big
96.5
96.8
97.0
96.9
95.6
Voceval
95.0
95.0
94.9
94.9
94.2
Entitysgg
87.2
87.1
87
87.5
$6.5
Emity
Entilyseg
Entityseg
90.8
90.7
90.9
90.9
90.2
Average
89.0
89.0
89.2
89.2
$8.4
Bodypart
84.1
85.0
85.2
$4.1
84.3
Easyportait
75.4
78.7
78.
78.5
79,0
Part
Fashionpedia
77.6
78.0
77.4
Fishioopedia subpart
71.8
72.3
72.3
72.4
71.6
Phcn
71.7
72.2
72.5
72.3
71.5
Average
76.0
77.2
77.4
77.1
76.8
All
Average
86.0
86.7
$6.8
86.7
$5.9
COIFT
92.4
92.7
92.8
79.9
Salient object
Thin(Oheeat5K
HRSOD
84.6
89.4
89.6
85.7
X5.5
89.9
$6.0
90.0
83.2
$8.6
Big
89.2
90.0
87.2
Viceval
85.1
83.7
Average
86.3
86.9
N7.4
W7.3
84.9
Entityseg
82.9
83.5
$2.0
Emity
Entityseg
85.4
85.1
85.7
85.5
$4.4
Eutilyseg
86.6
86.3
86.9
86.8
85.7
mBBoUU
Average
Bodypart
78.9
80.0
$0.3
78.9
79.3
Eacyponrait
70.6
70.8
Part
Fashionpedia
75.3
75.9
76.4
76.4
75.6
Fashionpedia subpart
70.5
71.1
71.1
71.1
70.2
Paco
Average
73.7
73.5
Averago
35

---

F,5 Threshold for the Uncertainty-aware Dice Loss
In Fig. we present an ablation study of the uncertainty-awan Dice loss threshold, denoted by A. smaller implies greater reliance on the pseudo labels. Our findings indicate that overly low values of f adversely affect the model's performance in salient object and entity segmentation due the effects of pseudo label noise. On the contrary, excessively high 0 values may prevent the model from effectively leveraging part segmentation masks from the pseudo labels.
Number point prompts:
Number point prompts:
mlou
Salient Object
Entity
Part
Salient Object
Entity
Part
Theta Value
0.2
0.5
0.6
0.8
Number of point prompts:
Number point prompts:
80
mBloU
70
60
50
Sallent Object
Entity
Part
Salient Object
Entity
Part
Value
0.2
0.6
Figure 16: Ablation study for the 0 threshold in the uncertainty-awan Dice loss. The figures compare the model's ' mean IoU and boundary IoU for various segmentation tasks and different values.
36

---

G Additional Quantitative Results
G.1
Performance of SUM under Various Annotation Budgets
In this section we evaluate our proposed model (SUM) when fine-tuned under various annotation budgets, utilizing the FT-Medium and FT-Large datasets, We focus on point prompts and mean IoU. Additional results for box-prompt and mean Boundary IoU are provided in Appendix G.2 More granular dataset-wise results are reported Appendix G.4
FT-Medium Fig. 17 shows comparison between the Human-Amnotation-Onn) baseline fine-tuned on HQ-Seg-44k and EntitySeg train set, SUM fine-tuned FT-Medium (including SA-250K) and SAM. The Numan-Annotation Onlly baseline and SUM outperform SAM on the segmentation tasks associated with the human-annotalee data (salient-object and entity segmentation). Although the Aomman-AAnotationn
baseline suffers a dramatic decrease in performance on part segmentation, the performance of SUM on part segmentation actually improves with respect to SAM by 3 points in mean IoU. This is notable because part segmentation is not explicitly represented in the human annotations, demonstrating SUM's capability to enhance accuracy across general tasks.
Number point prompts:
Number prompts:
90
80
SAM
70
60
Annotation Only
Human
50
SUM
40
30
Salient Object
Entity
Part
Sal Object
Entiyy
Part
Figure 17: Interactive segmentation mean IoU of SAM, SUM fine-tuned on the FT-Medium dataset, and AanomnnnnnnttiinnOiiii baseline fine- mieettnedd combination of the HQ-seg-44k and EntitySeg training sets. Fine-tuning only using human annotations improves performance for the tasks associated with the annotations (salient-object and entity segmentation), but deteriorates it dramatically for part segmentation.
FT-Large Fig. compares the Caiman-AnntaatinnOnne -Oll baseline fine-tuned on the large-scat scale human-annotated internal train set, SUM, and the Vanilla baseline fine-tuned FT-Large (including SA-250K). Human-Annotation Annotation-Only and SUM outperform Vanilla on salient-object segmentation. The three models perform similarly entity segmentation. SUM is clearly superior in part segmen- tation. This shows the advantages of leveraging unlabeled data via SUM, even when the available human-annotate dataset is large-scale and diverse.
Salient Object
Entity
Part
90
Human Annotation Only
80
70
Vanilla
SUM
60
1
2
3
4
5
6
6
2 3
Number point prompts
Figure 18: Comparison of the interactive segmentation mean IoU of SUM and Vanilla baseline fine-tuned on the FT-Large dataset, and a scale internal dataset. SUM outperforms the other models for part segmentation and matches the Mumaan-Annotation-Ont on the other tasks.
-only baseline fine-tuned on a large-
G.2 Box Prompt Evaluation and Complementary Metrics
In this section we report evaluations for box prompts and complementary evaluation metries (mean IoU or mean boundary IoU) to the figures in the main paper.
G.2.1 Interactive Segmentation Evaluation with Initial Box Prompt
37

---

Salient Object
Entity
Part
HQ-SAM
Vanilla
(HO-SAM arch.)
SUM (HQ-SAM arch.)
2
3
5
2
Number rounds
Sallem Object
Entity
Part
80
mBloU
HQ SAM
Vanilla
HAM arch.)
60
SUM (HO-SAM arch.)
Number of rounds
Figure 19: Interactive segmentation evaluation with initial box prompt. These results complement those in the top row of Fig. 4 It compares the performance HQ-SAM (fine-tuned on HQ-Seg-44k) with that of Vanilla and SUM, both of which have undergone fine-tuning using the same lightweight scheme as HQ-SAM on FT-Small. The top row of this figure displays the mean IoU, while the bottom row shows the mean Boundary IoU. SUM matches HQ-SAM and outperforms Vanilla in sabient-object segmentation, and is superior in entity and part segmentation.
Sallent Object
Entity
Part
90
SAM
SUM (Encoder
80
Adapter arch.]
SUM (HQ SAM arch.)
70
Fine tuning arch.)
60
2
3
5
2
2
Salient Object
Entity
Part
SAM
mBloU
(Encoder
Adapter arch..)
70
(HQ-SAM arch.) SUM (Full
60
Fine-tuning arch.)
Number of rounds
Figure 20: Interactive segmentation evaluation with initial box prompt. These results com- plement those in the bottom row of Fig. 4 Comparison of SAM and different versions of SUM fine-tuned on the FT-Small dataset. The top row of this figure displays the mean loU, while the bottom row shows the mean boundary loU.
38

---

Number rounds:
Number rounds:
90
mlou
80
70
60
Salient Object
Entity
Part
Salient Object
Entity
Part
Only
Number of rounds: 1
Number of rounds: 6
80
mBloU
70
60
Sallent
Entity
Sallent Object
Entity
SAM
Human Annotation Only
MB SUM
Figure 21: Box-prompt assessment corresponding to Fig, 17 This figure shows the interactive segmentation evaluation, where the first prompt is a box. We compare SAM, SUM fine-tuned the FT-Medium dataset, and Human-Ammotation-Onn baseline fine-tuned on combination of the HQ-seg-44k and EntitySeg training sets. Fine-tuning only using human annotations improves performance for the tasks associated with the annotations (salient-object and entity segmentation), but deteriorates it for part segmentation in the case of one prompt. Since the box prompt has less semantic ambiguity compared to point prompts, the decrease not as dramatic.
39

---

G.2.2
Complementary Evaluation Metrics
Salient Object
Multiple Entities
Part
90
90
80
80
70
Prompe::
50
50
50
40
Number point prompts
Number point prompts
Number point prompts
SAM
-B- HQ-SAM
SUM
Figure 22: Mean IoU complementary to the mean Boundary IoU in Fig. SUM consistently outperforms SAM and HQ-SAM in quantitative comparisons, achieving the highest mean boundary IoU across diverse evaluation sets and interactive segmentation rounds. SUM model is fine-tuned on FT-Large dataset. Note that all the results here are single output results for a fair comparision with HQ-SAM.
Sallent Objact
Entity
Part
80
HQ SAM
mBloU
(HQ -AM arch.)
(HO-SAM arch.)
3
4
5
2
3
1
Number of point prompts
Figure 23: Mean boundary IoU supplementary to the mean IoU evaluation in Fig. Comparison of HQ-SAM (fine-tuned on HQ-Seg-44k) with Vanilla and SUM fine-tuned using the same lightweight scheme as HQ-SAM. SUM matches HQ-SAM and outperforms Vanilla salient-object segmentation, and is superior in entity and part segmentation.
40

---

8D
minnnu
SAN
LoRA
SUM
SAM
AugSeg Mixed Prompt
UZPL SUM Conficence-Uneet
Adopter FT
Decodare Pnempreooor
Vanilla
AugSeg Box-Promst
4009 SUM SUM Dicrepancy- Uncer
Figure 24: Mean boundary IoU supplementary to the mean IoU evaluation in Fig. 5 Left: Comparison of single point-prompt segmentation mean Boundary IoU for SUM versus models fine-tumed using various strategies on the HQSeg-44K dataset. All competing models improve on the salient-object segmentation task associated with this dataset but deteriorate other segmentation tasks. Right: Mean Boundary IoU for single point-prompt segmentation: comparisons of models fine-tuned on FT-Small dataset with various strategies. SUM clearly outperforms all other strategies on salient-object and part segmentation, while preserving a competitive performance on entity segmentation.
Sallent Object
Entity
Part
80
SAM
SUM (Encoder
mBtoU
Adapter arch.)
60
SUM (HQ-SAM arch.)
SUM [Full Fine-tuning arch.)
40
2
3
4
5
6
1
2
3
4
1
2
3
6
Number point prompts
Figure 25: Mean boundary IoU supplementary the mean IoU evaluation in Fig. 12 Comparison of SAM and different versions of SUM fine-tuned on the FT-Small dataset. Full parameter tuning shows the most significant improvement over SAM.
Number of point prompts:
Number of point prompts: 3
80
70
SAM
BBleu
60
Human
Annotation Only
50
40
30
Salient Object
Entity
Part
Salient Object
Entity
Part
Figure 26: Boundary IoU metrics corresponding to Fig. 17 This figure shows the interactive segmentation mean boundary IoU of SAM, SUM fine-tuned on the FT-Medium dataset, and a Human- Annotation-Only baseline fine-tuned on combination of the HQ-seg-44k and Entity training sets. Fine-tuning only using human annotations improves performance for the tasks associated with the annotations (salicnt-object and entity segmentation), but deteriorates it dramatically for part segmentation in the case of one prompt.
Salient Object
Entity
Part
Human Annotation Only
Vanilla
60
SUM
1
4
3
2
Number of point prompts
Figure 27: Mean Boundary IoU evaluation corresponding to Fig. 18 Comparison of the interactive segmentation mean boundary loU of SUM and Vanilla baseline fine-tuned on the FT-Large dataset, and /luman-AnnotttionnOOne outperforms the other models for part segmentation, and matches the performance of the Human- Annotation-Ona Only bascline on the other tasks.
baseline fine-tuned a large-scale internal dataset. SUM clearly
41

---

G.3 Dataset-wise Comparison with HQ-SAM
Here, we provide the dataset- wise performance comparison HQ-SAM (fine-tuned on HQ-Seg-44k) with Vanilla and SUM fine-tuned using the same lightweight scheme as HQ-SAM on FT-Small (HQ-Seg-44k and SA-250k).
We provide interactive segmentation results where the first prompt is point Table T0 (single- point-prompt segmentation) and Table 11 piix-point-rromppt segmentation).
We provide interactive segmentation results where the first prompt a box Table 12 (single-box- prompt segmentation) and Table 13 (starting with box prompt followed by five point prompts).
Table 10: Single-point-propee evaluation. This table shows the comparison of HQ-SAM (fine-tuned on HQ-Seg-44k) with Vanilla and SUM fine-tuned using the same lightweight scheme as HQ-SAM. SUM is superior in all the tasks.
Vanilla (HQ-SAM arch.)
SUM (HQ-SAM arch.)
HQ-SAM
COIFT
72.0
71.3
79.4
DIS-VD
Salient object
HRSOD
58.4
51.5
ThinObjeeL5K
44.6
45.2
54.7
Big
Voceval
54.5
51.0
60.3
Average
49.8
47.0
$7.4
Entityseg
47.3
54.4
57.7
Eutity
Entitysog
48.5
53.6
Entityseg 2
47.7
53.3
$7.4
mloU
Average
47.8
53.8
57.3
Bodypart
56.7
57.3
59.2
Easyportrait
22.8
26.4
20.7
Part
Fashionpedia
46.6
46.0
48.0
Fashionpedia subpart
Paco
38.1
38.5
39.3
Avenge
38.8
39.5
39.6
All
Average
45.5
45.8
$1.0
DIS-VD
18.6
16.0
23.4
Salient object
HRSOD
42.2
ThinObjectKK
32.9
34.1
Big
33.4
32.4
44.5
Voceval
Average
39.4
37.5
48.2
Entityseg
41.0
48.0
51.1
Eutity
Entityseg
42.3
47.8
50.6
Entityseg
mBHoU
Average
41.2
47.3
50.7
Bodypart
50.2
51.0
53.4
Easyportrait
13.8
18.1
12.1
Fashionpedia
Fashionpedia subpart
28.1
27.7
29.5
Paco
36.6
37.2
37.9
Average
34.6
35.5
Avenge
42

---

Table 11: Six rounds interactive segmentation evaluation: first prompt with initial point prompt. This table shows the comparison of HQ-SAM (fine-tuned on HQ-Seg-44kk with Vanilla and SUM fine-tuned using the same lightweight scheme HQ-SAM. SUM matches HQ-SAM and outperforms Vanilla salient-object segmentation, and is superior in entity and part segmentation.
Metries
Task
Dataset
HQQ-AM
Vanilla (HQ-SAM
SUM (HQ-SAM
arch.)
arch.)
COIFT
95.8
95.5
95.6
DIS VD HRSOD
84.3
77.7
78.4
94.5
92.7
93.7
object
Thin:Oject5KK
92.9
91.7
91.5
Big
94.0
93.5
95.7
Voceval
Average
92.4
90.8
91.5
Entityse
70.5
81.5
84.1
Entity
Entityseg
74.4
84.0
86.6
Entityseg
mloU
Average
72.8
83.8
86.4
Bodypart
74.3
80.5
82.7
Easyportrait
50.1
47.6
61.2
Part
Fashioopedia
64.9
68.6
Fashionpedia subpart
52.6
57.2
67.3
Paco
58.2
61.8
Average
All
Average
76.7
79.4
COIFT
91.6
91.3
91.4
DIS-VD
Salient object
HRSOD ThinObjeet5K
89.2
87.0
88.6
84.1
82.5
81.8
Big
Vaceval
81.4
83.5
84.5
84.5
83.1
Entityseg Entityseg
63.2
76.0
Eutity
67.6
79.0
Entityeeg 2
66.1
80.2
83.0
mBloU
Average
65.6
78.4
Bodypart
67.6
74.8
77.3
Easyportrait
40.0
38.5
52.7
Part
Pashionpedia
62.1
66.4
72.5
Fashionpedia subpart
Paco
56.5
60.4
67.2
Average
55.3
59.1
67.1
All
Average
70.0
73.5
77.5
43

---

Table 12: Single-box-prong evaluation. This table shows the performance comparison of HQ-SAM (fine-tuned on HQ-Seg-44k) with Vanilla and SUM fine-tuned using the same lightweight scheme as HQ-SAM. SUM matches HQ-SAM and outperforms Vanilla in salient-object segmentation, and is superior in entity and part segmentation.
Metries
Task
Dataset
HQQ-AMM
Vanilla (HQ-SAM
SUM (HQ-SAM
arch.)
arch.)
COIFT DIS VD HRSOD
95.4
94.9
95.2
79,5
70.4
77.1
92.9
90.3
92.7
object
Thin:Oject5KK
89.8
86.5
88.7
Big
88.6
81.8
89.4
Voceval
88.2
Average
89.1
85.2
83.7
Entitysg
60.0
73.5
76.3
Entity
Entityseg
64.6
75.8
78.8
Entityseg
Average
62.2
74.5
78.2
Bodypart
67.9
68.5
68.0
Easyportrait
38.0
37.0
36.2
Part
Fashioopedia
66.7
67.1
Fashionpedia subpart
55.2
56.7
62.5
Paco
56.2
56.3
59.9
Average
56.8
57.1
All
Average
71.8
COIFT DIS-VD
90.9
90.3
90.9
Salient object
HRSOD ThinObjeet5K
86.1
83.0
86.8
78.8
75.4
77.8
Big
Vaceval
74.7
74.4
76.5
79.7
76.2
Entityseg Entityseg
54.2
66.9
Eutity
58.8
69.6
73.1
Entityeeg 2
55.8
67.2
72.7
mBloU
Average
56.3
Bodypart
61.8
63.0
63.6
Easyportrait
31.0
31.0
29.3
Part
Pashionpedia
64.2
64.7
66.5
Fashionpedia subpart
Paco
5499
54.8
58.4
Average
53.0
53.7
55.7
All
Average
65.2
66.4
69.6
44

---

Table 13: Six rounds interactive segmentation evaluation with initial box prompt. This table shows the comparison HQ-SAM (fine-tuned on HQ-Seg-44k) with Vanilla and SUM fine-tuned using the same lightweight scheme HQ-SAM. SUM matches HQ-SAM and outperforms Vanilla in salient-object segmentation, and is superior in entity and part segmentation.
Metries
Task
Dataset COIFT
HQQSAM
Vanilla (HQ-SAM
SUM (HQ-SAM
arch.)
arch.)
95.9
95.8
DIS VD HRSOD
85.9
82.6
82.4
object
Thin:Oject5KK
94.7
93.2
93.5
92.5
94.1
92.6
Big
94.5
95.2
96.2
Voceval
Average
92.7
92.2
92.6
Entitysg
68.7
82.2
85.2
Entity
Entityseg
72.2
84.6
87.5
Entityseg
mloU
Average
70.5
83.8
87.2
Bodypart
79,3
82.6
84.5
Easyportrait
50.8
54.0
66.0
Part
Fashioopedia
72.3
73.3
Fashionpedia subpart
62.4
64.9
74.4
Paco
62.5
65.4
Average
65.5
All
Average
78.2
COIFT
91.5
91.7
91.6
DIS-VD
Salient object
HRSOD ThinObjeet5K
89.6
87.9
89,3
84.3
83.2
83.1
Big
Vaceval
80.8
83.7
85.4
84.8
84.7
85.7
Entityseg Entityseg
62.2
76.9
Eutity
65.9
79.9
Entityeeg 2
63.8
79.3
83.8
mBloU
Average
64.0
78.7
82.3
Bodypart
72.5
Easyportrait
41.7
45.3
Part
Pashionpedia
69.4
71.2
76.7
Fashionpedia subpart
Paco
61.0
64.1
72.5
Average
61.0
64.2
71.9
All
Average
71.8
76.1
80.1
45

---

G.4 Dataset-wise Comparison with SAM
Here, we provide the dataset-wise performance comparison of SAM with SUM fine-tuned full parameter fine-tuning manner on different train sets: FT-Small, FT-Medium, and FT-Large.
We provide interactive segmentation results where the first prompt is point in Table 14 (singl--point prompt segmentation), Table 13 five-point-prompt segmentation) and Table T6 (six (six-point-promg) segmentation).
We provide interactive segmentation results where the first prompt is a box in Table 17 (single- box-prompt segmentation) and Table 18 (Prompts: box as first prompt and following 5 point prompts).
Table 14: Single-point-romptt evaluation Comparison of SAM with SUM fine-tuned on FT-Small, FT FT-Medium, and FT-Large. SUM achieves consistent overall improvement compared SAM, improves more when more human- -amman--nnotttdd data is included.
SUM (FT-Meddim
Metrics
Tisk
Dataset
SAM
SUM (FT-small)
SUM (FT-large)
COIFT DIS-VD
93.7
95.0
93.1
94.8
51.0
72.6
69.1
74.1
Salient objoct
HRSOD
85.1
91.9
90.1
92.1
ThinObject5K
79.4
90.2
87.5
88.1
Big
Voceval
81.1
85.4
84.8
85.5
78.7
86.4
85.2
86.9
Entityseg
75.4
78.1
Fatity
Entityseg
77.2
74.9
79.3
75.3
Entilyseg
79.6
76.8
81.9
77.1
mloli
Average
77.4
75.1
79.8
75.8
Bodypart
66.6
65.9
68.5
Easyportrait
40.6
56.4
55.3
75.1
Part
Fashionpedia
58.6
58.2
61.8
Fashionpedia suhpart
44.7
44.2
45.3
46.8
Avenge
52.2
55.0
55.3
60.5
All
Average
69.0
72.8
73.4
75.1
COIFT
DIS-VD
41.3
64.7
60.8
66.6
HRSOD
77.3
85.2
83.0
86.1
Salient objoct
ThinOhject5K
64.5
79.7
76.1
77.7
Big
70.9
72.5
75.2
74.9
Voceval
67.7
70.9
70.0
70.6
Average
68.3
77.3
75.6
77.7
Entityseg 0
68.3
66.1
71.8
67.6
Eatity
Entityseg
Entityseg 2
71.5
68.2
75.0
68.9
mBloU
70.
67.3
73.3
68.2
Bodypurt
Easypontrai
32.6
47.9
47.4
69.5
Part
Fashionpedia
56.5
55.8
55.9
59.2
Fashionpedia subpust
42.9
42.3
43.3
44.8
Paco
49.0
48.7
49.9
48.5
Avenge
Averag
46

---

Table 15: Five rounds interactive segmentation evaluation with initial point prompt. Comparison of SAM with SUM fine-tuned on FT-Small, FT-Medium, and FT-Large. SUM consistently outper- forms SAM, showing even greater improvement as the budget human-annotated data increases.
Metrics
Task
Dataset
SAM
SUM (FT-small)
SUM (TT-Medium)
SUM (FT--Large)
COIFT DIS-VD
95.2
96.0
96.1
96.4
76.2
84.1
84.3
84.1
Salieat objoct
ThinOhject5K
HRSOD
90.1
92.7
93.7
92.6
94.2
93.3
94,8
93.5
Big
Voceval
93.2
93.8
94.6
94,7
93.4
Entityseg
86.3
Entity
Entityseg
86.3
86.2
88.7
87.8
Entityseg
88.3
88.1
90.4
89.3
mloU
Averuge
86.1
86.1
88.7
87.8
Bodypart
78.3
81.9
82.3
85.6
Easyportrait
68.6
77.7
76.1
86.2
Part
Fashionpodia
75.8
75.2
76.4
78.1
Fashionpedia subpurt
67.6
68.5
70.1
70.6
Paco
Avergee
71.9
74.4
75.0
78.4
All
Averggo
82.8
84.7
85.7
86.9
COLFT
DIS-VD
68.0
77.4
78.5
78.6
Salient object
HRSOD
87.1
90.9
ThinOhject5K
79.0
84.0
85.6
86.2
Big
85.2
85.6
89.1
90.1
84.5
84.8
Average
81.9
85.0
86.7
$7.3
Entityseg 0
78.1
78.2
82.9
81.6
Entily
Entityseg
83.4
Entityseg 2
82.5
82.1
86.1
84.2
mBloU
Averaga
80.6
Bodypart
80.3
Easyportrait
60.7
69.5
67.2
81.4
Part
Fashionpedia
74.0
73.2
74.7
76.6
Fashionpedia subpart
66.3
67.0
68.7
69.4
Paco
66.7
67.2
69.0
70.4
Average
75.6
All
Averago
76.7
80.8
82.2
47

---

Table 16: Six rounds interactive segmentation evaluation with initial point prompt. Comparison of SAM with SUM fine-tuned on FT-Small, FT-Medium, and FT-Large. Comparison of SAM with SUM fine-tuned on FT-Small, FT-Medium, and FT-Large. SUM consistently outperforms SAM, showing even greater improvement as the budget of human-annone annotated data increases.
Metries
Task
Dataset
SAM
SUM (FT-small)
(FT. Medium)
SUM (FT-Large)
96.1
96.5
DIS-VD
78.2
84.6
85.4
84.8
Salicat objoct
ThinOhject5K
HRSOD
91.0
92.8
93.5
93.6
Big
95.4
95.5
96.9
97.2
Voceval
93.6
94.4
94.9
95,0
Average
91.2
92.8
93.5
93.7
Entityseg 0
84.6
84.8
87.5
86.8
Entity
Entityseg
87.1
88.5
Entityseg 2
88.8
88.7
90.9
90.0
mfot
Average
Bodypart
Easyportrat
73.9
79.9
78.5
87.2
Fashionpedia
77.6
79.7
Fashionpedia subpart
70.4
70.7
72.4
72.8
Paco
70.4
70.5
72.3
73,9
Average
74.5
77.1
All
Averago
84.3
86.7
87.7
COIFT
91.2
92.3
92.8
93.3
DIS-VD
70.2
77.9
79.9
79.4
Salient object
HRSOD
88.0
88.5
90.0
91.0
ThinOhject5K
Big
86.4
86.4
90.0
90.3
Voceval
82.1
83.1
85.1
85.4
Averaee
82.9
87.3
87.6
Entityseg 0
79.1
79.2
83.5
82.4
Entity
Entilyseg
82.2
81.7
85.5
84.2
Entityseg 2
83.3
82.8
86.8
85.2
mBloU
Average
81.5
81.2
85.3
83.9
Bodypart
75.2
78.5
78.9
81.5
Easyportrait
70.1
82.8
Part
Fashionpedia
75.9
74,8
76.4
78.3
Fashionpedia subpurt
Paco
69.1
69.1
71.0
72.7
Average
70.9
77.4
Averaga
78.3
83.2
48

---

Table 17: Mingle-box-prompe segmentation comparison SUM consistently outperforms SAM, showing even greater improvement as the amount of human-annotatee data increases.
Metrics
Task
Datasel
SAM
SUM (FTsmall)
SUM (FT-Meedium)
SUM (FT-Large)
COIFT HRSOD
94.3
95.7
95.0
95.6
Salient object
ThinOhject5K
91.3
80.0
91.5
93.0
92.8
93.5
86.8
Big
85.8
90.5
91.5
90.9
Voceval
87.3
89.7
90.7
89.5
Averaee
84.5
89.8
89,2
Entityseg 0
80.8
83.1
81.7
Entity
Entityseg
82.7
81.4
84.9
83.2
Entityseg
mloU
Average
82.6
81.3
84.8
82.8
Bodypart
74.5
73.7
74.7
75.8
Eayportrai
Part
Fashionpedia
74.8
72.3
72.5
74.1
Fashionpedia subpurt
69.6
66.8
67.3
68.2
Paco
67.0
64.9
65.6
67.2
Average
66.6
64.8
65.8
68.7
All
Average
77.7
79.2
80.2
80.5
COIFT
89.6
92.1
90.9
91.4
DIS-VD
59.9
72.0
71.8
71.8
Salient object
ThinObject5K
HRSOD
67.1
81.4
78.8
86.8
88.0
76.4
Big
76.0
Voceval
74.9
76.6
77.6
76.6
75.4
81.5
81.1
80.8
Entityseg 0
74.5
72.9
77.6
75.6
Entity
Entityseg
77.1
74.9
79.6
77.4
Entityseg 2
77.6
75.2
80.6
76.8
mBIoU
Averaga
76.4
79.3
76.6
Bodypart
69.3
68.9
69.3
Easyportrait
40.3
39.0
41.8
53.3
Fashionpedia
Fashiompedia subpurt
68.0
65.1
65.6
66.6
Paco
65.5
63.2
65.5
Avenge
63.2
62.1
65.4
All
Averge
71.3
72.7
73.9
74,4
49

---

Table 18: Six rounds interactive segmentation evaluation with initial box prompt. Comparison of SAM with SUM fine-tuned FT-Small, FT-Medium, and FT-Laagee SUM consistently outperforms SAM, showing greater improvement as the amount of human-annotated data increases.
SUM (FT-Meedium)
Metrics
Task
Dataset
SAM
SUM (FT-small)
SUM (FT--Large)
COIFT
95.7
96.
96.1
96.4
DIS-VD
80.3
85,3
86.6
86.1
Salieat objoct
ThinOhject5K
HRSOD
91.2
93.7
93.0
94.3
93.7
95.1
93.9
Big
97.3
Voceval
94.0
94.5
95.1
95.3
Entityseg
85.8
87.7
Entity
Entityseg
88.0
87.6
89.7
89.2
Entityseg
89.4
89.1
91.3
90.6
mloU
Averuge
87.7
87.5
89.7
89.2
Bodypart
82.8
84.4
85.7
87.3
Easyportrait
63.9
71.6
76.8
86.4
Part
Fashionpedia
81.0
78.8
79.9
81.2
Fashionpedia subpurt
77.6
75.2
76.2
78.2
Paco
Avergee
76.
76.8
78.8
82.1
All
Averggo
85.3
86.0
87.6
88.7
COLFT
DIS-VD
72.5
78.5
81.1
80.6
Salient object
HRSOD
88.3
91.2
ThinOhject5K
79.8
84.4
86.2
86.7
Big
86.5
86.2
90.5
90.5
82.7
85.9
Average
83.5
85.6
87.7
88.0
Entityseg 0
80.4
80.3
84.2
83.4
Entily
Entityseg
Entityseg 2
84.0
83.3
87.3
86.0
mBloU
Averaga
82.5
82.0
84.9
Bodypart
77.7
82.5
Easyportrait
55.3
63.9
69.0
82.5
Part
Fashionpedia
79.4
76.9
78.3
79.7
Fashionpedia subpart
76.4
73.8
75.0
77.2
Paco
74.0
72.4
74.1
76.3
Average
79.6
All
Averago
79.4
80.4
82.9
84.4
50

---

G.5 SUM vs SAM under the Single Output Evaluation Setting
In this section, we provide a single output evaluation of SAM and SUM fine-tuned on different train sets (FT-Small, FT-Medium, and FT-Large). We provide the IoU evaluation in Fig. 28 and boundary IoU evaluation in Fig. 29 In this setting, only one output is provided when single prompt is provided, as specified in Appendix B.3
Salient Object
Entity
Part
First Prompt: Box
80
mlou
60
40
4
2
3
First Prompt: Point
80
mlou
60
40
1
2
4
6
2
4
5
1
3
5
Number of rounds
SAM
FT-Small
FT-Medium
FT-Large
Figure 28: Evaluation of mean IoU for the single output. This figure displays the mloU values for the single output case, segregated into three segmentation tasks: Salient Object, Entity, and Part. The performance of the SAM and models fine- tneettned on datasets of varying annotation budgets -mall (FT-Small), medium (FT-Medium), and large (FT-Large) plotted over six iterative rounds.
G.6 Results on Additional Evaluation Sets
We organize evaluation tasks in the experiment according the hierarchical level of granularity, covering various levels (part, entity, multiple instances). Each task is evaluated using several diverse datasets designed to encompass wide range of images and subtasks. For instance, our part segmentation evaluation utilizes five diverse datasets: Fashionpedia, Fashionpedia Subpart, Paco, Multi-Human Parsing, and Portrait. The first two include comprehensive ontology of fashion elements and different levels of part granularity. Paco covers 75 object classes and the last two focus on different human-specific part segmentation subtasks.
To further strengthen the robustness of our proposed methods, we have extended our evaluation by testing SUM and SAM on additional image types. For reproducibility, SUM is fine-tuned the Public dataset FT-Medium. We selected 7 datasets from the evaluation sets used in SAM to complement our existing 14 public evaluation sets. Additionally, have included part one of synthetic dataset, GTAV 82). These additional evaluation sets encompass various image types e.g. driving. synthetic, egocentric, irregular shapes, paintings, underwater animals, drones, and underwater trash.
The mloU comparison results, reported in the Table 19 confirm that SUM consistently outperforms SAM.
51

---

Salient Object
Entity
Part
First Prompt: Box
80
mBloU
60
40
1
4
6
2
3
5
First Prompt: Point
80
mBloU
60
40
1
2
3
4
6
2
4
5
6
1
2
3
5
6
Number of rounds
SAM
FT-Small
FT-Medium
FT-Large
Figure 29: Evaluation of mean boundary IoU for the single output. This figure displays the mean boundary IoU values for the single output cases, segregated into three segmentation tasks: Salient Object, Entity, and Part. The performance of the SAM and models fine-tuned on datasets of varying annotation budgets small (FT-Small), medium (FT-Medium), and large (FT-Large) plotted over six iterative rounds.
Table 19: Point-prompted interactive segmentation This table reports the point-prompt mean IoU over interactive segmentation rounds. SUM consistently outperforms SAM across additional evaluation datasets over interactive rounds.
Dataset
Image type
Method
Round
Round
Round
Round 9
SAM
44.1
50.9
$8.6
64.6
Cityscapes
Driving
SUM
46.4
57.3
62.5
67.1
SAM
77.9
85.4
90.2
91.9
EgoHOS
84
Egocentric
SUM
79.0
90.0
92.3
93.5
SAM
68.4
81.3
87.9
91.1
DRAM 8
Paintings
SUM
73.6
85.8
89.2
91.4
SAM
41.4
60.4
75.7
84.6
ishape 86
Irregular shapes
SUM
74.9
87.3
92.6
94.4
SAM
44.8
47.0
$2.2
57.0
GTAV 82
Synthetic
SUM
45.7
52.8
56.7
59.7
SAM
86.2
88.3
90.2
91.3
NDD20 87
Underwater Animal
SUM
87.9
91.3
92.2
93.1
SAM
63.3
69.1
76.7
82.6
TrashCan 88
Underwater
SUM
64.9
76.8
82.0
86.1
SAM
78.8
84.5
91.2
93.6
IBD 89
Drones
SUM
78.4
87.6
91.4
93.9
52

---

Additional Qualitative Results
H.1 Task Prompt
Fig.
32, and shows the multiple outputs of SUM for the same input and point prompt when we vary the task prompt. When provided with task prompt I, associated with 'Salient Object', the model is able to generate segmentation masks that encompass multiple salient objects within the image.
30 31
Inpot Image and point prompt
Task prorept
Output
Output
Ouiput
(Default)
(Salion Object)
2 (Fmiryy)
Figure 30: Outputs of SUM for different task prompts.
Input Image and poine prompt
Task prompp
Curpan
Ourput
Outpot
(Defaul))
(Salient Object)
(Entiry)
Figure 31: Outputs of SUM for different task prompts.
53

---

Input Image and point prompt
Task prontee
Ourput
Output
(Default)
(Salient Object)
(Emity)
Figure Outputs of SUM for different task prompts.
Input Image and point prompt
Tisk promutt
Ourput
Output
(Default)
(Salion Object)
(Entity)
Figure 33; Outputs of SUM for different task prompts.
54

---

Input Image
SAM
SUM
Ground Truth
Figure 34: Comparative visualization of segmentation outcomes using single-box prompts. SUM produces predictions of substantially higher quality than those of SAM..
H.2
SUM versus SAM
The following examples compare SUM to SAM for different segmentation tasks: salient object segmentation Fig 34 Fig. 35 and Fig, 36 entity segmentation in Fig. 37, and part segmentation in
Fig. 38
55

---

Input Image
SAM
SUM
Ground Truth
Figure 35: Comparative visualization of segmentation outcomes using single-box prompts. SUM
produces predictions of substantially higher quality than those of SAM.
56

---

Input Image
SAM
SUM
Ground Truth
Figure 36: Comparative visualization of segmentation outcomes using single-box prompts. SUM
produces predictions of substantially higher quality than those of SAM.
57

---

Input Image
SAM
SUM
Ground Truth
Figure 37: Comparative visualization of segmentation outcomes using single-box prompts. SUM
produces predictions of substantially higher quality than those of SAM.
58

---

Input Image
SAM
SUM
Ground Truth
FROOOR
Figure 38: Comparative visualization of segmentation outcomes using point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM produces predictions of substantially higher quality than those of SAM.
Input Image
SAM
SUM
Ground Truth
Figure 39: Comparative visualization of segmentation outcomes using point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM produces predictions of substantially higher quality than those of SAM.
59

---

H.3 SUM versus HQ-SAM
In this section we show a comparison between HQ-SAM and a version of SUM that is constrained to utilize the same lightweight fine-tuning scheme for fair comparison. We compare (1) the proposed SUM method fine-tuned on FT-Small (i.e. HQ-Seg-44k and SA-250k): (2) The original HQ-SAM, fine-tuned following the Nluman-Anntation
baseline approach (i.e. fine-tuning exclusively on
HQ-Seg-44k). Fig. 40. Fig. 43 and Fig. [42 show different examples. Additionally, we show several
examples for the part segmentation task in Fig. 45
Input Image
HQ-SAM
SUM (HQ-SAM arch.)
Ground Truth
Figure 40: Comparative visualization of entity segmentation outcomes using 5-point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM achieves better entity segmentation performance than HQ-SAM.
60

---

Input Image
HQ-SAM
SUM (HO-SAM arch.)
Ground Truth
Figure 41: Comparative visualization of entity segmentation outcomes using 5-point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM achieves better entity segmentation performance than HQ-SAM.
Input Image
HQ-SAM
SUM (HQ-SAM SAM areh.)
Ground Truth
Figure 42: Comparative visualization of entity segmentation outcomes using 5-point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM achieves better entity segmentation performance than HQ-SAM.
61

---

Input Image
HQ-SAM
SUM (HQ-SAM arch.)
Ground Truth
apankacHoice
apunkactdlce
apunkacHoice
apimkacctoice
Figure 43: Comparative visualization of part segmentation outcomes using 5-point prompts, where blue points signify positive prompts and red points indicate negative prompts. SUM achieves better part segmentation performance than HQ-SAM.
62

---

Human Evaluation
I.1 SAM Pseudo Label Quality
In this section, we report a comprehensive human evaluation on a subset of the SA-1B dataset, consisting of 20,000 randomly selected masks. The evaluation protocol involves series of questions designed to assess the quality of the segmentation masks in terms of coverage, boundary precision, and object part identification.
The criteria for evaluation are defined as follows:
This criterion determines whether mask accurately encompasses the target object or region within an image. It deemed 'correct" the mask captures the entire object or region without fragmentation object's edges is not mandatory, the mask must include all components of the object or specified area. For objects that part of larger entity, correct coverage is achieved when the mask fully encloses the object's s subset. The coverage labeled as 'undecided" the object's boundaries are indiscernible, such in conditions of poor visibility when edges are occluded.
Coverage:
inclusion of adjacent non-target entities. While exact adherence to the
Boundary: This aspect examines how well a mask's edges align with the actual boundary of the target entity. A boundary is considered "accurate' the mask's periphery matches the entity's 's contours throughout. Conversely, boundary is 'inaccurate' if there is a noticeable discrepancy between the mask's ' edges and the entity's true outline, irrespective of overall coverage adequacy. Similar to coverage, the boundary is rated as 'undecided" when the true edges of the entity are not clearly distinguishable, which may occur in conditions of low visibility or obstruction.
Part: Human participants respond with 'yes' or 'no' to determine if the masked region represents a portion of another object. The term 'part' specifically refers to a segmented region that is a constituent of larger object. In contrast, standalone entities such person, animal, not considered 'parts.' Similar to the previous two categories, we also provide 'undecided' option in case it is not clearly distinguishable.
Boundary
Coverage
Part
Coverage
Undecided
Undec ded
Others
Undecided
4.6%
Inwcourate Boundlary
46%%
0.1%
39.8%
Accurate Boundary
$1.2%
7%
Port
Accurata Coverage
Figure 44: Human evaluation of SAM pseudo label quality. We conducted human evaluation on randomly selected 20,000 masks in the SA-IB dataset. While the overall coverage the object masks is satisfactory, the boundaries are predominantly of poor quality.
Fig. shows some key findings:
1. Approximately 55.6% of the pseudo labels were identified with boundary inaccuracies.
2. About 81.2% of the labels were adjudged to have accurate coverage.
3. 52.7% were classified part masks.
This analysis yields valuable insights, indicating that while the overall coverage of the object masks is satisfactory, the boundaries predominantly of poor quality. Consequently, refining masks emerges as a viable approach to enhance mask quality without necessitating significant alterations to the original pseudo labels in many cases.
In addition, we studied the sample filtering metrics used in SAM. SAM utilizes two key metries, Predicted IoU and Stability Score, for pscudo label filtering, For more detailed description of the
63

---

two metrics, please refer to Appendix B (automatic mask generation details) in M. Here, our analysis specifically focused on examining the role of the filtering metrics in further discerning the quality of pseudo labels that have already been subjected to initial filtering phase.
Fig- 45 and Fig- show the results. For boundary accuracy, the Predicted loU did not show significant variance between accurate and inaccurate classifications (p-value = 0.548), unlike the Stability Score (p-value 0.00087). In terms of coverage accuracy, both Predicted IoU (p-value 0.037) and Stability Score (p-value = 5.83e-06) displayed significant differences between correct and incorrect categorizations. However, the observed overlap in both metrics across quality categories suggests limitations in their discriminative ability to accurately assess boundary accuracy and coverage after the initial filtering.
Distribution Predicted IoU by Boundary
1,075
1,000
1.025
Predicted leU
1.000
0.975
0.950
0.925
0.900
Scourate
inacurrae
undecided
Boundary
Distribution of Stability Score by Boundary
0.98
Stabiity
0.07
accurate
Inaccurate
undecaded
Boundary
Figure 45: Top: Distribution of Predicted Intersection over Union (IoU) by Boundary Accuracy. This boxplot illustrates the distribution of predicted IoU values across images categorized by boundary accuracy. The categories include accurate, inaccurate, and undecided boundaries. The median, quartiles, and outliers are represented within each category. indicating the variability of the IoU metric in relation to boundary accuracy. Bottom: The results for the Stability Score are similar.
64

---

Distribution of Predicted IoU Coverage
1.075
1.060
1,025
Presicted IeU
1.000
0.975
0.050
0.925
0.900
0.675
incorrect
Coverage
Distribution of Stability Score by Coverage
0.99
0.96
Stability
Coverage
Figure 46: Top: Distribution of Predicted Intersection over Union (IoU) by Coverage Correctness. Presented here are the predicted IoU values for images, grouped by the correctness of coverage. The categories include correct, incorrect, and undecided coverage. Bottom: Distribution of Stability Scores by Coverage Correctness. This boxplot showcases stability scores across images with varying levels of coverage correctness.
65

---

Comparison of Boundary Accuracy
Comparison of Coverage
35%
Ours Refined
SAM Pseudo Label
Ours Refined
SAM Pseudo Label
Both Labels are Bad
Both Labels Good
Both Labels are
Both Labels are Good
Semantic Correspondence
Figure 47: This figure presents the outcomes of human evaluation comparing segmentation masks. total of 2,000 mask pairs, randomly selected from the SA-1B dataset, were assessed for boundary accuracy and coverage. Participants were tasked with determining the accuracy of mask boundaries and the adequacy of target area coverage, choosing between the SAM Pseudo Label and Ours Refined for each metric, or indicating if both labels were satisfactory or unsatisfactory, The results demonstrate a clear preference for masks refined by our module, highlighting improvements in boundary quality without compromising the comprchensive coverage of the original SAM masks. Semantic correspondence between the original and refined masks was also verified.
66

---

1.2 Mask-refinement Module
We tested whether the proposed mask-refinement module improves the quality of SAM pseudo labels via human evaluation. Participants were presented with the task of comparing segmentation masks generated by the SAM pseudo label approach against those refined by our mask-refinement module. We randomly selected 2,000 mask pairs from the SA-1B dataset to undergo comparison. Too ensure fair comparison, we randomized the listing order of the two segmentation masks for each pair. The evaluation was focused on two key quality metrics:
Boundary Accuracy: Participants were asked to assess the accuracy of the mask's boundary in relation the true outline of the target object. The options provided for the judgment included: (i) SAM Pseudo Label, indicating that the original SAM-generated mask had more accurate boundaries, (ii) Ours Refined, denoting more accurate boundary of our refined mask, and (iii) Both Labels are Bad Both Labels are Good, to reflect situations where neither or both approaches were satisfactory.
Coverage:
For the coverage metric, participants evaluated the extent which the mask encompasses the intended target area without including extraneous regions or omitting portions of the object. The same response options were provided as for the boundary accuracy assessment.
Participants reviewed whether the masks align in terms of their semantic categories. For instance, part mask in one should correspond to part mask in the other; similarly, an instance mask should not encompass multiple entities if its counterpart does not.
Semantic Correspondence:
presents the human evaluation results, which demonstrate the improved accuracy in boundary quality of our refined masks over the original SAM pseudo labels. Our approach the extensive coverage of the SAM model's masks. Additionally, there is confirmed semantic correspondence between the pseudo and refined masks.
Fig. 47
also preserves
67

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
we provide the context and the experiments support the main claims of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we provide discussions on the limitations of the proposed method in the main paper section and further discussion in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
68

---

Answer:
[NA]
Justification:
the paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: we provide detailed descriptions of the implementations, including the dataset, and training resources, optimization functions etc in Section 4.1 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
69

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: all the public dataset are available according to separate licenses as detailed in Appendix A..I] Some internal dataset licensed be public. Some parts of the code made available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic//uuiee/CodeSuuuiii oow.eeeisabmmssssoooooooco. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/aips.cc/public/gu..ees
sssssssssiniiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: provide detailed descriptions of the settings of the experiment both in the main paper and the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: we follow the common practices of the segmentation domain and the previous papers, which do not provide the error bar due to extensive computational requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
70

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide such information the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: we have reviewed the Code of Ethics and fully obey
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
we provide that in the Appendix A.2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
71

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: we currently do not release the model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we provide the license and the requirements in the Appendix A.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, wwpprswithcooeeeooo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
72

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: no new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
73


Reviews:
[{"review_id": "VpzsIUOyYq", "reviewer": "Reviewer_NsqR", "summary": "This paper proposes the Segment with Uncertainty Model (SUM), a fine-tuning framework for existing foundational segmentation models like SAM. Specifically, SUM consists of two main components: an uncertainty-aware training pipeline and the task prompt concept to reduce ambiguity.\n\nTechnically, the uncertainty-aware training pipeline comprises three distinct techniques. First, the uncertainty maps generation module is fine-tuned from SAM using filtered high-quality images from human-annotated datasets. With uncertainty maps generated from this module, the uncertainty-aware prompt sampling strategy is then proposed to increase the probability of selecting prompts from high-confidence regions. Finally, the SUM is trained in a semi-supervised manner, and the uncertainty-aware focal and dice loss is applied to the unlabeled branch. Additionally, the task prompt concept is introduced to differentiate data from various sources, reducing ambiguity related to different segmentation types of interest across datasets.\n\nThe experimental evaluation in this paper assessed the performance of the proposed strategy on various datasets. The results indicate that the proposed SUM achieved superior performance across multiple benchmarks compared to the other methods.", "questions": "I thank the author for their detailed experiments and analysis. Nevertheless, there are a few questions I wonder about.\n\n- The uncertainty-based strategy is mainly applied to unlabeled data. What would happen if a similar strategy were applied to the labeled data as well?\n- It appears that the task prompt is used to distinguish the type of input images. In Table 2, the SUM Continuous TP setting employs the task prompt during inference as well. I am curious why the performance is better without using the task prompt during inference and how the performance would be affected by using different task prompts.", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper provides a clear explanation and detailed analysis of the proposed method. Also, it is good that even the supplementary material was faithfully written, allowing me to check most of the things I was curious about while reading the main text.\n- The ideas behind uncertainty-based training strategy are considered intuitive and novel.\n- The experimental analysis is comprehensive. In addition to the evaluation of multiple benchmarks across various datasets, the ablation studies and quantitative results analyses presented in the supplementary materials further enhance credibility.\n- The discussion on semi-supervised baselines makes the approach by which SUM utilizes unlabeled data more convincing.", "weaknesses": "There are a few concerns about this paper:\n- Compared to SAM fine-tuning methods, SUM requires an additional training phase to obtain uncertainty maps. This introduces extra training burden.\n- SUM demonstrates outstanding performance on numerous benchmarks. However, as shown in Table 6, the performance improvement is not substantial with the increase in the number of parameters of the backbone. Does this indicate that SUM may face challenges in terms of scaling up the model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback. We believe it will improve the manuscript. We respond in detail below.\n\n### **Q1 Extra training burden** \nThe reviewer is correct that the additional training phase for obtaining uncertainty maps in the SUM framework adds some computational overhead, but it is manageable (relative to the large size of the backbone model). It involves tuning a small number of parameters and can be completed within 4 hours using 8 A100 GPUs. We will mention this in the revised version of the paper.\n\n### **Q2 Scaling up with parameter numbers**\n\nThe reviewer is correct that the performance gains eventually saturate when scaling the backbone, but this already occurs in the original SAM framework, as was noted in the SAM paper [1] (arXiv version, page 12): “ViT-H improves substantially over ViT-B, but has only marginal gains over ViT-L. Further image encoder scaling does not appear fruitful at this time.”  Our results in Table 6 align with this observation, showing similar performance trends. \n\nWe reorganized Table 6 to highlight the boundary IoU performance of the original SAM model, confirming that our results are consistent with the conclusions of the SAM paper:\n\n| Backbone   | Metrics   |   Points |   Salient Object SAM |   Entity SAM |   Part SAM |\n|:-----------|:----------|---------:|---------------------:|-------------:|-----------:|\n| ViT-B      | mBIoU     |        1 |                 57.7 |         64.4 |       48.0 |\n| ViT-L      | mBIoU     |        1 |                 67.4 |         69.5 |       48.8 |\n| ViT-H      | mBIoU     |        1 |                 68.3 |         70.1 |       48.3 |\n\nOur SUM framework consistently improves SAM performance across different tasks and backbones, we display the Gains of SUM with respect to SAM  in the following table:\n\n\n| Backbone   | Metrics   |   Points | Gain on Salient Object| Gain on Entity| Gain on Part|\n|:-----------|:----------|---------:|----------------------:|--------------:|------------:|\n| ViT-B      | mBIoU     |        1 |                   8.9 |           2.3 |         2.3 |\n| ViT-L      | mBIoU     |        1 |                   7.5 |           1.6 |         0.2 |\n| ViT-H      | mBIoU     |        1 |                   7.3 |           3.2 |         2.9 |\n\n\n| Backbone   | Metrics   |   Points | Gain on Salient Object| Gain on Entity| Gain on Part|\n|:-----------|:----------|---------:|----------------------:|--------------:|------------:|\n| ViT-B      | mBIoU     |        6 |                   2.7 |           2.4 |         1.7 |\n| ViT-L      | mBIoU     |        6 |                   3.9 |           2.9 |         1.3 |\n| ViT-H      | mBIoU     |        6 |                   4.4 |           3.8 |         2.6 |\n\nIn summary, while the SAM framework provides limited gains from scaling the backbone, our SUM framework consistently enhances performance across various tasks and backbones.\n\n\n### **Q3 Applying uncertainty-aware training for the labeled data**\n\nWe designed the uncertainty-aware training strategy to mitigate the influence of noisy annotations in pseudo-labels during model training, as our labeled data are of high quality. However, the reviewer's suggestion is very intriguing. In scenarios where the quality of the available annotations is not assured, our uncertainty quantification module could be applied to these labels to enhance training. We will mention this in Section 5. In the rebuttal pdf, we provide a proof of concept that the uncertainty map can be applied to human-labeled data.\nThe uncertainty map corresponding to an image from Cityscapes [2] training set with coarse mask is provided in **Figure 4 of the rebuttal PDF**. This map accurately highlights the boundary regions where the human annotations tend to be less precise. \n\n\n\n### **Q4 continuous TP in SUM** \n\nThis is a good point. SUM only adds the task prompt in the first round (i.e. single prompt) during training and inference. Our results indeed indicate that adding the task prompt in all rounds (i.e. both single prompt and multi-prompt scenario) during the interactive training and inference may be counterproductive. A possible explanation is that the task prompt provides an implicit prior to the model regarding the desired output mask. When only one prompt is provided, this prior is useful and improves performance. However, when multiple prompts are provided, this already sufficiently constrains the desired mask, in a way that may slightly contradict the prior associated with the task prompt for some images. We will mention this in the revised manuscript, and point out that correctly balancing different user prompts is an important topic for future research. \n \n \nIn order to illustrate the effect of providing different task prompts, we report the results of applying SUM (FT-Medium) to the salient object segmentation task (on the same evaluation sets as the main paper) for three different round-1 task prompts. The results show that task prompt 1 enables the model to achieve the best performance in round 1, which makes sense since task prompt 1 is associated with the salient object task. However, for later rounds, the difference in performance is very small.\n\n| Task | Task prompt | Round 1 | Round 3 | Round 5 |\n|---|---|---|---|---|\n| Salient Object | 0 | 77.7 | 90.4 | 93.3 |\n|  | 1 | 85.2 | 91.6 | 93.5 |\n|  | 2 | 81.9 | 91.1 | 93.5 |\n\n\n\n[1] Kirillov et al. Segment Anything. arXiv preprint arXiv:2304.02643 (2023).\n\n[2] Cordts et al. The cityscapes dataset for semantic urban scene understanding. CVPR. 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPFcbfyfXX", "reviewer": "Reviewer_cnn3", "summary": "In this paper, the authors proposed the Segmentation with Uncertainty Model (SUM) which combines high-quality annotated data with a large unlabeled dataset. This novel framework improves the performance of the large-scale foundation model without forgetting.", "questions": "Please see the weaknesses parts and the following questions. \n\n1. Can the authors provide the uncertainty map mathematically?\n\n2. Is the uncertainty map similar to the attention in the transformer?", "limitations": "The authors adequately addressed their work's limitations in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Paper clarity. The paper is overall well-written and structured. The appendix is\n\nGood results. The method improves over the original SAM method and achieves SoTA results. \n\nAdequate appendix. The quantitative and qualitative results in supplementary material support the paper's idea and concept.", "weaknesses": "Although the paper is overall well-written and structured, the figures are confusing. \n\n1. The legend in Figure 1 looks confusing to me.\n\n2. Figure 2 misses the emphasized part to me.\n\n3. Figure 3 missed the uncertainty map.\n\n\nBesides, I would suggest the authors use bold to highlight the context they want to emphasize in the Intro Sec.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. We respond in detail below.\n\n\n### **Q1 Clarification on the figures**\n\nWe appreciate the reviewer’s suggestion to make the figures clearer. We have included updated figures in the rebuttal PDF (see **Figure 1, 2, 3 in the rebuttal PDF**) and will incorporate them into the final version of our paper. \n\nIn Figure 1, we have allocated individual legends to each corresponding sub-figure to clarify the relationship between the figures and their respective legends.\n\nIn Figure 2, we highlight the novel components of our proposed framework, making them easier to identify. \n\nIn Figure 3, we have emphasized the uncertainty map and toned down the colors to avoid distraction.\n\n\n### **Q2 Highlight the context in intro**\nWe appreciate the reviewer's suggestion. We will highlight key concepts in the introduction and method sections in our revised draft.\n\n\n### **Q3 Definition of the uncertainty map** \nWe will add the following mathematical definition of the uncertainty map to Section 3.3. The uncertainty map is obtained from the absolute difference between the original SAM input and the refined output by the Mask-refinement Module. Both the sigmoid-transformed probabilities of the SAM logits and the refined prediction have values in the range $[0,1]$ and share the same spatial dimensions. The uncertainty map is calculated as the absolute difference between them for each pixel. Let $u_i$ represent the uncertainty value for the  $i$-th pixel in the uncertainty map, it is equal to:\n\n\n$u_i =  \\left|  \\sigma(\\mathbf{S}_i)  - \\sigma(\\mathbf{R}_i )\\right| $\n\n\nwhere $ \\sigma $ is the sigmoid function, $\\mathbf{S} $ denotes the SAM logits, and $\\mathbf{R} $ denotes the refined prediction.\nThis yields values between 0 (no difference, indicating low uncertainty) and 1 (large difference, indicating high uncertainty). \n\n\n### **Q4 The similarity of uncertainty map with respect to the attention in the transformer**\n\nThe attention mechanism in transformers serves multiple crucial functions, such as capturing long-range dependencies and assigning importance weights to tokens. Similarly, the uncertainty map in our method assigns importance to regions of the pseudo labels, which is somewhat analogous in spirit to attention. However, there are important differences.\n\nThe uncertainty map is specifically used to guide the training process of the segmentation model, modifying the training cost function and the prompt sampling process. In contrast, attention is a component of the function implemented by transformers,  which enables the model to weigh the relevance of different tokens within the sequence, facilitating better context understanding and representation. It does not modify the cost function (or the prompt sampling process).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrdN3DPsxo", "reviewer": "Reviewer_HNZW", "summary": "The paper introduces a novel framework for enhancing the accuracy of the Segment Anything Model (SAM) while maintaining its generalization capabilities. SAM is a foundational model for interactive binary segmentation, but it struggles with segmenting intricate structures accurately. Fine-tuning SAM with high-quality annotated data often leads to overfitting, degrading its generalization abilities.\n\nThe proposed framework, called Segmentation with Uncertainty Model (SUM), addresses these challenges by combining high-quality annotated data with a large unlabeled dataset. Key innovations include: Uncertainty-aware Fine-tuning: The framework quantifies uncertainty in SAM’s pseudo labels and incorporates this into the fine-tuning process to improve segmentation accuracy without losing generalization capabilities. Task Prompts: SUM uses task prompts to specify the segmentation task for each training example, reducing ambiguity and improving the model’s performance on diverse tasks. Uncertainty-aware Prompt Sampling: This technique avoids misleading prompt locations by focusing on regions with high confidence. Experiments demonstrate that SUM consistently outperforms SAM and other fine-tuning strategies across various datasets, achieving significant improvements in mean Intersection over Union (mIoU) and mean boundary IoU (mBIoU).", "questions": "see the weakness", "limitations": "see the weakness", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Improved Accuracy:\n\nSUM significantly enhances the segmentation accuracy of SAM, particularly for complex structures. The uncertainty-aware fine-tuning process focuses on regions with high confidence, leading to more precise segmentation results.\n\nMaintained Generalization:\n\nUnlike traditional fine-tuning methods that can lead to overfitting, SUM maintains SAM’s generalization abilities. This is achieved by effectively combining high-quality annotated data with a large, diverse set of unlabeled data.\n\nVersatility and Flexibility:\n\nThe use of task prompts allows SUM to handle various segmentation tasks, including salient-object, entity, and part segmentation. This flexibility makes SUM suitable for a wide range of applications.\n\nEfficiency in Handling Uncertainty:\n\nBy incorporating uncertainty maps and an uncertainty-aware loss function, SUM effectively manages the noise and inaccuracies in pseudo labels. This leads to more reliable training and improved overall performance.\n\nRobust Performance Across Datasets:\n\nSUM demonstrates robust performance on multiple public benchmarks and internal datasets, consistently outperforming existing methods in mIoU and mBIoU. This robustness underscores the model’s effectiveness in diverse settings.\n\nInnovative Use of Uncertainty Maps:\n\nThe generation and utilization of uncertainty maps to guide the training process is an innovative approach. It not only enhances segmentation accuracy but also improves the quality of the pseudo labels used in training.", "weaknesses": "Training Complexity:\n\nThe SUM framework introduces significant complexity into the training process. Incorporating uncertainty maps, task prompts, and uncertainty-aware prompt sampling requires additional computation and fine-tuning, which may be challenging and time-consuming for implementation.\nDependency on Initial Model Performance:\n\nThe effectiveness of the SUM framework heavily depends on the initial performance of the SAM model. If SAM’s initial pseudo labels are highly inaccurate, the overall performance of SUM could be compromised, as the refinement process might not fully correct these inaccuracies.\nComputational Overhead:\n\nThe additional steps involved in generating and utilizing uncertainty maps, as well as the iterative prompt sampling, add computational overhead. This could be a barrier for practical deployment in resource-constrained environments where computational resources are limited.\nEvaluation on Specific Datasets:\n\nThe paper primarily evaluates SUM on a limited set of datasets focused on specific segmentation tasks. While results are promising, the generalizability of the framework to other datasets and segmentation tasks remains uncertain. Further validation on a broader range of datasets is necessary to confirm its robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. It will help us improve the manuscript. We respond in detail below.\n\n### **Q1 Training complexity and  Q3 computational overhead**\nWe acknowledge the reviewer's concern about the additional training phase for obtaining uncertainty maps in the SUM framework. While the framework adds steps, they remain manageable within practical constraints, as detailed below. We will mention this in the paper and add a subsection to the appendix with a more detailed explanation.\n\n**Training the Uncertainty Map Generation Module:**\nTraining the uncertainty map generation module is relatively efficient on the human-labeled samples. It involves tuning a small number of parameters and can be completed within 4 hours using 8 A100 GPUs.\n\n**Generating uncertainty map for the unlabeled image**\nThe computational overhead introduced by the SUM framework is minimal. The process of generating pseudo labels and uncertainty maps for unlabeled data utilizes the existing SAM encoder, thereby sharing the primary computational workload. Beyond pseudo-label generation, the additional time required to produce refined masks using a lightweight decoder is negligible compared to the time taken by the image encoder. ViT-H image encoder in SAM has 632M parameters while the prompt-based decoder only has 3.87M parameters. As per the reference [1], on an RTX4090, the decoder adds only approximately 2% more computational time compared to the encoder.\n\n**Fine-Tuning the SAM Model:**\nOnce the uncertainty map is generated, the fine-tuning SAM model using the uncertainty map is similar to the standard training used in SAM. \n\n- *Iterative Point Prompt Sampling*: This method is used in standard SAM training as well, and replacing the original uniform sampling with weighted (uncertainty-aware) sampling results in a negligible training burden. Sampling a point from a 1024x1024 candidate pool using both methods can be completed within 0.006-0.009 seconds (average of 1000 runs) on the CPU of a MacBook.\n\n- *Task Prompt*: The task prompt, a learnable single vector, is combined via element-wise addition with the embeddings from the SAM image encoder and is used only in the first round of interactive segmentation. The element-wise addition of two tensors is relatively fast.\n\n- *Uncertainty-aware Loss Computation*: This involves thresholding and a weighted loss computation, which requires a similar running time to the original loss. \n\n**Inference Phase:** Uncertainty maps are utilized only during training. Once our model is trained, inference operates similarly to the SAM model, without additional computational burden. This results in negligible computational overhead.\n\n### **Q2 Dependence on the initial model performance**\nAs shown in Figure 9, although the quality of refined results depends on the initial model performance, the gains are positive for most examples. We provide **Figure 5 in the rebuttal PDF** to demonstrate that even when the initial SAM input quality is suboptimal, our mask refinement module still enhances the input and improves performance. \n\n\n\n### **Q4 Evaluation set**\nWe organize our evaluation tasks according to the hierarchical level of granularity, covering various levels (part, entity, multiple instances). **Each task** is evaluated using several diverse datasets designed to encompass **a wide range of images and subtasks**. \n\nFor instance, our part segmentation evaluation utilizes five diverse datasets: Fashionpedia, Fashionpedia Subpart, Paco, Multi-Human Parsing, and Portrait. The first two include a comprehensive ontology of fashion elements and different levels of part granularity. Paco covers 75 object classes and the last two focus on different human-specific part segmentation subtasks.\n\nThat said, we acknowledge that evaluating a broader range of segmentation tasks would further strengthen the robustness of our proposed methods. To address this, we have extended our evaluation by testing SUM and SAM on additional image types. For reproducibility, SUM is fine-tuned on the Public dataset FT-Medium.\n\nWe selected 7 datasets from the evaluation sets used in SAM to complement our existing 14 public evaluation sets. Additionally, we have included part one of a synthetic dataset, GTAV [2]. These additional evaluation sets encompass various image types e.g. driving, synthetic, egocentric, irregular shapes, paintings, underwater animals, drones, and underwater trash.\n\nThe mIoU comparison results, reported in the following tables, confirm that SUM consistently outperforms SAM. We appreciate the reviewer’s suggestion and will include these additional results in the Appendix of the final version.\n\n\n| Dataset | Image type | Method | Round 1 | Round 3 | Round 5 | Round 9 |\n|---|---|---|---|---|---|---|\n| Cityscapes | Driving | SAM | 44.1 | 50.9 | 58.6 | 64.6 |\n|  |  | SUM | **46.4** | **57.3** | **62.5** | **67.1** |\n| EgoHOS | Egocentric | SAM | 77.9 | 85.4 | 90.2 | 91.9 |\n|  |  | SUM | **79.0** | **90.0** | **92.3** | **93.5** |\n| DRAM | Paintings | SAM | 68.4 | 81.3 | 87.9 | 91.1 |\n|  |  | SUM | **73.6** | **85.8** | **89.2** | **91.4** |\n| ishape | Irregular shapes | SAM | 41.4 | 60.4 | 75.7 | 84.6 |\n|  |  | SUM | **74.9** | **87.3** | **92.6** | **94.4** |\n| GTAV | Synthetic | SAM | 44.8 | 47.0 | 52.2 | 57.0 |\n|  |  | SUM | **45.7** | **52.8** | **56.7** | **59.7** |\n| NDD20 | Underwater Animal | SAM | 86.2 | 88.3 | 90.2 | 91.3 |\n|  | | SUM | **87.9** | **91.3** | **92.2** | **93.1** |\n| TrashCan | Underwater | SAM | 63.3 | 69.1 | 76.7 | 82.6 |\n|  |  | SUM | **64.9** | **76.8** | **82.0** | **86.1** |\n| IBD | Drones | SAM | **78.8** | 84.5 | 91.2 | 93.6 |\n|  |  | SUM | 78.4 | **87.6** | **91.4** | **93.9** |\n\n[1] Songa et al. SAM-Lightening: A Lightweight Segment Anything Model with Dilated Flash Attention to Achieve 30 times Acceleration.\" arXiv preprint arXiv:2403.09195 2024.\n\n[2] Richter et al. Playing for data: Ground truth from computer games. ECCV 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnPC9rkMzG", "reviewer": "Reviewer_MMBZ", "summary": "This paper introduces the Segmentation with Uncertainty Model (SUM) that combines high-quality annotated data with a large unlabeled dataset to improve performance without forgetting. First, the authors quantify the uncertainty in the SAM pseudo labels associated with the unlabeled data and leverage it to perform uncertainty-aware fine-tuning. Second, the task prompt encodes the type of segmentation task associated with each training example to reduce ambiguity. The proposed method is evaluated on different test sets consisting of 14 public benchmarks.", "questions": "1.The Related Work Section should be divided into subheadings. Summarizing the related work separately is conducive to improving the readability of the paper. In addition, emphasizing the insightful advantages of the proposed method in each section of related work is conducive to a better understanding of the proposed method.\n\n2. The introduction of the uncertainty-aware segmentation method is relatively broad, and a more detailed analysis of the related work should be provided.", "limitations": "As above", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of the paper, which aims to improve the accuracy of SAM without affecting generalization, is interesting.\n2. The paper verifies the effectiveness of the proposed method through a large number of experiments.", "weaknesses": "1.The proposed uncertainty-aware fine-tuning method has limited innovation. The uncertainty-aware fine-tuning strategy has been widely used and has been proven to effectively purify pseudo-labels [1,2]. The proposed method aims to improve the initial prediction of SAM. Although the framework has many complex modules, it does not reveal any insightful views or lacks key theoretical analysis.\n\n2.The method section lacks theoretical introduction. Only the text introduction reads like a technical document. There should be more forms of introduction to help understand the principle of the method, such as formulas or figures.\n\n3.The presentation of the method is unclear and not very readable. For example, in the first sentence of section 3.2, it is introduced that \"SUM applies the same prompt-sampling strategy as SAM for the human-annotated data during interactive training\". In this case, should this module be placed in the Preliminary instead of a separate section in the method section? This makes it confusing to understand the proposed method.\n\n[1]Yuxi Wang, Junran Peng, and ZhaoXiang Zhang. Uncertainty-aware pseudo label refinery for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9092–9101, 2021.\n[2]Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision, 129(4):1106– 1120, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. We respond in detail below. \n\n### **Q1 Difference with previous uncertainty-aware segmentation methods**\n\nOur approach **differs fundamentally from previous approaches in both 1) the generation of uncertainty maps and 2) the utilization of these uncertainty maps**, as detailed below. We will clarify this in the revised manuscript. \n\nThe uncertainty-aware fine-tuning strategies in [1][3] generate uncertainty via model prediction logit values. [2][4] use prediction differences from different heads to generate uncertainty. They are designed for domain-adaptive semantic segmentation or semi-supervised semantic segmentation. In contrast, our proposed strategy is designed for interactive binary segmentation, which has not been tackled by the previous method.\n\n**Uncertainty map generation:**  Our **main insight** is to utilize external supervision to detect and correct systematic biases that accumulate during training. This design logic is fundamentally different from relying on the model self-training itself, as most previous models including [1] and [2]. SAM undergoes several rounds of self-training, leading to the accumulation and overfitting of errors in pseudo-labels. In our evaluation, the model frequently predicts **erroneous regions with high confidence, and different heads concur on these incorrect areas**. Therefore, traditional methods to generate uncertainty maps, including [1][2][3][4], do not capture this uncertainty effectively.\n\n**Utilization of the uncertainty map:** Our method is the first to introduce uncertainty-aware prompt sampling for training interactive segmentation models, representing a novel contribution to the field.  In contrast, the semantic segmentation tasks addressed in [1] and [2] do not involve prompt sampling. Additionally, we have tailored the uncertainty-aware focal and Dice loss to train the binary segmentation foundation model. They are not proposed in [1] and [2]. We will report this in Section 2.\n\n**Quantitative comparison with previous methods:**  As reported in Table 1 of the paper, the proposed SUM method outperforms existing methods based on uncertainty quantification, including [1] and [2], \"SUM Confidence-Uncer\"  corresponds to  [1], and \"SUM Discrepancy-Uncer\" corresponds to  [2].  Table 1 also provides a comparison with UP2L [3], an uncertainty-aware semi-supervised segmentation method that uses uncertainty maps in the feature space via contrastive learning, which also performs worse than  SUM (see also Figure 5). \n\n### **Q2 Key insights for module design**\nWe will modify the introduction and method sections to clearly explain the three main novel components of our framework: \n1. A module for uncertainty map generation, which is trained by external supervision to correct the systematic bias in the foundation model training. The module accurately quantifies the uncertainty in pseudo-labels, generalizing effectively across different tasks.  We will also mention novel design considerations, including data-pair filtering, training mask generation, and model tuning design (now in Appendix Section D) in the main paper.\n2. An uncertainty-aware cost function, which leverages the uncertainty map.\n3. A strategy for uncertainty-aware prompt sampling during training, also leveraging the uncertainty map.\n\nWe will highlight these contributions more clearly in the introduction, comparing them in more detail to the existing uncertainty-aware segmentation methods, as explained below, modifying the paper structure as explained below. We will also provide a more mathematical formulation of the uncertainty map, as suggested by the reviewer, and highlight our contributions more clearly in the figures (see the provided modified figures). \n\n### **Q3 Discussion about other uncertainty-aware methods in related work** \nWe will expand the related work section by providing more detailed explanations of relevant techniques. Previous uncertainty quantification approaches such as [1][2][3][4] are **not designed for training interactive foundation models**. Our experiments validate that they are not effective in generating effective uncertainty maps in our setting. Additionally, the way these methods utilize uncertainty maps differs from our setting.\n\n[1][2][4] are designed for domain-adaptive semantic segmentation, while [3] is for semi-supervised semantic segmentation. The concept of uncertain classes explored in [1] is not applicable to binary segmentation scenarios. Our strategy, however, is tailored for interactive binary segmentation. Please also see Q1.\n\n\n\nWe have already conducted comprehensive experiments and provided **a detailed discussion of related work in Appendix E.1 and E.2**, respectively. We will further expand the related work section to include these additional insights.\n\n### **Q4 Paper organization.** \n- **Related Work Section:** We will reorganize the Related Work section into subheadings, summarizing related work in distinct categories to improve readability and highlight the novel aspects of our method.\n- **Method Section - Theoretical Introduction:** We will add formulas and figures to the Method section to clarify the principles behind our approach. \n- **Method Section - Presentation:** We will move the description of the prompt-sampling strategy to the Preliminary section and add figures to enhance clarity and readability.\n\n\n\n\n[1]Wang et al. Uncertainty-aware pseudo label refinery for domain adaptive semantic segmentation. CVPR 2021. \n\n[2]Zheng et al. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV 2021.\n\n[3] Wang et al. Semi-supervised semantic segmentation using unreliable pseudo-labels. CVPR 2022.\n\n[4] Wu et al. Upl-sfda: Uncertainty-aware pseudo label guided source-free domain adaptation for medical image segmentation. IEEE transactions on medical imaging 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Response for all the reviewers:
We thank the reviewers for their thoughtful comments and are encouraged by their positive feedback. We appreciate the recognition of our paper's soundness, contributions, and presentation. 

**Positive Feedback:**
- **Soundness:** Excellent (Reviewer NsqR), Good (Reviewers cnn3, HNZW, MMBZ)
- **Contribution:** Excellent (Reviewers NsqR, cnn3), Good (Reviewer HNZW)
- **Presentation:** Excellent (Reviewer NsqR), Good (Reviewer HNZW)

**Novelty and Methodology:**
- We are pleased that the reviewers found our uncertainty-based training strategy intuitive and novel (Reviewer NsqR), and appreciated the innovative use of uncertainty maps (Reviewer HNZW), efficiency in handling uncertainty (Reviewer HNZW), and the versatility and flexibility of task prompts (Reviewer HNZW).
- The use of unlabeled data was found convincing (Reviewer NsqR), and the overall framework was seen as novel (Reviewers cnn3, HNZW) and well-motivated (Reviewer MMBZ).

**Experimental Validation:**
- Reviewers noted the comprehensiveness of our experiments (Reviewer NsqR), the outstanding performance validated on numerous benchmarks (Reviewer NsqR), credible ablation studies (Reviewer NsqR), and significant improvements and robust performance across datasets (Reviewer HNZW).
- The effectiveness in a large number of experiments (Reviewer MMBZ) and achieving state-of-the-art results (Reviewer cnn3) were also highlighted.

**Writing and Presentation:**
- We appreciate the positive feedback on the clarity and detailed analysis of our paper with faithfully written appendix (Reviewer NsqR), as well as its well-written and structured nature with adequate appendices (Reviewer cnn3).


We address the comments of the reviewers in detail in our responses below. Specifically, we elaborate on the additional computational cost of the proposed approach, provide improved figures, and report results on 8 additional evaluation sets to further validate the robustness and generalization of the approach. The additional experiments and clarifications will be added to the revised version of the paper.

We provide all figures mentioned in the rebuttal in the rebuttal PDF.

Meta-review: Most of the reviewers land on the positive side. Only one reviewer (Reviewer MMBZ) gave “Borderline reject”, but the reviewer claimed to increase the score in the final rating and probably forgot to do so. Therefore, the AC suggests acceptance.  The authors should include the discussion and new results in the rebuttal into the final version. Especially, they should release the code as they promised in the rebuttal.
Decision: Accept (poster)