Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM
Authors: Chenxin Li
Yuzhihuang
Wuyang Li
Hengyu Liu
Xinyu Liu
Qing Xu
Zhen Chen
Yue Huang
Yixuan Yuan
Venue: NeurIPS 2024 poster (2024)
Keywords: Foundation Model
Ambiguous Segmentation
Uncertainty
Primary area: machine_vision

Abstract: As the vision foundation models like the Segment Anything Model (SAM) demonstrate potent universality, they also present challenges in giving ambiguous and uncertain predictions. Significant variations in the model output and granularity can occur with simply subtle changes in the prompt, contradicting the consensus requirement for the robustness of a model. While some established works have been dedicated to stabilizing and fortifying the prediction of SAM, this paper takes a unique path to explore how this flaw can be inverted into an advantage when modeling inherently ambiguous data distributions. We introduce an optimization framework based on a conditional variational autoencoder, which jointly models the prompt and the granularity of the object with a latent probability distribution. This approach enables the model to adaptively perceive and represent the real ambiguous label distribution, taming SAM to produce a series of diverse, convincing, and reasonable segmentation outputs controllably. Extensive experiments on several practical deployment scenarios involving ambiguity demonstrates the exceptional performance of our framework. Project page: \url{https://a-sa-m.github.io/}.

Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM
Chenxin Li Yuzhi Huang?; Wuyang Li', Hengyu Liu',
Xinyu Liu', Qing Xu², Zhen Chen Yue Huang", Yixuan Yuan'
The Chinese University of Hong Kong Xiamen University University of Nottingham Ningbo China Yale University
Abstract
As the vision foundation models like the Segment Anything Model (SAM) demon- strate potent universality, they also present challenges in giving ambiguous and uncertain predictions. Significant variations in the model output and granularity can occur with simply subtle changes in the prompt, contradicting the consensus requirement for the robustness of model. While some established works have been dedicated to stabilizing and fortifying the prediction of SAM, this paper takes a unique path to explore how this flaw can be inverted into an advantage when modeling inherently ambiguous data distributions. We introduce an optimization framework based on a conditional variational autoencoder, which jointly models the prompt and the granularity of the object with latent probability distribution. This approach enables the model to adaptively perceive and represent the real ambiguous label distribution, taming SAM to produce series of diverse, convincing, and reasonable segmentation outputs controllably. Extensive experiments on several practical deployment scenarios involving ambiguity demonstrates the exceptional performance our framework. Project page: https //aa sa- m. github. io/
Introduction
The advent of Visual Foundational Models (VFMs) such as the Segment Anything Model (SAM) 22 has been unprecedented, largely due to the availability vast datasets and computational resources. These models have exhibited remarkable generalization capabilities in zero-shot scenarios and the capacity to interact with human feedback. SAM, in particular, employs specialized data engine to manage 11 million image masks, using a unique prompt- based segmentation framework to generate accurate masks for any object within a visual context, largely extending the capacity and generality of segmentation [27]. Such successes have been widely extended to various domains, such as medical
imaging analysis 35 28) remote sensing etc. etc.
However, it has been observed that SAM suffers from severe predictive ambiguity to segment desired concepts 2 in practical scenarios. To uncover the factual basis, we delve into this ambiguity and detail it into two flaws according to experimental insight. Specifically, the first flaw lies in that SAM prediction sensitive slightly different prompt variants. As shown in Fig. (a), we ground SAM into realistic clinical scenario to segment the lesion in CT images. Even though three medical experts uniformly give rational box prompts covering the lesion, SAM, unfortunately, makes enormous differences among them, even including one wrong case. Further, we provide detailed statistics regarding IoU over the small perturbation (only pixels) of box prompts, as shown in Fig.
"Equal Contribution: (chenxinli @ link.cukk.cca. cuhk.cdu.hk, yzhuang 13 @ ).tu..mu.cducc). 1 Corresponding Author: [yxyuan www.e.ccccccckk.......
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
(c)
Prompt
aecorpeee Seg Okrip
0.6
Variant
0.
Canonical Prompt
0.5
Full Granularity
Segment Anything Model [SAM]
0.3
0.4
Irput Image
0.2
0.3
Gramularity Voriont
0.1
0.2
0.0
15px
Giy Variation of Object Granuaritty
Variation Pronge
Figure I: Analysis of Inherent Ambiguity in SAM. (a): Feeding SAM with slightly different prompts from multiple experts for a single image can significantly alter the segmentation output. (b)(c): We evaluate SAM using canonical box prompts and various perturbed versions, measuring the mean and variance of segmentation IoU on LIDC. Perturbations involve shifting the box five pixels in different directions and employing various granular outputs within SAM. Results highlight SAM's sensitivity to prompt variations and granularity.
(b). We can observe a significant IoU fluctuation small prompt differences, which reveals that SAM prediction is highly sensitive to the prompt variants caused by such small perturbation
The second flaw lies in the susceptibility of SAM output to the inherent structural granularity of the object. Despite the fact that VFMs like SAM gain generalizable knowledge and abilities from extensive datasets, they frequently forfeit the capacity to segment specific visual concepts as they are class agnostic, making the model unable to discern the difference between objects with different levels of semantic granularity. Consequently, for targets that challenging to define, particularly those with rich internal hicrarchical granularity, they are inclined to produce multiple candidate results at different granularities and amalgamate them, rather than directly outputting definitive result. As depicted in Fig. [ (c), we also discern that the multiple candidate results captured by SAM often exhibit significant differences, and the segmentation precision of outputs at different granularities diverges greatly when compared to the final integrated SAM output that incorporates multiple candidates.
Nonetheless, every coin has two sides. While these observations reveal SAM's faults regarding the output sensitivity, we explore different perspective: could this sensitivity flaw become advantage in other cases, such as ambiguous segmentation, which requires the model to learn from set of ambiguous object annotations caused by imaging noise, ambiguous tissue boundaries, and different annotator preferences? Specifically, we are particularly interested in the following: O Given the segmentation result's sensitivity to prompts, variation to tame prompt-sensitive SAM to controllably yield multiple likely results close to the actual fuzzy distribution? @ Considering the segmentation result's its sensitivity to object structure, probabilisticall model this granularity variation to tame granularity-sent -sessiivee SAM to controllably yield multiple likely results close to the actual ambiguous distribution?
probabilistically model this prompt
Driven by these questions, we propose an innovative strategy, which flips the inherent category- agnostic ambiguity induced in SAM into a controlled ability to generate a range of feasible results for ambiguous segmentation tasks. Specifically, to simulate segmentation ambiguity under different prompts, we introduce context-aware prompt ambiguity modelling. This method probabilistically models the uncertainty the prompts inputted into SAM using a latent learnable distribution, which adaptively perceives the specific ambiguity of different contexts. Furthermore, to simulate ambiguity caused by complex object structures at different granularities, we introduce granularity object ambiguity modelling. This method introduces and enhances the visual ambiguity of objects at different granularities into SAM's original image embedding via a learnable embedding distribution. While establishing these two levels of ambiguity modelling, we introduce an efficient optimization strategy based on posterior constraints, allowing the model to mimic models that can perceive the actual ambiguous distribution. Our contributions can be summarized as follows:
In practice, such perturbation commonly exists and cannot be controlled by users, for medical experts.

---

We explore the inherent ambiguity in SAM to flip this commonly seen disadvantage in deter- ministic segmentation tasks into an advantage for more practical ambiguous segmentation tasks that allow multiple possible outputs.
We propose A-SAM framework that employs a learnable latent distribution to encapsulate the ambiguity at two strata, from prompts and object granularity.
We introduce optimization architecture based on variational autoencoders, which effec- tively represents ambiguity by constraining the sample embedding to align with those from series of practically feasible annotations.
Rigurous benchmarked experiments across a wide range of potential scenarios demonstrate that our method produces more accurate, diverse, and reasonable segmentation outputs.
2 Related Work
Prompting Foundational Models for Segmentation. There has been a surge in the advancement of laage-scale vision models for image segmentation, drawing inspiration from language foundation Segmentation Foundation Models (SFMs), such as the Segment Anything 22 and SEEM 166), have delivered significant segmentation results across various
models 662 4 25 39].
Model (SAM)
38 54]. SAM, leveraging a data engine incorporating model-in-the-lovw annotation, learns a promotable segmentation framework that generalizes to downstream scenarios in a zero-shot manner. Meanwhile, other models like Painter (56) and SegGPT [57] introduce robust in-context learning paradigm that enables segmenting any images given image-mask prompt. On the contrary. SEEM [66] presents a general segmentation model prompted by multi-modal references, such language and audio, thereby incorporating wide range of semantic knowledge. These advancements in SFMs, driven by promptable segmentation design, involve two types of prompts: semantic prompts (e.g., free-form texts) and spatial prompts (e.g., points bounding boxes)
downstream datasets 17
12 57
40
43 26
Recently, the practice of adapting vision foundation models such SAM (2 for application in medical image segmentation is garnering increasing interest [46] 12 44 33), A prevalent and cost- effective approach involves adapter techniques, which necessitate the inclusion of bottleneck modules with a finite number of parameters within the model. By fine-tuning these diminutive adapters, SAM can bridge the domain discrepancy between medical and natural images while preserving stellar performance. For instance, models like MSA (58), SAM-Med2D (9] and SAM-adapter [6] utilize adapter strategies to transfigure SAM for medical imaging, thereby achieving significant segmentation outcomes. Despite these advancements, acquiring suitable prompts for SFMs remains largely under-explored. This work aims to explore generating effective prompts for SAM, focusing on harnessing pre-training knowledge to complete ambiguous image segmentation.
Ambiguous Image Segmentation. Ambiguous image segmentation aims to model a range of, A wealth of existing research has proposed various techniques to quantify uncertainty. Initial research focused enhancing traditional U- Net(ST 16, [29] (5] with a probabilistic component to generate multiple predictions for the same image. This was typically achieved by incorporating conditional variational autoencoder (c (AVA)) (53), with the fow-dimensional latent space encoding potential segmentation variations. Subsequent work further extended this setup a hierarchical variant 3
rather than single, segmentation labels [3
65 45
24
64
15].
Other research has utilized normalizing to represent a discrete latent space 44 or
flows to allow for distribution in the cVAE 52
55
incorporated variational dropout and directly used inter-grader variability as a training target. Several other methods [34] do not rely on the Probabilistic U-Net [47 21 32 [1 [60] . Monteiro et al. [47] proposed network utilizing low-rank multivariate normal distribution model the logit distribution. Kassapis et al. (21] leveraged adversarial training to learn potential label maps based on the logits of a trained segmentation network. Zhang et al. (63 employed an autoregressive PixelCNN to model the conditional distribution between pixels 36). Finally, Gao et al. [13] used mixture of stochastic experts, where each expert network estimates a mode of uncertainty, and a gating network predicts the probabilities of an input image being segmented by one of the experts. Unlike previous efforts 42 (5 7 our approach signifies the first exploration of leveraging the inherent properties in vision foundation models for ambiguous image segmentation.
3

---

(a) Context-aware Prompt Ambiguity Modeling (Top)
Clone
PP
Prompt Encoder
PGN
o
Posterior PGN
Prompt Embed.
Prompt
Input Prompts
Sampling
Embed.
Mask Decoder
CE
M
Ground-truth
Image
Sampling
Image
Input Image
Embed.
Embed
Image Encoder
IGN
Posterior
IGN
Concat;
'AM
Clone
(b) Granularity Object Ambiguity Modeling (Bottom)
Figure 2: A-SAM Training Pipeline. We probabilistically model the prompt and object-level ambiguity by jointly probabilities the SAM embeddings with PGN and IGN, respectively.
3 Method
3.1 Revisiting SAM by Probabilistic Perspective
SAM: Segment Anything Model. Given an image I and set of user-given prompts P, which could be a point, a box, or a rough mask, the Segment Anything Model (SAM) [22] employs a vision transformer-bas image encoder Enc, to extract salient image feature F, and deploy a prompt encoder Encp with length & to encode prompt embeddings Tp, which are denoted as follows,
Fj = (Tc Tp = Encp(P),
where F7 E Rxxxxx and Tp E Rxc, where the resolution of the image feature map is represented by h, w, and the feature dimension is denoted by c. Subsequently, the encoded image and prompts are introduced into the decoder Decи for interaction based on attention mechanisms. SAM constructs the decoder's input tokens by concatenating several learnable mask tokens Tas as prefixes to the prompt tokens Tp, These mask tokens accountable for generating the mask output M
M DocM Tp, TM),
A-SAM: Lifting SAM to Distributional Space. Unlike the one-to-one deterministic mapping in SAM, we formulate probabilistic latent distribution enable the one-to-many ambiguous mapping named A-SAM, with each observation being a sample from this hidden distribution. To this end, the prompt and image embedding can be probabilistically formulated distribution:
~ Pr(O), F P)())
where Pp and Pr denote a latent distribution for prompt embedding and image embedding, respec- tively. Tp and Fj denotes a prompt at one sampling from the defined latent distribution at one time. construct distributional mapping
Formally, by implementing multiple rounds of sampling, of segmentation outputs with respect to their prompts, formulated as the format of expectation,
M DCCM [F,, Tp, Tar). st. Tp Pp P))
where M denotes the SAM output corresponding to one prompt sampling, which can also be interpreted as the sampling from a virtual distribution Par(1) for the segmentation results obeying parameters Ω. As result,
construct optimized probability distribution Тpp Pr(O) and F P) by narrowing the gap between M Par (S2) and ground-truth distribution.
A-SAM: Inference Stage. After the training, A-SAM can model two types latent distribution, representing the ambiguity of the prompt variation and the varied object granularity, respectively.

---

Based on this formulation, each latent sample drawn from the distribution represents a segmentation candidate. Concretely, predict set of m segmentations, we apply the network times to the same input image. In each iteration i E (1..... m,,, we draw a random sample regarding prompt embedding T R and image embedding Fr IRN, respectively from Pp(0) and Pr(Φ). The final prediction maps M (12) can be obtained by Eq. 4 In what follows, will primarily focus on the methodology of training our overall ambiguous segmentation framework.
3.2 Context-aware Prompt Ambiguity Modeling
Distributional Prompt Representation. To model the distribution of prompt embedding, it is imperative to estimate the parameters θ of this distribution. We adopt an axisymmetric Gaussian distribution to characterize the prompt embedding, which is dictated by two erucial parameters, including mean μ and standard deviation ø. Then, we can sample prompt embedding from the given Gaussian distribution, which is shown
Tp ~ Pr(0) = (ur, diag(ar)),
(5)
where Hi and dp denotes the parameters characterized for prompt P. μP and o respectively denote the mean and standard deviation of the Axis Gaussian Distribution generated by the network. where μP, op E RN This simple yet effective formulation enables the discrete prompt to be continuously represented in the probabilistic latent space, making the uncertainty estimation available.
Context-aware Prompt Embedding Generation. To parameterize the latent prompt distribution, we propose a prompt generation network (PGN) to effectively model the aforementioned Gaussian related to prompt embedding. This network is simply designed to include several convolution blocks. Considering the variation in salient regions within the image context, the required prompt positions and sizes should also be varied. Therefore, we incorporate image context as prior knowledge into PGN during the forward inference process. By integrating this prior knowledge, the network can customize a unique prompt-associate associated axial Gaussian distribution for each image I, thereby achieving adaptive and infinite sampling in the latent prompt distribution:
[up.op] apl FPGN Fr;
(6)
where the parameters of the prompt generation network FPON are modeled by the parameters O of our desired distribution, as each set generated mean and variance uniquely specifies a distribution. Previous research indicates that allowing the model to conditionally perceive the ground truth label distribution during the training phase enhances training stability for such tasks that exhibit significant uncertainty. Thus, a posterior version for prompt generation network Expoet parameterized by @T, is further introduced during training, learning to generate the effective distribution for prompt embedding when accessing the ground-truth label distribution:
[nF, of = PPeN (Tp, F1, GT
We only employ this posterior network during training and guide the standard network, which cannot perceive the true labels during testing, to achieve viable performance via a KL loss.
LPKL = DKL (N(Hi, diag((F)) ( diag(op)))
(8)
3.3 Granularity- Object Ambiguity Modeling
Distributional Object Representation. The model integrates object ambiguity from a probabilistic perspective, enhancing its ability to solve problems in innovative ways. We instantiate various segmentation labels to represent ambiguity levels and incorporate them as priors in visual feature extraction, influencing object-related feature modeling for the input image I,
Fj = Con
Fy1 ( = N(ur, diag(ar)),
where μ4r and σy respectively denote the mean and standard deviation the Axis Gaussian Distribu- tion generated by the network, where (4r a, E Nr This approach enhances our understanding and depiction of object diversity and uncertainty in a more profound and lucid manner, enabling better adaptability and handling of variation and uncertainty.
5

---

Granularity-awan Image Embedding Generation. SAM generates multiple candidate segmen- tation masks for the same object at different granularities and levels, demonstrating the inherent ambiguity of SAM related to object granularity. This inspires us to leverage this aspect to enhance the model's perception of ambiguous objects. We subsequently introduce an image generation network (IGN) for object embedding to model this distribution:
(10) where the parameters of the image generation network FIGN are modeled by the parameters of desired distribution, as each set of generated mean and variance uniquely specifies a distribution. Similar to the previous introduction of a posterior network to enhance learning in modeling prompt ambiguity, we introduce a posterior version of IGN, denoted as IGN. for perceptible labels.
[ur,or] = @Fr;
= Faa iaN((FF GT;
(11)
We only employ this posterior network during training and guide the standard network, which cannot perceive the true lahels during testing, achieve viable performance via KI. loss.
LIKL DRL (NiNT diag(o7)) ((rz,
(12)
3.4 Overall Optimization
In line with current SAM techniques that generate segmentation masks for the same object at different granularity levels, the present approach utilizes this feature to enhance the model's capture of multi-level object concepts through a common ensemble strategy. Specifically, given the multiple candidate outputs from SAM, represented as {MT M/2,,,,, Mr), where n is the number of scales. By introducing of learnalle mask weights W [W1, W2....; W,, } E R the final mask output can be fine-tuned and obtained through a weighted sum calculation:
M = "" M=
④ M'
(13)
where w,, W2, W. are initialized to 1 and subsequently fine-tuned to enable the model effectively being aware of the object scales. By adaptively integrating multiple scale masks, the model's perception and modeling capabilities for complex target diversity are further enhanced.
When the representation from SAM is combined with the ground -ruth segmentation GT from the training samples, guide-providing teacher prediction segmentation M is created. A cross- entropy loss CE(.,-) is employed to penalize the discrepancies between the distribution of MT and GT. i.e. Par (S2) and Per, where the distribution of GT is a constant value that does not need to be parameterised, as: Lisey = (GT, M). Additionally, the KT. losses introduced to regularize training in prompt ambiguity and image ambiguity, respectively, in Eq 8 and Eq. 12 are amalgamated into a weighted sum with weight coefficient of ap and ay:
LAU Lee + ap- LPKL + o' LIKL.
(14)
The model is trained from scratch using randomly initialized weights. Parameters requiring update include the prompt encoder and image encoder within SAM as well as PGN, posterior PGN, IGN, and posterior IGN. The KL loss during training aligns the distribution of perceptually true segmentation labels (encoded segmentation variants) with the distribution that is imperceptible at inference time. Adhering to this training objective, the eventual distribution is adjusted encompass all segmentation variants for specific input image.
4 Experiment
4.1
Experimental Setup
Four datasets are utilized for comparison. The LIDC-IDRI dataset (2) is used for lung lesion segmentation, consisting of lung computed tomography scans from 1010 subjects with annotations
Dataset.
from four domain experts. This dataset accurately captures the typical ambiguity found in CT imaging. The BS 2017 dataset [18] is used for 3D brain tumor segmentation, comprising 285 cases of 3D MRI images. Each image includes 155 slices and four modes (TI, Tlce, T2, and Flair). These slices

---

SAM w/ Box Shift
SEEM w/ Mask Shift
Ground
Figure Qualitative comparison with prompted segmentation models adapted for ambiguous segmentation. Examples include three tround-truth expert labels and sampled segmentation masks.
Table 1: Comparison with prompted segmentation models adapted for ambiguous segmentation.
Metric
GED | HM-IOUT D Dmox
D mean
GED 1
HM-IOUT Dmar
Deenn f
Method
LIDC
BRATS
SegGPT w/ Point shift
0.462
0,280
0.573
0.153
0,451
0.032
0.144
0.046
SegGPT w/ Box shift
0.392
0.354
0.638
0.325
0.348
0.082
0.224
0.146
SEEM w/ Mask shift
0.381
0,401
0.692
0.272
0.210
0.228
0.281
0.194
SAM w/ Point shift
0.377
0.365
0.650
0.337
0.252
0.169
0.334
0.238
SAM w/ Box shift
0.361
0.380
0.673
0,253
0.239
0.242
0.344
0.246
A-SAM (Ours)
0.228
0.717
0.948
0.356
0.193
0.610
0.864
0.423
Method
ISBI
Sim10K
SPgGPT w/ Point shift
0.649
0.662
0.659
0.323
0.259
0.128
0.151
0.127
SegGPT w/ Box
0.527
0.772
0.874
0.536
0,272
0.152
0.220
0.183
SEEM w/ Mask shift
0.522
0.821
0.908
0.760
0,238
0.271
0.344
0.246
SAM w/ Point shift
0.513
0.782
0.886
0.681
0,265
0.155
0.229
0.189
SAM w/ Box shift
0.491
0.792
0.896
0.685
0,255
0.160
0.239
0.199
A-SAM (Ours)
0.276
0.835
0.926
0.904
0,233
0.637
0.851
0.327
annotated into four classes: Background, Non-enhancinn/NNerroiii Tumor Core, Edema, and Enhancing Tumor Core. The ISBI 2016 dataset [14] contains 900 dermoscopic images for training and 379 images for testing, all annotated by expert with the lesion area. The images resized and padded to maintain a uniform scale. The SIM 10k dataset [19] consists of 10,000 images rendered by the gaming engine Grand Theft Auto, providing bounding boxes of 58,701 cars training images.
Implementation Details. For the LIDC dataset, we use the included four expert annotations to represent different ambiguous segmentation labels. In the case of the BraTS dataset, amalgamate annotations from different categories into a binary mask, creating multiple segmentation masks to mimic real-world ambiguous segmentation scenarios. For the ISBI dataset, provided. All three datasets are optimized using the Adam optimizer, with a learning rate of le-4, 100 epochs. For the SIM 10k dataset, select images where pixels from two instances overlap, creating three potential masks. Optimization for this dataset carried out with Adam optimizer over 500 epochs, with a learning rate of lc-4. The trade-off coefficients are as ap aj
the single label
Evaluation Metrics. Four metrics are used for evaluation: Generalized Energy Distance (GED), Haungarian-Matched Intersection over Union (HM-loU), Maximum Dice Matching (Dmaz), and Average Dice Matching (Dmean)) GED is metric used ambiguous image segmentation tasks that compares the distribution of segmentations. It leverages the distance between observations, where lower energy signifies better match between prediction and the ground truth. HM-IoU calculates the optimal match of Intersection over Union (IoU) between annotations and predictions using Hungarian algorithm, providing an accurate representation of sample fidelity. Dmaa and Dmean represent the best and average Dice scores between each prediction result and each ground truth, respectively.
4.2 Comparison Prompted Segmentation Models
Tab. I presents the quantitative results four datasets, offering comparison with the current state-of of-te-rft prompting-ban based segmentation models adapted for ambiguous segmentation tasks. Specifically, we have adapted SegGPT [57], SAM-like prompt-based segmentation approach that

---

PixelSeg
Mose
Ours
Ground Truth
Figure Qualitative comparison with efforts specially designed for ambiguous segmentation. Examples include four ground-truth expert labels and sampled segmentation masks.
Table 3: Quantitative comparison on ambiguous
segmentation on the BraT'S dataset.
Table 2: Quantitative comparison on ambiguous segmentation the LIDD dataset.
Method
GED
HM-IOU
Dmaz²
Dmcon f
Method
GED
HM-IOU
Daner
Prob UNet
0.225
0.521
0.645
0.364
PixelSeg
0.419
0.528
0.785
0.361
Prob UNet
0.324
0,423
A-SAM (Ours)
0.192
0.603
0.886
0.438
HProb UNet
0.270
0,530
PHiseg
0.262
0,595
Table 4: Quantitative comparison on ambiguous segmentation on the ISBI dataset.
SSN
0.259
0,555
CAR
0.252
0,549
0.732
PixelSeg
0.243
0,614
0.814
Method
GED
HM-IOU1
D Dmaz
Dmnon 1
CIMD
0.234
0.587
UNet
0.815
0.902
0.902
Mose
0.234
0.623
0.702
Prob UNet
0.329
0.824
0.914
0.894
A-SAM (Ours)
0.230
0.763
0.959
Plliseg
0.289
0.788
0.912
0.871
cFlow
0.306
0.822
0.918
0.892
A-SAM (Ours)
0.267
0.834
0.918
0.905
supports multimodal prompt input we use the smallest box containing all segmentation labels in each image's mask as the standard prompt. The standard prompt then randomly perturbed multiple times by scaling [0.8,1.2] and shifting up, down, left, or right by [-8.8] pixels to obtain different ambiguous segmentation results. We have also adapted SEEM [66], segmentation model that follows the in-context learning paradigm. Given a reference image and reference segmentation mask, it segments the object in the query image. We select an image with multiple mask labels and apply random perturbations on reference segmentation mask by shifting [-8.8] pixels left. right, up. down, resulting in multiple different results on the query image. Comparing with SAM, we employ the same paradigm for prompt acquisition and perturbation SegGPT. We find that A-SAM outperforms the state-of-the-a segmentation foundational models based on point or box in terms of diversity and accuracy. In addition, A-SAM surpasses SEEM, segmentation paradigm directly based masks, in all aspects. This indicates that our designed strategy accurately captures the ambiguous attributes present in different images and objects, effectively achieving balance between diversity and accuracy in ambiguous segmentation tasks. Fig. 3 further illustrates the qualitative results of our method in comparison with existing techniques. Our proposed A-SAM yields segmentations that preserve greater degree of accurate object detail, particularly boundary specifics, and offers an exceptional visual representation of potential diversity, as compared other technologies.
simulate the actual scenario of prompt-based segmentation,
4.3
Comparison to Conventional Ambiguous Segmentation Models
The numerical outcomes across the four datasets are delineated in Tab. 2 and 5) where we draw comparisons with con- temporary leading-edge classical ambigu- ous segmentation methodologies. Precisely, comparative method results on the LIDC and ISBI datasets are direct quotations from their respective papers, while outcomes on other datasets are predicated our reimple-
Table Quantitative comparison ambiguous seg- mentation on the SIM 10k dataset.
Method
GED J HM-IOU
Dmar t
D Dmean
Prob UNet
0.292
0.391
0.462
0.229
PixelSeg
0.398
0.525
0.644
0.358
A-SAM (Ours)
0.241
0.596
0.833
0.421

---

mentation of their official code. The methods compared encompass recent ambiguous segmentation methodologies that amalgamate a conditional variational autoencoder and UNet: Probabilistic U- Net (33 Hierarchical Probabilistic U-Net (HProb UNet) 24] PhiSeg [3. Stochastic Segmentation
seeeeeeeeeeeeee segmentation endeavor of Calibrated Ad- versarial Refinement (CAR) (20) and Collective Intelligence Medical Diffusion (CIMD) (50) that integrate generative models. It also includes ensemble-ber
Networks (SSN) 47 PixelSeg (63, and the ambiguous
techniques utilizing a Mixture-of- expert like the Mix of Stochastic Experts (Mose) [13]. We observe that the A-SAM transcends state-of-the-art methodologies that amalgamate a conditional variational autoencoder and UNet in terms of diversity and precision. Additionally, the A-SAM outstrips segmentation paradigms based on generative models or ensembles in all dimensions. This suggests that our strategy accurately apprehends the ambiguous characteristics inherent in varied images and objects, effectively attaining equilibrium between diversity and precision in the ambiguous segmentation task. Fig. further elucidates the qualitative outcomes of our methodology juxtaposed with extant technologies. In contrast to other technologies, the segmentations engendered by our proposed A-SAM preserve a higher degree of exact object detail, particularly boundary details, and provide distinctive visual representation of potential diversity.
4.4 Further Empirical Results
Table 6: Ablation study on the proposed key components.
Ablation Study. Tab. 6 delineates the consequences of eliminating vari- ous principal strategies of the A-SAM. No Ambiguity Modeling precludes all ambiguous modeling, including both prompt embedding and image embed- ding levels. At this stage, we employ the smallest box encompassing all seg- mentation labels within each image mask as the standard prompt. This standard prompt is then randomly perturbed numerous times by scaling [0.8, 1.2] and shifting up, down, left, or right by [-8, 8) pixels, yielding different ambiguous segmentation outcomes. No Object Ambiguity and No Prompr Ambiguity either eradicates the ambiguity related to the object the prompt, that is, it alters its object embedding or prompt embedding to a regular deterministic rather than an ambiguous characteristic. No Posterior Distillation eliminates a process of network training guided by teacher network that can perceive the actual labels. We discover that when any component is excised, the performance correspondingly deteriorates, which underscores the effectiveness of our proposed several strategies.
No Key Components
GED J
HM-IOU1
Douzz" f
No Ambiguity Modeling
0.361
0,380
0.673
0.253
No Object Ambiguity
0.370
0.389
0.691
0.230
No Prompt Amhiguity
0.308
0.674
0.930
0.336
No Posterior Distillation
0.266
0.385
0.805
0.341
A-SAM (Ours)
0.228
0.717
0.948
0.356
Robustness Analysis. Fig. reeports the loU perfor- mance of A-SAM under variety of instantaneous perturbations. The blue and red solid lines respec- tively illustrate the performance changes of the SAM model and our method when prompted by different disturbances, while the dashed lines depiet the perfor- mance of both models under standard prompts, serv- ing as an upper bound. We selected both light and severe degrees of perturbation. Specifically, 'Shift' indicates random offset of the box by [0.5] pixels. 'Scale' represents a random scaling of the box by 15. 'Shift+" denotes random offset of the box by [0,8] pixels, and 'Scale+' implies a random scaling of the box by [0.7.1.3]. Compared to the vanilla SAM baseline, A-SAM demonstrates robust- ness against various instantaneous perturbations.
0.85
0.75
0.70
0.65
Blosed promots for /Ors
0.60
Shift
Scate
ShiftSScale
Shiftt
Scalet
Shifts
Figure 5: Robustness analysis of A-
SAM framework over the SAM baseline against prompt perturbation.
5 Conclusion
The continued evolution of vision foundation models like the Segment Anything Model (SAM) demonstrates impactful universality, while also posing challenges in producing ambiguous and uncer- tain predictions. Minor changes in the prompt can cause significant variations in the model's output, challenging its required robustness. While many works aim to stabilize SAM's prediction capabilities,

---

this paper uniquely explores leveraging this perceived flaw to advantageously model inherently ambiguous data distributions. We introduce an innovative optimization framework grounded in conditional variational autoencoder, which cohesively models the prompt and the object granularity with latent probability distribution. This approach endows the model with the capacity adaptively perceive and represent the genuine ambiguous label distribution, thereby enabling SAM to generate controlled series of diverse, persuasive, and reasonable segmentation outputs. Our comprehensive experiments across multiple practical deployment scenarios involving ambiguity underscore the exceptional performance our framework, thereby illuminating the need for increased focus on addressing related challenges and opportunities.
Acknowledgement. This work was supported by the Hong Kong Research Grants Council (RGC) General Research Fund under Grant 1 11211221, 14204321. This work was also supported in part by the National Natural Science Foundation of China under Grant 82172033, Grant 82272071 and in part by the Dreams Foundation of Jianghuai Advance Technology Center, and in part by the Open Fund of the National Key Laboratory of Infrared Detection Technologies.
10

---

References
[1] S. Ali, N. Ghatwary, D. Jha, E. Isik-Polat, G. Polat, C. Yang, W. Li, A. Galdran, M.-Á. G. Ballester,
V. Thambawita, ct al. Assessing gencralisability of deep learning-basse based polyp detection and segmentation methods through a computer vision challenge. Sci. Rep., 2024.
[2] S. G. Armato MI, G. McLennan, I.. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, A. Hoffman, et al. The lung image datahase consortium (lide) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics, 38(2):115 9,19,,1,2222 2011.
[3] C. F. Baumgartner, K. C. Tezcan, K. Chaitanya, A. M. Hötker, U. J. Muehlematter, K. Schawkat, A. S. Becker, O. Donati, and E. Konukoglu. Phiseg: Capturing uncertainty in medical image segmentation. In Medical Image Computing and Computer Assisted Intervention MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part II 22, pages 119-127. 2019.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Amanda, S. Agarwal, A. Herbert- Voss, G. Krueger, H. Tom, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, C. Benjamin, J. Clark, C. Berner, M. Sam, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, arXiv: Computation and Sanguage.arXiv. Computation and Language, May 2020.
[5] H. Cao, Y. Wang. J. Chen, D. Jiang, X. Zhang, Q Tian, and M. Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference on computer vision, pages 218. Springer, 2022.
[6] T. Chen, I.. Zhu, C. Deng, R. Cao, Y. Wang. S. Zhang, Z. Li, Sun, Y. Zang, and P. Mao. Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3367-3375, 2023.
[7] Z. Chen, W. Li. X. Xing. and Y. Yuan. Medical federated learning with joint graph purification for noisy label learning, MedlA, 2023.
[8] Z. Chen, Q. Xu, X. Liu, and Y. Yuan. Un-sam: Universal prompt-frce segmentation for generalized nuclei images. ArXiv, abs/2402.16663,, 2024.
[9] J. Cheng, J. Ye, Z. Deng. J. Chen, T. Li, H. Wang, Y. Su, Z. Huang, J. Chen, L. Jiang, et al. Sam-med2d. arXiv preprint arXiv:2308. 16184, 2023.
[10] R. Deng, C. Cui, Q. Liu, T. Yao, I.. W. Remedios, S. Bao, B. A. Landman, I.. E. Wheless, L. A. Coburn, K. T. Wilson, al. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation 2023.
on whole slide imaging. arXiv preprint arXiv:2304.0411
[11] Z. Ding, Q Dong, H. Xu, C. Li, X. Ding, and Y. Huang. Unsupervised anomaly segmentation for brain lesions using dual semantic- manifoll reconstruction. In International Conference on Neural Information Processing, pages 133-144 144. Springer, 2022.
[12] Y. Gao, W. Xia, Hu, and Gao. Desam: Decoupling segment anything model for generalizable medical image segmentation. arXiv preprint arXiv:2306.0040 00499, 2023.
[13] 7. Gao, Y. Chen, C. Zhang, and X. He. Modeling multimodal aleatoric uncertainty in segmentation with mixture of stochastic expert. arXiv preprint arXiv:2212.07328. 2022.
[14] D. Gutman, N. C. F. Codella, E. M. Celebi, B. Helba, M. Marchetti, N. K. Mishra, and A. Halpern. Skin lesion analysis toward melanoma detection: challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic). ArXiv, abs/ (tds//605.0199, 2016.
[15] Z. Hc, W. Li, Y. Jiang, Z. Peng, P. Wang, X. Li, T. Liu, J. Han, T. Zhang, and Y. Yuan. F2tnet: Fmri to tlw mri knowledge transfer network for brain multi-phenotype prediction. In MICCAI, pages 265-275, 2024.
[16] H. Huang, I.. Lin, R. Tong, H. Hu, Q Zhang, Y. Iwamoto, Han, Y.WW: T.W. Chen, and J. Wu. Unet 3+: A full-scale connected unet for medical image segmentation. In ICASSP 2020-2020 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 1055-1059, IEEE, 2020.
[17] Y. Huang, C. Li, Z. Lin, H. Liu, H. Xu, Y. Liu, Y. Huang. X. Ding. X. Tu. and Y. Yuan. P2sam: Probabilistically prompted sams are efficient segmentator for ambiguous medical images. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9779-97787 2024.
11

---

[18] F. Isensee, P. Kickingereder, W. Wick, M. Bendszus, and K. H. Maier-Hein. Brain tumor segmentation and radiomics survival prediction: Contribution the brats 2017 challenge. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuriex: Third International Workshop. BrainLes 2017, Held in Conjunction with MICCAI 2017, Quebec City, QC, Canada, September 14, 2017, Revised Selected Papers 3. pages 287-297. Springer, 2018.
[19] M. Robeess,,, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? arXiv preprint arXiv:161001188. 2016.
[20] E. Kassapis, Dikov, D. K. Gupta, and C. Nugteren. Calibrated adversarial refinement for stochastic semantic segmentation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7037-7041 2020.
[21] F. Kassapis, G. Dikov, D. K. Gupta, and C. Nugteren. Calibrated adversarial refinement for stochastic semantic segmentation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 7057-7067, 2021.
[22] A. Kirillov, Mintun, N. Ravi, H. Mao, Rolland, L. Gustafson, Xiao, S. Whitchead, A. C. Berg. W.-Y. Lo, el al. Segment anything. arXiv preprint arXiv:2304.0264. 2023.
[23] S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. Eslami, D. Jimenez Rezende, and O. Ronneherger. A probabilistic u-net for segmentation of ambignous im- ages. Advances in neural information processing systems, 31, 2018.
[24] S. A. Kohl, B. Porera,P K. H. Maier-Hein, D. J. Rezende, S. Eslami, Kohli, Zisserman, and O. Ronneberger, A hicrarchical probabilistic u-net for modeling multi-scale ambiguities. arXiv preprint 13077, 2019.
[25] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv: 104. 104.0:91,, 2021.
[26] C. Li. B. Y. Feng. Y. Liu, H. Liu, C. Wang. W. Yu, and Y. Yuan. Endosparse: Real-time sparse view synthesis endoscopic seenes using gaussian splatting. arXiv preprint urXiv:2407.01029 2024.
[27] C. Li, X. Lin, Mao, W. Lin, Q Qi, X. Ding, Huang, D. Liang, and Yu. Domain generalization on medical imaging classification using episodic training with task augmentation. Computers in biology and medicine, 141:105144 2022.
[28] C. Li, H. Liu, Y. Liu, B. Y. Feng, W. Li, X. Liu, Z. Chen, J. Shao, and Y. Yuan. Endora: Video generation models endoscopy simulators. arXiv preprint arXiv:2403. 10050. 2024.
[29] C. Li, X. Liu, W. Li, C. Wang, H. Liu, and Y. Yuan. U-kan makes strong backbone for medical image segmentation and generation. arXiv preprint 2xXiv:2406.02918 2024.
[30] C. Li, X. Liu, C. Wang, Y. Liu, W. Yu, J. Shao, and Y. Yuan. Gtp-4o: Modality-promple heterogencous graph learning for modil biomedical representation. arXiv preprint arXiv:2407.05540, 2024.
[31] C. Li, W. Ma, L. Sun, X. Ding, Y. Huang, G. Wang, and Y. Yu. Hierarchical deep network with uncertainty- aware semi semi-supervised learning for vessel segmentation. Neural Computing and Applications, pages 14
[32] C. Li, Zhang. J. Li, Y. Huang, and X. Ding. Unsupervised anomaly segmentation using image- cycle translation. arXiv preprint wxXi:2103.0000)) 2021.
[33] C. Li, Y. Zhang, Z. Liang, W. Ma, Y. Huang, and X. Ding. Consistent posterior distributions under vessel- mixing: regularization for cross-domain retinal artery/vein classification. In 2021 IEEE International Conference on Image Processing (ICIP), pages 61 65. IEEE, 2021.
[34] W. Li, X. Guo, and Y. Yuan. Novel scenes classes: Towards adaptive open-set object detection. In ICCV, pages 15780-15790, 2023.
[35] W. Li, X. Liu, Q Yang, and Y. Yuan. From static to dynamic diagnostics: Boosting medical image analysis via motion-informed generative videos. In MICCAI, 2024.
[36] W. Li, X. Liu, X. Yao, and Y. Yuan. Scan: Cross domain object detection with semantic conditioned adaptation. In AAAL, pages 1421-1428, 2022.
12

---

[37] W. Li. X. Liu. and Y. Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection. In CVPR, 2022.
[38] H. Liu, Y. Liu, C. Li, W. Li, and Y. Yuan. Lgs: A light-weight 4d gaussian splatting for efficient surgical scene reconstruction. arXiv preprint arXiv:2406. 16073, 2024.
[39] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 2023.
[40] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0542 2023.
[41] X. Liu, W. Li, and Y. Yuan. Intervention interaction federated abnormality detection with noisy clients. In MICCAI, pages 309-319, 2022.
[42] X. Liu, W. Li, and Y. Yuan. Diffrect: Latent diffusion label rectification for semi-supervised medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 56-66. Springer. 2024.
[43] Y. Liu, C. Li, C. Yang, and Y. Yuan. Endogaussian: Gaussian splatting for deformable surgical scene reconstruction. arXiv preprint arXiv:2401.12566 2024.
[44] Y. Liu, W. Li, C. Wang, H. Chen, and Y. Yuan. When 3d partial points meets sam: Tooth point cloud segmentation with sparse labels. In MICCAI, pages 778-788, 2024.
[45] Z. Lou, Q. Xu, Z. Jiang, X. Hc, Z. Chen, Y. Wang. C. Li, M. M. Hc, and W. Duan. Nusegdg: Integration of heterogeneous space and gaussian kernel for domain-generaliin muclei segmentation. arXiv preprint arXiv:2408. // 1787, 2024.
[46] J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang. Segment anything in medical images. Nature Communicutions, 15(1):654, 2024.
[47] M. Monteiro, I.. Le Folgae, D. Coelho de Castro, N. Pawlowski, B. Marques, K. Kamnitsas, M. van der Wilk, and B. Glocker. Stochastic segmentation networks: Modelling spatially correlated aleatoric uncer- tainty. Advances in Neural Information Processing Systems, 33:1 33:12756-12767, 2020.
[48] I. P. Osco, Q. Wu, E. L. de Lemos, W. N. Gonçalves, A. P. M. Ramos, J. Li, and J. M. Junior. The segment anything model (sam) for remote sensing applications: From zero to one shot, ArXiv, abs/2306.16600 16623, 2023.
[49] D. Qiu and L. M. Lui. Modal uncertainty estimation via discrete latent representation. arXiv preprint arXiv:2007.12888 2020.
[50] A. Rahman, J. M. J. Valanarasu, I. Hacihatiloglu, and V. M. Patel. Ambiguous medical image segmentation using diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1153-11546 2023.
[51] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assister intervention, pages 234- 241. Springer, 2015.
[52] R. Selvan, F. Faye, J. Middleton, and A. Pai. Uncertainty quantification in medical image segmentation with normalizing flows. In Machine Learning in Medical Imaging: I1 International Workshop, MLMI 2020, Held in Conjunction with MICCAI 2020. Lima, Peru, October 4, 2020, Proceedings 11. pages 80-90. Springer, 2020.
[53] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[54] L. Sun, C. Li, X. Ding, Y. Huang, Z. Chen, G. Wang, Y. Yu, and J. Paisley. Few-shot medical image segmentation using a global correlation network with discriminative embedding. Computers in biology and medicine, 1400100 2022.
[55] M. A. Valiuddin, C. G. Viviers, R. J. van Sloun, P. H. de With, and F. van der Sommen. Improving aleatoric uncertainty quantification in multi-annotated medical image segmentation with normalizing flows. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging. Placental and Preterm Image Analysis: 3rd International Workshop, UNSURE 2021, and 6th International Workshop. PIPPI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021. Proceedings 3, pages 75 88. Springer, 2021.
13

---

[56] X. Wang, W. Wang, Y. Cao, C. Shen, and T. Huang. Images speak in images: A generalist painter for in-context visual learning. Dec 2022.
[57] X. Wang, X. Zhang. Y. Cao, W. Wang, C. Shen, and T. Huang. Seggpt: Segmenting everything in context. ArXiv, abs/2304.03222 03284, 2023.
[58] J. Wu, R. Fu, H. Fang, Y. Liu, Z. Wang, Y. Xu, Y. Jin, and T. Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304. /2620, 2023.
[59] J. M. Y. Y. Wuyang Li, Xinyu Liu. Cliff: Continual latent diffusion for open-vocabulary object detection. In ECCV, 2024.
[60] H. Xu, Y. Zhang. L., Sun, C. Li, Y. Huang, and X. Ding. Afse: Adaptive fourier space compression for anomaly detection. arXiv preprint arXiv:2204.0777 07966, 2022.
[61] Y. Yuan, M. Chao, and Y.-. Lo. Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance. IEEE Transactions Medical Imaging, 36:1876-1886, 2017.
[62] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao. Ilama-adapter:: Efficient. fine-tuning of language models with zero-init attention.
[63] W. Zhang, X. Zhang, S. Huang. Y. Lu, and K. Wang. Pixelseg: Pixel-by-pixee stochastic semantic segmentation for ambiguous medical images. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4742-4750, 2022.
[64] W. Zhang, X. Zhang. Huang, Lu, and K. Wang. A probabilistic model for controlling diversity and accuracy of ambiguous medical image segmentation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4751-4759, 2022.
[65] Y. Zhang, C. Li, Lin, L. Sun, Zhuang, Y. Huang, X. Ding,
X. Liu, and Yu. Generator versus segmentor: Pseudo-healthy synthesis. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg. France, September 27-October 1. 2021, Proceedings, Part VI 24, pages 150-160 160. Springer, 2021.
[66] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang. L. Wang. J. Gao, and Y. Lee. Segment everything everywhere all once. Advances in Neural Information Processing Systems, 36, 2024.
14

---

A Appendix
The following contents are provided in the supplements:
Limitation and Border Impact.
More experimental details (Sec. 4.1 in main paper).
Details of network architecture of A-SAM.
More visualization about our experiments. (Sec. 4.2 and Sec. 4.3 in the main paper),
A.1 Limitation and Border Impact
Limitation. The current approach is constrained by the uncontrolled and unquantifiable nature of uncertainty, This limitation means that the accuracy of handling uncertainty varies across different scenarios. Further systematic analysis is required to comprehend the underlying factors that result in some scenarios being more manageable than others in terms of uncertainty.
Broader Impacts. Ambiguous segmentation and uncertainty handling essential in fields such as image processing and medical diagnosties. Fuzzy segmentation improves our understanding of complex or unclear image content. Proper uncertainty management can enhance prediction accuracy and decision decision-making, especially in medical diagnostics, aiding in precise disease diagnosis and treatment planning. Ensuring image processing safety is also critical for user privacy and data security, thereby building trust and satisfaction.
A.2 Detailed Experimental Setup
Dataset. Four datascts are used for comparison. LIDC-IDRI 2 is a datasct for lung lesion segmentation, which encompasses voluminous collection of lung computed tomography scans from 1010 distinet subjects, with manual annotations provided by panel composed of four domain experts. A diversified panel of 12 radiologists leveraged their expertise to provide annotation masks for the dataset, a characteristic that allows the dataset to reflect the typical ambiguity frequently encountered in CT imaging, thereby ensuring comprehensive, accurate annotations that represent a broad range of expert opinions. The resolution of all images is 128% x 128. BraTS 2017 [8 a dataset for 3D brain tumor segmentation, which consists of 285 cases of 3D MRI images, each image comprising 155 slices. Each slice exhibits four modes (TI. Tlce T2. and Flair) and meticulously annotated by professional radiologists into four classes: Background (BG), Noon-ehhnciin/Necoott Tumor Core (NET), Edema (OD), and Enhancing Tumor Core (ET). The resolution of all images is 240x240. ISBI 2016 [14 is dermoscopy dataset containing 900 dermoscopic images for training and 379 images for testing. Each image is 8-bit RGB and is annotated by an expert with the lesion area. To keep each image at the same scale, we follow [6] to resize the images to 256 192 and pad the top and bottom with 32 pixcls, respectively, to get 256x256 images. SIM 10k [1]] consists of 10,000 images that are rendered by the gaming engine Grand Theft Auto. In SIM 10k, bounding boxes of 58,701 cars are provided in the 10,000 training images. All images used in the training.
Implementation Details. For LIDC, we directly employ the four expert annotations included in the dataset represent four different ambiguous segmentation labels (2). For BraTS, we overlay and amalgamate annotations from disparate categories, subsequently transforming the outcome into a binary mask that comprises solely the foreground and background [18). This process is geared towards generating multiple segmentation masks to mimic real-world ambiguous segmentation scenarios, thereby augmenting the rigor and reliability of the experiment. For ISBI, we directly use the single label included in the dataset 14 For the aforementioned three datasets, we carry out optimization using the Adam optimizer, with learning rate of Ic-4, over 100 epochs. For the SIM 10k dataset, we contemplate a practical overlap setting. Specifically, we select images in which pixels from two instances within frame overlap, thereafter creating three potential masks from the two overlapping instances. These masks could represent the first object, the second object, the union of both. For SIM 10k [19]. we earry out optimization using the Adam optimizer with learning rate of Ic-4, over 500 epochs. The trade-off coefficients Eq. 14 are set as ap = az
Evaluation Metries. Four metrics are used for strict evaluation. Generalized Energy Distance (GED) is commonly used metric in ambiguous image segmentation tasks that leverages distance between observations by comparing the distribution of segmentations [3]. as De (Pat. Pous) ( Y)] E[d(S, E[d(Y, Y" where d corresponds to the distance measure d(r, y) 1 IoU v). Y and Y' are independent samples of P.t and S and S" are sampled from Paut. Lower energy indicates better agreement between Slungarian-Mtthhe
prediction and the ground truth distribution of segmentations. (HM-loU) is used by calculating the optimal match of Intersection over Union (IoU) between annotations and predictions, which is searched by Hungarian algorithm. This metric offers more accurate representation of
Intersection over Union
15

---

Inper
Pro. U-ad
PaceBog
Grond Truth
PioeBieg
PSAM
Figure 6: More visualization on the LIDC dataset, displaying only the first samples.
sample fidelity. contrasting Generalized Energy Distance (GED) which tends over-reward sample diversity. The Hungarian algorithm identifies the best one-to-one correspondence between objects in two sets. In this context, we utilize ToU (Y, Y") to determine the similarity between two samples. Maximum & Average Dice Matching (Dmax & Dmean) is respectively the best and average results over the Dice scores between each prediction result and each ground truth.
A.3 Network Architecture
A.3.1 Prompted SAM
In the experiment, we adopted the Vit-b version of the SAM model and accommodated Ency by reducing the size of the output feature map F/ by compared with the original. This change expected to reduce the required memory usage during the training process accelerate inference speed the model. addition. we adjusted the SAM model multi output mode with 8 outputs, and set pixel mean and pixel std parameters to 0 and 1, respectively.
A.3.2 Prompt Generation Network (PGN)
The network mainly consists of two parts. (1) Encoder: This part contains convolutional blocks, each with 3 convolutional layers inside. These convolutional blocks have channel numbers of 32, 64, 128 and 192, respectively, to gradually extract and deepen features. (2) Axis Gaussian Generation Network: This network consists Ixl convolutional layer with 256 channels and an axial Gaussian distribution generator. This design first increases dimensionality of the feature map output by the Encoder through a convolutional layer to obtain 256 dimensional and a, and then these two parameters are fed into a Gaussian generator to generate the distribution of Tp.
A.3.3 aavere Assembling Module
the number of mask weights W to 8 and initialize each weight J. This setting aims to correspond outputs SAM model. the first stage of the experiment, these weight W will be trained to meet the model requirements. In the second stage. we will freeze these weights. This is to enable the prompt generation network to generate more diverse and representative segmentation results, thereby effectively guiding the modeling of Tr.
experimental design,
A.4 More Visualization about Experiments
As demonstrated Fig. 6 and Fig. these illustrations provide extensive visualization of research outcomes. These figures meticulously depict various aspects of our data, aiding readers gaining profound understanding of our research findings.
16

---

Inpur
Ground Trush
PiselSeg
PSAM
Ingut
Ground Truth
F2
Pro. U-nee
PixelSeg
P'SAM
Figure 7: More visualization on the BraTS2017 dataset, displaying only the first samples.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction sections offer comprehensive discussion of the manuscript's context, intuition, and ambitions, well its contributions.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results he expected generalize other setrings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are discussed by authors at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
17

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification. asympiotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tes -oxttttt system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, the paper provides the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The pipeline of the methods and the details of experiments are presented with corre- sponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
18

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(h) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may he tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may he that access the model is limited in some way (e.g., registered users), but should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All utilized data sourced from open-access platforms. The code, which will be made publicly available, is uploaded as a zip file.
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines http:///nipssccppbllie guides/Codee htte///oooSSeiiseeee wttsSSummisioooooiioooooo for more details.
While we encourage release of code and data, we understand that might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines lhttps://nipe htt:://ips.ccppuble/ cc/public/ htt////ooeeeiiaioinnnonao
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Detail
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer. etc.) necessary understand the results?
Answer: [Yes]
Justification: The pipeline of the methods and details of experiments are presented with corre- sponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results contain the standard deviation of the results over several random runs.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mcan.
It is OK report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The details experiments are presented with corresponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does research conducted the paper conform, every respect, NeurlPS Code of Ethics http:////errrpp...cc/cpee///eeee
Question:
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethies.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimatc to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow controlled use of the model, by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original owners of assets, including data and models, used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL.
The name of license (e.g.. CC-BY 4.0) should be included each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, (paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors encouraged to reach out to the asset's creators.
13. New Assets
21

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced in the paper well documented and provided alongside the assets,
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the dctails of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity applica- ble), such as the institution conducting the review.
22


Reviews:
[{"review_id": "lEVAcm7qmv", "reviewer": "Reviewer_vXP5", "summary": "This paper aims to convert the flaws in the vision foundation model (e.g., SAM) into advantages for ambiguous object segmentation. To this end, the authors propose a novel framework that employs latent distribution and an optimization architecture. The authors validated the performance of the proposed methods through comprehensive experiments.", "questions": "1. What is the difference between PGN and posterior PGN?", "limitations": "The authors address limitations of this work and broader impact properly.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Unlike existing approaches that aim to stabilize the sensitivity to ambiguous objects in SAM, this paper suggests leveraging the vulnerability for ambiguous object segmentation. The proposed approach seeks to harness SAM's sensitivity, redeemed as a weakness, to address ambiguous and uncertain predictions.", "weaknesses": "1. The explanations are unclear and hard to follow. Specifically, it needs further explanation of how to extract the mean and standard deviation from the convolution blocks and how to utilize the ground truth labels in the posterior version of the prompt generation network.\n2. Some symbols are used without explanation (e.g., Θ, Φ, N_i, N_p).\n3. Missing reference: Previous research at line 169.\n4. Since this paper focuses on clinical scenarios for ambiguous object segmentation, it seems unfair to compare the performance without including existing medical segmentation methods such as OM-Net [1], DC-UNet [2], and CE-Net [3].\n\n[1] https://arxiv.org/pdf/1906.01796v2\n[2] https://arxiv.org/pdf/2006.00414v1\n[3] https://arxiv.org/pdf/1903.02740v1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our paper as harnessing SAM's sensitivity, redeemed as a weakness, to address ambiguous and uncertain predictions. We provide pointwise responses to your concerns below.\n\n## Q1. Method details\n\n**<How to extract the mean and standard deviation from networks?>**\nAs noted in Lines 155-157, the mean and standard deviation of the Axis Gaussian distribution are the two vectors output by the neural network. The input to the network can be the image or prompt embedding, and the network is trained to directly output the mean and standard deviation parameters. This allows for efficient extraction of the distributional statistics from the network outputs, which is crucial for downstream probabilistic modeling tasks.\n\n**<How to utilize the ground truth labels in posterior prompt generation network?>**\nAs described in Eq.(7), for each iteration of the training process, the method randomly samples one of the possible ambiguous labels from the label set. This sampled label is then input into the posterior prompt generation network, which is responsible for producing prompts that capture the semantics of the given ambiguous label. By iteratively sampling from the set of ambiguous labels and feeding them to the prompt generation network, the model is able to learn to produce effective prompts that can handle the inherent ambiguity in the training data.\n\n## Q2. Symbol details\n\nThank you for your pointing out. The parameters Θ and Φ that you mentioned represent the hyperparameters characterizing two distributions, such as the mean and variance of a Gaussian distribution. $N_i$ and $N_p$ are hyperparameters denoting the dimensions of the vectors. We will provide a more detailed description of these parameters in the revision. \nWe appreciate your attention to detail and the opportunity to clarify this point.\n\n## Q3. Missing reference\n\nThank you! The authors will add the line \"Previous research [1] has described...\" at Line 169 to provide more context on how the current approach builds upon prior work in this area, as outlined in the referenced publication [1].\n\n[1] A Probabilistic U-Net for Segmentation of Ambiguous Images\n\n## Q4. Comparison with more existing models\n\nWe appreciate your insightful suggestion! It's worth noting that our method is designed for ambiguous segmentation, while the comparison approach you proposed is designed for conventional deterministic segmentation. To enable a fair comparison, we have adapted our method for deterministic segmentation by ensemble averaging the segmentation results from three sampling iterations into a single output.\n\nWe have conducted further comparisons on the overlap dataset and the BraTS 2017 dataset, as mentioned in the references OM-Net [1], DC-UNet [2], and CE-Net [3] you cited. The results of these comparisons are listed as below. As we can see from these results, our method still achieves superior segmentation performance under this setting. We believe this additional evaluation addresses your concern and provides a more comprehensive assessment of our method's performance across different segmentation paradigms.\n\n| Metrics | |Dice↑ |  | |Hausdorff95↓ |  |\n|:----------|:--------:|:------------:|:--------:|:------------:|:--------:|:------:|\n| **Category** | **Core** | **Enh. Core** | **Whole** | **Core** | **Enh. Core** | **Whole** |\n| OM-Net | 0.842 | 0.785 | 0.907 | 7.561 | 3.299 | 4.382 |\n| nnU-Net | 0.819 | 0.776 | 0.903 | 8.642 | 3.163 | 6.767 |\n| Wang et al. | 0.838 | 0.786 | 0.905 | 6.479 | 3.282 | 3.890 |\n| Kamnitsas et al | 0.797 | 0.738 | 0.901 | 6.560 | 4.500 | 4.230 |\n| $\\mathcal{A}-SAM$ (Ours) | **0.863** | **0.803** | **0.921** | **6.463** | **3.084** | **3.741** |\n\n[1] https://arxiv.org/pdf/1906.01796v2 [2] https://arxiv.org/pdf/2006.00414v1 [3] [https://arxiv.org/pdf/1903.02740v1](https://arxiv.org/pdf/1903.02740v1)\n\n## Q5. Difference between PGN and posterior PGN?\n\nAs noted in Lines 171-173, a posterior version for the prompt generation network $F^{post}_{PGN}$, parameterized by $\\Theta^{\\mathcal{T}}$, is further introduced during the training process. This posterior prompt generation network learns to generate the effective distribution for the prompt embedding when **accessing the ground-truth label distribution**. The introduction of this additional network component allows the model to better capture the semantics of the ground-truth labels and generate more targeted prompts, improving the performance of models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSthAlrX3T", "reviewer": "Reviewer_YPmC", "summary": "This paper builds a framework for amigous object segmentation on top of SAM prompted with bounding boxes, which is known to be sensitive to small prompt changes. \n\nThe framework is based on a VAE, and the main idea is to jointly model the prompt and the object granularity with a latent probability distribution to gain more control over SAM’s output. In practice, the prompt  embeddings  and image embeddings (controlling granularity) are formulated as a distribution.\n \nThe method is evaluated on 3 medical imaging datasets and on a synthetic driving dataset, showing superior performance over the baselines.", "questions": "## General remarks\n1. Evaluation on a real-world (not synthetic) non-medical dataset would help to show the generality of the method.\n2. It would help readability if it was mentioned that the evaluation metrics are defined in the appendix, also it would help to see the related references in the main paper\n3. Is there some intuition/more details on why the granularity is modelled within the image embedding?\n\n## Reproducibility\n3. How were the trade-off coefficients tuned?\n4. More details on how the three masks from overlapping instances on the SIM10k dataset were obtained should be provided.\n5. How was the best checkpoint selected? Was there any hyper-parameter tuning?\n6. What does  'achieving significant segmentation outcome‘ mean on line 97? Improvement in segmentation performance over SAM without adapter?\n\n## Fig. 1:\n7.  SAM outputs multiple predictions for a prompt, how is this handled in Fig. 1a and 1b?\n8.  Medical domain is not in the training domain of SAM so higher uncertainity/instability of prediction is expected, maybe it is not the best example to showcase the behaviour.\n9.  What are canonical box prompts from description of Fig. 1? Ground truth bounding boxes?\n10.  I assume granularities in 1c correpsond to the three output masks of SAM, what is full granularity then?\n11.  The prompt variation experiment depicted in Figure 1 includes bounding boxes that do not cover the whole region to be segmented. It is not unrealistic to control for that in real scenarios, and it would be interesting to see how the figure would change since SAM seems to be quite sensitive to whether the whole object is covered or not – making the bounding box smaller than an object impacts segmentation more than making it larger.\n12.  It would be helpful to see how the experts annotate  the example \n\n## Fig 2: \n13.  Why is image embedding concatenated with the IGN sample, but the prompt embedding is the output of PGN directly?\n14.  Incomplete description – ‚by jointly probabilities‘‘\n\n## Add Weakness 1. – unclear statements and missing details \n15. How were the trade-off coefficients set?\n16. How exactly was the three masks generated from overlapping instances on the SIM10k dataset?\n17. How was the best checkpoint selected? Was any hyper-parameter tuning performed (if yes, on what data)?\n18. Line 153 – parameters of axisymmetric Gauss. Distribution „including mean and std“ – the gauss. distirbution does not have any other parameters.\n19. What does  'achieving significant segmentation outcome‘ mean on line 97? Improvement in segmentation performance over SAM without adapter?\n20. What is meant by the 'final integrated SAM output that integrates multiple candidates‘ on line 44? The only part of SAM that integrates multiple predictions I am aware of is SamAutomaticMaskGenerator class provided by the authors (it features non-maxima suppression) but it prompts SAM with a uniform grid of points while the paper discusses bounding box prompts.\n21. The explanation of GT generation for the datasets is confusing since it is incomplete in the paper, it would be nice to at least have a link to the appendix for more details.\n22. On lines 38-42, it would help to see an example of such behaviour – what is meant by SAM amalgamating the candidates at different granularities? AFAIK, SAM outputs multiple predictions for each prompt specifically to deal with ambigous prompts.\n23. What does 'diminutive adapters‘ on line 93 mean?\n24. What is meant by encoder lenght in line 123?\n \n## Add Weakness 2. – evaluation \n25. Evaluation on a real-world (not synthetic) non-medical dataset would help to show the generality of the method.\n26. Why is original SAM not included in the comparison from subsection 4.2?\n\n## Typos: \n27. Line 83 – promotable instead of promptable", "limitations": "Limitations are addressed in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is the first to use a promptable large-scale pretrained model like SAM for ambiguous image segmentation\n2. The methodology is in general clearly written and easy to follow, figure 2 provides a great overview of the method\n3. Extensive evaluation and ablations were performed, showing the method’s superior performance compared to baselines on all of the datasets. (the method is not evaluated on any non-medical real dataset though, see weaknesses)\n4. The joint modeling of promts and image embeddings of the proposed method is efficient since the probability sampling is only performed after the SAM encoder and thus the image embedding needs to be computed only once (SAM decoder is lightweight)", "weaknesses": "1. The paper contains several unclear statements or missing details, which make the reproducibility of the method difficult.\n2. The evaluation is carried on a niche domain (medical) or on synthetic datasets only. It is hard to judge the performance of this method in general real-world setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive initial impression and valuable feedback. We look forward to revising our manuscript based on your suggestions. Below are our point-by-point responses to your concerns. For brevity, we address recurring issues only once.\n\n## Q1. General remarks\n\n**<Applicable to real-world non-synthetic and non-medical datatset?>**\n As shown in Fig. 3 & Tab. 1 in initial submission, we evaluated our method on the Sim10k dataset, featuring synthetic non-medical street scenes with ambiguous semantics. This challenging context demonstrates our approach's broader applicability beyond medical imaging.\n\nFollowing your suggestion, we further conducted experiments on KINS dataset for instance segmentation in real-world street scenes. More details and results are shown in **Gloabl Table 2 and Global Response 2**.\nOur method performs competitively on KINS, achieving the superior performance.\n\n**<More Details about metrics>**\nWe appreciate your suggestion. In the revision, we will include the following content in the appendix:\n\n**Generalized Energy Distance (GED)**: A metric for ambiguous image segmentation comparing segmentation distributions:\n\\begin{equation}\nD^2_{GED}(P_{gt},P_{out}) = 2\\mathbb{E}[d(S,Y)]-\\mathbb{E}[d(S,S')]-\\mathbb{E}[d(Y,Y')]\n\\end{equation}\nwhere $d(x, y) = 1 - IoU(x, y)$, $Y, Y'$ are samples from $P_{gt}$, and $S, S'$ from $P_{out}$. Lower energy indicates better agreement between prediction and ground truth distributions.\n\n**Maximum and Mean Dice Matching ($D_{max}, D_{mean}$)**: For medical diagnoses, we define Dice score as:\n\\begin{equation}\nDice(\\hat{Y},Y) = \\begin{cases}\n\\frac{2 |Y \\cap \\hat{Y}|}{|Y|+|\\hat{Y}|}, &\\text{if } Y \\cup \\hat{Y} \\neq \\emptyset\\\n1, & \\text{otherwise}\n\\end{cases}\n\\end{equation}\nWe calculate $D_{max}$ for each ground truth $Y_i$ as:\n\\begin{equation}\n\\{D}_{max} = max \\\\{Dice(\\hat{Y}_1,Y_i),Dice(\\hat{Y}_2,Y_i),..., Dice(\\hat{Y}_N,Y_i) \\\\}\n\\end{equation}\n\n**Hungarian-Matched Intersection over Union (HM-IoU)**: This metric calculates the optimal 1:1 match between annotation and prediction using the Hungarian algorithm, better representing sample fidelity. It uses $IoU (Y, Y')$ to determine similarity between samples.\n\n**<Insights of modeling granularity within image embedding?>**\nAs noted in Lines 185-189, SAM generates multiple candidate masks at different granularities, demonstrating inherent ambiguity related to object granularity. This inspired us to enhance the model's perception of ambiguous objects. We introduce an image generation network (IGN) for object embedding to model this distribution, injecting granularity-aware priors.\n\n## Q2. Implementation\n\n**<Trade-off coefficients>**\nAs noted in Lines 238, the trade-off coefficients are set as α_P = α_I = 1.\n\n**<How three masks from SIM10k dataset were obtained?>**\nAs noted in Lines 236-237, for the SIM 10k dataset, we select images where pixels from two instances overlap (e.g., instances A and B), and derive the three potential segmentation masks (e.g., only region A, only region B, the union region of A and B).\n\n**<Model selection>**\nFollowing current best practices [1], we employ a 6:2:2 ratio for training, validation, and test sets. Results are reported based on best performance on the validation set. We will include this information in our revision.\n\n[1] A probabilistic u-net for segmentation of ambiguous images\n\n## Q3. Fig1\n\n**<What’s full granularity?>**\nFull granularity refers to the ensemble of various granular outputs within SAM. This approach leverages the multi-scale nature of SAM's internal representations, allowing for a more comprehensive and nuanced segmentation result.\n\n**<Bboxs that are not covering whole region are not considered?>**\nWe respectfully disagree, as our approach accounts for this scenario. Due to potential bounding box offsets, the complete segmentation region may not always be fully contained. Our method ensures robust performance even when the initial box doesn't perfectly encapsulate the entire region of interest\n\n## Q4. More details\n\n**<How GT generated?>**\nWe will add the following details to the revised appendix:\n\nThe LIDC dataset features manual annotations from four domain experts, accurately reflecting CT imaging ambiguity. Twelve radiologists provided annotation masks. We use the version after the second reading, where experts reviewed and adjusted annotations based on peer feedback, ensuring comprehensive and diverse expert opinions.\n\nThe BraTS 2017 dataset includes 285 cases of 3D MRI images (155 slices each) in four modalities (T1, T1ce, T2, Flair). Expert radiologists annotated four classes: background, non-enhanced/necrotic tumor core, oedema, and enhanced tumor core. We overlay these annotations to create binary masks, simulating ambiguous segmentation scenarios.\n\nThe ISBI 2016 dataset contains 900 training and 379 testing dermoscopic images. A dermatology expert annotated lesion boundaries in all images, providing a gold standard for segmentation tasks.\n\nFor the SIM 10k dataset, we focus on images where pixels from two instances overlap (e.g., instances A and B). From these, we derive three potential segmentation masks: the region exclusive to A, the region exclusive to B, and the union region of A and B.\n\n**<What’s SAM amalgamating different granularities? >**\nWe acknowledge that the original SAM outputs multiple predictions per prompt to address ambiguity, without integration. Ensemble integration was introduced in Per-SAM [1]. We will clarify this distinction in our revision.\n\n[1] Personalize segment anything model with one shot. \n\n## Q5. Evaluation\n\n**<Why original SAM not included in the comparison from Sec. 4.2?>**\nTable 1 (Sec. 4.2) compares SAM variants adapted for ambiguous segmentation. As suggested, we also compared with the original SAM, as shown in **Global Table 1 in Global Response 1**. Results show the original SAM's multi-output capability is insufficient for complex, diverse scenarios, leading to suboptimal performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCyj4cRCCe", "reviewer": "Reviewer_2dZP", "summary": "This paper proposes a SAM-based framework to address the ambiguous image segmentation problem. The authors present an optimization framework based on a conditional variational autoencoder, which simultaneously models the prompt and the granularity of the object using a latent probability distribution. This approach allows the model to adaptively perceive and represent the real ambiguous label distribution, enabling SAM to controllably produce a series of diverse, convincing, and reasonable segmentation outputs. Experiments on multiple datasets and metrics demonstrate the effectiveness of the method.", "questions": "1. Will the proposed method perform better than the original SAM if there is no point/box/mask shift?\n\n2. Why is the proposed method trained from scratch using randomly initialized weights? Would it be better to finetune from the pre-trained SAM?", "limitations": "Please see the weaknesses section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. To the best of my knowledge and as indicated by the authors, this paper is the first work that leverages the inherent properties in vision foundation models (SAM) for ambiguous image segmentation.\n2. The experimental results demonstrate impressive advantages. Compared to the original SAM, the proposed method shows significantly better performance in the presence of prompt shifts. This high level of robustness is extremely valuable in practical applications.", "weaknesses": "1. The task setup of ambiguous image segmentation in this paper is somewhat confusing for me. I have read some referenced and comparative works cited in the paper, such as [a], and found that their task objective is providing multiple segmentation hypotheses for ambiguous images. However, this paper seems to focus more on increasing the accuracy and stability of the model's output when the input prompt has noise or shifts. More explanation about the task setup is needed. Accordingly, it is recommended to include a section in the main text that introduces the task setup, which can help readers who are not experts in this research area understand the paper better.\n\n2. The comparison with conventional ambiguous segmentation models seems unfair because most of the compared methods do not use a network structure as large as SAM. Therefore, it is unclear whether the performance advantage comes from the increased number of network parameters in SAM or from the innovative designs proposed in this paper. I noticed that some of the compared methods, such as [b], can be applied with any encoder-decoder-based segmentation models. Thus, the results of these methods using SAM as the segmentation model should also be reported and compared. This would help evaulate whether the effectiveness of the proposed model is solely due to SAM's larger number of parameters.\n\n3. The writing structure of the paper is somewhat unclear, making it a little difficult to read. For example, the inference method is illustrated in Section 3.1, but the training method is introduced in Section 3.4. It is recommended to create a section titled “Training and Inference,” which contains two subsections that respectively introduce the training and inference methods.\n\nMinor Problem:\n\n1. In Line 169, `Previous research indicates that...' should have corresponding citations added.\n\n[a] A Probabilistic U-Net for Segmentation of Ambiguous Images\n\n[b] MODELING MULTIMODAL ALEATORIC UNCERTAINTY IN SEGMENTATION WITH MIXTURE OF STOCHASTIC EXPERTS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very glad and appreciate that you had a positive initial impression. Thanks for appreciating our paper as the first work that leverages the inherent properties in vision foundation models for ambiguous image segmentation, demonstrating impressive advantages and value in practical applications. We provide pointwise responses to your concerns below.\n\n## Q1. Task setting of ambiguous segmentation\n\nYour assessment of the ambiguous segmentation task setting is entirely correct, namely, providing multiple segmentation hypotheses for ambiguous images.\n\nThe key insight of this paper stems from revealing an inherent characteristic of SAM: its non-robust output in response to input prompt perturbations. We innovatively leverage this apparent weakness, transforming it to serve the objective of ambiguous segmentation tasks. Relevant experiments are presented in Figures 3, 4 and Tables 1, 2, 3, 4, 5 in our submission.\n\nFurthermore, we discovered that this design for ambiguous segmentation tasks also incidentally improves the model's robustness to prompt perturbations. Related experiments are shown in Figure 5 of our submission.\n\n## Q2. Performance comparison with Mose\n\nThank you for your valuable suggestion! We implemented Mose [1] in conjunction with SAM as you recommended. The results of quantitative comparsion for ambiguous segmentation on the LIDC datasetare presented as below. We also include the qualitative comparison in **Figure 1 in uploaded PDF of Global Response**.\n We can see that our method still achieves superior performance compared to this approach, demonstrating the advantages of our strategy design beyond the use of SAM.\n\n\n| Method | GED ↓ | HM-IOU ↑ | D_max ↑ |\n|:--|--:|--:|--:|\n| Prob UNet | 0.324 | 0.423 | - |\n| CAR | 0.252 | 0.549 | 0.732 |\n| PixelSeg | 0.243 | 0.614 | 0.814 |\n| CIMD | 0.234 | 0.587 | - |\n| Mose | 0.234 | 0.623 | 0.702 |\n| Mose + SAM | 0.236 | 0.662 | 0.777 |\n| $\\mathcal{A}$-SAM (Ours) | **0.230** | **0.763** | **0.959** |\n\n\n[1] MODELING MULTIMODAL ALEATORIC UNCERTAINTY IN SEGMENTATION WITH MIXTURE OF STOCHASTIC EXPERTS\n\n## Q3. Writing structure\n\nThank you for your suggestion. We agree with this perspective and will reorganize the paper structure as recommended. We will create a new section titled \"Training and Inference\" in the revision.\nWe will ensure this new section clearly articulates both the training and inference processes, highlighting the connections between them while maintaining the integrity of each part. This change will facilitate readers' understanding of our approach.\nWe appreciate your valuable feedback once again, as it will significantly enhance the overall quality and clarity of our paper.\n\n## Q4. Results of original SAM without point/box/mask shift?\n\nTable 1 (Sec. 4.2) compares SAM variants adapted for ambiguous segmentation. As suggested, we also compared these with the original SAM, as shown in **Global Table 1 in Global Rebuttal**. Results show the original SAM's multi-output capability is insufficient for complex, diverse scenarios, leading to suboptimal performance.\n\n## Q5. Trained from scratch or finetuned?\n\nWe strongly agree with your assertion that fine-tuning a pre-trained SAM would be superior to training from scratch, as it would be impractical to expend enormous resources to build a large model like SAM from the ground up. Therefore, our approach indeed involves fine-tuning a pre-trained SAM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fUQM9wfPxg", "reviewer": "Reviewer_V2bw", "summary": "The paper presents a novel approach to handling the inherent ambiguities in the SAM used for image segmentation. SAM, despite its robustness, often exhibits sensitivity to slight variations in prompts and object granularity, leading to inconsistent predictions. The authors propose a new framework leveraging a conditional variational autoencoder to model these ambiguities probabilistically. This approach enables SAM to produce diverse and reasonable segmentation outputs by adapting to the inherent ambiguities in the data. The paper details extensive experiments demonstrating the effectiveness of this framework across various practical scenarios involving ambiguous segmentations.", "questions": "1.\tEquation 13 mentions learning weights to assemble multiple masks into a final output. Where are these weights predicted from? Does the method obtain multiple results through random sampling or a weighted averaging process? If it's the latter, how does it learn multiple sets of weights? If it's random, how does it correspond to the ground truth?\n\n2.\tWhat is the average inference speed for the entire dataset? What percentage of the images contain reasonable masks?\n\n3.\tCan you elaborate more on why those specific datasets were being chosen?\n\n4.\tPlease refer to the weakness section, can you be more specific on what datasets were used in Ablation and Robustness studies?", "limitations": "The authors have addressed their limitations and discussed the broader impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis work addresses a critical challenge in image segmentation, especially in medical imaging and other fields where ambiguous data is common. By turning SAM's sensitivity into an advantage, the paper contributes to the advancement of robust and adaptable segmentation models.\n\n2.\tprovides a thorough analysis of SAM's sensitivity to prompt variations and object granularity, backed by detailed experiments and statistical evaluations.\n\n3.\tThe paper is well-structured, with clear definitions and explanations of the proposed methods. The use of figures and tables enhances the understanding of the framework and its performance.", "weaknesses": "1.\tThe paper primarily tests the framework on specific medical imaging and synthetic datasets. There is a lack of diverse real-world datasets, such as those from different domains (e.g., natural scenes, industrial applications), which might exhibit different types and degrees of ambiguity.\n\n2.\tI have a concern that the framework might be overfitted to the specific characteristics of the tested datasets. This concern is evidenced by Table 6, where the \"No Prompt Ambiguity\" configuration demonstrated metrics comparable to those of A-SAM. Would it be possible that the test datasets might be biased, exhibiting little ambiguity in prompts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our paper as addressesing a critical challenge, contributing to the advancement of robust and adaptable segmentation models. We provide pointwise responses to your concerns below.\n\n## Q1. Applicability to real-world non-synthetic and non-medical datasets\n\n As shown in Fig. 3 & Tab. 1 in initial submission, we evaluated our method on the Sim10k dataset, featuring synthetic non-medical street scenes with ambiguous semantics. This challenging context demonstrates our approach's broader applicability beyond medical imaging.\n\nFollowing your suggestion, we further conducted experiments on KINS dataset for instance segmentation in real-world street scenes. More details and results are shown in Gloabl Table 2 and Global Response 2. Our method performs competitively on KINS, still achieving the superior performance. We will include the results in the revision. Thank you for your insightful suggestions!\n\n## Q2. In Table 6, the performance of \"No Prompt Ambiguity\" is comparable to the full model?\n\nWe respectfully point out that this is a factual error based on the empirical evidence presented in our paper. In Table 6, the results for the \"No Prompt Ambiguity\" variant are significantly lower across all metrics compared to our full model $\\mathcal{A}-SAM (Ours)$, clearly demonstrating the necessity and effectiveness of all components in our proposed design. For example, GED↓: 0.308 vs. 0.228, HM-IOU↑: 0.674 vs. 0.717, etc. These substantial differences consistently observed across various performance indicators provide strong support for the critical role of each element in our model, including the prompt ambiguity mechanism.\n\n## Q3. Where are the ensembling weights in Eq. 13 obtained?\n\nThe original SAM outputs multiple predictions for each prompt to address ambiguity. We manually design an ensemble weight for each prediction and make this weight vector learnable. This approach allows our model to dynamically adjust the importance of each prediction, optimizing the combination of multiple outputs to produce more accurate final segmentations for ambiguous cases.\n\n## Q4. Inference speed\n\nThank you for your valuable suggestion! We've added the inference speed for different methods, as shown below. We find that our method achieves better performance while using less or comparable inference time, demonstrating the superiority and practicality of our approach. This balance of high accuracy and computational efficiency highlights the real-world applicability of our $\\mathcal{A}$-SAM model.\n\n| Method | GED ↓ | HM-IOU ↑ | D_max ↑ | Avg. Inference Time (s) |\n|:--|--:|--:|--:|--:|\n| Prob UNet | 0.324 | 0.423 | - | 0.0277 |\n| PixelSeg | 0.243 | 0.614 | 0.814 | 3.6093 |\n| Mose | 0.234 | 0.623 | 0.702 | 0.0955 |\n| $\\mathcal{A}$-SAM (Ours) | **0.230** | **0.763** | **0.959** | 0.0847 |\n\n## Q5. What percentage of the images contain reasonable masks? \n\nThese diverse ambiguous masks are obtained from expert multi-annotations provided with the established datasets. Currently, there are no existing standards or methods to evaluate their reasonableness. This lack of standardized evaluation metrics for ambiguous annotations is a recognized challenge in the field. Exploring methods to assess the validity and quality of ambiguous masks could indeed be valuable future work. We appreciate your insightful suggestion, as it highlights an important area for further investigation in ambiguous image segmentation.\n\n## Q6. Why select these datasets?\n\nFollowing the current common setting established in existing wisedom [1-4], we selected datasets specifically designed for evaluating ambiguous segmentation tasks. This approach ensures our work aligns with the standard practices in the field and allows for meaningful comparisons with existing methods. These datasets are widely recognized in the research community for their ability to challenge models with ambiguous segmentation scenarios.\n\n[1] A probabilistic u-net for segmentation of ambiguous images.\n\n[2] A hierarchical probabilistic u-net for modeling multi-scale ambiguities.\n\n[3] Phiseg: Capturing uncertainty in medical image segmentation.\n\n[4] Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty.\n\n## Q7. What datasets used in Ablation and Robustness studies?\n\nAblation and Robustness studies are conducted on the LIDC dataset. This dataset was chosen for its comprehensive collection of lung CT scans, each annotated by multiple expert radiologists, providing a rich source of ambiguous segmentations. These studies evaluate the contribution of each model component and assess performance stability under various conditions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response 1. Results of original SAM

As suggested by **Reviewer YPmC** and **Reviewer 2dZP**, we have added the results of the original SAM for comparison. As shown in the figure below, SAM (point) and SAM (box) represent the results of the original SAM obtained using different prompts. It can be seen that, compared to these added original SAM methods, our $\mathcal{A}-SAM$ still exhibits superior results.

| Global Table 1. Comparison with different SAM variants for ambiguous segmentation. |
|:-----------------------------------------------------------------------------|

| Metric | GED↓ | HM-IOU↑ | D_max↑ | D_mean↑ | GED↓ | HM-IOU↑ | D_max↑ | D_mean↑ |
|:--|--:|--:|--:|--:|--:|--:|--:|--:|
| Method | LIDC | | | | BRATS | | | |
| SAM (Point) | 0.385 | 0.347 | 0.664 | 0.335 | 0.274 | 0.167 | 0.341 | 0.225 |
| SAM (Box) | 0.376 | 0.372 | 0.695 | 0.248 | 0.248 | 0.235 | 0.368 | 0.238 |
| SAM w/ Point shift | 0.377 | 0.365 | 0.650 | 0.337 | 0.252 | 0.169 | 0.334 | 0.238 |
| SAM w/ Box shift | 0.361 | 0.380 | 0.673 | 0.253 | 0.239 | 0.242 | 0.344 | 0.246 |
| $\mathcal{A}$-SAM (Ours) | **0.228** | **0.717** | **0.948** | **0.356** | **0.193** |**0.610**| **0.864** | **0.423** |
| Method | ISBI | | | | Sim10K | | | |
| SAM (Point) | 0.538 | 0.793 | 0.891 | 0.709 | 0.294 | 0.162 | 0.249 | 0.197 |
| SAM (Box) | 0.526 | 0.806 | 0.899 | 0.713 | 0.312 | 0.176 | 0.261 | 0.208 |
| SAM w/ Point shift | 0.513 | 0.782 | 0.886 | 0.681 | 0.265 | 0.155 | 0.229 | 0.189 |
| SAM w/ Box shift | 0.491 | 0.792 | 0.896 | 0.685 | 0.255 | 0.160 | 0.239 | 0.199 |
| $\mathcal{A}$-SAM (Ours) | **0.276** | **0.835** | **0.926** | **0.904** | **0.233** | **0.637** | **0.851** | **0.327** |



## Global Response 2. Further results on real-world non-synthetic non-medical dataset.


We appreciate the suggestions from  **Reviewer vXP5** and **Reviewer YPmC** to evaluate our method on a real-world non-synthetic non-medical dataset. To address this, we conducted additional experiments on the KINS dataset [1], which is specifically designed for amodal instance segmentation. KINS is derived from the KITTI dataset [2] and includes instance-level semantic annotations. The dataset comprises 7,474 training images and 7,517 testing images across seven object categories.
We followed the same data processing method as in our initial submission to create ambiguous image-label pairs, selecting images with pixel overlap between two instances and creating three potential masks (instance 1, instance 2, and their union). We then evaluated our approach on KINS using the same metrics. The results are presented in the table below:

| Global Table 2. Comparison Results on A  real-world non-synthetic non-medical dataset, KINS. |
|:-----------------------------------------------------------------------------|

| KINS | GED↓ | HM-IOU↑ | D_max↑ | D_mean↑ |
|----------------------|-------|-------|-------|-------|
| SegGPT w/ Point shift| 0.427 | 0.584 | 0.691 | 0.396 |
| SegGPT w/ Box shift  | 0.385 | 0.640 | 0.758 | 0.472 |
| SEEM w/ Mask shift   | 0.362 | 0.695 | 0.812 | 0.518 |
| SAM w/ Point shift   | 0.340 | 0.612 | 0.735 | 0.445 |
| SAM w/ Box shift     | 0.254 | 0.482 | 0.587 | 0.331 |
| $\mathcal{A}$-SAM (Ours)| **0.237** | **0.633** | **0.839** | **0.445** |


These results demonstrate that our method performs competitively on the KINS dataset, a real-world non-synthetic non-medical dataset. Notably, our method achieves the best performance in terms of GED and D_{max} metrics, while maintaining comparable results with other top-performing methods in HM-IOU and D_{mean} metrics.


[1] Amodal instance segmentation with kins dataset, CVPR, 2019

[2] Vision meets robotics: The kitti dataset, The International Journal of Robotics Research


## Notes on added Figures in the uploaded PDF
As suggested by **Reviewer 2dZP**, we include the visualized comparison between the proposed method  with the Mose+SAM (equipping Mose [3] with SAM backbone) as **Figure 1 in the uploaded PDF**. We can see that the compared to Mose+SAM, the segmentations engendered by our proposed A-SAM preserve a higher degree of exact object detail, particularly boundary details, and provide a distinctive visual representation of potential diversity.

[3] Modeling multimodal aleatoric uncertainty in segmentation with mixture of stochastic expert, in ICLR, 2023

Meta-review: This paper received generally positive feedback, including one weak accept and three borderline accepts. The reviewers commend the authors for addressing the ambiguous image segmentation problem with clear and novel methodologies, as well as for the extensive evaluation and ablation studies. The rebuttal effectively clarified many concerns raised in the initial reviews, providing detailed explanations and solid evidence that convinced the reviewers to increase their ratings. The AC concurs with the reviewers and recommends accepting this paper.

However, as noted by Reviewers vXP5 and YPmC, the revised camera-ready version requires further improvements based on the detailed responses in the rebuttal and the discussion. These enhancements will make the paper more robust. Additionally, the inclusion of more discussion and new results in non-medical tasks is necessary in the camera-ready version to further verify its broader applications. We look forward to seeing this work presented at NeurIPS.
Decision: Accept (poster)