Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM

Authors: chenxin  li
yuzhi  huang
wuyang  li
hengyu  liu
xinyu  liu
qing  xu
zhen  chen
yue  huang
yixuan  yuan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-26 10:32:15

Abstract: As the vision foundation models like the Segment Anything Model (SAM) demonstrate potent universality, they also present challenges in giving ambiguous and uncertain predictions. Significant variations in the model output and granularity can occur with simply subtle changes in the prompt, contradicting the consensus requirement for the robustness of a model. While some established works have been dedicated to stabilizing and fortifying the prediction of SAM, this paper takes a unique path to explore how this flaw can be inverted into an advantage when modeling inherently ambiguous data distributions. We introduce an optimization framework based on a conditional variational autoencoder, which jointly models the prompt and the granularity of the object with a latent probability distribution. This approach enables the model to adaptively perceive and represent the real ambiguous label distribution, taming SAM to produce a series of diverse, convincing, and reasonable segmentation outputs controllably. Extensive experiments on several practical deployment scenarios involving ambiguity demonstrates the exceptional performance of our framework. Project page: \url{https://a-sa-m.github.io/}.

Main paper: 
 introduction: 
 The advent of Visual Foundational Models (VFMs) such as the Segment Anything Model (SAM) [22] has been unprecedented, largely due to the availability of vast datasets and computational resources. These models have exhibited remarkable generalization capabilities in zero-shot scenarios and the capacity to interact with human feedback. SAM, in particular, employs a specialized data engine to manage 11 million image masks, using a unique prompt-based segmentation framework to generate accurate masks for any object within a visual context, largely extending the capacity and generality of segmentation [27] . Such successes have been widely extended to various domains, such as medical imaging analysis [8, 35, 28] , remote sensing [48] , etc. However, it has been observed that SAM suffers from severe predictive ambiguity to segment desired concepts [22] in practical scenarios. To uncover the factual basis, we delve into this ambiguity and detail it into two flaws according to experimental insight. Specifically, the first flaw lies in that SAM prediction is sensitive to slightly different prompt variants. As shown in Fig. 1 (a), we ground SAM into a realistic clinical scenario to segment the lesion in CT images. Even though three medical experts uniformly give rational box prompts covering the lesion, SAM, unfortunately, makes enormous differences among them, even including one wrong case. Further, we provide detailed statistics regarding IoU over the small perturbation (only 5 pixels) of box prompts, as shown in Fig. (b). We can observe a significant IoU fluctuation over small prompt differences, which reveals that SAM prediction is highly sensitive to the prompt variants caused by such small perturbation 3 .1The second flaw lies in the susceptibility of SAM output to the inherent structural granularity of the object. Despite the fact that VFMs like SAM gain generalizable knowledge and abilities from extensive datasets, they frequently forfeit the capacity to segment specific visual concepts as they are class agnostic, making the model unable to discern the difference between objects with different levels of semantic granularity. Consequently, for targets that are challenging to define, particularly those with rich internal hierarchical granularity, they are inclined to produce multiple candidate results at different granularities and amalgamate them, rather than directly outputting a definitive result. As depicted in Fig. 1 (c), we also discern that the multiple candidate results captured by SAM often exhibit significant differences, and the segmentation precision of outputs at different granularities diverges greatly when compared to the final integrated SAM output that incorporates multiple candidates.Nonetheless, every coin has two sides. While these observations reveal SAM's faults regarding the output sensitivity, we explore a different perspective: could this sensitivity flaw become an advantage in other cases, such as ambiguous segmentation, which requires the model to learn from a set of ambiguous object annotations caused by imaging noise, ambiguous tissue boundaries, and different annotator preferences? Specifically, we are particularly interested in the following: ➊ Given the segmentation result's sensitivity to prompts, can we probabilistically model this prompt variation to tame prompt-sensitive SAM to controllably yield multiple likely results close to the actual fuzzy distribution? ➋ Considering the segmentation result's sensitivity to object structure, can we probabilistically model this granularity variation to tame granularity-sensitive SAM to controllably yield multiple likely results close to the actual ambiguous distribution?Driven by these questions, we propose an innovative strategy, which flips the inherent categoryagnostic ambiguity induced in SAM into a controlled ability to generate a range of feasible results for ambiguous segmentation tasks. Specifically, to simulate segmentation ambiguity under different prompts, we introduce context-aware prompt ambiguity modelling. This method probabilistically models the uncertainty of the prompts inputted into SAM using a latent learnable distribution, which adaptively perceives the specific ambiguity of different contexts. Furthermore, to simulate ambiguity caused by complex object structures at different granularities, we introduce granularity-aware object ambiguity modelling. This method introduces and enhances the visual ambiguity of objects at different granularities into SAM's original image embedding via a learnable embedding distribution. While establishing these two levels of ambiguity modelling, we introduce an efficient optimization strategy based on posterior constraints, allowing the model to mimic models that can perceive the actual ambiguous distribution. Our contributions can be summarized as follows:• We explore the inherent ambiguity in SAM to flip this commonly seen disadvantage in deterministic segmentation tasks into an advantage for more practical ambiguous segmentation tasks that allow multiple possible outputs.• We propose a A-SAM framework that employs a learnable latent distribution to encapsulate the ambiguity at two strata, from prompts and object granularity.• We introduce an optimization architecture based on variational autoencoders, which effectively represents ambiguity by constraining the sample embedding to align with those from a series of practically feasible annotations.• Rigurous benchmarked experiments across a wide range of potential scenarios demonstrate that our method produces more accurate, diverse, and reasonable segmentation outputs. 
related work: 
 Prompting Foundational Models for Segmentation. There has been a surge in the advancement of large-scale vision models for image segmentation, drawing inspiration from language foundation models [62, 4, 25, 39] . Segmentation Foundation Models (SFMs), such as the Segment Anything Model (SAM) [22] and SEEM [66] , have delivered significant segmentation results across various downstream datasets [17, 38, 54] . SAM, leveraging a data engine incorporating model-in-the-loop annotation, learns a promotable segmentation framework that generalizes to downstream scenarios in a zero-shot manner. Meanwhile, other models like Painter [56] and SegGPT [57] introduce a robust in-context learning paradigm that enables segmenting any images given an image-mask prompt. On the contrary, SEEM [66] presents a general segmentation model prompted by multi-modal references, such as language and audio, thereby incorporating a wide range of semantic knowledge. These advancements in SFMs, driven by promptable segmentation design, involve two types of prompts: semantic prompts (e.g., free-form texts) and spatial prompts (e.g., points or bounding boxes) [22, 57, 40, 41, 43, 26] .Recently, the practice of adapting vision foundation models such as SAM [22] for application in medical image segmentation is garnering increasing interest [46, 10, 12, 44, 33] . A prevalent and costeffective approach involves adapter techniques, which necessitate the inclusion of bottleneck modules with a finite number of parameters within the model. By fine-tuning these diminutive adapters, SAM can bridge the domain discrepancy between medical and natural images while preserving stellar performance. For instance, models like MSA [58] , SAM-Med2D [9] , and SAM-adapter [6] utilize adapter strategies to transfigure SAM for medical imaging, thereby achieving significant segmentation outcomes. Despite these advancements, acquiring suitable prompts for SFMs remains largely under-explored. This work aims to explore generating effective prompts for SAM, focusing on harnessing pre-training knowledge to complete ambiguous image segmentation.Ambiguous Image Segmentation. Ambiguous image segmentation aims to model a range of, rather than single, segmentation labels [31, 65, 45] . A wealth of existing research has proposed various techniques to quantify uncertainty. Initial research focused on enhancing a traditional U-Net [51, 16, 29, 5] with a probabilistic component to generate multiple predictions for the same image. This was typically achieved by incorporating a conditional variational autoencoder (cVAE) [53] , with the low-dimensional latent space encoding potential segmentation variations. Subsequent work further extended this setup to a hierarchical variant [3, 24, 64, 15] . Other research has utilized normalizing flows to allow for distribution in the cVAE [52, 55] to represent a discrete latent space [49] or incorporated variational dropout and directly used inter-grader variability as a training target. Several other methods [34] do not rely on the Probabilistic U-Net [47, 21, 32, 11, 60] . Monteiro et al. [47] proposed a network utilizing a low-rank multivariate normal distribution to model the logit distribution. Kassapis et al. [21] leveraged adversarial training to learn potential label maps based on the logits of a trained segmentation network. Zhang et al. [63] employed an autoregressive PixelCNN to model the conditional distribution between pixels [37, 36] . Finally, Gao et al. [13] used a mixture of stochastic experts, where each expert network estimates a mode of uncertainty, and a gating network predicts the probabilities of an input image being segmented by one of the experts. Unlike previous efforts [1, 42, 59, 7] , our approach signifies the first exploration of leveraging the inherent properties in vision foundation models for ambiguous image segmentation. Given an image I and a set of user-given prompts P , which could be a point, a box, or a rough mask, the Segment Anything Model (SAM) [22] employs a vision transformer-based image encoder Enc I to extract salient image feature F I and deploy a prompt encoder Enc P with length k to encode prompt embeddings T P , which are denoted as follows,EQUATIONwhere F I ∈ R h×w×c and T P ∈ R k×c , where the resolution of the image feature map is represented by h, w, and the feature dimension is denoted by c. Subsequently, the encoded image and prompts are introduced into the decoder Dec M for interaction based on attention mechanisms. SAM constructs the decoder's input tokens by concatenating several learnable mask tokens T M as prefixes to the prompt tokens T P . These mask tokens are accountable for generating the mask outputEQUATIONA-SAM: Lifting SAM to Distributional Space. Unlike the one-to-one deterministic mapping in SAM, we formulate a probabilistic latent distribution to enable the one-to-many ambiguous mapping named A-SAM, with each observation being a sample from this hidden distribution. To this end, the prompt and image embedding can be probabilistically formulated as a distribution:EQUATIONwhere P P and P I denote a latent distribution for prompt embedding and image embedding, respectively. TP and FI denotes a prompt at one sampling from the defined latent distribution at one time. Formally, by implementing multiple rounds of sampling, we can construct a distributional mapping of segmentation outputs with respect to their prompts, formulated as the format of expectation, M = Dec M FI , TP , T M , st. TP ∼ P P (Θ), FI ∼ P I (Φ),where M denotes the SAM output corresponding to one prompt sampling, which can also be interpreted as the sampling from a virtual distribution P M (Ω) for the segmentation results obeying parameters Ω. As a result, we can construct an optimized probability distribution TP ∼ P P (Θ) and FI ∼ P I (Φ) by narrowing the gap between M ∼ P M (Ω) and ground-truth distribution.A-SAM: Inference Stage. After the training, A-SAM can model two types of latent distribution, representing the ambiguity of the prompt variation and the varied object granularity, respectively.Based on this formulation, each latent sample drawn from the distribution represents a segmentation candidate. Concretely, to predict a set of m segmentations, we apply the network m times to the same input image. In each iteration i ∈ {1, . . . , m}, we draw a random sample regarding prompt embedding TP ∈ R N P and image embedding FI ∈ R N I respectively from P P (Θ) and P I (Φ). The final prediction maps M ∼ P M (Ω) can be obtained by Eq. 4. In what follows, we will primarily focus on the methodology of training our overall ambiguous segmentation framework.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of the work are discussed by authors at the end of the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
context-aware prompt ambiguity modeling: 
 Distributional Prompt Representation. To model the distribution of prompt embedding, it is imperative to estimate the parameters Θ of this distribution. We adopt an axisymmetric Gaussian distribution to characterize the prompt embedding, which is dictated by two crucial parameters, including mean µ and standard deviation σ. Then, we can sample a prompt embedding from the given Gaussian distribution, which is shown asEQUATIONwhere µ P and σ P denotes the parameters characterized for prompt P . µ P and σ P respectively denote the mean and standard deviation of the Axis Gaussian Distribution generated by the network, where µ P , σ P ∈ R N P . This simple yet effective formulation enables the discrete prompt to be continuously represented in the probabilistic latent space, making the uncertainty estimation available.Context-aware Prompt Embedding Generation. To parameterize the latent prompt distribution, we propose a prompt generation network (PGN) to effectively model the aforementioned Gaussian related to prompt embedding. This network is simply designed to include several convolution blocks.Considering the variation in salient regions within the image context, the required prompt positions and sizes should also be varied. Therefore, we incorporate image context as prior knowledge into PGN during the forward inference process. By integrating this prior knowledge, the network can customize a unique prompt-associated axial Gaussian distribution for each image I, thereby achieving adaptive and infinite sampling in the latent prompt distribution:EQUATIONwhere the parameters of the prompt generation network F P GN are modeled by the parameters Θ of our desired distribution, as each set of generated mean and variance uniquely specifies a distribution. Previous research indicates that allowing the model to conditionally perceive the ground truth label distribution during the training phase enhances training stability for such tasks that exhibit significant uncertainty. Thus, a posterior version for prompt generation network F post P GN , parameterized by Θ T , is further introduced during training, learning to generate the effective distribution for prompt embedding when accessing the ground-truth label distribution:EQUATIONWe only employ this posterior network during training and guide the standard network, which cannot perceive the true labels during testing, to achieve viable performance via a KL loss.EQUATIONDistributional Object Representation. The model integrates object ambiguity from a probabilistic perspective, enhancing its ability to solve problems in innovative ways. We instantiate various segmentation labels to represent ambiguity levels and incorporate them as priors in visual feature extraction, influencing object-related feature modeling for the input image I,EQUATIONwhere µ I and σ I respectively denote the mean and standard deviation of the Axis Gaussian Distribution generated by the network, where µ I , σ I ∈ R N I . This approach enhances our understanding and depiction of object diversity and uncertainty in a more profound and lucid manner, enabling better adaptability and handling of variation and uncertainty.Granularity-aware Image Embedding Generation. SAM generates multiple candidate segmentation masks for the same object at different granularities and levels, demonstrating the inherent ambiguity of SAM related to object granularity. This inspires us to leverage this aspect to enhance the model's perception of ambiguous objects. We subsequently introduce an image generation network (IGN) for object embedding to model this distribution:EQUATIONwhere the parameters of the image generation network F IGN are modeled by the parameters Φ of our desired distribution, as each set of generated mean and variance uniquely specifies a distribution. Similar to the previous introduction of a posterior network to enhance learning in modeling prompt ambiguity, we introduce a posterior version of IGN, denoted as F post IGN , for perceptible labels.EQUATIONWe only employ this posterior network during training and guide the standard network, which cannot perceive the true labels during testing, to achieve viable performance via a KL loss.EQUATIONIn line with current SAM techniques that generate segmentation masks for the same object at different granularity levels, the present approach utilizes this feature to enhance the model's capture of multi-level object concepts through a common ensemble strategy. Specifically, given the multiple candidate outputs from SAM, represented as { M 1 , M 2 ..., M n }, where n is the number of scales. By introducing a set of learnable mask weights W = {w 1 , w 2 ..., w n } ∈ R n , the final mask output can be fine-tuned and obtained through a weighted sum calculation:EQUATIONwhere w 1 , w 2 , ..., w n are initialized to 1 n and subsequently fine-tuned to enable the model effectively being aware of the object scales. By adaptively integrating multiple scale masks, the model's perception and modeling capabilities for complex target diversity are further enhanced.When the representation from SAM is combined with the ground-truth segmentation GT from the training samples, a guide-providing teacher prediction segmentation M T is created. A crossentropy loss CE(•, •) is employed to penalize the discrepancies between the distribution of M T and GT , i.e., P M (Ω) and P GT , where the distribution of GT is a constant value that does not need to be parameterised, as: L Seg = CE(GT, M ). Additionally, the KL losses introduced to regularize training in prompt ambiguity and image ambiguity, respectively, in Eq. 8 and Eq. 12, are amalgamated into a weighted sum with weight coefficient of α P and α I :EQUATION)The model is trained from scratch using randomly initialized weights. Implementation Details. For the LIDC dataset, we use the included four expert annotations to represent different ambiguous segmentation labels. In the case of the BraTS dataset, we amalgamate annotations from different categories into a binary mask, creating multiple segmentation masks to mimic real-world ambiguous segmentation scenarios. For the ISBI dataset, we use the single label provided. All three datasets are optimized using the Adam optimizer, with a learning rate of 1e-4, over 100 epochs. For the SIM 10k dataset, we select images where pixels from two instances overlap, creating three potential masks. Optimization for this dataset is carried out with Adam optimizer over 500 epochs, with a learning rate of 1e-4. The trade-off coefficients are set as α P = α I = 1.Evaluation Metrics. Four metrics are used for evaluation: Generalized Energy Distance (GED), Hungarian-Matched Intersection over Union (HM-IoU), Maximum Dice Matching (D max ), and Average Dice Matching (D mean ). GED is a metric used in ambiguous image segmentation tasks that compares the distribution of segmentations. It leverages the distance between observations, where lower energy signifies a better match between prediction and the ground truth. HM-IoU calculates the optimal match of Intersection over Union (IoU) between annotations and predictions using Hungarian algorithm, providing an accurate representation of sample fidelity. D max and D mean represent the best and average Dice scores between each prediction result and each ground truth, respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: For each theoretical result, the paper provides the full set of assumptions and a complete (and correct) proof. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
comparison to prompted segmentation models: 
 Tab. 1 presents the quantitative results on four datasets, offering a comparison with the current state-of-the-art prompting-based segmentation models adapted for ambiguous segmentation tasks. Specifically, we have adapted SegGPT [57] , a SAM-like prompt-based segmentation approach that We have also adapted SEEM [66] , a segmentation model that follows the in-context learning paradigm. Given a reference image and a reference segmentation mask, it segments the object in the query image. We select an image with multiple mask labels and apply random perturbations on the reference segmentation mask by shifting [-8,8] pixels left, right, up, or down, resulting in multiple different results on the query image. Comparing with SAM, we employ the same paradigm for prompt acquisition and perturbation as SegGPT. We find that A-SAM outperforms the state-of-the-art segmentation foundational models based on point or box in terms of diversity and accuracy. In addition, A-SAM surpasses SEEM, a segmentation paradigm directly based on masks, in all aspects. This indicates that our designed strategy accurately captures the ambiguous attributes present in different images and objects, effectively achieving a balance between diversity and accuracy in ambiguous segmentation tasks. Fig. 3 further illustrates the qualitative results of our method in comparison with existing techniques. Our proposed A-SAM yields segmentations that preserve a greater degree of accurate object detail, particularly boundary specifics, and offers an exceptional visual representation of potential diversity, as compared to other technologies. The numerical outcomes across the four datasets are delineated in Tab. 2, 3, 4, and 5, where we draw comparisons with contemporary leading-edge classical ambiguous segmentation methodologies. Precisely, the comparative method results on the LIDC and ISBI datasets are direct quotations from their respective papers, while outcomes on other datasets are predicated on our reimple-mentation of their official code. The methods compared encompass recent ambiguous segmentation methodologies that amalgamate a conditional variational autoencoder and UNet: Probabilistic U-Net [23] , Hierarchical Probabilistic U-Net (HProb UNet) [24] , PhiSeg [3] , Stochastic Segmentation Networks (SSN) [47] , PixelSeg [63] , and the ambiguous segmentation endeavor of Calibrated Adversarial Refinement (CAR) [20] and Collective Intelligence Medical Diffusion (CIMD) [50] that integrate generative models. It also includes ensemble-based techniques utilizing a Mixture-ofexpert like the Mix of Stochastic Experts (Mose) [13] . We observe that the A-SAM transcends state-of-the-art methodologies that amalgamate a conditional variational autoencoder and UNet in terms of diversity and precision. Additionally, the A-SAM outstrips segmentation paradigms based on generative models or ensembles in all dimensions. This suggests that our strategy accurately apprehends the ambiguous characteristics inherent in varied images and objects, effectively attaining equilibrium between diversity and precision in the ambiguous segmentation task. Fig. 4 further elucidates the qualitative outcomes of our methodology juxtaposed with extant technologies. In contrast to other technologies, the segmentations engendered by our proposed A-SAM preserve a higher degree of exact object detail, particularly boundary details, and provide a distinctive visual representation of potential diversity.Table 6 : Ablation study on the proposed key components.No Ambiguity either eradicates the ambiguity related to the object or the prompt, that is, it alters its object embedding or prompt embedding to a regular deterministic rather than an ambiguous characteristic.No Posterior Distillation eliminates a process of network training guided by a teacher network that can perceive the actual labels. We discover that when any component is excised, the performance correspondingly deteriorates, which underscores the effectiveness of our proposed several strategies. Robustness Analysis. Fig. 5 reports the IoU performance of A-SAM under a variety of instantaneous perturbations. The blue and red solid lines respectively illustrate the performance changes of the SAM model and our method when prompted by different disturbances, while the dashed lines depict the performance of both models under standard prompts, serving as an upper bound. We selected both light and severe degrees of perturbation. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The pipeline of the methods and the details of experiments are presented with corresponding reproducible credentials. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 The continued evolution of vision foundation models like the Segment Anything Model (SAM) demonstrates impactful universality, while also posing challenges in producing ambiguous and uncertain predictions. Minor changes in the prompt can cause significant variations in the model's output, challenging its required robustness. While many works aim to stabilize SAM's prediction capabilities, this paper uniquely explores leveraging this perceived flaw to advantageously model inherently ambiguous data distributions. We introduce an innovative optimization framework grounded in a conditional variational autoencoder, which cohesively models the prompt and the object granularity with a latent probability distribution. This approach endows the model with the capacity to adaptively perceive and represent the genuine ambiguous label distribution, thereby enabling SAM to generate a controlled series of diverse, persuasive, and reasonable segmentation outputs. Our comprehensive experiments across multiple practical deployment scenarios involving ambiguity underscore the exceptional performance of our framework, thereby illuminating the need for increased focus on addressing related challenges and opportunities.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All utilized data are sourced from open-access platforms. The code, which will be made publicly available, is uploaded as a zip file.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The pipeline of the methods and the details of experiments are presented with corresponding reproducible credentials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The results contain the standard deviation of the results over several random runs. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original owners of assets, including data and models, used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The new assets introduced in the paper are well documented and provided alongside the assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The following contents are provided in the supplements:• Limitation and Border Impact.• More experimental details (Sec. 4.1 in main paper).• Details of network architecture of A-SAM.• More visualization about our experiments. (Sec. 4.2 and Sec. 4.3 in the main paper).Limitation. The current approach is constrained by the uncontrolled and unquantifiable nature of uncertainty. This limitation means that the accuracy of handling uncertainty varies across different scenarios. Further systematic analysis is required to comprehend the underlying factors that result in some scenarios being more manageable than others in terms of uncertainty.Broader Impacts. Ambiguous segmentation and uncertainty handling are essential in fields such as image processing and medical diagnostics. Fuzzy segmentation improves our understanding of complex or unclear image content. Proper uncertainty management can enhance prediction accuracy and decision-making, especially in medical diagnostics, aiding in precise disease diagnosis and treatment planning. Ensuring image processing safety is also critical for user privacy and data security, thereby building trust and satisfaction.Dataset. Four datasets are used for comparison. LIDC-IDRI [2] is a dataset for lung lesion segmentation, which encompasses a voluminous collection of lung computed tomography scans from 1010 distinct subjects, with manual annotations provided by a panel composed of four domain experts. A diversified panel of 12 radiologists leveraged their expertise to provide annotation masks for the dataset, a characteristic that allows the dataset to reflect the typical ambiguity frequently encountered in CT imaging, thereby ensuring comprehensive, accurate annotations that represent a broad range of expert opinions. The resolution of all images is 128×128. BraTS 2017 [18] is a dataset for 3D brain tumor segmentation, which consists of 285 cases of 3D MRI images, each image comprising 155 slices. Each slice exhibits four modes (T1, T1ce, T2, and Flair) and is meticulously annotated by professional radiologists into four classes: Background (BG), Non-enhancing/Necrotic Tumor Core (NET), Edema (OD), and Enhancing Tumor Core (ET). The resolution of all images is 240×240. ISBI 2016 [14] is a dermoscopy dataset containing 900 dermoscopic images for training and 379 images for testing. Each image is 8-bit RGB and is annotated by an expert with the lesion area. To keep each image at the same scale, we follow [61] to resize the images to 256×192 and pad the top and bottom with 32 pixels, respectively, to get 256×256 images. SIM 10k [19] consists of 10,000 images that are rendered by the gaming engine Grand Theft Auto. In SIM 10k, bounding boxes of 58,701 cars are provided in the 10,000 training images. All images are used in the training.Implementation Details. For LIDC, we directly employ the four expert annotations included in the dataset to represent four different ambiguous segmentation labels [2] . For BraTS, we overlay and amalgamate annotations from disparate categories, subsequently transforming the outcome into a binary mask that comprises solely the foreground and background [18] . This process is geared towards generating multiple segmentation masks to mimic real-world ambiguous segmentation scenarios, thereby augmenting the rigor and reliability of the experiment. For ISBI, we directly use the single label included in the dataset [14] . For the aforementioned three datasets, we carry out optimization using the Adam optimizer, with a learning rate of 1e-4, over 100 epochs. For the SIM 10k dataset, we contemplate a practical overlap setting. Specifically, we select images in which pixels from two instances within a frame overlap, thereafter creating three potential masks from the two overlapping instances. These masks could represent the first object, the second object, or the union of both. For SIM 10k [19] , we carry out optimization using the Adam optimizer with a learning rate of 1e-4, over 500 epochs. The trade-off coefficients in Eq. 14 are set as αP = αI = 1.Evaluation Metrics. Four metrics are used for strict evaluation. Generalized Energy Distance (GED) is a commonly used metric in ambiguous image segmentation tasks that leverages distance between observations by comparing the distribution of segmentations [23] , asD 2 GED (Pgt, Pout) = 2E[d(S, Y )] -E[d(S, S ′ )] - E[d(Y, Y ′ )],where d corresponds to the distance measure d(x, y) = 1 -IoU (x, y), Y and Y ′ are independent samples of Pgt and S and S ′ are sampled from Pout. Lower energy indicates better agreement between prediction and the ground truth distribution of segmentations. Hungarian-Matched Intersection over Union (HM-IoU) is used by calculating the optimal match of Intersection over Union (IoU) between annotations and predictions, which is searched by Hungarian algorithm. This metric offers a more accurate representation of Justification: The abstract and introduction sections offer a comprehensive discussion of the manuscript's context, intuition, and ambitions, as well as its contributions.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In practice, such perturbation commonly exists and cannot be controlled by users, even for medical experts. 


Decision: This paper received generally positive feedback, including one weak accept and three borderline accepts. The reviewers commend the authors for addressing the ambiguous image segmentation problem with clear and novel methodologies, as well as for the extensive evaluation and ablation studies. The rebuttal effectively clarified many concerns raised in the initial reviews, providing detailed explanations and solid evidence that convinced the reviewers to increase their ratings. The AC concurs with the reviewers and recommends accepting this paper.

However, as noted by Reviewers vXP5 and YPmC, the revised camera-ready version requires further improvements based on the detailed responses in the rebuttal and the discussion. These enhancements will make the paper more robust. Additionally, the inclusion of more discussion and new results in non-medical tasks is necessary in the camera-ready version to further verify its broader applications. We look forward to seeing this work presented at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper aims to convert the flaws in the vision foundation model (e.g., SAM) into advantages for ambiguous object segmentation. To this end, the authors propose a novel framework that employs latent distribution and an optimization architecture. The authors validated the performance of the proposed methods through comprehensive experiments.
strength_weakness: strength_weakness: 1. The explanations are unclear and hard to follow. Specifically, it needs further explanation of how to extract the mean and standard deviation from the convolution blocks and how to utilize the ground truth labels in the posterior version of the prompt generation network.
2. Some symbols are used without explanation (e.g., Θ, Φ, N_i, N_p).
3. Missing reference: Previous research at line 169.
4. Since this paper focuses on clinical scenarios for ambiguous object segmentation, it seems unfair to compare the performance without including existing medical segmentation methods such as OM-Net [1], DC-UNet [2], and CE-Net [3].

[1] https://arxiv.org/pdf/1906.01796v2
[2] https://arxiv.org/pdf/2006.00414v1
[3] https://arxiv.org/pdf/1903.02740v1
questions: questions: 1. What is the difference between PGN and posterior PGN?
limitations: limitations: The authors address limitations of this work and broader impact properly.
paper_summary: paper_summary: This paper builds a framework for amigous object segmentation on top of SAM prompted with bounding boxes, which is known to be sensitive to small prompt changes. 

The framework is based on a VAE, and the main idea is to jointly model the prompt and the object granularity with a latent probability distribution to gain more control over SAM’s output. In practice, the prompt  embeddings  and image embeddings (controlling granularity) are formulated as a distribution.
 
The method is evaluated on 3 medical imaging datasets and on a synthetic driving dataset, showing superior performance over the baselines.
strength_weakness: strength_weakness: 1. The paper contains several unclear statements or missing details, which make the reproducibility of the method difficult.
2. The evaluation is carried on a niche domain (medical) or on synthetic datasets only. It is hard to judge the performance of this method in general real-world setting.
questions: questions: ## General remarks
1. Evaluation on a real-world (not synthetic) non-medical dataset would help to show the generality of the method.
2. It would help readability if it was mentioned that the evaluation metrics are defined in the appendix, also it would help to see the related references in the main paper
3. Is there some intuition/more details on why the granularity is modelled within the image embedding?

## Reproducibility
3. How were the trade-off coefficients tuned?
4. More details on how the three masks from overlapping instances on the SIM10k dataset were obtained should be provided.
5. How was the best checkpoint selected? Was there any hyper-parameter tuning?
6. What does  'achieving significant segmentation outcome‘ mean on line 97? Improvement in segmentation performance over SAM without adapter?

## Fig. 1:
7.  SAM outputs multiple predictions for a prompt, how is this handled in Fig. 1a and 1b?
8.  Medical domain is not in the training domain of SAM so higher uncertainity/instability of prediction is expected, maybe it is not the best example to showcase the behaviour.
9.  What are canonical box prompts from description of Fig. 1? Ground truth bounding boxes?
10.  I assume granularities in 1c correpsond to the three output masks of SAM, what is full granularity then?
11.  The prompt variation experiment depicted in Figure 1 includes bounding boxes that do not cover the whole region to be segmented. It is not unrealistic to control for that in real scenarios, and it would be interesting to see how the figure would change since SAM seems to be quite sensitive to whether the whole object is covered or not – making the bounding box smaller than an object impacts segmentation more than making it larger.
12.  It would be helpful to see how the experts annotate  the example 

## Fig 2: 
13.  Why is image embedding concatenated with the IGN sample, but the prompt embedding is the output of PGN directly?
14.  Incomplete description – ‚by jointly probabilities‘‘

## Add Weakness 1. – unclear statements and missing details 
15. How were the trade-off coefficients set?
16. How exactly was the three masks generated from overlapping instances on the SIM10k dataset?
17. How was the best checkpoint selected? Was any hyper-parameter tuning performed (if yes, on what data)?
18. Line 153 – parameters of axisymmetric Gauss. Distribution „including mean and std“ – the gauss. distirbution does not have any other parameters.
19. What does  'achieving significant segmentation outcome‘ mean on line 97? Improvement in segmentation performance over SAM without adapter?
20. What is meant by the 'final integrated SAM output that integrates multiple candidates‘ on line 44? The only part of SAM that integrates multiple predictions I am aware of is SamAutomaticMaskGenerator class provided by the authors (it features non-maxima suppression) but it prompts SAM with a uniform grid of points while the paper discusses bounding box prompts.
21. The explanation of GT generation for the datasets is confusing since it is incomplete in the paper, it would be nice to at least have a link to the appendix for more details.
22. On lines 38-42, it would help to see an example of such behaviour – what is meant by SAM amalgamating the candidates at different granularities? AFAIK, SAM outputs multiple predictions for each prompt specifically to deal with ambigous prompts.
23. What does 'diminutive adapters‘ on line 93 mean?
24. What is meant by encoder lenght in line 123?
 
## Add Weakness 2. – evaluation 
25. Evaluation on a real-world (not synthetic) non-medical dataset would help to show the generality of the method.
26. Why is original SAM not included in the comparison from subsection 4.2?

## Typos: 
27. Line 83 – promotable instead of promptable
limitations: limitations: Limitations are addressed in the appendix.
paper_summary: paper_summary: This paper proposes a SAM-based framework to address the ambiguous image segmentation problem. The authors present an optimization framework based on a conditional variational autoencoder, which simultaneously models the prompt and the granularity of the object using a latent probability distribution. This approach allows the model to adaptively perceive and represent the real ambiguous label distribution, enabling SAM to controllably produce a series of diverse, convincing, and reasonable segmentation outputs. Experiments on multiple datasets and metrics demonstrate the effectiveness of the method.
strength_weakness: strength_weakness: 1. The task setup of ambiguous image segmentation in this paper is somewhat confusing for me. I have read some referenced and comparative works cited in the paper, such as [a], and found that their task objective is providing multiple segmentation hypotheses for ambiguous images. However, this paper seems to focus more on increasing the accuracy and stability of the model's output when the input prompt has noise or shifts. More explanation about the task setup is needed. Accordingly, it is recommended to include a section in the main text that introduces the task setup, which can help readers who are not experts in this research area understand the paper better.

2. The comparison with conventional ambiguous segmentation models seems unfair because most of the compared methods do not use a network structure as large as SAM. Therefore, it is unclear whether the performance advantage comes from the increased number of network parameters in SAM or from the innovative designs proposed in this paper. I noticed that some of the compared methods, such as [b], can be applied with any encoder-decoder-based segmentation models. Thus, the results of these methods using SAM as the segmentation model should also be reported and compared. This would help evaulate whether the effectiveness of the proposed model is solely due to SAM's larger number of parameters.

3. The writing structure of the paper is somewhat unclear, making it a little difficult to read. For example, the inference method is illustrated in Section 3.1, but the training method is introduced in Section 3.4. It is recommended to create a section titled “Training and Inference,” which contains two subsections that respectively introduce the training and inference methods.

Minor Problem:

1. In Line 169, `Previous research indicates that...' should have corresponding citations added.

[a] A Probabilistic U-Net for Segmentation of Ambiguous Images

[b] MODELING MULTIMODAL ALEATORIC UNCERTAINTY IN SEGMENTATION WITH MIXTURE OF STOCHASTIC EXPERTS
questions: questions: 1. Will the proposed method perform better than the original SAM if there is no point/box/mask shift?

2. Why is the proposed method trained from scratch using randomly initialized weights? Would it be better to finetune from the pre-trained SAM?
limitations: limitations: Please see the weaknesses section.
paper_summary: paper_summary: The paper presents a novel approach to handling the inherent ambiguities in the SAM used for image segmentation. SAM, despite its robustness, often exhibits sensitivity to slight variations in prompts and object granularity, leading to inconsistent predictions. The authors propose a new framework leveraging a conditional variational autoencoder to model these ambiguities probabilistically. This approach enables SAM to produce diverse and reasonable segmentation outputs by adapting to the inherent ambiguities in the data. The paper details extensive experiments demonstrating the effectiveness of this framework across various practical scenarios involving ambiguous segmentations.
strength_weakness: strength_weakness: 1.	The paper primarily tests the framework on specific medical imaging and synthetic datasets. There is a lack of diverse real-world datasets, such as those from different domains (e.g., natural scenes, industrial applications), which might exhibit different types and degrees of ambiguity.

2.	I have a concern that the framework might be overfitted to the specific characteristics of the tested datasets. This concern is evidenced by Table 6, where the "No Prompt Ambiguity" configuration demonstrated metrics comparable to those of A-SAM. Would it be possible that the test datasets might be biased, exhibiting little ambiguity in prompts?
questions: questions: 1.	Equation 13 mentions learning weights to assemble multiple masks into a final output. Where are these weights predicted from? Does the method obtain multiple results through random sampling or a weighted averaging process? If it's the latter, how does it learn multiple sets of weights? If it's random, how does it correspond to the ground truth?

2.	What is the average inference speed for the entire dataset? What percentage of the images contain reasonable masks?

3.	Can you elaborate more on why those specific datasets were being chosen?

4.	Please refer to the weakness section, can you be more specific on what datasets were used in Ablation and Robustness studies?
limitations: limitations: The authors have addressed their limitations and discussed the broader impacts.

Comments: title: Further Clarification for Emphasized Issues
Further Clarification for Emphasized Issues
Thank you for your response
comment: Dear Reviewer YPmC,

We sincerely appreciate your prompt response, valuable suggestions, and raising the rating for our work. We look forward to including the suggested changes and modifications and hope the paper can inspire a broader audience thanks to your constructive feedback!

Yours,

Authors
Dear Reviewer V2bw,

We sincerely appreciate your prompt response, valuable suggestions, and rasing the rating for our work. We look forward to including the suggested changes and hope the paper can inspire a broader audience thanks to your constructive feedback!

Yours,

Authors
Thank you for the detailed response and additional evaluation - after reading it and after careful consideration, I am raising my score.

The authors have shown they are willing to work on improving the clarity of the paper and I think it can benefit the community. 

I would like to point out that because of the amount of things that were not clear, I had a hard time reading the paper and couldn't have reproduced it. This seems to be a common concern among the reviewers, it is not only about the points raised by me specifically. I trust the authors to significantly improve the writing based on their responses in the rebuttal/during the discussion.

Final note re non-medical: Thanks, now it is much clearer. Even though it was established by prior work, I am still not sure why merging two objects that are overlapping in 2D is useful, the task seems better motivated within the medical domain to me.
We appreciate your **acknowledgment on our method** and that this is *the first work leveraging the inherent properties of vision foundation models (SAM) for ambiguous image segmentation**. We will make the following revisions in our updated submission:
* According to your suggestions, we are commited to **fine-tuning the structure in Method**, by intergrating the related contents of our training and inference pipelines into **an unified sub-section ``Training and Inference''**.
* Also, we are committed to **modifying the statement in Line 213** by: *"We train all weights from scratch except for the SAM components, including the modules of PGN, IGN, posterior-PGN, and posterior-IGN. For the components including prompt encoder, image encoder, and mask decoder in SAM, we initialize them with SAM's original weights before commencing training."*

We sincerely look forward to**incorporating these suggested changes from you** and hope that this paper will **inspire a broader audience thanks to your constructive feedback**.

Yours,

Authors
**Last but not least**, we have to acknowledge **the challenge of elucidating all details of the proposed method regarding data, model and metrics** within the **confined space of a submission**. Nevertheless, we beg to point out that the majority of the content addressed in this rebuttal was $\textcolor{blue}{\textbf{already existed in our initial submission}}$, both in the **main text and the appendix**. **For instance**, questions regarding **<more details about metrics>** were addressed in the $\textcolor{blue}{\textbf{appendix, Section A.2, Lines 513-525}}$. The inquiry about **<How multiple GT generated?>** was covered in the $\textcolor{blue}{\textbf{appendix, Section A.2, Lines 485-500}}$. It's worth noting that most of the datasets used in this paer, including LIDC, BraTS, ISIBI, etc., are widely recognized benchmark datasets for ambiguous segmentation, following **established practices [1-5]**. We simply utilized the pre-defined multiple GTs provided by these datasets. Additionally, **<Trade-off coefficients>** were mentioned in $\textcolor{blue}{\textbf{Line 238}}$ in main text. **Our responses to these queries primarily  $\textcolor{blue}{\textbf{emphasize where they are in the original text}}$ rather than introducing new clarification or content.**

Hence, we sincerely believe that the **areas requiring further clarification in the revision are primarily limited to the following several aspects**:
* Adding a definition & clarification of **full granularity** in **Figure 1's caption and in the introduction**.
* Including **a sentence in Section 4.1 (Dataset, Line 230)** to tell the readers to **refer to more details in the appendix**.
* Expanding **Section A.2 in the Appendix (Dataset, after Line 500)** with **a sentence to account for how the ambiguous segmentation masks are made**. We will also add a figure to clarify potential misunderstandings about spatial overlap.

$\textcolor{blue}{\textbf{Most other clarifications are extracted from the existing content in the initial submission and appendix.}}$ Furthermore, we sincerely appreciate $\textcolor{blue}{\textbf{your ``Applause" that this paper is interesting}}$ and we sincerely hope that **some "Flaws" related to clarification issues in this paper can be effectively addressed** thanks to your constructive feedback.

Yours,

Authors

[1] A probabilistic u-net for segmentation of ambiguous images. 

[2] Phiseg: Capturing uncertainty in medical image segmentation.

[3] Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty

[4] Pixelseg: Pixel-by-pixel stochastic semantic segmentation for ambiguous medical images.

[5] Ambiguous medical image segmentation using diffusion models
Thank you for your thoughtful comments and the opportunity to clarify our work. We are pleased that our initial response addressed some of your concerns. We will now respectfully address the points you further emphasized.

> C1: Full granularity refers to the ensemble of various granular outputs within SAM - how is this ensemble computed?

Regarding the full granularity ensemble within SAM, as shown in **Eq. (13) and Lines 199-205**, we implement a ensemble strategy to $\textcolor{blue}{\textbf{weighted summing the multi-granular outputs of SAM}}$ to get an ensembled output map. The texts in initial submission is provided as follow: “*Specifically, given multiple candidate outputs from SAM, represented as {M¹, M², ..., M^n}, where n is the number of scales, we introduce a set of learnable mask weights W = {w₁, w₂, ..., w_n} ∈ R^n. The final mask output is obtained through a weighted sum calculation:
$$
\tilde{M}=\Sigma_{i=1}^n w_i \odot \tilde{M}^i,  
$$
where w₁, w₂, ..., w_n are initialized to 1/n and subsequently fine-tuned to enable the model to effectively perceive object scales. By adaptively integrating masks at multiple scales, the model's perception and modeling capabilities for complex target diversity are further enhanced.*”

> C2: I do not think I stated 'Bboxs that are not covering whole region are not considered' in any part of my review so I am quite confused about this reponse

We apologize for the confusion. This response actually **corresponds to Question 11 in your initial review regarding Figure 1**: “The prompt variation experiment depicted in Figure 1…making the bounding box smaller than an object impacts segmentation more than making it larger”. We initially misinterpreted your point, as it requires some efforts to accurately understand. You were inquiring about the **impact of bounding box scaling (both enlargement and reduction) on SAM's segmentation performance, particularly when the box may or may not cover the entire object**, right?

First, in **Table 1 of the initial submission**, both SegGPT and SAM use box shifts that include random pixel offsets in all directions and size scaling from [0.8, 1.2], as stated in **Lines 253-254 and 259-260**. We found that our method $\mathcal{A}-SAM$ can achieve superious results compared with them.

To further specifically address your interest in **how box scaling affects segmentation performance when the box may not fully cover the object**, we conducted additional experiments in LIDC dataset (which follows the same setting in Fig. 1 in initial submission). We scale down and up the canonical box prompt for foreground in each image by different scaling factors from 0.8~1.20.  The results are as follows. We observed that **when the box doesn't fully cover the entire object (i.e., $\textcolor{blue}{\textbf{scaling factor < 1}}$), the performance change is more dramatic compared to the standard box**. This aligns with intuition. We appreciate your insightful suggestion!

| Scaling Factor of Box Prompt | 0.80  | 0.85  | 0.90  | 0.95  | 1.00 (Canonical prompt)   | 1.05  | 1.10  | 1.15  | 1.20  |
|----------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|Seg. IOU | 0.227 | 0.284 | 0.352 | 0.450 | 0.592 | 0.524 | 0.471 | 0.413 | 0.372

> C3: I do not understand the methodology of creating masks for Sim10k and KINS... . 'the region exclusive to A, the region exclusive to B, and the union region of A and B' - why is the intersection of A, B excluded from the first two masks? …

Regarding the methodology for creating masks for Sim10k and KINS, we refer to region of instance A, region of instance B, and their union A∪B. We didn't extensively discuss the intersection of A and B in the last response, because, for segmentation, there typically aren't strictly overlapping regions at the pixel level. **If a local region of A spatially overlaps and occludes a local region of B, only the local region of A is visible**. Thus, we approach this from a 2D segmentation, pixel classification perspective when referring to 'the region exclusive to A, the region exclusive to B, and the union region of A and B'. From a spatial relationship viewpoint, **$\textcolor{blue}{\textbf{the overlapping region belongs to the occluder}}$ rather than the occluded object**. Thank you for pointing this out!
Thanks to the authors for the response. They have addressed all my concerns. Thus I will raise my initial rating.
Thank you for the response!

The following concers remain/arise from your response:
* C1: Full granularity refers to the ensemble of various granular outputs within SAM - how is this ensemble computed?
* C2: I do not think I stated 'Bboxs that are not covering whole region are not considered' in any part of my review so I am quite confused about this reponse
* C3: I do not understand the methodology of creating masks for Sim10k and KINS... .  'the region exclusive to A, the region exclusive to B, and the union region of A and B' - why is the intersection of A, B excluded from the first two masks? I thought the prompt is ambiguous because pixels on the intersection belong to both instances, but here you choose to exclude those pixels... Maybe some examples would help.

I think the work is interesting and the method seems to perform well on different datasets. What prevents me from raising my score is that overall, I am worried that the amount of clarifications and rewriting needed is too big.
Thanks for the response. I think the method is good, but as also indicated by other reviewers (vXP5, YPmC), the writing of this paper needs further improvement. Therefore, I keep my rating unchanged.
Thanks the authors for the responses. They've addressed my concerns. Thus I'll raise my rating.
Dear Reviewer 2dZP,

Thank you for your comment. Regarding line 213 of the main paper, when we stated "training the model from scratch with randomly initialized weights," we meant to convey that we train all weights outside the SAM component (e.g., PGN, IGN, posterior PGN, posterior IGN, etc.) from scratch, while the weights used for the SAM component (e.g., prompt encoder, image encoder, mask decoder, etc.) are initialized with SAM's original weights. We will clarify this point in the revision. Thank you for bringing this to our attention!

Best,

Authors of Submission 830
Thank you for your detailed response. A minor question: Line 213 of main paper indicates that "The model is trained from scratch using randomly initialized weights". So is this a writing mistake?
Dear Reviewer V2bw,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,
Dear Reviewer 2dZP,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,
Dear Reviewer YPmC,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,
Dear Reviewer vXP5,

We would greatly appreciate it if you could review our response by Aug 13 AoE. After that date, it might be challenging for us to engage in further discussions. If you have any follow-up questions, please don't hesitate to reach out. We deeply value your expertise and time.

Best,