Zero-Shot Medical Image Segmentation Based on Sparse Prompt Using Finetuned SAM

Authors: tal  shaharabany
lior  wolf
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 10:24:20

Abstract: Segmentation of medical images plays a critical role in various clinical applications, facilitat- ing precise diagnosis, treatment planning, and disease monitoring. However, the scarcity of annotated data poses a significant challenge for training deep learning models in the medical imaging domain. In this paper, we propose a novel approach for minimally-guided zero-shot segmentation of medical images using the Segment Anything Model (SAM), orig- inally trained on natural images. The method leverages SAM’s ability to segment arbitrary objects in natural scenes and adapts it to the medical domain without the need for labeled medical data, except for a few foreground and background points on the test image it- self. To this end, we introduce a two-stage process, involving the extraction of an initial mask from self-similarity maps and test-time fine-tuning of SAM. We run experiments on diverse medical imaging datasets, including AMOS22, MoNuSeg and the Gland segmen- tation (GlaS) challenge, and demonstrate the effectiveness of our approach.

Main paper: 
 introduction: 
 Manual segmentation of regions of interest in medical images by experts is time-consuming and prone to inter-observer variability. Therefore, the development of automated segmentation algorithms has become increasingly important. Indeed, automated medical image segmentation plays a crucial role in diagnosis (Devunooru et al., 2021) , treatment planning (Sharma and Aggarwal, 2010) , and screening (Norman et al., 2018) .Deep learning techniques, particularly convolutional neural networks (CNNs), have shown remarkable success in various image analysis tasks, including medical image segmentation (Wang et al., 2021; Patel et al., 2021) . Recently, transformer based models have shown improved performance (Dosovitskiy et al., 2020) . Such deep models can learn complex patterns and features from large-scale annotated datasets, enabling them to achieve state-of-the-art performance in segmentation tasks. However, the success of deep learning methods heavily relies on the availability of annotated training data, which is often limited and expensive to acquire in the medical domain.Zero-shot segmentation offers a promising solution to address the challenges associated with data scarcity in medical imaging. Unlike traditional supervised learning approaches, zero-shot segmentation aims to perform segmentation tasks without relying on annotated data from the target domain. Instead, it leverages pre-trained models trained on related domains or tasks and generalize to a test image from an unseen domain. This approach has the potential to significantly reduce the annotation burden and accelerate the development of segmentation solutions in medical imaging.In this paper, we propose a novel method for zero-shot segmentation in medical images by adapting a deep learning architecture originally trained on natural images. Specifically, we rely on the seminal SAM foundation model by (Kirillov et al., 2023) .Our approach aims to bridge the gap between natural and medical image domains, enabling accurate and robust segmentation of anatomical structures and pathological regions without the need for extensive labeled medical data. It does require a sparse prompt in the form of a handful of forground and background points that are marked on the test image.We demonstrate the effectiveness of our method through comprehensive experiments on diverse medical imaging datasets. As we show, while SAM itself has zero-shot capabilities, those obtained by our method far surpass these baseline capabilities. 
related works: 
 In the domain of medical imaging, segmentation plays a crucial role. Numerous deep learning-based approaches have been proposed for medical image segmentation (Zhou et al., 2018; Xiao et al., 2018; Wang et al., 2021; Patel et al., 2021; Shaharabany and Wolf, 2022) . Utilizing convolutional neural networks (CNNs), these methods automatically delineate anatomical structures and pathological regions from medical images. While these algorithms offer substantial improvements in accuracy and efficiency compared to manual segmentation, they necessitate a significant amount of annotated data. Our solution addresses this challenge by harnessing the power of large pretrained "foundation" models, allowing for effective segmentation without reliance on labeled data.The Segment Anything Model (SAM) has emerged as a powerful tool for segmentation tasks in computer vision. Developed by Kirillov et al. (2023) , SAM introduces a versatile architecture capable of segmenting arbitrary objects in natural images. SAM utilizes a combination of sparse and dense prompts to guide the segmentation process, enabling accurate delineation of objects of interest. Several studies have explored the effectiveness of SAM in medical domain under fully-supervised settings tasks (Shaharabany et al., 2023; Wu et al., 2023; Xie et al., 2024; Ma et al., 2024) , demonstrating its ability to outperform traditional methods in segmenting complex scenes. Utilizing SAM, our method operates under zero-shot conditions, indicating the absence of labeled data and training sets. (Shi et al., 2023; Mattjie et al., 2023; Roy et al., 2023) employ SAM also in zero-shot settings, without engaging in weight fine-tuning, while our methodology involves fine-tuning of the SAM image encoder based on its self-similarity map.Learning to distinguish between the foreground and background of medical imaging with few marked points, positive and negative, can be used with a contrastive loss (Zhai et al., 2023) or, as we do, employing transfer learning from a method that already was trained on such a prompt (Xie et al., 2024) . Our approach builds upon the sparse prompt encoder of SAM, but unlike previous work, achieves this in the zero-shot setting, without performing fine-tuning on the training set.Zero-shot learning has emerged as a promising paradigm for addressing the challenge of data scarcity by leveraging knowledge transfer from related domains (Bucher et al., 2019; Narayan et al., 2020; Tewel et al., 2022; Mahapatra et al., 2021) , especially zero-shot segmentation methods (Bian et al., 2021; Ma et al., 2021) which offer a cost-effective and efficient solution for medical image segmentation, paving the way for automated analysis of medical images without the need for extensive annotation efforts. These previous work employ a different setting and cannot be compared directly with our results.Recent work have used self-similarity maps for unsupervised and weakly supervised localization tasks for natural images (Siméoni et al., 2021; Wang et al., 2023) . These identify a pattern that is well supported at multiple spatial locations in the latent space. Our approach is based on a two-stage solution, in the first stage we extract an initial mask from the self-similarity maps based on the locations provided in the sparse prompt directly. Friebel et al. (2022) ; Berg et al. (2019) ; Pachitariu and Stringer (2022) and others integrate human-machine interactions to accelerate the annotation process for medical objects. These methods typically leverage pretrained models from medical domains in conjunction with label sketches. In a manner akin to our methodology, these techniques can be customized for utilization in a single-image context, employing point-prompt based strategies without requiring further adjustments or refinement. In order to compare with such approaches, we ignore the utilization with pretrained models from medical domains and compare with a baseline that incorporates two components that are frequent in these methods: a superpixel technique and color clustering. 
method: 
 In this section, we describe the application of the Segment Anything Model (SAM) (Kirillov et al., 2023) to perform zero-shot segmentation on medical images, which is an out-ofdistribution domain. Our method consists of two stages (i) the extraction of initial mask from self-similarity maps, and (ii) a test-time image-specific fine-tuning of SAM. The Segment Anything Model S generates an output mask M ∈ R H×W given an RGB input image I ∈ R H×W ×3 (for gray scale medical images we duplicate the image across the three channels) and two types of prompts: (i) a sparse prompts P s which related to bounding boxes B ∈ R 4 or two sets of positive and negative points P ∈ R 2 , and (ii) a dense prompts P d ∈ R W ×H which is an estimated segmentation map. The output map is obtained as M o = S(M, P s , P d ).SAM consists of three main components. The first component is the visual encoder E i , which is based on a visual transformer (ViT) (Dosovitskiy et al., 2020) with a patch size of 16 × 16 and one of three architectures (i) "Base" with 12 transformers layers, (ii) "Large" with 24 transformers layers, or (iii) "Huge" with 32 transformers layers. The encoder maps the input image I to a tensorZ i ∈ R d i × W 16 × H 16, where d i is 768. The second component of SAM is the two prompt encoders: one for the sparse prompt P s and one for the dense prompt P d . These encoders E s , E d map the corresponding input prompt to tensorsZ s ∈ R ds× W 16 × H 16 and Z d ∈ R ds× W 16 × H16 , respectively. For each encoder, there is a null prompt for not using this specific prompt.The last component of SAM is the mask decoder D m , which receives the visual representation Z i together with the prompts representations Z s and Z d and generates an outputs mask M o . In the setting of the zero-shot medical segmentation, the model receives the image and several points that indicate foreground P pos and background P neg pixels.The straightforward option is to use SAM's sparse prompt encoder E s with P s that is the concatenation of the positive and the negative points, together with the original encoder E i in order to get the output mask from D m . However, since the data is out-of-distribution, the output mask is inaccurate and requires further refinement.We, therefore, propose a two-stage method that first extracts an initial mask J from the visual embeddings Z i returned by E i , and then applies test time fine-tuning of SAM's image encoder to bring SAM's output mask closer J. The other encoders and the decoder are not tuned (frozen networks).First stage -From Self Similarity Maps to Initial Mask We consider the normalized visual latent space as Z i and calculate the cosine similarity between the vectors in R d i in each pair of spatial locations. The output similarity map A ∈ R cwc h ×cwc h , where c h and c w are H 16 and W 16 , respectively. For SAM cases, the input image size is fixed to 1024 × 1024, so c w and c h equals to 64.The map A has a diagonal equal to 1 and A uv stands for the cosine similarity betweenZ i (⌊ u c h ⌋, u mod c h ) and Z i (⌊ v cw ⌋, v mod c w ), where indexing into Z i by spatial dimensions and provide a vector in d i .For each input point (x, y), whether it P pos or P neg , we calculate the corresponding latent pixel by dividing the pixel values by 16, and sampling the u = x * 64 + y row of A. The obtained row is denoted as A(u) and can be viewed as a spatial map of size c w × c h .We then define two maps: a positive maps M pos , and a negative map M neg (both contain only positive numbers) by a simple summation: M pos = (x,y)∈Ppos A(x * 64 + y), M neg = (x,y)∈Pneg A(x * 64 + y)In the case in which no negative points are used, the algorithm normalizes the value of M pos by subtracting the minimum value in M pos from each pixel in M pos and dividing by a d = max M pos -min M pos . Then, the initial mask J is given by thresholding the obtained normalized map at a value of 0.5.In the case of mixed label point, the algorithm obtains the initial mask J by comparing each pixel in the two maps, i.e., J(x, y) = 1 2 (sign(M pos (x, y) -M neg (x, y)) + 1) . A visualization of J is presented in App. A.Second Stage -Test time Optimization of SAM In the second stage, we utilize the initial mask J, which indicates foreground and background pixels, and finetune SAM's visual encoder E i while the decoder weights are frozen and both prompts are null, by considering M o = S(I, null, null). During the optimization, for each test image separately, the algorithm initializes S with the pre-trained weights and optimizes for that single test image independently of any other test images.The following loss terms are used during this fine-tuning processL(I) = L BCE (M o , J) + L dice (M o , J), where the BCE loss is defined as L BCE (X, Y ) = -Y * log(X)-(1-Y ) * log(X), the Soft-Dice loss is defined as: L dice (X, Y ) = 1- (2 * ij X i j * Y i j)+1 2 * ij X i j * Y i j+ ij (1-X i j) * Y i j+ ij X i j * (1-Y i j)+1 
experiments: 
 In this section, we detail the experimental setup used to validate the efficacy of our proposed zero-shot sparse-prompt segmentation method employing the Segment Anything Model (SAM) (Kirillov et al., 2023) . Our study is structured around two primary stages: (i) the extraction of the initial mask J from self-similarity maps and (ii) the test-time finetuning of SAM. We, therefore, also provide multiple alternative results, which apply only some of these elements: the SAM model with the point prompt, SAM model on top of the initial mask J, and the initial mask itself.To assess the effectiveness of our network in image-based segmentation tasks, we utilized the commonly employed metrics: mean Intersection-over-Union (IoU) and the Dice-Score.Three datasets are used: AMOS22 (Ji et al., 2022) Abdominal CT Organ Segmentation with 240 test images and 14 different categories. We adopted the 2D slicing configuration In the experiments, positive points are sampled randomly from coordinates where the pixel value in the ground-truth mask equals 1, while negative points are sampled from coordinates where the pixel value is 0. The random sampling means that the points are, in many cases, less typical than those a human would select as guidance. Since the sampling process introduces a variation, each experiment result is averaged over five iterations on each test image and then averaged over the entire test set. Naturally, the same points are used for comparing the various algorithms.Inspired by previous work, we add a strong zero-shot baseline ("Super-Clustering") that combines superpixel segmentation and color clustering techniques for zero-shot segmentation. Additional details are available in App. C.Optimization details During the test-time fine-tuning of S, we employ the ADAM optimizer with an initial learning rate of 1 • 10 -5 . A batch size of 1 is utilized, and no further augmentation has been implemented. We conduct training on NVIDIA A5000 with 24GB GPU RAM. The number of fine-tuning iterations for S network fine-tuning was set to 100. The SAM pre-trained weights that we utilized were based on the ViT 'base' and 'large' architecture, Due to resource constraints, we opt not to use the Huge ViT-based encoder, which necessitates more than 48GB GPU memory. The SAM image encoder is provided with input images of size 1024 × 1024. We adhere to the pre-processing protocol outlined by SAM (resize, padding, normalization) to ensure optimal performance. The test time finetuning operation takes 40 and 80 seconds per image for Vit-base and Vit-large respectively.Tab. 1 presents zero-shot segmentation results for the AMOS22 (Ji et al., 2022) dataset across different numbers of positive points (P) and various methods. Notably, Table 2 : MoNu results for different number of positive points n pos and negative points n neg .All methods are zero-shot methods with a handful of points on the test image used for supervision. SAM w/P uses the points as the sparse prompt. SAM w/J uses the map J obtained from the first phase of our method as a dense prompt. The first number indicates the dice score and the second the IoU. Best (2nd best) results are indicated as bold (underlined). For reference, the fully supervised AutoSAM method, trained on the entire training dataset, which is not used in zero-shot at all, obtains a Dice score of 82.43 and an IoU of 70.17 (Shaharabany et al., 2023) . .90/3.87 31.40/18.85 35.88/22.16 40.84/26.30 11.76/6.60 33.59/20.37 36.34/22.43 39.19/24.76 38.43/24.17 5 0 9.42/5.37 30. 79/18.36 33.59/20.35 36.23/22.40 12.57/6.98 33.85/20.56 36.90/22.95 39.20/24.88 38.53/24.10 3 3 18.80/11.31 32.13/19.45 42.28/28.10 42.48/28.65 7.34/4.46 34.03/20.70 42.91/27.88 49.09/33.82 21.81/15.81 4 4 15.16/9.14 32.74/19.86 46.43/30.67 54.98/39.49 11.49/6.97 34.86/21.33 40.85/26.22 46.69/31.69 3.62/2.42 5 5 18.13/10.82 33.57/20.50 45.14/29.43 52.37/36.34 8.25/4.92 34.10/20.76 43.76/28.44 50.45/34.80 4.61/3.41 our proposed method, labeled as "ours w/P=1," achieves the highest Dice scores for most anatomical structures at P=1, surpassing the average performance of super-clustering by more than 0.2 dice score, and outperforming competing methods SAM w/P=1 and J w/P=1 by 0.02 and 0.06 dice score on average, respectively. This highlights the effectiveness of our approach even with minimal positive points. As positive points increase to P=3, our method maintains superior performance, particularly evident in Dice scores for organs like the spleen, right kidney, and left kidney, outperforming SAM w/P=3 and J w/P=3. On average for P=3, our algorithm outperforms super-clustering, SAM w/P=3, and J w/P=3 by more than 0.24, 0.02, and 0.06 dice score, respectively. At P=10, our method (Ours w/P=10) sustains high performance across most structures, achieving the highest overall Dice score and surpassing super-clustering by more than 0.15 dice score on average, as well as baseline methods SAM w/P=10 and J w/P=10 by more than 0.005 and 0.04 dice score on average, respectively. Moreover, our method competes favorably with the supervised stateof-the-art represented by nnUNet 3D, demonstrating its potential for practical deployment in scenarios with limited annotated data. The improvement of J during the finetune for images from AMOS22 presented in App. B. The results from the MoNu dataset are presented in Tab. 2. It is evident that our method achieves the best performance, attaining a mean IoU of 39.49 with ViT-base, which significantly outperforms the top result obtained by SAM using J in the mask prompt encoder, which achieved a mean IoU of 21.33 with ViT-base. Across most input marked point modes, both ViT architectures demonstrate superior performance compared to standard zero-shot with SAM and super-clustering, except in cases where a single positive point is sampled without a negative point for ViT-large. This discrepancy could be attributed to the inherent inaccuracies associated with relying on a single random representation in our algorithm, whereas the mask encoder of SAM effectively manages this issue. Consequently, for multi-point input scenarios, our algorithm demonstrates superior performance. As a general trend, given more points, the performance improves. However, going from four to five points of each type does not increase performance. Typically results on the Monu dataset using ViT-Base are presented in Fig. 1 for Vit-Base SAM. The masks obtained by SAM, using either the point prompt or a dense prompt that relies on the initial mask we extract are inaccurate, while our full method provides adequate segmentation maps. The results for the GlaS dataset are depicted in Tab. 3. Among these, the highest performance is achieved by ViT-large, with a mean IoU of 58.84, surpassing the best performance of the standard SAM which yields a mean IoU of 51.43, and super-clustering, which obtained mean IoU of 50.75. Our algorithm demonstrates superior performance in scenarios involving four marked points; however, its sensitivity to outliers leads to decreased performance in cases with fewer points. Notably, the point encoder's ability to handle negative points appears inadequate, suggesting potential limitations in SAM's capacity to discern foreground from background on this data. Figure 2 showcases typical results on the GlaS dataset utilizing Vit-Base SAM. Similarly to Figure 1 , our solution enhances the segmentation mask in comparison to SAM's results using either prompt and also greatly improves over J. 
conclusions: 
 We demonstrate that SAM is not effective in zero-shot segmentation of out-of-domain medical images with multiple foreground segments, when using the sparse prompt. However, when relying on SAM's embedding, one can extract a reasonable initial mask. SAM's output when using this initial mask as a dense prompt is not better than this mask. However, when applying a test-time fine-tuning of SAM, the obtained model is considerably better on the test image at hand. As future work we would like to study the generalization ability of these finetuned networks (a simple experiment that we did not yet run) and also train a feed-forward model using meta-learning (Lutati and Wolf, 2023) .  
 
 Appendix:  


Decision: This paper presents a zero-shot image segmentation method by fine-tuning SAM and using very sparse prompts.

The paper received rigorous reviews and discussion, and based on the concerns of the reviewers, mostly involving more details/related works, as well as quite a bit of experimental improvement. The original reviews were borderline. Following this, the authors went through careful revision of the work and responded with cleaner experiments and manuscript. Overall, all reviewers were pleased. 

Although there were substantial changes from the original manuscript, I think the paper is now in shape to be insightful to the community and should be accepted. As reviewers note, there are still some concerns left, and I encourage the authors to take these into account when they present the work at the conference.

Reviews: review: paper_summary: paper_summary: The authors propose a method to take advantage of the progresses of fundamental segmentation models in the medical domain, by reducing the number of prompt inputs needed when working on images with multiple foreground segments. To this end, they do a retraining of the encoder of the SAM model to help it overcome the domain shift when working on zero-shot learning, this retraining being based on the cosine similarity between the embeddings of each pair of pixels of the image.
main_review: main_review: Interesting method that seems to bring improvement on some medical segmentation tasks, where we're constrained by the quantity of annotated data available, with few effort to make to define prompts for the model to use. A good effort was made to answer the questions of the reviewers and to complete and clarify the work.
strength_weakness: strength_weakness: - Confronting the proposed method to other types of data of different scales (ex : CT/PET images, MRI, ...) would give a better understanding of its capabilities.
- A figure explaining the model would be of a great help to understand the process at hand.
questions: questions: I would be interested in seeing if this progress is generalizable on other types of data, maybe on tasks where the background is more variable.
paper_summary: paper_summary: The paper introduces a fine-tuning approach that utilizes initial masks from self-similarity maps to enhance visual encoding for zero-shot segmentation tasks. Unlike other methods that fine-tune the decoder, this paper proposes fine-tuning the visual encoder. The results indicate that this fine-tuning method surpasses alternatives that only provide pseudo points or masks.
main_review: main_review: The author addressed my concerns during the rebuttal period, leading me to increase the score for the final judgment. The only concern is the quality of the fine-tuned results, which still do not accurately follow the object morphology. However, the general idea of the paper is interesting and promising for future endeavors.
strength_weakness: strength_weakness: 1. The quality of the fine-tuned segmentation results is poor. Figures 1, 2, and 3 show segmentation masks with coarse shapes and boundaries, appearing to lose basic morphological patterns.
2. There is no comparative analysis of segmentation performance between fine-tuning the visual encoder and the mask decoder. The rationale and benefits of the proposed fine-tuning approach are not clearly articulated.
3. The paper does not provide SAM segmentation results with full prompting to establish the upper bound of segmentation tasks.
questions: questions: Please address the questions in weaknesses and detailed comments.
paper_summary: paper_summary: The authors present an approach to adapt the Segment Anything Model for zero-shot adaptation for the task of interactive / semi-automatic nuclei and gland segmentation. They present two main contributions: First, they derive a self-similarity map from point inputs, which they then use to prompt SAM, and second, they fine-tune the encoder during test-time such that the output of SAM better matches this self-similarity map. The authors report improved results compared to vanilla SAM and compared to using the self-similarity map alone on two microscopic datasets.
main_review: main_review: The authors responded in detail to most of questions, and spend substantial effort during the rebuttal to add additional comparisons to the paper. I am therefore happy to increase my rating to some extent. Some concerns remain, i.e., with regard to the drop in performance for some comparison approaches and the performance of vanilla SAM. As it still may be interesting to discuss these aspects at MIDL, I'm assigning a borderline accept rating to this paper.
strength_weakness: strength_weakness: - Lack of comparison to other (zero-shot) approaches. It is likely that the approach is successful because the target structures are repeating patterns, which makes it difficult for single-object approaches like random walker and more complex methods that are based on that to succeed. Still, a number of pre-trained models exist that could be used for comparison. Good candidates from my perspective would be, for example, Cellpose 2.0 [1], ilastik [2], and potentially a clustering-based approach as featured in [3]. While these methods may tackle the problem from a different side or may start from different starting conditions and training dataset, from my perspective this provides important insights into whether the current approach is an exploratory experiment or actually competitive with ready-to-use tools for microscopic image analysis. 
- Following this, it is unclear to what extent this approach would work also for larger objects that feature a stronger heterogeneity, e.g., organ segmentation or whether the authors present here an after all very targeted approach for a rather specific problem (which is not an issue by itself, but should be discussed).
- The literature review is rather brief on the existing studies of the (also zero-shot) capabilities of SAM for medical images. It may be interesting to further consider the by now relatively vast literature on medical image segmentation and SAM, e.g., [4,5], which have evaluated the performance of zero-shot SAM.
- The output of (vanilla) SAM looks unexpectedly bad, and I am wondering whether there could be a mistake in the implementation. Having tried out similar images with the SAM demo [6], I obtained less than perfect but rather decent nuclei segmentation masks (I understand that there may be some post-processing, but the presented segmentation maps, e.g., in Fig. 1 look completely off and also feature extremely strong checkerboard artifacts).


[1] https://www.nature.com/articles/s41592-022-01663-4
[2] https://www.nature.com/articles/s41592-019-0582-9
[3] https://academic.oup.com/bioinformatics/article/38/19/4622/6670619
[4] https://arxiv.org/abs/2304.05396
[5] https://arxiv.org/abs/2305.00109
[6] https://pubmed.ncbi.nlm.nih.gov/37296799/
questions: questions: - What is the reason that the segmentation maps for SAM look so unexpectedly bad? Could there be any issues with the normalization?
- Why did the authors not compare to any existing unsupervised or zero-shot approach? What would be the result, e.g., with CellPose (unsupervised/zero-shot) or ilastik (zero-shot)?
- To what extent do the authors believe that their method is only suitable for texture-like segmentation tasks with very strong repeating patterns?

Comments: title: Reply to author comments
General Response
comment: We truly appreciate the detailed reply and are thankful for the ongoing dialog.

> Lack of comparison to other zero-shot approaches

We appreciate the feedback and acknowledge the valid point raised regarding the alignment of the suggested methods with the proposed setting. Upon reconsideration, we agree that both CellPose and Ilastik can indeed be adapted for use in a single-image, point-prompt based approach without requiring additional adaptation or selection. In response to this, we revised the related work section to better reflect this aspect. Thank you for bringing this to our attention.

> Superpixel approach 

In our experimentation, we observed that introducing negative points into the model's process in some cases, did not lead to accuracy improvements, but rather resulted in performance deterioration. This phenomenon was particularly noticeable in scenarios where the model lacked the ability to differentiate between foreground and background. This observation underscores the significance of foreground-background separation, especially in medical datasets where color clustering may not inherently convey meaningful foreground-background distinctions.

> Larger objects

We appreciate the reviewer's inquiry regarding the performance discrepancy between SAM alone and SAM combined with J (self-similarity maps) on the AMOS22 dataset. The lower resolution of SAM+J, which is 16 times lower than the full image resolution and 4 times lower than the original SAM resolution, could be a contributing factor. In datasets where the shape of individual objects is well-defined, like in AMOS22, the low resolution of SAM+J may not provide any advantage over using SAM with points as prompts. However, in cases where objects are repetitive and their shapes are not well-defined, SAM with the original prompts might perform poorly compared to using the self-similarity map.

> The output of (vanilla) SAM

We apologize for any confusion caused. By stating "Obtained similar findings," we intended to convey that we replicated the performance results reported in the SAM.MD short paper using our framework. As a result, we did not encounter any discrepancies or issues.

> Renaming to soft-Dice loss:

Thank you for the clarification regarding your comment. We appreciate your attention to detail. We revised the equation on page 5.

> Layout of the paper:

In response to the feedback regarding the layout of the manuscript, we have taken into account the suggestion to disregard the “vspace” and have increased the font size of all small text throughout the tables. Additionally, we have considered relocating some tables.
I would like to thank the authors for their detailed comments and providing the revised version of their paper, the additional experiments and explanations. It is clear that the authors spend a lot of effort on this revision.

I would like to briefly reply to the points mentioned by the authors:
> Lack of comparison to other zero-shot approaches

I appreciate the additional superpixel / clustering baseline, but see two issues:
1) I disagree with the general statement that the suggested methods do not align with the proposed setting. While the motivation of the approaches is broader, both cell-pose and ilastik can be used in a single-image, point-prompt based approach without further adaptation / selection. This request should not prompt the authors to adapt their comparison method, but from my perspective, this is not suitable represented in the related work section.
2) the selected superpixel approach seems to tremendously deteriorate when even a single negative point is provided (similarly actually vanilla SAM). Do the authors have an explanation for that?

> Larger objects

I would like to thank the authors for adding this dataset. Do the authors have an explanation why SAM + J performs considerably worse compared to SAM alone (in contrast to the other datasets)?

> The output of (vanilla) SAM

I thank the authors for the additional check within their own data. After re-reading the SAM.MD short paper, I did not find any comment on this or comparable artifacts. Could the authors point me to this statement?

> Fine-tuning of SAM

Thanks, that makes it more clear!

> Renaming to soft-Dice loss: 

It seems that the authors may have misunderstood my comment. I don't really have an issue with the name (which is often called "Dice loss" but with the equation on pg. 5, which uses binarized TP / FP / FN. Since this makes gradient computation very sparse, I kindly ask the authors to revisit this equation.

I completely understand where this comes from, but small font sizes in the tables and potentially changes in the vspacing makes the paper rather cramped. The authors may want to consider moving some tables / part of the tables also into the supplementary material.
We thank the reviewer for the comprehensive feedback.

> Lack of comparison to other zero-shot approaches

We recognize the significance of benchmarking against established methodologies in the field. While the setting of several of the existing existing references [1, 2, 3] may not directly align with our approach, we have added a baseline method with the specified components (superpixels and color clustering). 

> Larger objects

Following the review, we now benchmark our method also on the AMOS22 dataset, which encompasses 14 distinct categories featuring varied sizes of medical objects. The results on the Amos22 dataset serve as compelling evidence that our method extends beyond texture-like segmentation tasks with strong repeating patterns. It also shows that the method performs well across a diverse range of medical objects with varying sizes and complexities.

> The output of (vanilla) SAM

We appreciate the reviewer for bringing this to our attention. The segmentation maps for SAM appear unexpectedly poor because SAM was not trained on medical data or designed for medical image segmentation tasks. Upon further investigation, we did not identify issues within our framework that may have exacerbated this problem. We note that another contribution (SAM.MD; [4]) obtained similar findings. It is worth noting that in the Meta demo, the initial SAM utilized was ViT-Huge (while we employed Large and Base versions), and additional post-processing was applied to the output mask, potentially leading to slightly improved results.

> Additional literature review


In Table 1, we compare our performances with findings from SAM.MD[4] which similarly employs SAM in zero-shot scenarios (the baseline results for SAM with the points prompt is exactly what was done in [4]). As part of our revision, we have also updated the related work section. 

> Fine-tuning of SAM

We chose not to optimize the decoder because doing so could cause it to simply follow $J$ and converge to it without refining it. By freezing the decoder and solely optimizing the encoder, the output mask is not able to precisely follow $J$ and therefore refines it, as demonstrated in Appendix B.
We thank the reviewers for their valuable feedback and suggestions. Following the reviews, we have added a benchmark (AMOS22) that is not a cell segmentation task, as well as an additional baseline method. The added results strengthen the utility of our method.

We have uploaded a revised version. A summary of the changes is provided below:
* Following reviewer aU6b, we have added more content to the related work section.
* Following all reviews, the AMOS22 benchmark was added.
* Following reviewer aU6b, a new baseline ‘’Super-cluster’’ was added.
* We have fixed the typos raised by the reviewers
* Following reviewer k7S5, a graph that presents the improvement of the finetune process was added as appendix B. 
* Following reviewer aU6b, the dice loss has been renamed to soft-dice loss.
We thank the reviewer for the comprehensive feedback.

> Lack of Comparative Analysis

In Appendix B, we included an analysis of the fine-tuning process in comparison to the initial mask $J$. The figure highlights the significance of this procedure in refining the output mask for the majority of the data points. 

> Fine-tuning the visual encoder and the mask decoder

We chose not to optimize the decoder because doing so could cause it to simply follow $J$ and converge to it without refining it. By freezing the decoder and solely optimizing the encoder, the output mask cannot follow $J$ precisely and therefore refines it, as demonstrated in Appendix B.

>SAM with a complete prompt:

For each table presented in the results section, we included a reference to the state-of-the-art supervised algorithm. As observed in the AMOS22 dataset, the gap between our method and the supervised algorithm is relatively narrow.
We thank the reviewer for the comprehensive feedback.

> Comparison with Other Data Types

Following your request, we have incorporated the AMOS22 dataset as a new benchmark that has a different type of data. Our method demonstrates superior performance compared to the other baselines, as evidenced by the results presented in Table 1.