ICL-SAM: Synergizing In-context Learning Model and SAM in Medical Image Segmentation

Authors: jiesi  hu
yang  shang
yanwu  yang
xutao  guo
hanyang  peng
ting  ma
y  hu
x  yang
h  guo
t ma peng
sam  encoder
assign  conv
  sigmoid
sam  decoder
Venue: MIDL_io_2024_Conference
Published: 2024-01-22 07:43:29

Abstract: Medical image segmentation, a field facing domain shifts due to diverse imaging modal- ities and biomedical domains, has made strides with the development of robust models. The In-Context Learning (ICL) model, like UniverSeg, demonstrates robustness to domain shifts with support image-label pairs in varied medical imaging segmentation tasks. How- ever, its performance is still unsatisfied. On the other hand, the Segment Anything Model (SAM) stands out as a powerful universal segmentation model. In this work, we intro- duce a novel methodology, ICL-SAM, that integrates the superior performance of SAM with the ICL model to create more effective segmentation models within the in-context learning paradigm. Our approach employs SAM to refine segmentation results from ICL model and leverages ICL model to generate prompts for SAM, eliminating the need for manual prompt provision. Additionally, we introduce a semantic confidence map gener- ation method into our framework to guide the prediction of both ICL model and SAM, thereby further enhancing segmentation accuracy. Our method has been extensively eval- uated across multiple medical imaging contexts, including fundus, MRI, and CT images, spanning five datasets. The results demonstrate significant performance improvements, particularly in settings with few support pairs, where our method can achieve over a 10% increase in the Dice coefficient compared to cutting edge ICL model. Our code will be publicly available.

Main paper: 
 introduction: 
 Image segmentation represents a pivotal challenge in medical image analysis, and deep learning has increasingly become the predominant approach for this task (Ronneberger et al., 2015; Isensee et al., 2021; Chen et al., 2021b; Dolz et al., 2018) . The field grapples with domain shift due to the heterogeneity of imaging modalities and the diversity within biomedical domains (Liu et al., 2023) . While existing segmentation tools are technologically advanced, they are often restricted to specific tasks or related domains. This specialization limits their ability to address domain shifts and diverse scenarios, particularly in the context of evolving clinical and scientific requirements in medical imaging. Despite various finetuning techniques being proposed (Liu et al., 2023; Chen et al., 2021a; Yang et al., 2022; Hu et al., 2023) , the necessity for extensive computational resources and specialized machine learning expertise poses significant barriers in actual applications.Originating from the field of natural language processing, In-Context Learning (ICL) has recently emerged as a promising methodology for developing universal segmentation models robust to domain shifts. Models such as UniverSeg (Butoi et al., 2023) and Neuralizer (Czolbe and Dalca, 2023) applied this approach, constructing models that can adapt to new tasks or domains by leveraging the support set present in input data. This circumvents the need for extensive retraining and achieves promising results in few-shot scenarios. However, current state-of-the-art ICL models like UniverSeg (Butoi et al., 2023) still face challenges due to their suboptimal performance, particularly with limited support set data.On the other hand, the Segment Anything Model (SAM) (Kirillov et al., 2023) represents another universal segmentation model capable of generating satisfactory segmentation masks with bounding box or point prompts. Since SAM's introduction, numerous studies have explored its application in image segmentation (Zhang et al., 2023) . Huang et al. (Huang et al., 2023 ) assessed SAM's effectiveness across various medical datasets, noting that bounding box prompts generally outperform point prompts. Ma et al. (Ma and Wang, 2023) achieved promising results by fine-tuning SAM with a substantial annotated medical segmentation dataset. However, the usage paradigm of SAM differs from that of ICL models. SAM necessitates providing prompts for each query image, whereas ICL models require only a set of fixed image-label pairs.In this paper, we propose a method called ICL-SAM that utilizes SAM to refine the suboptimal segmentation results produced by ICL model and allows ICL model to provide targeted prompts for SAM, thereby eliminating the necessity for manual prompt provision. Furthermore, we harness semantic confidence maps derived from SAM's feature maps, which are replete with rich semantic information, to guide the generation of results and, thus, improve accuracy. ICL-SAM has been comprehensively evaluated in various contexts, including fundus, MRI, and CT images. It achieves significant performance improvements over cutting edge ICL models, notably in few-shot scenarios, where it achieves an increase over 10% in the Dice coefficient. The key contributions of our work are summarized as follows:• We introduce an innovative method that employs the powerful SAM model to bolster and refine in-context learning model in medical image segmentation.• We propose a novel semantic confidence map generation technique and its integration within our framework to enhance segmentation performance, based on the in-context learning framework.• Our methodology is extensively evaluated across three types of images and five datasets, demonstrating significant improvements in model performance, especially in situations with limited support sets.  
method: 
 Consider the set {(x i , y i )} n i=1 of image-label pairs for a segmentation task. We utilize the advanced ICL model UniverSeg (Butoi et al., 2023) in our experiments. The UniverSeg model learns a function ŷICL = Dec ICL (Enc ICL (x, S)), utilizing a CNN encoder and decoder architecture. It predicts a label map for an input x in accordance with the task-specified support set S = {(x j , y j )} m j=1 , which consists of available example image-label pairs, where m denotes the number of pairs in the support set. For the SAM model, the prediction map is defined as ŷSAM = Dec SAM (Enc SAM (x), prompt). Figure 2 shows the workflow of these two models during inference.The architecture of our cooperative methodology is depicted in Figure 1 . This module operates by iteratively generating bounding boxes from the segmentation outputs of the ICL model. These bounding boxes are then employed as prompts within the SAM framework, thereby achieving refined segmentation outcomes. Empirical studies, such as those by (Ma and Wang, 2023; Huang et al., 2023) , have substantiated the effectiveness of bounding boxes as potent prompts for medical image segmentation. In addition, to fully harness the capabilities of the SAM encoder, we generate a semantic confidence map from its feature map outputs, specifically designed to emphasize the foreground regions. This confidence map plays a pivotal role in concurrently enhancing the segmentation performance of both the ICL model and SAM.The final output map fuses results from both SAM and ICL models, formulated by the following equation:EQUATIONwhere ŷ denotes the prediction of our method. Considering that the performance of the ICL model is enhanced with an increase in support set size m, we dynamically adjust the γ value to γ = a(1 -e -τ m ) to increase the weight of ICL, where τ is a temperature factor.Given that the support set comprises labeled data, it is feasible to employ the universal features extracted by SAM to train a Logistic Regression (LR) model. This model is adept at generating a semantic confidence map for the query image that accentuates the target region, as delineated in Figure 1 . In the training phase, features Enc SAM (S) ∈ R m×64×64×256 from the support set are utilized as training data, where m denotes the size of the support set. These features are reshaped into Enc SAM (S) ∈ R 4096m×256 to serve as the input for the LR model, with the corresponding labels in the support set acting as the ground truth Y ∈ R 4096m . The SAGA algorithm (Defazio et al., 2014) , known for its efficiency with large training sets, alongside binary cross-entropy loss, is utilized for parameter optimization.After training, the semantic confidence map can be computed as follows:EQUATIONwhere δ represents the parameters of the LR model. Furthermore, since the dot product in feature dimensions can be replaced by a convolution layer with a 1×1 kernel, the parameters of the trained logistic regression model are assigned to a convolution layer during actual inference. This approach enables the model to exploit GPU parallelism, thus ensuring inference speed. During inference, the confidence map for a query image x is computed byC = sigmoid(conv 1×1 (Enc SAM (x))), where the confidence map is subsequently interpolated to C ∈ R h×w , where h and w represent the spatial size of the target layer to which C will be applied. Note that other pretrained models can be used to build confidence maps instead of SAM using this method.The bounding boxes provided by the ICL model may lack precision. To alleviate this, we introduce explicit semantic guidance into the decoder of SAM, enhancing its focus on foreground regions. As depicted in Figure 1 , we incorporate the generated confidence map into the second cross-attention block of the image-to-token attention and the final tokento-image attention blocks within SAM's decoder. While there are other attention blocks in SAM's decoder, we have found that changing just these two blocks suffices. Specifically, we have modified the original attention matrices I ∈ R h×w corresponding to the mask token for these blocks as follows:EQUATIONwhere Z-Norm(C) denotes the z-score normalization, and β = 2 serves as a balancing factor. A larger value of β exerts greater influence of the confidence map on the original attention matrices. This modification allows SAM to more effectively concentrate on foreground features even when the bounding box is not accurate, thereby enhancing overall segmentation accuracy.The spatial information provided by the semantic confidence map can be beneficial for the ICL model, particularly when its segmentation is imprecise. Therefore, we incorporate the confidence map as spatial attention into the decoder of the ICL model. Specifically, the process of adding the confidence map in UniverSeg is defined as follows:EQUATIONwhere Dec i ICL (x) represents the feature map at the ith decode layer of the UniverSeg model. The function Norm is employed to ensure the consistency of the L2 norm of the feature map, defined as Norm(B|A) = B||A|| 2 ||B|| 2 . This attention mechanism is incorporated at each layer of the decoder in UniverSeg, analogous to the Attention U-Net (Oktay et al., 2018) . The larger the value of α, the more significant is the impact of the confidence map on the UniverSeg model. When α = 0, the outcome is equivalent to the vanilla UniverSeg. We dynamically adjust the α value as α = be -τ m . When the support set size is larger, we reduce the effect of the semantic confidence map.We generate the bounding box prompts for SAM using the output pseudo-label of the ICL model. To eliminate noise within the prediction map, we first apply morphological shrinkage to the pseudo-label, reducing it to reduce the area of the pseudo-labels to 90% of their original size. This step helps eliminate some of the finer noise. Subsequently, we use morphological inflation to restore the retained pseudo-labels to their original size, ensuring the accuracy of the bounding boxes generated afterward. Subsequently, we iteratively select foreground components and generate bounding boxes around the component, which are then inputted as prompts into the SAM decoder. The final output map from the SAM decoder unifies the maps generated for each bounding box: Dec SAM (Enc SAM (x)) = i Dec SAM (Enc SAM (x), prompt i ). This method can effectively handle scenarios involving multiple separated targets within an image. 
datasets: 
 Our methodology was evaluated across three distinct scenarios, encompassing the segmentation of fundus, brain MRI, and kidney CT images. Each dataset consists of designated meta support set and query sets. We randomly selected image-label pairs to constitute the support set from the meta support set and conducted inference on the query set.(1) For the segmentation of the optic disc and cup in retinal fundus images, we utilized datasets from the REFUGE challenge (Orlando et al., 2020), RIM-ONE-r3 (Fumero et al., 2011) , and Drishti-GS (Sivaswamy et al., 2015) . The composition of the meta support and query sets for these datasets was 320/80, 99/60, and 50/51 images, respectively.(2) Whole tumor segmentation was performed on T1, T1ce, T2, and FLAIR modalities using the BraTS2020 dataset (Bakas et al., 2018) , focusing on low-grade glioma cases. The meta support set and the query set were randomly divided with 53 and 23 cases.(3) The Kits23 dataset (Heller et al., 2023) was utilized for combined segmentation of kidney and tumor, with the dataset being randomly partitioned into meta support and query subsets containing 245 and 244 cases, respectively.For the 3D MRI and CT datasets, we extracted 2D slices that contained the segmentation targets. The preprocessing of images was aligned with the protocols in UniverSeg (Butoi et al., 2023) and SAM (Kirillov et al., 2023; Ma and Wang, 2023) . Detailed description of the datasets could be found in appendix.Our experiments were conducted on NVIDIA V100 GPUs equipped with 32GB of memory. For each inference scenario, we randomly selected support sets 10 times, calculating their mean results to derive the final outcome. The Dice coefficient, which quantifies the overlap between the predicted segmentation and the ground truth, was employed for evaluation. In these experiments, the parameters a, b, τ were set to 0.5, 0.3, and 0.1, respectively, values that were determined to be optimal through our testing. Note that our model does not require fine-tuning the parameters of the ICL and SAM models. Thus, ICL-SAM can be directly applied to other tasks without retraining.To ascertain the efficacy of our approach, comparisons were made with both Uni-verSeg (Butoi et al., 2023) and Neuralizer (Czolbe and Dalca, 2023) , which are state-of-theart in-context learning models in the realm of medical imaging. UniverSeg is a universal segmentation model, and Neuralizer, trained on neuroimaging data, is versatile in performing a variety of tasks beyond segmentation. Regarding the SAM model, we evaluated both the original SAM (Kirillov et al., 2023) checkpoint and the MedSAM (Ma and Wang, 2023) checkpoint of the ViT-B model.Table 1 displays the Dice score of our model across various support set sizes and datasets. In the case of the Fundus dataset, we present the mean segmentation values for the optic disc and cup across three datasets. For the BraTs2020 dataset, the average segmentation outcomes across all modalities are shown. For the Kits23 dataset, we illustrate the results of the combined segmentation of the kidney and tumor. Detailed results are available in the appendix. The SAM+GT demonstrates the performance of using ground truth map to generate bounding box input into SAM, which demonstrates the upper bound of the corresponding SAM model. In the appendix, we also present the Average Symmetric Surface Distance (ASSD) of our model. A notable enhancement in performance is evident when the support set size is small. Employing our methodology with a support set size of 1 yields improvements of 15.94%, 13.09%, and 19.68% in the Dice coefficient across the three datasets, respectively. Furthermore, our approach consistently outperforms across all support set sizes and datasets. The integration of SAM and MedSAM demonstrates differential impacts in various scenarios. MedSAM is particularly advantageous for fundus and brain MRI, and the addition of SAM is more effective for CT kidney segmentation. This disparity is likely due to SAM's limited specialization in the medical field, leading to suboptimal performance in cases with less distinct boundaries like Fundus and BraTs, and superior results in kidney segmentation where boundaries are more obvious. For the Fundus data set, although the performance of SAM+GT is poor, UniverSeg+SAM can still bring improvements. We attribute this is to the benefits brought by the proposed confidence map. Although the current efficacy is yet to match that of MedSAM+GT, our method has successfully narrowed this gap, showcasing potential within the in-context learning framework.Figure 3 presents our results in the form of a curve graph, where the shaded areas represent the standard deviation of the corresponding performances. Notably, for the Fundus and Kits23 datasets, our method with only 16 support instances achieves performance comparable to that of UniverSeg with a support size of 64. In the context of the BraTs2020 dataset, a support size of 32 achieves performance equivalent to UniverSeg with 64 support instances. This underscores the potential of our approach to significantly reduce the annotation burden on clinicians for in-context inference while maintaining robust performance. Figure 4 demonstrates the enhancement process of UniverSeg through our methodology. It shows that UniverSeg's segmentation can be incomplete and coarse. The incorporation of the semantic confidence map leads to better segmentation outcomes. By processing the bounding boxes through SAM, we achieve a more comprehensive and accurate segmentation. Table 2 presents an ablation study, showcasing the mean performance across all three datasets and for every support set size. The task-specific ablation analyses are in the appendix. It is evident that incorporating SAM enhances the UniverSeg model's performance. Furthermore, the addition of semantic confidence into either UniverSeg or SAM markedly elevates the overall model efficacy. The Fusion approach, which combines the predictive outcomes of both the ICL and SAM models, also demonstrates improvement, particularly when the support set is large, because, in such scenarios, UniverSeg is capable of providing an relatively accurate segmentation mask. 
conclusion: 
 Our proposed methodology capitalizes on SAM's high precision in segmentation and the ICL model's ability to provide contextual support. The integration of a semantic confidence map further enhances segmentation accuracy. Our comprehensive evaluations demonstrate the effectiveness of the proposed framework, particularly in scenarios with limited support sets. Additionally, this framework reduces the need for extensive manual input for SAM. Our research highlights the significant potential of the in-context learning paradigm, suggesting opportunities for future enhancements. Furthermore, there is a gap between our approach and MedSAM+GT, suggesting potential for further enhancement of in-context learning models. Figure 4 presents additional visualization results, primarily illustrating how our outcomes evolve from the original ICL model (UniverSeg). The evolution process adheres to the naming convention of the ablation study in Tables 2. To distinctly showcase the effects, we mainly selected smaller context sizes (2 for the fundus dataset and 4 for others). From the figure, it is evident that incorporating the confidence map into the ICL model or integrating SAM significantly enhances the results. This also reveals that the main reason for the improvement offered by our method is the utilization of the powerful SAM to rectify inaccuracies produced by the ICL model. Moreover, our approach tends to yield good results when the segmentation from UniverSeg provides an approximate location of the target and generates precise bounding boxes. However, cases where UniverSeg's segmentation map misses the target result in SAM's inability to produce accurate results as well. This limitation is a primary factor in the persisting performance gap between our method and MedSAM+GT.Figure 6 displays the hyperparameter sensitivity analysis. We assessed the effects of different hyperparameters on fundus datasets and found that optimal performance is attained when a, b, τ , and β are set to 0.5, 0.1, and 2.0, respectively. Given their effectiveness on the fundus datasets, these hyperparameters were also applied to other datasets, demonstrating the robustness and generalizability of our algorithm.  
 
 Appendix:  


Decision: This paper presents a method that combined in-context learning (e.g. universeg) for segmentation with interactive foundation models (e.g. SAM). Essentially the users use initial segmentations provided by universeg to then segment the target using SAM. 

The reviewers all found the work intuitive and interesting, in an important and growing area. The discussion brought about some clarity for the reviewers, although none of them changed their scores. Overall, a solid paper that should be discussed at the conference. 

In terms of presentation type, the paper received two votes for Poster (WA) and one for Oral (WR), and I do believe the decision on that is still borderline and either would be reasonable.

Reviews: review: paper_summary: paper_summary: The authors propose to integrate the strong points of two growing fields in the segmentation domain: the robustness of the In-Context Learning approach to domain shift and the performance of the strong fundamental models such as SAM. They innovatively fuse these two approaches in a single framework, with a confidence map generation method to boost the performances of the whole. Extensive quantitative and qualitative studies on three modalities of data and five datasets prove the interest of the method.
main_review: main_review: This paper is qualitative, being clear and easy to read, and it presents an innovative method that brings great improvement on multiple segmentation tasks on variable datasets. It has been studied in depth, with great effort to answer the reviewers' questions and remarks.
strength_weakness: strength_weakness: - It would be interesting to consider at least another metric in addition to the Dice score, considering its properties/limitations (cf Reinke, Annika et al. “Common Limitations of Image Processing Metrics: A Picture Story.” ArXiv abs/2104.05642 (2021)), to get a clearer understanding of the behavior of the model.
questions: questions: Cf above
paper_summary: paper_summary: The authors show that their method, in context learning (the authors use UniverSeg and the semantic segmentation part of Neuralizer) combined with a foundation model (here: SAM and MedSAM) outperforms ICL-only methods. The authors test their method on three independent datasets and could show that SAM+ICL constantly outperforms ICL-only methods. Using an ablation study, the authors could show which elements of their method quantitatively contribute to the presented result.
main_review: main_review: I am happy with the response of the authors and my rating stays the same as in the preliminary assessment. This is a nice paper, and especially with the amendments that the authors provided can be a very interesting story to be presented at MIDL.
strength_weakness: strength_weakness: From the text and table 1, I did not fully get the SAM+GT comparison. I understood that the bounding box was provided by the GT, and then the Dice score was computed with the GT itself. In Table 1, these values constantly outperform the ICL+SAM models. Does this mean, that accurate bounding boxes with the manual (?) prompt of SAM would create high-quality semantic segmentations?

As the authors claim that there is no need for manual prompts as in SAMs and the prompts are then provided by the ICL model, I would be interested in the prompts. How do they differ from the manual ones, what is the "salt" of the prompts allowing the performance gains? On the other hand, I thought SAMs do not need necessarily prompts but rather bounding boxes with a generic prompt. If the authors could give examples and show (maybe in the appendix) those, I would be very happy and intrigued.
questions: questions: See Weaknesses.
paper_summary: paper_summary: This paper provided an in-context learning model called ICL-SAM, which is based on UniverSeg and SAM. The main objective of this model is to enhance the performance of in-context segmentation compared to UniverSeg. The authors have also introduced a novel technique for generating a semantic confidence map and integrated it into the combined framework. The experiments conducted on three different datasets demonstrate the effectiveness of the proposed methods. Overall, this paper provides valuable insights into improving in-context segmentation.
main_review: main_review: This paper is well-written and provides enough experiments to evaluate the method that presents the improvement to the in-context segmentation. Although some details are not fully provided, I think this paper could be accepted after revising the existing issues stated in the weakness.
strength_weakness: strength_weakness: 1. In the background, the authors claim SAM requires prompts and thus can be labor-intensive. Indeed this is true, but the proposed in-context model requires some label samples which might be more labor-intensive. Therefore, I recommend the authors reconsider their focus on the problem and it is better not to point out that SAM is “labor-intensive and time-consuming”.
2. The authors should pay more attention to the figure. For example, in Fig. 1, “token-to-image” is not unified because of letter capitalization, and the patch in the figure of “Confidence map” seems not unified with the “Attention matrix”. It might be better to check again.
3. In section 2.5, the sentence “the union of the maps produced by each bounding box” might mean that the results output unifies the maps with the three boxes, but it might be obscure from the description. In section 3.3, it is better to demonstrate the purpose of presenting the result of SAM+GT, which might indicate the upper bound of the setting.
4. In the datasets, the authors claim they conducted inference on the test set, but they do not describe clearly where they got the support set and query in the training stage and test stage.
5. In the implementation, I wonder whether the support sets are the same for all the comparison methods since different images in the support sets might bring unstable factors to the evaluation.
6. In Table 2, the authors might need to reorganize, since it might cause misunderstanding. For example, “Confidence map in MedSAM”  means MedSAM with confidence map, and “Fusion” should be explained before being abbreviated.
questions: questions: I recommend the authors provide more details and revise some of the referring arguments as stated in the section on weakness to improve the quality of the paper.

Comments: comment: Thank you for your understanding. We performed two-class classification using logistic regression with the binary cross-entropy as the loss function. This information has been added to Section 2.2 of our manuscript.

Furthermore, we have incorporated a new metric—Average Symmetric Surface Distance (ASSD)—into Table 6. ASSD measures the average distance between corresponding points on the surfaces of two objects and is widely utilized in fundus and tumor segmentation. The results from ASSD further demonstrate that our method significantly improves UniverSeg, especially at small context sizes.
Thank you for your efforts in clarifying and completing this paper, including re-doing your experiments to complete your metrics, which I can understand is a tedious task. The answers you provided seems clear to me, so I'd like to thank you again for that. However, I understand that you do not fine-tune the foundational models used, thus having no need for differenciability in your morphological operations ; but don't you still need to train your Logistic Regression to generate your Semantic Confidence Map (and thus to use a loss function) ?
Thank you for your comments. I will address your points in the order of weaknesses mentioned:

1.	We highly value your suggestion concerning the labor-intensive and time-consuming aspects of our methodology. We have taken your advice into consideration and have made revisions to the relevant sections (introduction & abstract) in our paper, thereby redirecting our focus towards enhancing the efficacy of the in-context learning model.

2.	Thank you for your meticulous observation on Figure 1. We have addressed the capitalization issue in Figure 1 and rectified some minor problems as well.

3.	We have provided additional clarification in Sections 2.5 and 3.3 as per your suggestion. In short, the final output map from the SAM decoder unifies the maps generated for each bounding box. This method can effectively handle scenarios involving multiple separated targets within an image. In addition, the SAM+GT demonstrates the performance of using the ground truth map to generate bounding box input into SAM, which demonstrates the upper bound of the corresponding SAM model.

4.	Our description regarding the source of the support set and query set was indeed unclear. Consequently, we have renamed the training set and test set to meta support set and query set, respectively. Our support set is randomly selected from the meta support set, and evaluation is conducted on the images within the query set. Moreover, we did not fine-tune the weights of the SAM and ICL models. Therefore, we do not have a traditional "training stage," and our models can be directly applied to new tasks without the need for fine-tuning, with all evaluations occurring during the "test stage." We have emphasized this point in Section 3.2.

5.	You are correct that variations in the support set can influence the outcomes, particularly when the support set size is small. To address this concern in our experiments, we ensured that all comparison methods utilized the same support sets. Furthermore, to mitigate instability caused by the selection of support sets, we randomly drew 10 sets of support sets and calculated the average.

6.	We have reorganized Table 2, dividing each task into three sections: methods comparison, ablation study, and GT+SAM. We have also clarified the descriptions of Table 2 in the appendix for better understanding.

We truly appreciate the time and effort you invested in reviewing our manuscript and offering insightful comments.
Thank you for your suggestions, and I will respond to your points in order, from weaknesses to detailed comments:
Weaknesses:

1.	We are grateful for your insightful inquiry regarding SAM+GT. Your understanding is correct. The term "SAM+GT" refers to the scenario where we use the ground truth to generate perfect bounding boxes to provide to SAM, and then calculate the Dice coefficient for SAM's output in this context. Indeed, providing accurate bounding box prompts does result in better segmentation from SAM. However, this outcome is expected since the bounding box prompt effectively offers strong supervisory information. Moreover, it confines the model's predictions within a very narrow scope and effectively reduces false positives in the segmentation.

2.	For your question about the prompt, bounding box prompts are derived from pseudo-labels generated by the ICL model, thus not necessarily more accurate than manually provided bounding box prompts. However, the advantage lies in eliminating the need for manual prompts for each query image, saving considerable human effort. The performance gains in our model are primarily relative to the ICL model (UniverSeg), with improvements mainly stemming from the semantic confidence maps generated by SAM and SAM's refinement of the output from the ICL model. Figure 4 demonstrates how ICL generates bounding box prompts and how SAM refines its outputs to some extent. Following your suggestion, we have included more examples in the appendix (Figure 5) to provide further clarity.

Detailed Comments:

3.	For your inquiry regarding the coefficients mentioned in our paper, we have detailed the process of selecting our hyperparameters in Figure 6 of the appendix. In brief, we assessed the impact of various hyperparameters on fundus datasets and subsequently applied these hyperparameters to other datasets. We found them to be effective across datasets, thereby demonstrating the robustness and broad applicability of our algorithm.

We extend our sincerest gratitude for your valuable suggestions, which have contributed to refining our manuscript.
Thank you for your suggestions. I will respond to your points in order, from weaknesses to detailed comments:

Weaknesses:

1.	For your suggestion to include additional metrics beyond the Dice score, we are actively working on incorporating extra metrics. However, due to the extensive number of experiments required in our study, approximately 3900 in total. For each task and each setting, we run 10 times. Given the constraints of our computational resources, we may not be able to complete this before the rebuttal deadline. We will update the results as soon as we finished.

Detailed Comments:

2.	For your question regarding morphological operations, I would like to clarify that in our method, we have not fine-tuned the parameters of SAM (and MedSAM) and the ICL model (UniverSeg) on any specific dataset, meaning that we have not conducted backward propagation. This approach stems from the fact that SAM and UniverSeg have been extensively trained on a wide range of datasets, allowing us to utilize their parameters directly for segmentation across various medical domains. Consequently, the differentiability of morphological operations like shrinkage and inflation does not impact our work. To elaborate, for the pseudo-labels output by UniverSeg, we first apply morphological shrinkage to reduce the area of the pseudo-labels to 90% of their original size. This step helps eliminate some of the finer noise. Subsequently, we use morphological inflation to restore the retained pseudo-labels to their original size, ensuring the accuracy of the bounding boxes generated afterward. We clarify this in our revised manuscript.

3.	For your question about loss function, as mentioned in response to the previous query, we have not fine-tuned the parameters of the foundational models utilized in our study. Consequently, the application of a loss function was not a part of our methodology.

4.	Thank you for highlighting the need for clarification regarding the definitions of w and h. We apologize for the lack of clarity in our paper regarding this matter. Due to the varying spatial sizes associated with different layers in the network, it is necessary to continuously resize the semantic confidence map obtained from a specific layer for application to other layers. In this process, the spatial size of the target layer is denoted by h×w. We clarify this point in revisions of the paper. Regarding the quantity of data for the fundus datasets, this information is detailed in the main body of the paper as well.

Your constructive feedback has been instrumental in enhancing the quality of our work, for which we are deeply thankful.