Prompting to Adapt Foundational Segmentation Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 15:08:20

Abstract: Foundational segmentation models, predominantly trained on scenes typical of natural environments, struggle to generalize across varied image domains. Traditional "training-to-adapt" methods rely heavily on extensive data retraining and model architectures modifications. This significantly limits the models' generalization capabilities and efficiency in deployment. In this study, we propose a novel adaptation paradigm, termed "prompting-to-adapt", to tackle the above issue by introducing an innovative image prompter. This prompter generates domain-specific prompts through few-shot image-mask pairs, incorporating diverse image processing techniques to enhance adaptability. To tackle the inherent non-differentiability of image prompts, we further devise an information-estimation-based gradient descent strategy that leverages the information entropy of image processing combinations to optimize the prompter, ensuring effective adaptation. Through extensive experiments across nine datasets spanning seven image domains (\emph{i.e.}, depth, thermal, camouflage, endoscopic, ultrasound, grayscale, and natural) and four scenarios (\emph{i.e.}, common scenes, camouflage objects, medical images, and industrial data), we demonstrate that our approach significant improves the foundational models' adaptation capabilities. Moreover, the interpretability of the generated prompts provides insightful revelations into their image processing mechanisms. Our source code will be publicly available to foster further innovation and exploration in this field.

Main paper: 
 related work: 
 Foundational Segmentation Models. Foundational segmentation models have marked a significant advancement, showcasing remarkable capabilities in accurately segmenting diverse objects. These models, such as the segment anything model (SAM) [16] and the segment everything everywhere model (SEEM) [43, 44] , stand out for their utilization of handcrafted prompts to generate detailed segmentation masks. By integrating various types of prompts, including points, boxes, or textual information, these models have proven adept at identifying and segmenting objects across a myriad of images. Despite these advancements, the potential of using the input images themselves as adaptive prompts represents an intriguing avenue for enhancing model adaptability. Our study delves into this concept, exploring the development and application of effective image prompts tailored for foundational segmentation models Central to our approach is the employment of a few-shot learning strategy aimed at facilitating the models' seamless adaptation to a diverse range of image domains. This strategy underscores our effort to broaden the applicability and versatility of foundational segmentation models, enabling their tailored use in specific tasks or domains that extend beyond their initial training distribution. Prompt Engineering. Prompt engineering has emerged as a strategic approach to repurpose foundation models for novel downstream tasks without necessitating model fine-tuning or modifications. This technique has seen increasing exploration within both the neural language processing and visual foundation model domains, aiming at the automatic derivation of optimal templates for task adaptation [9, 23] . In the context of neural language processing, the methodology for automatic template learning bifurcates into offline and continuous strategies. Offline methods include prompt mining or prompt paraphrasing [13] , gradient-based search [31] , prompt generation [8] , and prompt scoring [5] . On the other hand, continuous methods primarily consist of prefix tuning [20] , tuning initialized with discrete prompts [40] , hard-soft prompt hybrid tuning [25] , and P-Tuning [24] . Similarly, for visual foundation models, several prompt engineering methods have been proposed. These methods include continuous text token optimization, conditional text token optimization [41] , and other approaches [14, 28, 38, 42] .In the segmentation foundation models sector, an approach that parallels our own is the introduction of SAMAug [4] , a method that leverages visual point augmentation to boost interactive image segmentation capabilities within the SAM framework. This study introduces a tailored prompt engineering strategy specifically for foundational segmentation models. Our methodology distinguishes itself by examining various image processing techniques to craft effective prompts, aiming to enhance the adaptability and performance of these models.Test-Time Adaption. Test-Time Adaptation (TTA) for image recognition has emerged as a pivotal approach to address the challenges posed by distribution shifts between training and testing data [21] . This technique allows for the fine-tuning of pre-trained models during the inference phase, thereby enhancing their robustness and accuracy when confronted with novel data distributions. TTA is particularly useful in scenarios where the test data distribution is unknown or significantly different from the training data distribution, which is common in real-world applications. TTA methods are categorized into several types, including source-free domain adaptation (SFDA) [17, 19, 22] , test-time batch adaptation (TTBA) [30, 32, 39] , online test-time adaptation (OTTA) [27, 35] , and test-time prior adaptation (TTPA) [29] . image processing techniques for the inference stage of a pre-trained segmentation foundation model. 
method 3.1 problem formulation: 
 The core aim of the "prompting-to-adapt" strategy is to discern optimal image prompts that effectively alter input images, thereby facilitating the adaptation of pre-trained foundational segmentation models to atypical domains. These image prompts are crafted through the strategic combination of various image processing techniques, enabling a seamless adaptation without necessitating alterations to the models' parameters or architecture. We conceptualize the problem as identifying image prompts that adeptly modify input images. This modification aims to produce accurate segmentation masks, leveraging few-shot image-mask ground-truth pairs for precision. Fig. 3 illustrates the comprehensive framework of our approach, which integrates three principal components: a search space, an image prompter, and an information-estimation-based gradient descent strategy. The search space encompasses an extensive array of image processing techniques along with their potential intensities, offering a broad spectrum for prompt construction. Functioning within the defined search space, the image prompter meticulously samples and combines image processing techniques to formulate an image prompt. This prompt specifies the sequence and intensity of the applied image processing techniques. Once the prompt is applied to the input images, generating modified images, these are then utilized in lieu of the original inputs for the foundational segmentation models. The predictions yielded from this process are evaluated against the ground-truth masks using the designated loss function. In the subsequent sections, detailed elucidations of each component integral to the adaptation process will be provided, shedding light on their respective roles and interactions within the overarching framework.In formulating the search space for image prompts, a meticulous examination of various image processing techniques was undertaken, leading to their categorization into two distinct groups: semanticdestructive and semantic-preserving. Semantic-destructive image processing techniques modify the core semantic content of an image. They include operations such as random cropping, rotation, the introduction of noise, and crop-outs. Although these manipulations alter the image's contextual information, they play a crucial role in testing the robustness of segmentation models to changes in image semantics. In contrast, semantic-preserving techniques adjust the visual representation of an image without impacting its semantic integrity. Such adjustments may involve modifications to the brightness, contrast, or saturation levels, serving to enhance or diminish specific image features without altering the underlying content. Despite the potential of semantic-destructive techniques to alter an image's context significantly, their incorporation alongside semantic-preserving techniques is essential for a holistic exploration within the search space. Thus, the constructed search space encompasses a diverse array of operations, including but not limited to, brightness enhancement, contrast reduction, saturation adjustment, box filtering, Gaussian blur, logarithmic transformation, gamma correction, Gaussian kernel sharpening, bilateral filtering, median filtering, and random rotation. To facilitate the search for optimal operation intensities, we have quantified the intensity range for each operation, dividing it into ten uniformly spaced intervals. This discretization enables a structured approach to exploring the search space, ensuring a comprehensive assessment of various image processing combinations and their effects on the adaptability and performance of segmentation models.Within the defined search space, we have developed an image prompter tasked with identifying the optimal combinations of image processing techniques to effectively adapt input images for segmentation tasks. The image prompter is designed to sequence and intensity of the selected image processing operations. Initially, a randomly-initialized query embedding, denoted as 𝑟 0 , is input into a long short-term memory (LSTM) layer [10] , producing an operation embedding 𝑟 1 . This embedding subsequently passes through a multilayer perceptron (MLP) layer, yielding the probability 𝑝 (𝑎 1 |𝑟 0 ) of selecting a specific image processing technique and its corresponding intensity. A softmax activation function is applied to normalize this probability distribution. The process iteratively continues, leveraging the operation embedding to sequentially generate a comprehensive prompt 𝜏 𝑡 , incorporating 𝑡 = 1, ...,𝑇 image processing techniques. While the LSTM layer serves as an illustrative example in our architecture, alternative structures, such as transformers [34] , can also be employed to enhance model design flexibility. Parameterizing the image prompter 𝑝 (•) with parameters 𝜃 , we define the formulated image prompts as a probabilistic combination of image processing techniques:EQUATIONwhere 𝑡 = 1, ...,𝑇 and 𝑅(•) assigns the highest probability image processing techniques and their intensities. Subsequently, the original input images 𝐼 undergo processing as follows:EQUATIONwhere 𝐼 𝑝 represents the processed, or prompted, images. The evaluation of the predicted masks is conducted through a loss function defined as:EQUATIONwhere 𝑓 (•) denotes the foundational segmentation model, 𝑚 ∼ 𝑀 refers to the ground-truth masks, and m to the predicted masks, and 𝐼𝑜𝑈 (•, •) quantifies the intersection over union between the predicted and ground-truth masks.Given the defined loss function as shown in Eq. 3, the conventional approach to optimize the parameters of the image prompter is expressed by: minEQUATIONHowever, the inherently non-differentiable nature of the sampling operation within 𝜏 ∼ 𝑝 𝜃 precludes the direct application of the gradient of the loss function for parameter updates in the image prompter. To circumvent this, we employ an information-estimationbased gradient descent strategy for the optimization task, focusing on modulating the information entropy of the output probabilities from the image prompter relative to the magnitude of the loss function. This process is formulated as:EQUATIONwhere 𝐻 (𝑎) represents the information entropy of the output probabilities for the selected combinations of image processing techniques. The rationale is that a lower loss indicates that the prompted images can predict precise masks, suggesting a need to decrease the entropy of the prompt probabilities to diminish the randomness in selecting prompts. Conversely, a higher loss necessitates increasing the randomness to explore more combinations. Accordingly, the gradient of 𝜃 is computed as:EQUATIONThis enables updating the parameter 𝜃 via gradient descent:EQUATIONwhere 𝛼 is the learning rate. In practice, we discover that the gradient stipulated in Eq. 6 can be directly derived from the gradient of Eq.3, elucidated as follows:EQUATIONThis revelation allows for the direct use of the gradient from Eq. 3 to update the image prompter effectively. Overall, the algorithm for adapting foundational segmentation models with image prompter is summarized in Alg. 1. 
experiments 4.1 datasets: 
 To evaluate the adaptability of our proposed method, we conducted a series of rigorous experiments across four categories: common scenes, camouflage objects, medical images, and industrial data, encompassing seven image modalities: depth, thermal, camouflage, endoscopic, ultrasound, grayscale, and natural images. Common Scenes. We utilized the NJU2K dataset [15] for depth imagery, featuring 2,000 stereo images with matching depth maps. Additionally, the VT1K dataset [33] provided 1,000 spatially aligned RGB and thermal infrared image pairs with ground truth annotations. The depth and thermal images from these datasets were directly used as input images for evaluating the adaptability. Camouflage Objects. Three datasets were used, namely the CAMO dataset [18] with 1,250 images, the COD10K dataset [7] with 3,040 training and 2,026 testing images, and the NC4K dataset [26] with 4,121 images, capturing objects in complex, camouflaged settings. Medical Images. The Kavsir dataset [12] provided 1,000 annotated endoscopic images for polyp segmentation, while the BUSI dataset [1] included 780 ultrasound images of breast tumors with detailed annotations. Industrial Data. The MTSD dataset [11] contained 388 magnetic tile images with defect labels, and the KoletorSDD2 dataset [2] included 356 images of surface defects with various imperfections.Search Spaces. The image processing techniques are systematically categorized into distinct subsets based on image intensity characteristics, as delineated in Fig. 3 . For methodologies pertinent to the manipulation of brightness, contrast, and saturation, their respective value spectra were bifurcated into two discrete segments: the first ranging from min value to half of the max value, designated for diminishing effects, and the second from half of the max value to max value, allocated for augmentative adjustments. Consequently, these methods can be denoted using an abbreviation format, where the abbreviation consists of an "id" subscript indicating the numerical values employed. Learned Image Prompts, ours BI:2-GC:5-CE:3 BE:6-GB:9-SR:5 SR:7-SR:3-GC:0 SR:7-SE:2-SE:0 BE:0-SE:4-CE:3 GS:3-BX:6-MF:0 GS:7-BI:8-CR:1 SE:2-BE:9-BX:3 CE:8-GS:9-BX:8Table 2 : Main results of 10-shot learning. 'FT' denotes the employment of fine-tuning techniques with adapter modules, while 'IP' represents the process of evaluating model performance through the application of image prompts during testing.assessment of segmentation outcomes. The mIoU for an individual image is derived by computing the average Intersection over Union (IoU) score, which is achieved by correlating each predicted mask with its respective ground truth mask. This procedure entails the calculation of the IoU for each predicted mask in relation to all the ground truth masks associated with a given image, culminating in the identification of the highest IoU value. Thereafter, the mIoU is ascertained by averaging these peak IoU values across the entire ensemble of predicted masks present within the image.Tab. 1, Tab. 2, Tab. 3 present the results for the 5-shot, 10-shot, and 10%-shot experimental configurations. Overall, it is observed that the accuracy of both fine-tuning (FT) and the proposed imageprompts (IP) methodology improves with an increase in training samples. Under conditions of limited samples, such as the 5-shot and 10-shot scenarios, fine-tuning exhibits lower performance compared to image-prompts, potentially due to an insufficient sample Learned Image Prompts, ours CE:8-GB:6-LT:5 GB:4-GS:5-BR:9 CR:4-GC:0-BX:1 SR:1-GC1-BX:2 CR:4-GC:0-BX:1 GS:2-GC:8-GB:5 MF:7-GC:9-GC:7 SR:1-GC:1-BX:2 CE:7-GS:1-GB:7 size to effectively train fine-tuning adapters. In contrast, imageprompts can still achieve commendable image pre-processing results. Regarding depth images, the analysis indicates a propensity for contrast enhancement operations to be prevalent in the segmentation process of depth images. Visualizations presented in Fig. 2 (a) demonstrate significant enhancements in the delineation of foreground and background elements within depth maps as a result of contrast adjustments. In the context of thermal images, the Gaussian blur operation is frequently applied, suggesting that there is a discernible advantage to moderating edge intensity in thermal imagery. Fig. through the employment of image prompts. For camouflage images, although enhancements were observed across various datasets and models as a result of image prompting, the extent of improvement was somewhat limited. This constraint is attributed to the high variability present within camouflage datasets, which presents challenges in identifying a uniform prompt applicable to all scenes. As depicted in Fig. 2(c ), the search methodology predominantly concentrates on adjusting color saturation to counteract the effects of color diversity, given the heterogeneous nature of the scenes and the color-dependent distinction between foreground and background elements in camouflage images. In the case of endoscopic images, the findings underscore the recognition and application of processing techniques such as blurring operations (e.g., Gaussian blur, box filter) and color adjustments (e.g., saturation, contrast) to alleviate the impact of illumination-induced bright spots within endoscopic imagery, as exemplified in Fig. 3 This section examines the capacity for image prompts to be transferred across diverse domains, frameworks, and models. The crossdomain experiments, as delineated in Tab. 4, revealed a significant degree of transferability between the Kavsir and KolektorSDD2 datasets. This observation can be ascribed to the common imperative of alleviating the impact of illumination variations present in both datasets. Such a shared necessity highlights the critical role of emphasizing salient objects to augment the precision of segmentation tasks. Moreover, the cross-framework experiments presented in Tab. 5 indicated a constrained transferability of prompts identified by the SAM and SEEM methodologies. This observed limitation is attributed to the variances in the training datasets and the architectural design of the models employed in these two frameworks, implying that these discrepancies present formidable barriers to the transferability of prompts. Finally, the cross-model experiments detailed in Tab. 6) showcased a reciprocal transferability among models within the SAM framework, whereas such transferability was not evident in the SEEM framework. We surmise that the divergent training paradigms among SEEM models may be responsible for this observed inconsistency.The ablation studies were meticulously conducted on the Kavsir dataset to dissect and comprehend the various components and their impact on the efficacy of our proposed approach. Impact of Searching Sample Quantity. Tab. 7 elucidates the relationship between the size of the training dataset and the subsequent performance of the image prompts. The study's outcomes underscore the robustness of our methodology, which yields commendable results even under the constraints of a limited training regime, specifically in scenarios where as few as eight images are utilized. This observation is particularly pertinent in scenarios where the availability of annotated data is restricted, thereby underscoring the method's adaptability and resilience. gathered indicates that the adoption of central point forms is associated with the most advantageous outcomes. This finding suggests that the strategic selection of point prompts, particularly those that are centrally located, can significantly enhance the precision and effectiveness of the image segmentation process.Influence of Image Processing Method Diversity. Tab. 9 articulates the benefits of exploring an array of image processing methodologies. The empirical analysis conducted reveals that there exists an optimal threshold in the number of image processing methods that can be effectively integrated into the segmentation algorithm. Specifically, the performance of the algorithm plateaus once the number of explored methods reaches three. This saturation point implies that beyond a certain limit, the incremental addition of image processing techniques does not necessarily translate to a proportional improvement in performance, thereby providing valuable insights into the efficient allocation of computational resources and the strategic selection of image processing methods for optimization purposes. 
conclusion: 
 This paper presents a significant advancement in the domain of image segmentation by introducing the "prompting-to-adapt" paradigm, which addresses the critical issue of limited generalization in foundational segmentation models. Our approach eschews the traditional "training-to-adapt" methods that demand extensive retraining and architectural changes, instead opting for a more efficient and adaptable solution. By employing an image prompter that leverages few-shot learning and diverse image processing techniques, our method significantly improves adaptation capabilities without extensive retraining. Our strategy also introduces an innovative gradient descent method to optimize prompts, ensuring effective domain adaptation. Experiments on nine datasets validate our approach's effectiveness and the interpretability of prompts provides insights into image processing mechanisms.  
: 
 (c) Our proposed "prompting-to-adapt" paradigm searches for the optimal combination of image processing prompts in a few-shot manner, adapting the models by altering the context of input images.to advancing diagnostic processes in medical imaging. Recent advancements have led to the development of robust foundational models for image segmentation, such as the segment anything model (SAM) [16] and the segment everything everywhere model (SEEM) [44] These models are distinguished by their use of sophisticated prompts ranging from semantic texts that provide contextual understanding to spatial cues that highlight specific areas of interest, which facilitates the accurate segmentation of objects within a diverse array of images. Despite these significant strides, the application of foundational segmentation models to images outside the scope of standard natural scenes presents substantial challenges [3, 36] . This limitation is especially pronounced in domains featuring highly variable conditions, such as medical imagery, industrial scenes, and camouflaged environments. These contexts often exhibit complex background textures, fluctuating lighting conditions, and partial object occlusions, all of which demand an enhanced level of adaptability and precision from segmentation models. Previous methods [3, 36] for adapting foundational segmentation models typically involve intensive training and architectural modifications, which, while effective to a degree, significantly impede the models' ability to generalize and operate efficiently across different domains.In this paper, we propose an novel adaptation paradigm, changing the perspective from "training-to-adapt" to "prompting-toadapt" (as illustrated in Fig. 1 ), to overcome the above challenges. Central to this approach is the development of an innovative image We show the potential to adapt the foundational segmentation models to a variety of image domains, such as depth, thermal, camouflage, endoscopic, ultrasound, and grayscale images. This adaptation is facilitated through the strategic application of traditional image processing techniques. These techniques serve as effective prompts, guiding the models to adjust to the specific characteristics of various target domains. Please zoom in this figure for additional details.prompter, which employs few-shot image-mask pairs to generate domain-specific prompts. These prompts incorporate a variety of image processing techniques, aiming to adjust the content of input images for the adaptability of segmentation models without necessitating comprehensive model overhauls or extensive finetuning. To overcome the intrinsic challenge of non-differentiable image prompts, we further devise an information-estimation-based gradient descent strategy. This strategy capitalizes on the information entropy of image processing combinations, enabling the efficient and effective optimization of the image prompter. Through extensive experiments across diverse domains, we demonstrate the profound impact of our "prompting to adapt" paradigm. This approach not only facilitates significant improvements in the adaptability of foundational segmentation models but also illuminates the interpretability and practical applicability of the generated prompts. For example, as shown in Fig. 2(d) , by using the generated prompts, we can modify the lighting conditions of endoscopic images to mitigate the impact of bright spots in original input images.Our contributions can be summarized as follows:• We propose a novel adaption paradigm termed "promptingto-adapt", which significantly enhances the flexibility of foundational segmentation models across a diverse array of image domains. This approach is notable for its departure from traditional, labor-intensive adaptation methods, instead relying on the strategic use of image-mask pairs to generate domain-specific prompts. These prompts, comprised with varied image processing techniques, facilitate model adaptation without necessitating modifications to model architectures or extensive retraining. • We propose an information-estimation-based gradient descent strategy that effectively addresses the challenge of non-differentiable image prompts by leveraging the information entropy inherent in the combinations of image processing techniques. Beyond the quantitative enhancements in model performance, we also show the interpretability of the generated prompts. These prompts not only aid in model adaptation but also offer valuable insights into the underlying image processing mechanisms, providing a deeper understanding of how segmentation models can be effectively adapted to new and challenging environments. • We conduct extensive experiments on nine datasets, i.e., NJU2K [15] , VT1K [33] , CAMO [18] , COD10K [7] , NC4K [26] , Kavsir [12] , BUSI [1] , MTSD [11] , and KolekorSDD2 [2] , which cover seven image domains including depth, thermal, camouflage, endoscopic, ultrasound, grayscale, and natural images, across four scenarios: common scenes, camouflage objects, medical images, and industrial data. By applying our image prompting approach to foundational segmentation models such as SAM [16] and SEEM [44] , we demonstrate significant improvements in segmentation performance across various image domains. The key to adaptation lies within an image prompter, meticulously trained to craft effective prompts for adaptation. This method is designed to safeguard the core architecture of the segmentation models, thereby maintaining their inherent, pre-trained capabilities. In the inference stage, the generated prompts, which embody a blend of image processing techniques, are directly applied to the input images. This ensures that the foundational segmentation models can adeptly adjust to new inputs, leveraging the specifically designed prompts to achieve enhanced performance without altering the structural essence of the models. 
 
 Appendix:  


Decision: The manuscript introduces a 'prompting-to-adapt' paradigm designed to enhance the generalization capabilities of foundational segmentation models across various image domains. 
Extensive experiments conducted on a diverse array of datasets have demonstrated the method's significant improvements in segmentation performance, underscoring the adaptability and robustness of the proposed approach. 
All reviewers commend the novelty of the method and the practicality of the solution.

Reviews: review: paper_summary: paper_summary: The authors propose a novel adaptation paradigm, termed "prompting-to-adapt", to tackle the limitations of generalization capabilities and efficiency in deployment by introducing an innovative image prompter. First, the image prompter generates domain-specific prompts through few-shot image-mask pairs, incorporating diverse image processing techniques to enhance adaptability. Then, the authors further devise an information-estimation-based gradient descent strategy to tackle the inherent non-differentiability of image prompts. Specifically, this strategy leverages the information entropy of image processing combinations to optimize the prompter, ensuring effective adaptation. Finally, extensive experiments demonstrate that prompting-to-adapt brings significant improvements in segmentation performance across various image domains.
strength_weakness: strength_weakness: 1. A key strength of this paper is its novel approach to enhancing the generalizability of foundational segmentation models across diverse image domains through the introduction of the 'prompting-to-adapt' adaptation paradigm.
2. The extensive experimental evaluation on nine datasets spanning four different scenarios robustly demonstrates the superior performance of the proposed method.
limitations: limitations: 1. It would be beneficial for the authors to provide a clearer definition and delineation of the task setting to aid the reader's understanding.
2. The comparative performance of the proposed method, as shown in Table 3, indicates a lower performance than the 'FT' method on certain datasets. An in-depth discussion and analysis from the authors regarding this discrepancy would be valuable.
3. Given the current landscape of research, a discussion on how the authors' work relates to and differentiates from existing works, such as [1] and [2], would be advantageous for situating this paper within the field.

[1] Learning to Prompt Segmentation Foundation Models
[2] Customizing Segmentation Foundation Model via Prompt Learning for Instance Segmentation
paper_summary: paper_summary: This paper introduces “prompting-to-adapt” paradigm in place of the conventional “training-to-adapt” paradigm. The novel paradigm searches for the optimal combination of diverse image processing prompts in a few-shot manner, adapting the models by altering the context of input images. For image prompter, long short-term memory (LSTM) layer is designed to sequence and intensity of the selected image processing operations. To directly use gradient to update the image prompter, this paper further proposes an information-estimation-based gradient descent strategy which leverages the information entropy inherent in the combinations of image processing techniques.
strength_weakness: strength_weakness: 1.	The novel “prompting-to-adapt” paradigm provides a new perspective on the adaptation of foundational segmentation models. Better generated prompts can achieve enhanced performance without changing the structure of the model.
2.	The search space comprehensively encompasses a wide variety of semantic destructive and semantic-preserving image processing techniques, ensuring a comprehensive assessment of effects on the adaptability and performance of segmentation models.
3.	 The information-estimation-based gradient descent strategy has been designed to allow for the direct use of the gradient to update the image prompter effectively. This design addresses the challenge of non-differentiable image prompts by leveraging the information entropy inherent in the combinations of image processing techniques. 
4.	A comprehensive numerical study has been conducted based on nine benchmarks spanning seven image domains, and results can validate the advantages of the proposed paradigm against the conventional “training-to-adapt” paradigm.
limitations: limitations: 1.	The use of LSTM might not be the most efficient approach to design image prompter. Although the paper mentions the use of transformer to design the image prompter, the lack of comparisons with models of other structures leaves room for improvement in the robustness and effectiveness of the methodology.
2.	Tab. 5 shows a constrained transferability of prompts identified by the SAM and SEEM methodologies. While the analysis of the limitation is reasonable, this point proves that the transferability of the paradigm needs to be further improved.
3.	It is not explicitly indicated whether the segmentation results of the original images are obtained by the fine-tuning model or directly from the model without fine-tuning. Visualization of the results obtained in different ways can provide a more intuitive illustration of the effectiveness of the paradigm.
paper_summary: paper_summary: To address the poor generalization ability of foundational segmentation models on varied image domains, the authors proposed to generate domain-specific prompts to modify the input images. These prompts represent a series of image processing techniques to be applied on the input images, aiming to enhance the model adaptation performance without needing elaborate architecture or extensive re-training. Extensive experimental results across various image domains showed that the proposed method greatly boosts the adaptation performance of foundation segmentation models.
strength_weakness: strength_weakness: - I like the idea of formulating prompts for vision models as a sequence of image processing techniques; it explores a more interpretable prompting technique, and is in general novel and reasonable to me.
- The reported performance gain of the proposed method over fine-tuning with adapters is promising.
- The authors have conducted experiments on a wide range of datasets involving distinct image domains (camouflage, medical, industrial, etc), showing wide applicability of the proposed method.
limitations: limitations: - Details on the experiment
  - The authors didn't seem to disclose the details on the adapter modules they used for they experiments (SAM-FT and SEEM-FT in the tables).
  - There are too few words explaining "cross-domain transferability" experiment in Table 4, making it confusing. Taking Row 3, Column 2 (-6.76) as an example, did the authors train the prompter for SAM-H on VT1K and then directly use the prompter for SAM-H on NJU2K, resulting in a performance drop when compared to the mIOU on VT1K dataset?
  - Adding a comparison between the proposed IP method and full-parameter fine-tuning method can enhance the demonstration of the proposed method's practical applicability. That said, I understand that it may be difficult to fully tune models so large as SAM, so it is totally fine if such comparison is not feasible due to practical factors.

- Discussion on the image prompts
  - This work transforms the input images with image processing prompts. Other works like VPT[1] and EVP[2] that also modulates the inputs to achieves parameter-efficient tuning. What are the merits/insights of learning such transformation on the proposed search space over learning on the pixel space as in [1,2], apart from better interpretability?
- Necessity of "Gradient via Information Estimation"
  - There are a few common-practice tricks, e.g., Gumbel-softmax, that handle the gradient problem when learning undifferentiable processes. Could the authors clarify the advantages of the proposed "gradient via information estimation" method over these existing approaches?

- Insights for specific samples
  - In Fig. 2, some prompts seem to make the target object more difficult to recognize. (e.g., the second grid in Fig. 2(b), the grids in Fig. 2(c), etc.) Would the authors provide some explanation or insights on these counter-factual samples?
  - Also, for some samples, very small difference is obseved between original images and prompted images. However, they lead to significant improvement in the resulted mIOU. So I am curious how sensitive the method is/brings to the perturbations? How much of this sensitivity comes from the foundation model itself?

- Minor
  - Formula. In Eq. (1), the definition of $a$is missing. The definition of $R$and the relationship between $a$and $\tau$, are not easy to follow. In the following text, the mixed use of$a\sim p$, $\tau\sim p$and $\tau\sim a$make it even more confusing. I think it will be much better to clarify these symbols. Also, adding subscripts for $\arg\max$ in Eq. (1) will improve the readability. 
  - The designs in "Search Spaces" part (Line 554-560) are quite confusing. Just to confirm: the prompter will maximize the brightness of the image when given an intensity value of 1.0, while minimize it when given intensity value of either 1.0 or 1.9, is that right?

[1]  Visual prompt tuning. (ECCV' 22)

[2] Unleashing the Power of Visual Prompting At the Pixel Level. (TMLR' 24)