Part-aware Prompted Segment Anything Model for Adaptive Segmentation
Authors: Chenhui Zhao
Liyue Shen
Venue: Accepted by TMLR ()

Abstract: Precision medicine, such as patient-adaptive treatments assisted by medical image analysis, poses new challenges for image segmentation algorithms due to the large variability across different patients and the limited availability of annotated data for each patient. In this work, we propose a data-efficient segmentation method to address these challenges, namely $\textit{\textbf{P}art-aware}$ $\textit{\textbf{P}rompted}$ $\textit{\textbf{S}egment}$ $\textit{\textbf{A}nything}$ $\textit{\textbf{M}odel}$  ($\mathbf{{P}^{2}SAM}$). Without any model fine-tuning, $\text{P}^2\text{SAM}$ enables seamless adaptation to any new patients relying only on one-shot patient-specific data. We introduce a novel part-aware prompt mechanism to select multiple-point prompts based on part-level features of the one-shot data, which can be extensively integrated into different promptable segmentation models, such as SAM and SAM 2. To further promote the robustness of the part-aware prompt mechanism, we propose a distribution-guided retrieval approach to determine the optimal number of part-level features for a specific case. $\text{P}^2\text{SAM}$ improves the performance by $\texttt{+} 8.0\%$ and $\texttt{+} 2.0\%$ mean Dice score for two different patient-adaptive segmentation applications, respectively. In addition, $\text{P}^2\text{SAM}$ also exhibits impressive generalizability in other adaptive segmentation tasks in the natural image domain, $\textit{e.g.}$, $\texttt{+} 6.4\%$ mIoU within personalized object segmentation task. Code will be released upon acceptance.

Published in Transactions on Machine Learning Research (05/2025)
Part-aware Prompted Segment Anything Model for Adaptive Segmentation
Chenhui Zhao
chuishao@humich.oo edu
Deportment of Computer Science and Engineering, University of Michigan
Liyue Shen
liyues@umich. edu
Department of Electrical and Computer Engineering, University of Michigan
Reviewed OpenReview: https: //tp::///ppereveeu.. net/ formm? id= ccc.cc/hhhcoffrPPoooo////2--1-1.
Abstract
Precision medicine, such patient-adaptive treatments assisted by medical image anal- ysis, poses challenges for segmentation algorithms in adapting to new patients, due to the large variability across different patients and the limited availability of annotated data for each patient. In this work, we propose a data-efficient segmentation algorithm, namely Part-aware Prompted Segment Anything Model (P2SAM), Without any model fine-tuning, P'SAM enables seamless adaptation to any new patients relying only on one- shot patient-specific data. We introduce a novel part-aware prompt mechanism to selects multiple-point prompts based on the part-level features of the one-shot data, which can be extensively integrated into different promptable segmentation models, such as SAM and SAM 2. Moreover, to determine the optimal number of parts for each specific case, we propose distribution the part-aware prompt mechanism. PºSAM improves the performance by +8.0% and +2.0% mean Dice score for two different patient-adaplive segmentation applications, respectively. In addition, P2SAM also exhibits impressive generalizability in other adaptive segmentation tasks in the natural image domain, e.g., +6.4% mloU within personalized object segmenta- tion task. The code is available at https: sttps://github http:hubbccccccccc/1////a/aaaaa
retrieval approach that further enhances the robustness of
1 Introduction
Advances in modern precision medicine and healthcare have emphasized the importance of patient-adaptive treatment (Hodson 2016). For instance, in radiation therapy, the patient undergoing multi-fraction treat- ment would benefit Trom longitudinal medical data analysis that helps timely adjust treatment. plan- ning (Sonke al., 2019). To facilitate the treatment procedure, such analysis demands timely and accurate automatic segmentation of tumors and critical organs from medical images, which has underscored the role of computer vision approaches for medical image segmentation tasks (Hugo et al. 2016 Jha et al. 2020 Despite the great progress made by previous works (Ronneberger et al. 2015 Isensee et al. 2021 their focus remains on improving the segmentation accuracy within a standard paradigm: trained on a large num- ber of annotated data and evaluated on the internal validation set. However, patient-adaptive treatment presents unique challenges in adapting segmentation models to new patients: (1) the large variability across patients hinders dircct model transfer, and (2) the limited availability of annotated training data for each patient prevents fine-tuning the model on a per-palient basis (Chen al., 2023). Overcoming these obstacles requires segmentation approach that can reliably adapt to ezternal patients, in data-efficient manner.
In this work, we address the unmet needs of the patient-adaptive segmentation by formulating it as an in-context segnentation problem, where the conlent is the prior data from a specific patient. Such data can obtained in a standard clinical protocol (Chen et al. 2023), therefore will not burden clinician. To this end, we propose P²SAM: Part-aware Prompted Segment A Anything Model. Leveraging the promptable segmentation mechanism inherent Segment Anything Model (SAM) (Kirillov et al., (023), our method seamlessly adapts to any external patients relying only on one-shot patient-specific prior data without re-

---

Published in Transactions on Machine Learning Research (05/2025)
Natural domain Application
Positive Point
Radiation Therapy
Positive Point
Negative Point
SAM
PASAM
Target: Backpack
Result
Prior Data
Follow-up Visit
Result
Medical domain Application
Positive Point
Endoscopy Video
Positive Point
Negative Point
SAM
PÅSAM
e
Target: Polyp
Result
Prior Data
Follow-up Frame
Result
Figure 1: Illustration of SAM's ambiguity property. The ground truth is circled by a red dashed circle; the predicted mask is depicted by a yellow solid line.
Illustration of two patient-adaptive segmentation tasks. P²SAM can segment the follow-up data by utilizing one-shot prior data as multiple-point prompts. Prior and predicted masks are depicted by a solid yellow line.
Figure 2:
quiring additional training, thus in a data-efficient manner. Beyond patient-adaptive segmentation, P2SAM also demonstrates strong genoralizability in other adaptive segmentation tasks in the natural image domain, such as personalized segmentation (Zhang et al. 2023) and one-shot segmentation (Liu el al. 2023
In the original prompt mechanism of SAM (Kiilllov al., 2023), as illustrated in Figure ! single-point prompt may result in ambiguous prediction, indicating the limitation in both natural domain and medical domain applications (Zhang et al., 2023 Huang et al. 2024).. To alleviate the ambiguily, following the statement in SAM, "ambiguity is much rarer with multiple prompts", we propose novel part-aware prompt mechanism that meticulously presents the prior data as multiple-point prompts based on part-level features. As illustrated in Figure 2 our method enables reliable adaptation to an external patient across various tasks with one-shot patient-specific prior data. To extract part-level features, we cluster the prior data into multiple parts in the feature space and computing the mean for each part. Then, we select multiple- point prompts based the cosine similarity between these part-level features and the follow-up data. The proposed approach can be generalized to different promptable segmentation models that support the point modality, such as SAM and its successor, SAM 2 (Ravi et al.) 2024). Here, we primarily utilize SAM as the backbone model, and SAM 2 will be integrated within the specific setting.
On the other hand, when the number parts is set suboptimally, either more or less, the chance of encoun- tering outlier prompts may incrcase. In the extreme, assigning all image patches to a single part produces an ambiguity-aware prompl (Zhang et al. 2023). whereas assigning each image patch
different part yields many outlier prompts (Liu et al.) 2023 Determining the optimal number of parts is non-trivial, as it may vary across different cases. Here, introduced novel distribution-guided retrieval approach to investigate the optimal number of parts required by each case. This retrieval approach is based on the distribution distance wetween the foreground feature of the prior image and the resulting feature obtained under the current part count. This principle motivated by the fact that tumors and normal organs always lead to distinct feature distributions within medical imaging technologies SGarcía-Figucire et al. 2019).
With the aforementioned designs, PºSAM tackles fundamental challenge ambiguity whenn when adapting promptable segmentation models to specific applications. When ambiguity is not issue, P2SAM enhances model generality by providing curated information. The key contributions of this work lie in three-fold:
1. We formulate the patient-adaptive segmentation in data-efficient. segmentation approach, P²SAM, that requires only one-shot prior data and no model fine-tuning. P²SAM functions generic segmentation algorithm, enabling efficient and flexible adaptation across different domains, lasks, and models.
in-context segmentation problem, resulting
We propose a novel part-aware prompt mechanism that can select multiple-point prompts based on part-level features. Additionally, we introduce distribution-guides retrieval approach to determine the optimal number of part-level features required by different cases. These designs significantly enhance the gencralizability of promptable segmentation models.
2

---

Published in Transactions on Machine Learning Research (05/2025)
3. Our method largely benefits real-world applications like patient-adaptive segmentation, one-shot seg- mentation, and personalized segmentation. Experiment results demonstrate that P*SAM improves the performance by +8.0% and +2.0% mean Dice score in two different patient-adaptive segmenta- tion applications and achioves a new statc-of-the-art result, é.e,, 95.7% mIoU on the personalized segienttttio benchmark PerSeg.
2 Related Work
Segmentation Generalist. Over the past decade, various segmentation tasks including semantic segmen-
tation (Strudel et al. 2021 Li ct al. 2023a), instance segmentation (Hc et al. 2017 Li et al. 2022a), panoptic
segmentation Carion et al. 2020 Cheng al. 2021 Li et al. 2022b), and referring segmentation (Li et al.
2023b; Zou et al. 2024) have been extensively explored for the image and video modalities. Motivated by
the success of foundational language models (Radford et al.] 2018 2019 Brown et al. 2020 Touvron et al, 2023), the computer vision research community is increasingly paying attention to developing more gener- alized models that can tackle various vision or multi-modal tasks, or called foundation models (Li et al.
2022b; Oquab et al., 2023, Yan et al.) 2023; Wang et al. 2023a b Kirillov et al., 2023). Notably, Segment Anything model (SAM) (Kirillov et al., 2023) and its successor, SAM 2 (Ravi et al. 2024) introduces a promptable model architecture, including the positive- and negative-point prompt; the box prompt; and the
mask prompt. SAM and SAM 2 emerge with an impressive zero-shot interactive segmentation capability after pre-training on the large-scale dataset. The detail of SAM can be found in Appendix A
Medical Segmentation. Given the remarkable generality of SAM and SAM 2, researchers within the modical image domain have been socking to build foundational models for medical image segmentation (Wul et al.) 2023 Wong et al. 2023; Wu & Xu 2024 Zhang & Shen (024)) in the same interactive fashion. To date, SeribblePrompt (Wong et al., 2023) and One-Prompt. (Wu & Xul, (024) introduce a new prompt modality- scribble that provides more flexible option for clinician usage. MedSAM (Ma et al. 2024a] fine- tunes SAM on an extensive medical dataset, demonstrating significant performance across various medical image segmentation Lasks. Its successor (Ma el al. 2024b) incorporates SAM 2 to segment a 3D medical image volume as a video. However, these methods rely on clinician-povvided prompts for promising segmentation performance. Moreover, whether these methods can achieve zero-shot performance as impressive as SAM and SAM 2 remains an open question that requires further investigation (Ma et al. (024b)..
In-Context Segmentation. The concept of in-context learning is first introduced natural language processing (Brown et al. 2020), allowing the model to adapt to unseen input patterns with a few prompts and examples, without the need to fine-tune the model. Similar ideas (Rakelly et al. 2018 Sonke et al. 2019 Li et al. 2023b) have been explored in segmentation tasks. For example, few-shot sogmentation RRakelly ct al. 2018 Wang et al., 2019b Liu et al. 2020 Leng et al. 2024) like PANct (Wang jet al. 2019b); aims Lo segment new classes with only a lew examples: in adaptive therapy (Sonke et al. 2019).
new paradigm in
several works (Elmahdy et al. 2020 Wang et al., 2020 Chen et al. (023) attempt to adapt a segmentation
model to new patients with limited patient-specific data, but these methods require model fino-tuning in different manners. Recent advancements, such Painter (Wang et al.) 2023a) and SegGPT (Wang et al.
2023b) pioneer novel in-context segmentation approaches, enabling the timely segmentation of images based on specified image-mask prompts. SEEM (Zou et al.) 2024) further explores this concept by investigating different prompt modalities. More recently, PerSAM (Zhang et al. 2023) and Matcher (Liu et al., 2023) have utilized SAM to tackle few-shot segmentation through the in-context learning fashion. However, PerSAM prompts SAM with a single point prompt, causing ambiguity in segmentation results and therefore requires an additional fine-tuning stralegy. Matcher samples multiple sets of point prompts but based on patch-level features. This mechanism makes Matcher dependent on DINOv2 (Oquab et al. 2023) to generate prompts, which is particularly pre-trained under a patch-level objective. Despite this, Matcher still generates a lot of outlier prompts, therefore relies on a complicated framework to filter the outlier results.
In this work, we address the aatient-adaptive segmentation problem, also leveraging SAM's promptable ability. Our prompt mechanism is basod on part-level fcatures, which will not cause ambiguity and are more robust than patch-level features. The optimal number of parts for each case is determined by a distribution- guided retrieval approach, further enhancing the generality of the part-aware prompt mechanism.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Part-level Features
Test Image h
F
Prior Data Og. Mg3
Similarity Maps (S")==
Result My
Prompiable Decoder
Encadar
(RR) K F.
Pa[M2
Point- Peint-Prompts *, *]
Cosine Similarity
IPA
Figure 3: Illustration of the part-aware prompt mechanism. Masks are depicted by yellow solid line. We first cluster foreground features the reference image into part-level features. Then, we select multiple-point prompts based on the cosine similarity (& in the figure) between these part-level features and target image features. A colorful star, matching the color of the corresponding part, denotes a positive-point prompt, while a gray star denotes negative-point prompt. These prompts are subsequently fed into the promptable decoder to do prediction.
3
Method
In Section 3.1 we define the problem within the context of palient-adaptive segmentation. In Section 3.2 we present the proposed methodology. P'SAM, within a broader setting of adaptive segmentation. Section 3.3 we introduce an optional fine-tuning strategy when adapting the backbone model to medical image domain is required.
3.1 Problem Setting
Our method aims to adapt a promptable segmentation model to ezternel patients, with only one-shot patient- specific prior data. As shown in Figure 2 such data can be obtained in a standard clinical protocol, either from the initial visit of radiation therapy or the first frame of medical video. The prior data includes a reference image IR and mask MR delineating the segmented object. Given target image, Ir, our goal is to predict its mask Mr, without additional human annotation costs or model training burdens.
3.2 Methodology Overview
The setting described in Section 3.1 can be extended to other adaptive segmentation tasks in the natural image domain where the target image represents new view or instance of the object depicted in the prior data. As shown Figure 3 we illustrate our part-aware prompt mechanism using a natural image to clarify the significance of each part. Additional visualizations for parts in medical images are provided in Appendix D Since no part-level definitions exist for the two diseases studied in this work, we refer these parts data-driven parts.
Part-aware Prompt Mechanism. We utilize SAM (Kiillov et al. 2023) as the backbone model here, but our approach can be generalized to other promptable segmentation models that support the point prompt modality, such as SAM (Ravi et al.) 2024). Given the reference image-mask pair from the prior data, {IR, MR), P'SAM first apply SAM's Encoder to extract the visual features FR E Rxxwxd from the reference image LR. Then, we utilize the reference mask MR to select foreground features FR (FR[MR 1]) by:
F/ = (FRy Mny = 1, 1, E IhX")
(1)
where Thxw is the spatial coordinate set of FR. We cluster Ff with k-mean++ Arthur et al. 2007) into n parts. Then, we obtain n part-level features (P E guxd by computing the mean of each part. In Figure 3 we showcase an example of n=4. Each part-level feature Pa represented by colorful star
the foreground feature space. We further align the features of each part with pixels in the RGB space, thereby contouring the corresponding regions for each part in the image, respectively. We extract the
4

---

Published in Transactions on Machine Learning Research (05/2025)
Prior (U My]
Results [MT
(lg. Mg]
Results (MT
*:: Positive Point * Negative Point
1.44
0. 0.75
Part-aware Prompts
Polyp
=
3
n1
3
Distribution Distance
HHoustield Value Density
1.21
2. 44
Wasserstein Distance
Can
Prior
Figure 4: Illustration of P²SAM's improvement. Wasserstein distances between the priors and results are shown in white.
Figure 5: Illustration of the distribation-guided re- trieval approach.
features Fr E Rhxuxd from the target image Ir using the same Encoder, and compute similarity maps {S9 Jem1 E RRxxkx based on the cosine similarity between part-level features {PRte=I and Fr by:
Pf: Frij
S
(2)
| |
We determinc positive-point prompts {Pos"]n Fa with the highest similarity score cach similarity map S In Figure each prompt Pos is depicted colorful star the corresponding similarity map S
For natural images, the background of the reference image and the target image may exhibit little correlation. Thus, following the approach in PerSAM (Zhang et al. 2023). we choose one negative-point prompt {Neg) with the lowest score on the average similarity map > St. {Neg} depicted as the gray star in Figure 3 However, for medical images, the background of the reference image is highly correlated with the background of the target image, usually both representing normal anatomical structures. As a result, in medical images, shown Figure 2 in Section [
we identify multiple negative-point prompts {Neg")) Fe=1 from the background. This procedure mirrors the selection of multiple positive-poiis poiit prompts but background fcatures F (Fn[MR 0)). Finally, we send both positive- and negative-point prompts into SAM's Prompiable Decoder and get the predicted mask MM for the target image.
Distribation-Guudde Retrieval Approach. Improvements of the part-aware prompt mechanism are illustrated in Figure The proposod approach can naturally avoid the ambiguous prediction introducod by SAM (e.g., polyp) and also improve precision (e.g., can). However, this approach may occasionally result in outliers, observed in the segmentation example in Figure [ n=3. Therefore, propose distribution-guiieed retrieval approach to answer the question, "How many part-level features should choose for each case?". We assume the correct target foreground feature Fff (Fy[M, 1l), and the reference foreground feature E'f. should belong the same distribution. This assumption grounded in the fact that tumors and normal organs will be reflected in distinct distributions by medical imaging technologies (García- Figueiras et. al.] 2019), also observed by the densily of Hounsfield Unit value in Figure 5 To retriive the optimal number of parts for a specific case, we first define N candidate part counts, and obtain N part- aware candidate segmentation results (M+)N In-1- After that, we extract N sets of target foreground features 1 [I=((() Jn=1- Following WGAN (Arjovsky et al. 2017). we utilize Wasserstein distance Pw to measure the distribution distance between reference foreground features Fy and each set target foreground features rn(()) We determine the optimal number of parts by:
min Du Dw (f ((()), nE(1I--N)
(3)
where the details of - can be found in Appendix Equation The smaller distance value for the correct prediction in Figure] indicates this approach be extended to multiple image modalities.
5

---

Published in Transactions on Machine Learning Research (05/2025)
3.3 Adapt SAM to Medical Image Domain
Segment Anything Model (SAM) (Kirillov et al., 2023) is initially pre-trained on the SA-1B dataset. Despite the large scale, notable domain gap persists between natural and medical images. In more realistic medical scenarios, clinic researchers could have access to certain public datasets (Aerts et al.) 2015 Jha. et al., 2020) tailored to specific applications, enabling them to fine-tune the model. Nevertheless, even after fine- tuning, the model can still be limited to generalize across various external patients from different institutions because of the large variability in pationt population, demographics, imaging protocol, etc. PºSAM can then be flexibly plugged into the fine-tuned model to enhance robustness on unseen patients.
Specifically, when demanded, we utilize internal medical datasets (Aerts al., 2015 Jha et al. 2020) to fine- tune SAM. We try full fine-tune, and Low-Rank adaptation (LoRA) (Hu et al. 2021 for further efficiency. During the fine-tuning, similar to Med-SA (Wu et al.
2023 we adhere closely to the interactive training
strategy outlined in SAM to maintain the interactive ability. Details can be found in Appendix [ Then, we
employ external datasets (Bernal et al. 2015 Hugo et al. 2016 obtained from various institutions to mimic
new patient cases. Note that there is no further fine-tuning on these datasets.
4 Experiments
In Section
4.1 we introduce our experimental settings. In Section we evaluate the quantitative results of our approach. In Section 4.3] we conducted several ablation studies to investigate our designs. In Section 4.4
4.2
we show qualitative results.
4.1 Experiment Settings
Dataset. We utilize a total of four medical datasets, including two internal datasets: The ASCLC-Radionin dataset (Aerts et al.) 2015), collected for non-small cell lung cancer (NSCLC) segmentation, contains data from 422 patients. Each patient has a computed tomography (CT) volume along with corresponding seg- mentation annotations. The Kvasir-SEG dataset (Tha et al. 2020 contains 1000 labeled endoscopy polyp images. Two enternal datasets from different institutions: The LD-Lung dataset (Hugo et al. 20166, collected for longitudinal analysis, contains data from 20 patients, within which 13 patients underwent multiple visits, 3 to 8 visits for each patient. For each visit, CT volume along with corresponding segmentation labels is available. The CVC-CliniDB dataset (Bernal et al., 2015), contains 612 labeled polyp images selected from 29 endoscopy videos. During experiments, internal datasels serve as the training dataset to adapt SAM to the medical domain, while external datasets serve as unseen patient cases.
Patient-Adaptive Segmentation Tasks. We test P'SAM under two adaptive segmentation tasks: NSCLC segmentation in the patient-adaplive radiation therapy and polyp segmentation in the endoscopy video. For NSCLC segmentation, medical image domain adaptation will be conducted on the internal datasct, WSCLC-Radiomics For P²SAM, experiments are then carried out the external dataset, 4D- Lung. We evaluate P'SAM patients who underwent multiple visits during treatment. For each patient, we utilize the image-mask pair from the first visit as the patient-specific prior data. For polyp segmentation, domain adaptation will be conducted on internal dataset, Kvasir-EEG. For P'SAM, experiments are then carried out on ezternal datasct, CVC-ClinicDB, For each video, we utilize the image-mask pair from the first stable frame as the patient-specific prior data.
Implementation Details. All experiments are conducted on A40 GPUs For the NSCLC-Radiomies dataset, we extract 2-dimensional slices from the original computed tomography scans, resulting in total of 7355 labeled images. As for the Kvasir-EEG dataset,
utilize all 1000 labeled images. We process two
datasets following existing works (Hossain et al. 2019 Dumitru et al.) 2023). Each datasct was randomly split into three subsets: training, validation, and testing, with an 80: 10: 10 percent. ratio (patient-wise splitting for the -SCio-Radi dataset to prevent data leak). The model is initialized with the SAM's pro-trained weights and fine-tuned on the training splitting using the loss function proposed by SAM. We optimize the model by AdamW optimizer [Loshchilov & Hutter (017)) (31 (8,=0.9, 3,.0.99 with a weight decay of 0.05. We further penalize the SAM's encoder with drop path of 0.1. We fine-tune the model for
6

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Results of NSCLC segmentation for patient-adaptive radiation therapy. We show the mean Dice score. base indicates tuning 5.5M parameters of the base SAM the NSCLC-Radiomiss dataset before lesting on the 4D-Lung dataset. t indicates freeeing-free method; & indicates the method using SAM.
Meta
LoRA
Full-Fine- Tune
Method
hagee..n
basea.Bx larges. 5.M
basee3.88
lange212.55
direct-transfer" fine-lune
56.10
57.83
58.18
61.11
52.11
32.55
55.27
53.85
PANet
(Wang et al.
2019b
4.28
5.24
7.79
40.03
44.70
Matcherft
Liu et al. 2023
13.28
50.81
50.88
59.52
57.67
PerSAMit
(Zhang et al.
20233
9.84
63.63
64.69
62.58
64.45
P2SAM (Ours)
28.52
64.38
67.00
66.68
67.23
Table 2: Results of polyp segmentation for endoscopy video. We show the mean Dice score for each method. 5.5% indicates tuning 5. 5M5M parameters of the base SAM on the Kvasir-SEG dataset before testing on the CVC-ClinieBB dataset. t indicates training-free method; $ indicates the method using SAM.
Meta
LoRA
Full-Fine- Tune
Method
huuee.
base5.Ex
large5.90
20.8 large
direct- trans--ran
77.20
81.16
84.62
86.68
fine-tune
75.29
79.50
83.14
86.67
PANet
(Wang et al.
2019b
38.22
44.61
55.48
75.99
86.48
Matcher Liu et al., 2023
63.54
78.65
79.56
85.17
87.15
PerSAMI+
(Zhang et al.
2023
45.82
79.02
81.63
85.74
87.88
PºSAM IT (Ours)
66.45
80.03
82.60
86.40
88.76
36 epochs on the NSCLC- Radiomics dataset and 100 epochs on the Kvasir-SEC dataset with a batch size of 4. The initial learning rate 1e-4, and the fine-tuning process is guided by cosine learning rate decay, with linear learning rate warm-up over the first 10 percent epochs. More details are provided in Appendix C
Summary. We test SASAM on external datasets with three different SAM backbones: 1. SAM pre-trained on the SA-1B datasct Kirillov et al. 2023). denoted as Meta. 2. SAM adapted on internal datascts with LoRA (u et al. 2021 and 3. full fine-tune, denoted as LoRA and Full-Fine- Tune, respectively. We compare P2SAM against. various methods, including previous approaches such as the direct-transfer: fine-tune on the 2019a Elmahdy et al. 2020 Wang et al.] 2020 Chen et al.) 2023); the one-shot [Wang et al., 2019b):: and concurrent mothods that also utilize SAM, such as 2023) and Matcher (Liu et al. 2023). For PANet, utilize its align method for
prior data (Wang et al.
segmentation mothod, PANet
PerSAM (Zhang et. al.,
one-shot segmentation. For Matcher, adopt its setting of FSS-1000 (Li et al.] 2020). It is important to
note that all baseline methods share the same backbone model as P²SAM does for fairness.
4.2 Quantitative Results
Patient-Adaptive Radiation Therapy. As shown in Table 1 on the 4D-Lung dataset (Hugo et al., 2016), P²SAM outperforms all other baselines across various backbones. Notably, when utilizing Meta, P'SAM can outperform Matcher by +15.24% and PerSAM by +18.68% mean Dice score. This highlights P²SAM's superior adaptation to the out-of-domain medical applications. After domain adaptation, P"SAM outperforms the Sirecl-transfer by +8.01%, Matcher by +11.60% and PerSAM by +2.48% mean Dice score, demonstrating that P2SAM is a more effective method to enhance generalization on the external data.
Discussion. fine-tune susceptible overfitting with one-shot data, PANet fully depends on the encoder, and Matcher selects prompts based on patch-level features. These limitations prevent them from surpassing
7

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Comparison with existing baselines. indicates using human-given box prompt during the inference time.
Table 4: Results of one-shot semantic segmentation. We show the mean IoU score for each method. Note that all methods utilize SAM's encoder for fairness.
Method AD-Lung CVC-ClinieBB
Method COCO-20'
FSS-1000 LVIS-92' PerSeg
baseline 69.00
83.14
Matcher
25.1
82.1
12.6
90.2
61.11
86.68
PerSAM
23.0
71.2
11.5
89.3
PºSAM
67.23
88.76
PºSAM (Ours)
26.0
82.4
13.7
95.7
Table 5: Comparison with tracking methods. * indicates utilizing Full Fine- Tune,
Ablation study for the number of parts and the retrieval. Default settings are marked in Gray
Table 6:
Method
4D-Lung CVC-ClinieDB
parts (n)
CVC-ClinicDB
PerSeg
AOT P2SAM SAM 2
62.34
w.
w. retrieval w.o.
w. retrieval
67.23
1 (PerSAM)
45.8
45.8
89.3
89.3
81.98
SAM PASAM
84.43
2
53.9
59.5
83.7
92.9
3
53.6
61.9
91.0
95.6
propoppropngation
57.00
82.92
4
54.3
63.1
93.8
95.6
PPSAM *
67.23
88.76
5
56.6
64.2
93.3
95.7
the direct-transfer. On the other hand, NSCLC segmentation remains a challenging task. We consider MedSAM (Ma et al., 2024a), which has been pre-trained on a large-scale medical image dataset, as strong baseline method. In Table MedSAM achieves a 69% mean dice score on the 4D-Lanng dataset with human- given box prompt at each visit, while P²SAM achieves comparable performance only with the ground truth provided at the first visit.
Endoscopy Video. As shown in Table 2 on the CVC-ClinicDB dataset. Bernnl et al. 2015). P²SAM still achieves the best result across various backbones. When utilizing Meta, P-SAM can surpass Matcher by +2.91% and PerSAM by +20.63% mean Dice score. After domain adaptation, P2SAM can outperform by +2.03%, Matcher by +1.81% and PerSAM by +0.88% mean Dice score. Demonstrates P²SAM's generality to various patient-adaptive segmentation tasks.
Discussion. All methods demonstrate improved performance in datasets like CVC-ClinieDB, which exhibit smaller domain gap (Matsoukas et al. 2022) with SAM's pre-training dataset. In Table 3 we compare our results with Sanderson & Matuszowski (2022), which is reported as the method achieving the best performance in Dumitru et al. (2023) under the same evaluation objective: trained on Kvasir-EGG dataset and tested on the CVC-ClinieDB dataset. Our direct-transfer has already surpassed this result, which can be attributed to the superior generality of SAM and our P²SAM can further improve the generalization.
On the other hand, we observe that P'SAM's improvements over PerSAM become marginal after domain adaptation (LoRA and Full Fine-Tune u.s. Meta) on both datasets. This is because, as detailed in Ap- pendix B the ambiguity inherent in SAM, which is the primary limitation of PerSAM, is significantly reduced after fine-tuning on a dataset with a specific segmentation objective. Nevertheless, our method shows that providing multiple curated prompts can achieve further improvement.
Comparison with Tracking Algorithms. In Table 5 we additionally compared P²SAM with tracking algorithms: the label-propagation (Jabri et al.) 2020), AOT Yang et al. 2021), and SAM (Ravi et al., 2024). On the 4D-Lung dataset, we only test algorithms with Full Fine- Tune due to the large domain gap (Matsoukas et al. 2022). P2SAM outperforms the label-propagation, as the discontinuity in sequential visits where the interval between two CT scans can exceed a week leads to siguificant changes in tumor position and features. On the CVC-ClinicDB dataset, dramatic content shifts within the narrow field of view can also lead to discontinuity. Despite this, SAM achieves competitive results even without additional domain adaptation. However, as we have stated, P'SAM can be integrated into any promptable segmentation model. Indeed, we observe further improvements when applying PºSAM to SAM 2.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Ablation study for the distribution distance measurement. Default settings are marked in Gray
Table 8: Ablation study for model sizes. 1 indicates the improvement when compared with the same size PerSAM. Default settings are marked in Gray
Algorithm
CVC-ClinieDB PerSeg
Model
CVC-ClinieDB
PerSeg
W.O.
54.3
93.8
PerSAM fuuge
45.8
89.3
Hungarian
61.1
95.6
P²SAM brase
55.1
90.0
26.01
Jensen Shannon
Jensen-Shannon
58.1
94.0
P'SAM large
63.8
95.6 9.0t
Wasserstein
63.1
95.6
P2SAM Ruge
63.1
95.6 6.31
Existing One-shot Segmentation Benchmarks. To further demonstrate P*SAM can also be generalized to natural image domain, we evaluate its performance on existing one-shot semantic segmentation bench- marks: COCO-20' (Nguyen Todorovic, 2019). FSSS-100 (Li et al. 2020), LVIS-92' (Liu et al. 2023), and a personalized segmentation benchmark, PerSeg Zhang et al. 2023). We follow previous works (Zhang et al.,
2023 Liu et. al. 2023) for data pre-processing and evaluation. In Table 4 when utilizing SAM's encoder, P-SAM outperforms concurrent works, Matcher and PerSAM, all existing benchmarks. In addition, P'SAM can achieve a new state-of-the-art result, 95.7% mean IoU score, on the personalized segmentation
benchmark PerSeg (Zhang et al., 2023)
4.3 Ablation Study
Ablation studies are conducted on the PerSeg dataset (Zhang et al. 2023) and CVC-ClinicDB dataset (Bernal et al., 2015) using Meta. We explore the effects of the number of parts in the part-aware prompt mechanism: the retrieval approach; distribution distance measurements in the retrieval approach; and the model size, which can be considered a proxy for representation capacity.
Number of Parts n. To validate the efficacy of the part-aware prompt mechanism, we establish method without the retrieval approach. As shown in Table (w.o. retrieval), for both datasets, even solely relying on the part-aware prompt mechanism, increasing the number of parts n enhances segmentation performance. When setting n=5, our part-aware prompt mechanism enhances performance by +10.7% mean Dice score on CVC-ClinicDEL +4.0% mean IoU score on PerSeg. These substantial improvements underscore the effective- ness of our part-aware prompt mechanism.
Retrieval Approach. The effectiveness of our retrieval approach is also shown in Table G (w. retrieval). When setting n=5,, the retrieval approach enhances performance by +7.6% mean Dice score on the CVC- CliniDBB dataset, +2.4% mcan IoU score on the PerSog dataset. These substantial improvements show that our retrieval approach can retrieve an appropriate number of parts for different cases. Moreover, these suggest that we can initially define a wide range of part counts for retrieval, rather than tuning it meticulously as a hyperparameter.
Distribution Distance Measurements. The cornerstone of our retrieval approach lies in distribution distance measurements. To evaluate the efficacy of various algorithms, in Table 7 we juxtapose two distribution-related algorithms,, namely Wasserstein distance (Rüschendorf 1985) and Jensen Shannon di- vergence (Menéndez et al., 1997), alongside a bipartite matching algorithm, Hungarian algorithm. Given foreground features from the reference image and the target image, we compute: 1. Wasserstein distance following the principles of WGAN (Arjovsky et al., 2017): 2. Jensen-Shannon divergence basod on the first two principal components of each feature; Hungarian algorithm after clustering these two sets of features into an equal number of parts. All algorithms exhibit improvements in segmentation performance compared to the w.o. retrieval bascline, while the Wasserstein distance is better in our context. Note that, the efficacy of the Jensen-Shamnon divergence further corroborates our assumption that foreground features from the reference image and a correct target result should align in the same distribution, albeit it faces challenges when handling the high-dimensional data.
Model Size. In Table g we investigate the performance of different model sizes for our P'SAM, i.c., base, large, and huge, which can alternatively be viewed as the representation capacity of different. backbones. For
9

---

Published in Transactions on Machine Learning Research (05/2025)
Visit 02
Visit 03
Visit 04
Visit 05
Frame 04
Frame 12
Frame 16
Frame 17
Ground Truth
Ground Truih
Marcher
Matcher
PerSAM
PerSAM
Ouus
Olurs
Figure Qualitative results of NSCLC segmenta- tion on the 4D-Lung datasct, with Meta.
Figure 7: Qualitative results of polyp segmentation on the CVC-Clinich dataset, with Meta.
Visit 02
05
Visit 02
Visit 04
Frame 07
Frame 13
Frame 15
Frame 19
Ground
Ground Truths
Direct- Transfer
Direst- Trensfer
Outs
Ours
Figure 8: Qualitative results of NSCLC segmenta- tion from two patients on the 4D-Lung dataset, with Full-Fine- Tune.
Figure 9: Qualitative results of polyp segmentation from video on the CVC-ClinicDB datasct, with Full-Fine-The Tune.
Prior
Mutchor
FerSAM
Ours
Figure 10: Qualitative results of personalized seg- mentation on the PerSeg dataset, compared with Matcher.
Figure 11: Qualitative results of personalized seg- mentation on the PerSeg dataset, compared with PerSAM.
the CVC-ClinicB dataset, larger model size does not necessarily lead to better results. This result aligns with current conclusions (Mazurowski et al., 2023 Huang el al. 2024h: In medical image analysis, the huge SAM may occasionally be outperformed by the large SAM. On the other hand, for the PerSeg dataset, even utilizing the base SAM, P²SAM achieves higher accuracy compared PerSAM with the huge SAM. These findings further underscore the robustness of P"SAM, particularly in scenarios where the model exhibits weaker representation, circumstance more prevalent in medical image analysis.
10

---

Published in Transactions on Machine Learning Research (05/2025)
4.4 Qualitative Results
Figure 6 and 7 showcase the advantage of P2SAM for out-of-domain applications. As shown in Figure 6 by presenting sufficient negative-point prompts, we enforce the model's focus on the semantic target. Re- sults in Figure 7 further summarize the benefits of our method: unambiguous segmentation and robust prompts selection. Our P'SAM can also improve the model's generalization after domain adaptation. By providing precise foreground information, P'SAM enhances segmentation performance when the object is too small (e.g., the first two columns in Figure ( and when the segmentation is incomplete (e.g., the last two columns in Figure 9h. Figure and 11 showcase the qualitative results on the PerSeg dataset, compared with Matcher and PerSAM respectively. The remarkable results demonstrate that PºSAM can generalize
well to different domain applications.
5 Conclusion
We propose a data-cfficient segmentation method, PºSAM, to solve the patient-adaptive segmentation prob- lem. With novel part-aware prompt mechanism and distribation-guided retrieval approach, P'SAM can effectively integrate the patient-specific prior information into the current segmentation task. Beyond patient-adaptive segmentation, S*SAM demonstrates promising versatility in enhancing the backbone's gen- eralization across various levels: 1. At the domain level, P'SAM performs effectively in both medical and natural image domains. 2. At the task level, PºSAM enhances performance across different, patient-adaptive segmentation tasks. 3. At the model level, P²SAM can be integrated into various promptable segmentation models, such as SAM, SAM 2, and custom fine-tuned SAM. In this work, to meet clinical requirements, we choose to adapt SAM to the medical imaging domain with public datasets. We opted not to adopt SAM 2, as it requires video data for fine-tuning, which is more costly. Additionally, treating certain patient-adaptive segmentation tasks as vidco tracking is inappropriate. In contrast, approaching patient-adaptive segmen- tation as an in-context segmentation problem offers a more flexible solution for various palient-adaptive segmentation tasks. Additional discussions can be found in appendix. We hope our work brings attention to the patient-adaptive segmentation problem within the rescarch community.
Acknowledgments
The authors acknowledge support from University of Michigan MIDAS (Michigan Institute for Data Science) PODS Grant and University of Michigan MICDE (Michigan Institute for Computational Discovery and Enginocring) Catalyst Grant, and the computing resource support from NSF ACCESS Program.
11

---

Published in Transactions on Machine Learning Research (05/2025)
References
HJWL Aerts, E Rios Velazquez, RT Leijenaar, Chintan Parmar, Patrick Grossmann, S Cavalho, Johan Bussink, René Monshouwer, Benjamin Haine-Kains, Derek Rietveld, et al. Data from nsclo-radiomics. The cancer imaging archive, 2015.
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. Tn International conference on machine learning, pp. 214-223. PMLR, 2017.
David Arthur, Sergei Vassilvitskii, et al. k-means+++: The advantages of careful seeding. In Soda, volume 7, pp. 1027-1035, 2007.
Jorge Bernal, F Javier Sánchez, Gloria Bernández-Espparrcee Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Victor Ton Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 21438-21 21451,111 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213 229. Springer, 2020.
Yizheng Chen, Michael F Gensheimer, Hilary P Bagshaw, Santino Butler, Lequan Yu, Yuyin Zhou, Liyue Shen, Nataliya Kovalehuk, Murat Surucu, Daniel T Chang, et al. Patient-specific auto-segmentation on daily kvct images for adaptive radiotherapy. International Journal of Radialion Oncology* Biology Physics, 2023.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-1777 17875, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprinl 2rXiv:2010. 1 11929, 2020.
Razvan-Gabriel Dumitru, Darius Peteleaza, and Catalin Craciun. Using duck-net for polyp image segmen- 2023.
tation. Scientific Reports,
Mohamed S Elmahdy, Tanuj Ahuja, Uulke A van der Heide, and Marius Staring. Patient-specifi finetuning of deep learning models for adaptive radiotherapy in prostate ct. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 577-50 580. IEEE, 2020.
Mark Everingham, Lue Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303 338, 2010.
Roberto Carcia-Figueiies Sandra Baleato-González, Anwar R Padhani, Antonio Luna-Alcalá, Juan Antonio and Herbert Alberto
Vallejo-Casas, Evis Sala, Joan C Vilanova, Dow-Mu Koh, Michel Vargas. How clinical imaging can assess cancer biology. Insights inlo imaging, 10:1 351 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Richard Hodson. Precision medicine. Nature, 537 S49, 2016.
Shahruk Hossain, Suhail Najecb, Asif Shahriyar, Zaowad R. Abdullah, and M. Ariful Haque A pipeline for lung tumor detection and segmentation from ct scans using dilated convolutional neural nelworks. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1348-1352, 2019. doi: 10.1109/CCASSP220000.688
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 096855 2021.
Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al. Segment anything model for medical images? Medical Image Analysis, 92: 103061, 2024.
Geoffrey D Hugo, Elisabeth Weiss, William C Sleeman, Salim Balik, Paul J Keall, Jun Lu, and Jeffrey F Williamson. Data from 4d lung imaging of nscle patients. The Cancer Imaging Archive, 10:K9, 2016.
Fabian Isensee, Paul Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein nnt-net: a self-confiurrng method for deep learning- based biomedical image segmentation. Nature methods, 18(2): 203 211, 2021.
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems, 33:19545 19560, 2020.
Debesh Jha, Pia H Smedsrud, Michael A Riegler, P&l Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In MultiMedia Modeling: 26th International
Conference, MMM 2020, Daejeon, South Korea, January 5 5-8, 2020, Proceedings, Part IT 26, pp. 451-462. Springer, 2020.
Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in neural information processing systems, 35:36722-366 36732, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. WxXiv:234.0264/ 2023.
arXiv preprint
Tianang Leng, Yiming Zhang, Kun Han, and Xiaohui Xic. Sclf-sampling meta sam: Enhancing few-shot medical image segmentation with meta-learning. In Proceedings of the IEEE/CVI /CF Winter Conference on Applications of Computer Vision, pp.
2024.
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang. Pss-1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2869-2878, 2020.
Xiang Li, Jinglu Wang, Xiao Li, and Yan Lu. Hybrid instanco-aware temporal fusion for online vidco instance segmentation. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 36, pp. 1429 1437, 2022a.
Xiang Li, Shung-Ching Lin, Yinpeng Chen, Zicheng Liu, Jinglu Wang, and Bhiksha Raj. Paintseg: Training- free segmentation via painting. arXiv preprint arXiv:2305.19006 2023a.
Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023b.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pp. 280-290 20-296.222 Springer, 2022b.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Tsung-Y: Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-29000000 2017.
Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint (Xxu:2905.13310 2023.
Yongfei Liu, Xiangyi Zhang, Songvang Zhang, and Xuming He. Part-aware prototype network for few- shot semantic segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 142-158. Springer, 2020.
Lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. ar preprint 2017.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, :5(1):654, 2024a.
Jun Ma, Sumin Kim, Feifei Lá, Mohammed Baharoon, Reza Asakereh, Hongwei Lyu, and Bo Wang. Segment anything in medical images and videos: Benchmark and deployment. arXiv preprint orXiv:2408.03322 2024b.
Martin Maška, Vladimír Ulman, David Svoboda, Pavel Matula, Petr Matula, Cristina Ederra, Ainhoa Urbiola, Tomás España, Subramanian Venkatesan, Deepak MW Balak, et al. A benchmark for comparison of cell tracking algorithms. Bioinformatics, 30(11):1609 1617, 2014.
Christos Matsoukas, Johan Fredin Haslum, Mocin Sorkhei, Magnus Söderberg, and Kevin Smith. What makes transfer learning work for medical images: Feature reuse & other factors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9225 9234, 2022.
Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89:102918, 2023.
María Luisa Menéndez, JA Pardo, L Pardo, and MC Pardo. The jensen-shannon divergence. Journal of the Franklin Institute, 334(2):307-315 1997.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. Teee, 2016.
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 622-631, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacddin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXi::230.007193, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA7 7 blog, 188):9, 2019.
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. 2018.
Nikhila Ravi, Valentin Gabeur, Yuan Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et Sam 2: Segment anything in images and videos. arXiv preprint or XrXio:2408.000 00714, 2024.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Compuler-Assion Assisted Ialereeniion-MICCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part 111 18, pp. 234 241. Springer, 2015.
Ludger Rüschendorf.. The wasserstein distance and approximation theorems. Probability Theory and Related Fields, 70(1):117-129, 1985.
Edward Sanderson and Bogdan J Matuszewski. Fen-transformer feature fusion for polyp segmentation. In Annual conference medical image understanding and analysis, pp. 592-907. Springer, 2022.
Jan-Jakob Sonke, Marianne Aznar, and Coen Rasch. Adaptive radiotherapy for anatomical changes. Seminors in radiation oncology, volume 29, pp. 245 257. Elsevier, 2019.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7262- 7272, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprinl ar arx:22302. 13971, 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Chuang Wang, Neelam Tyagi, Andreas Rimner, Yu-Chi Hu, Harini Veeraraghavan, Guang Li, Margie Hunt, Gig Mageras, and Pengpeng Zhang. Segmenting lung tumors longitudinal imaging studies via patient- specific adaptive convolutional neural network. Radiotherapy and Oncology, 131:101
2019a.
Chuang Wang, Sadegh R. Alam, Siyuan Zhang, Yu-Chi Hu, Saad Nadeem, Neelam Tyagi, Andreas Rimner, Wei Lu, Maria Thor, and Pengpeng Zhang. Predicting spatial esophageal changes in multimodal lon- gitudinal imaging study via convolutional recurrent neural network. Physics in Medicine & Biology, 65 (23):235027, 2020.
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference computer vision, pp. 9197-9206, 2019b.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segment- ing everything context. ar preprint orXiv:2304.0328, 2023b.
Hallee E Wong, Marianne Rakie, John Guttag, and Adrian V Dalca. Scribbleprompt: Fast and flexible interactive segmentation for any medical image. arXiv preprint arXiv:2312.0724 2023.
Junde Wu and Min Xu. One-prompt to segment all medical images. In Proceedings of the IEEE/CVF 2024.
Conference on Computer Vision and Pattern Recognition, pp.
Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.1200 12620, 2023.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognilion, pp. 13325-15333 2023.
Zongxin Yang. Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmen- tation. Advances in Neural Information Processing Systems, 34:2491-281 12002, 2021.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Liu, and Yunchao Wei. Bridge the points: Graph-based few- shot segment anything semantically. Advances in Neural Information Processing Systems, 11:3232--3261) 2025.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXi::235.00344, 2023.
Yichi Zhang and Zhenrong Shen. Unleashing the potential of sam2 for biomedical images and videos: A survey. aixiv preprint orXiv:2408. 12889, 2024.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neurul Information Processing Systems, 36, 2024.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix
SAM Overview
SAM Adaptation Details
Test Implementation Details
D Additional Visualizations
B Discussions
Equations
A SAM Overview
Segment Anything Model (SAM) (Kirillov et al. 2023) comprises three main components: image encoder, prompt encoder, and mask decoder, denoted as Encr, Enep, and DeeM. As promptable segmentation model, SAM takes an image I and a set of human-given prompts P as input. SAM predicts segmentation masks Ms by:
Ms DeCM (Encr(T), Encr(P))
During training, SAM supervises the mask prediction with linear combination of focal loss (Lin et al. 2017] and dice loss (Milletari et al. 2016) in a 20:1 ratio. When only a single prompt is provided, SAM generates multiple predicted masks. However, SAM backpropagates from the predicted mask with the lowest loss. Note that SAM returns only one predicted mask when presented with multiple prompts simultaneously.
Enc, and Deem primarily employ the Transformer (Vaswani 2017 Dosovitskiy et al. 2020) architecture. Here, provide details components in Encp. Encp supports three prompt modalities input: the point, box, and mask logit. The positive- and negative-point prompts are represented by two learnable embeddings, denoted as Epos and Ezeg. respectively. The box prompt comprises two lcarnable embeddings representing the left-up and right-down corners of the box, denoted as Eup and Eao.- In cases where neither the point nor box prompt is provided, another learnable embedding Enot-a-point is utilized. available, the mask prompt encoded by a stack of convolution layers, denoted as Fmaak: otherwise, is represented by a learnable embedding Enot-a-mexk.
SAM employs interactive training strategy. In the first iteration, either a positive-point prompt, repre- sented by Epea, box prompt, represented by {Bpp Edoon,, is randomly selected with equal probability from the ground truth mask. Since there is no mask prompt in the first iteration, Epss or {Eap. Eaoin) is combined with Fnot-a-mask and fed into DecM- In the follow-up iterations, subsequent positive- and negative- point prompts are uniformly selected from the error region between the predicted mask and the ground truth mask. SAM additionally provides the mask logit prediction from the previous itcration as a supple- ment prompt. As result, {Epee, Eung: }msex} is fed into DeeM during each iteration. There are 11 total iterations: one sampled initial input prompt, 8 iteratively sampled points, and two iterations where only the mask prediction from the previous iteration is supplied to the model.
B SAM Adaptation Details
we propose adapt SAM the medical image domain when it is needed, with full fine- tune ( Full-Finc- Tune) and LoRA (Hu et al.) 2021) (LoRA). For Pull-Fine-Tune we fine-tune all parameters in SAM backbone. For LoRA, insert the LoRA module in the image encoder Enc, and only fine-tune parameters in the LoRA module and the mask decoder DeCM- Our fine-tuning objectives are as follows:
In Section 3.3
The model can accurately predict a mask even no prompt is provided.
2. The model can predict exact mask even if only one prompt is given.
17

---

Published in Transactions on Machine Learning Research (05/2025)
3. The model maintains promptable ability.
The training strategy outlined in SAM cannot satisfy all these three requirements: The mask decoder DeeM is not trained to handle scenarios where no prompt is given. 2. The approach to resolving the ambigu- ous prompt by generating multiple results is redundant as have a well-defined segmentation objective. Despite that, we find simple modification can meet all our needs:
In the imitial iteration, we introduce a scenario where no prompt is provided to SAM. As a result, (Enot-a-point; maat-a-maak) is fed into DeeM in the first iteration.
2. To prevent prpint and Enot-s-maxx from introducing noise when human-given prompts are avail- able, we stop their gradients every iteration.
3. We ensure that SAM always returns an exact predicted mask. As result, the ambiguity property does not exist in the model after fine-tuning,
C Test Implementation Details
Table 9: Retrieval range for the COCO-20', FSS-1000, LVIS-92', PerSeg dataset. Blue indicates the retrieval range for positive-point prompts. Red indicates the retrieval range for negative-point prompts.
COCO-20' FSS-1000
LVIS-92
PerSeg
105100/1 / 1,5-10 /1 1-5 /
Table 10: Retrieval range for the 4D-L.ung and CVC-ClinieDB dataset. Blue indicates the retrieval range for positive-point prompts. Red indicates the retrieval range for negative-point prompts.
Meta
LoRA
Full-Finee Tune
Datasct
huge
base
large
base
large
4D-Lung 1-2 45 1-3
1-3 1-3/1 1-3 1-3 / 1-5 1-3
CVC-ClinieDB 1-5 1-3 1-3 1-3/1-3 1-3 1-2 1211-33 1-3 1-2/1
In this section, for reproducibility, we provide the details of the retrieval range during the test time for the COCO-20' (Nguyen & Todorovic, 2019), FSS-1000 (Li et al. 2020), LVIS-92' (Liu et al. 2023), and Persog (Zhang et al.) 2023) datasct in Table 9 the 4D-Lung (Hugo et al. 2016) and CVC-ClinicDB (Bernal
et al. 2015) datasel in Table 10
The final number of positive-point and negative-poit prompts determined by our distribution retrieval approach. Below, we explain how the retrieval range is determined in Table 10 For LoRA and Fall-Pine-Tune, the retrieval range is determined based on the validation set of the indernal datasets. We uniformly sample positive-point and negative-point prompts on the ground-truth mask and perform interac- tive segmentation. The number of prompts is increased until the improvement becomes marginal, at which point this maximum number is set as the retrieval range for ezternal test datasets, On the 4D-Lung dataset, we consistently set the number of negative-point prompts to for these two types models. This decision is informed by conclusions from previous works (Ma al. 2024a Huang al. 2024)), which suggest that the background and semantic target can appear very similar in CT images, and using too many negative-point prompts may confuse the model. On the CVC-ClinicDB dataset, the endoscopy video is in RGB space, resulting in relatively small domain gap (Matsoukas et al.) 2022) compared to SAM's pro-trained dataset. Therefore, for Mela, we use the same retrieval range as the Full-Fine- Tune large model. In contrast, on the 4D-Lung dataset, CT images are in grayscale, leading to a significant domain gap (Matsoukas et al., 2022) compared to SAM's pro-trained dataset. Consequently, we set the retrieval range for positivc-point prompts to to avoid outliers and fixed the number of negative-point prompts to large constant (í.e., 45) rather than range, to ensure the model focuses the semantic target. These values were not further tuned.
18

---

Published in Transactions on Machine Learning Research (05/2025)
D
Additional Visualization
PorSAM
Ours
Figure 12: Additional qualitative results: (Columns 1-4) Full images from carlier illustrations; (Columns 5-6) Additional comparisons with PerSAM. Note that the negative-point prompt sometimes differ between P²SAM and PerSAM, as the similarity matrix changes when using part-level features.
n
72 = 3
n = 4
n 2
3
Figure 13: Visualization results on the 4D-Lung dataset, based on a varying number of part-level fea- tures.
Figure 14: Visualization results on the CVC- ClinieDB dataset, based on varying number of part-level features.
In this section, first provide the full images in Figure 12 that were presented in Scction to eliminate any possible confusion. Then, to provide deeper insight into our part-aware prompt. mechanism and distribution- guided retrieval approach, we present additional visualization results on the 4D-Lung (Hugo et al. 2016
dataset, the CVC-ClinicDB (Bernal et al., 2015) dataset, and the PerSeg (Zhang et al. 2023) dataset. These visualizations are based on varying number of part-level features, offering a clearer understanding of how the part-aware prompt mechanism adapts different segmentation tasks and domains. In Figure and 14
we observe that an appropriate number of part-level features can effectively divide the tumor into distinct parts, such as the body and edges for non-small cell lung cancer, and the body and light point (caused by the camera) for the polyp. This illustrates how PºSAM assist in cases of incomplete segmentation. In Figure 15 we observe that an appropriate number of part-level features can effectively divide the object into meaningful components, such as the pictures, characters, and aluminum material of can; the legs and platforms table; or the face, ears, and body of a dog. These parts can merge naturally based on texture features when using the appropriate number of part-level features, whercas using too many features may result in serreeggmnnttioon. Our retrieval approach, on the other hand, helps determine the optimal number of pevt- features for each specifie case.
19

---

Published in Transactions on Machine Learning Research (05/2025)
n=
n=
n=4
n=
Table
Dog
Figure 15: Visualization results on the PerSeg dataset, based on a varying number of part-level features.
E
Discussion
Table 11: Results of direct-trunsser CVC- ClinicDB. The model is trained on Kvasir- SEG with different pre-training weights.
Table 12: Results of interactive segmentation on the inter- nal Kvasir-SEG validation dataset and the external CVC- ClinieDB dataset. We use Full-Fine- FumeFFineTTume large
312. 5t here.
Dataset
No Prompt
1 Positive Box Point Prompt Prompt
P2SAM
Med-SAM 83.85
SAM 84.62
Kvasir-SEG CVC-ClinieDB
98.27
95.15 15
95.57
86.68
88.76
88.99
92.35
Table 13: Comparison with GF-SAM on the CVC-ClinicDB dataset. * indicates using DI- NOv2 for a better performance.
Table 14: Results of one-shot part segmentation on the PASCAL-Part dataset. Note that all methods utilize SAM's encoder for fairness.
Method
animals indoor person vehicles mean
Method Meta Full- Fune
Matcher
29.29
56.30
21.04
37.02
33.66
GF-SAM 60.55*
87.57
PerSAM
19.9
51.8
18.6
32.0
30.1
Matcher 63.54
87.15
P²SAM
20.29
54.82
19.62
34.21
32.24
P²SAM 66.45
88.76
P.SAM w. neg
20.54
54.65
20.59
36.91
33.17
Baseline Results. In this paper, we treat MedSAM Ma et al.
2024ah with a human-given box prompi.
the baseline for the 4D-Lung dataset (Hugo et al. 2016), DuckNet (Dunitru et al. 2023) as the baseline for the CVC-ClinieDB dataset (Bernal et al. 2015).
We acknowledge that MedSAM is widely used as a baseline across many benchmarks (Antonelli et al. 2022 Ji et al. 2022 However, these comparisons primarily focus on internal validation. MedSAM has the potential to outperform many models on external validation sets due its pre-training on large-scale medical image dataset. While there direct evidence to confirm this, DuckNet (Dumitru al. 2023) suggests that large-scale pre-trained models generally outperform others external validation sets,
if they lag behind internal validation. Among studies (Butoi et al. 2023 Wong et al. 2023 Ma et al. 2024a b Wu & Xu, 2024) that aim to develop promptable sogmentation models specifically for medical image segmentation, UniverSeg (Butoi el al. 2023 's performance may decline signifi- cantly with only one-shot support set, and both ScribblePrompt (Wong et al.) 2023) and One- tne-Prompt (Wu & Xul (024)) are trained on much smaller datascts. As we focus on segmenting external patient samples that lic outside the training distribution one-shot manner. Therefore, we argue that the model's generalization
ability is critical for achieving superior performance. The 4D-Lang dataset (Hugo et al. 2016) is relatively
20

---

Published in Transactions on Machine Learning Research (05/2025)
new benchmark for longitudinal data analysis, and no standard benchmark for comparison was available at the time this work was conducted. In addition, during evaluation, we supplemented MedSAM with a human-given box prompt, making it a very fair baseline for this work.
Baseline Methods. In this paper, we treat SAM-based methods such as PerSAM (Zhang et al.
2023 and
Matcher (Liu et a (023) as our primary baselines and also compare with PANet (Wang et al. do not include other backbone methods like SeribblePrompt (Wong et al. 2023) and One-Prompt (Wu & Xu 2024) because they primarily focus on interactive segmentation, just similar to MedSAM (Ma et al.) 2024a).
2019b We
which is the baseline we compare in Table g On the other hand, utilizing other prompt modalities, such scribble, mask, and box, presents challenges for solving the patient-adaptive segmentation problem, as it is difficult to represent prior data in these formats. In this work, we adopt a more flexible prompt modality: point prompts. Although it may be possible to convert our multiple-point prompts into a scribble prompt by connecting them together, we leave the exploration of this direction for future work. Consequently, the most relevant bascline methods remain SAM-base methods like PerSAM and Matcher.
Here, we evaluate more recent SAM-base method, GF-SAM (Zhang et al. 2025 Similar to Matcher, GF- method based on graph analysis. In Table 13 we evaluate GF-SAM on the CVC-ClinieDB dataset (Bernal et al. 2015 using both a natural image pre-trained encoder (Mela) and a medically adapted encoder (Full-Fine- Tune). With the natural image pre-trained encoder, P'SAM outperforms both GF-SAM and Matcher, since patch-level features are less robust than part-level features when there is domain gap between pre-training data and test data. However, GF-SAM fails to surpass Matcher in this task, which contrasts with its superior performance on natural image segmentation tasks. We hypothesize that this is because GF-SAM is hyper- parameter-free method, and factors such as the number of point prompts, the nunber of clusters, and the threshold value may be more sensitive when there is a domain gap between the pre-training data and the test data. GF-SAM outperforms Matcher with the medically adapted encoder, but still lags behind P'SAM, as the encoder is adapted for medical segmentation tasks and still lacks patch-lovel objectives. This result, along with the findings in Table S where P*SAM with base SAM outperforms PerSAM with huge SAM by 0.7% mloU and base SAM by 26.0% mloU on the PerSeg dataset further underscores that P'SAM is a more robust method when the model exhibits weaker reprosentations, scenario more prevalent in medical image analysis.
SAM utilizes DINOv2 to extract patch-level features; however, GF-SAM is hyper-parameter
Pre- Pre-trained Model. In this work, we choose to adapt SAM to the medical image domain using the SA-1B pre-trained model weights rather than weights from MedSAM for two reasons. First, although MedSAM fine-tunes SAM (SA-1B pre-trained) on a large-scale medical segmentation dataset, its fine-tuning dataset is still 1,000 times smaller than SAM's pre-training dataset. (1M vs. 1B). Since model generality after adaptation is crucial for our work, we assume that SAM remains a better starting point, despite MedSAM being strong option for zero-shot medical segmentation. Second, McdSAM only provides the SAM-Base pre-trained model, whereas our results in Table I Jand Table 2 demonstatte that larger models (iee. large) can further enhance performance across various tasks. In Table 11 we provide the direct-transfer result on the CVC-CliniDBB dataset, the model is trained on the Kvasir-SEG dataset with Med-SAM pre-trained weights and SA-1B pre-trained weights. The result follows our assumption and the discussion in MedSAM (Ma et al., 2024a) and its successor (Ma et al. 2024b), that with a specific task, maybe fine-tune from SAM is still a better choice.
Interactive Segmentation. As mentioned in Section 3.3 and detailed in Appendix B we closely adhere to SAM's interactive training strategy when adapting it with medical datasets. Therefore, our medically adapted model retains its interactive segmentation capability. In Table 12 we present both internal evalua- tion results on the Kvasir-SEG dataset's 's validation set and external evaluation results on the CVC-ClinieDB dataset. First, as discussed in Section 4.2 and Appendix B since we have a specific segmentation target, our adapted model does not need to be ambiguity-aware allowing a human-given single positive-point prompt to achieve good performance. P2SAM lags only slightly behind this result while operating fully automatically. For the human-given box prompt, it is not surprising that it outperforms PºSAM, as box prompt is strong prompt that essentially requires the provider to know the lesion's 's location.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Gound
Vich Box Reg
20
30
B. Bax
Figure 16: Qualitative results of single-cell segmentation on the PhC-C2DH-U373 dataset. The second row highlights the challenge P²SAM faces handling multiple similar objects. The third row demonstrates that PºSAM can overcome this challenge with a cost-free regularization.
Part Segmentation. We acknowledge that P2SAM's design was not initially focused part segmenta- tion but enhancing the medical image segmentation model's generality by providing more precise and informative prompts. We conduct the part segmentation task on the PASCAL-Part dataset (Everingham jet al. 2010 Note that all methods use SAM (Meta) as the backbone model. Part segmentation with SAM typically relies more on additional prompt modalities, such as box prompts, or diverse mask candidates. For example, Matcher employs random point-prompt sampling strategy to make their proposed mask candi- dates more diverse, potentially slowing down the algorithm. In Table [4 when compared with PerSAM, PºSAM consistently shows benefits (i.c., +2.23% mloU). However, P2SAM is surpassed by Matcher (i.c., 1.42% mloU). For PºSAM it is reasonable to provide additional negative-point prompts in part segmentation task because a portion of the background is correlated between the reference and target images (i.e., both re- fer to the rest of the object). Therefore, we additionally provide negative-point prompts to P'SAM (P'SAM w. neg), which further improves segmentation performance (i.c., +0.93% mloU) and brings PºSAM on par with Matcher. While achieve slightly better performance, Matcher utilizes 128 sampling iterations for the part segmentation task, making much slower (x3) than both PerSAM and P2SAM.
Similar Objects. P*SAM demonstrates improvements in the backbone's generalization across domain, task, and model levels. At the task level, we have already shown how P"SAM enhances performance for NSCLC segmentation in patient-adaptive radiation therapy and polyp segmentation in endoscopy vidcos. However, when addressing specific (asks that involve multiple similar targets, P"SAM may fail due to the lack of instance-leved objective. Although this scenario is uncommon patient-adaptive segmentation, we acknowl- edge that P2SAM faces the same challenge of handling multiple similar objects as other methods (Zhang et al. 2023 Liu al. ).23). In Figure 16 we present an example of single-cell segmentation on the PhC- C2DH-U373 dataset (Maška et al. 2014), which goes beyond the patient-specific setting. In Figure 16 the second row illustrates that P"SAM fails to segment the target cell due to the presence of many similar cells in the field of view. However, given the slow movement of the cell, we can leverage its previous information to regularize the current part-aware prompt mechanism. The third row in Figure 16 demonstrates that when using the bounding box from the last frame, originally propagated from the reference frame, to regularize the part-aware prompt mechanism in the current frame, PºSAM achieves strong performance on the same task. Since the bounding box for the first frame can be generated from the ground truth mask, which already available, this regularization incurs additional cost. Utilizing such tailored regularization incorporating various prompt modalities, showcase approach's 's flexible applicability to other applications.
F Equations
In this section, we provide details on the equation mentioned in Section 3.2
Wasscrstein Distance. In Equation 3 we use (:.)) to represent the Wasserstein Distance. Here we provide the details of this function. Suppose that features the reference image FR E Rexxd and features in the target image Fr E Roexd come from two discrete distributions, Fn P(FR) and Fr E P(Fz), where
22

---

Published in Transactions on Machine Learning Research (05/2025)
FR Ei= udy and F1- Eド= 836'; o,, being the Delta-Dirac function centered fr and ěy, being the Delta- function centered on ft. Since Fn and Fr are both probability distributions, sum of weight vectors is 1, E u4 = = E, Uj. The Wasserstein distance between FR and Pr is defined as:
FR F7
(FRRR.PF
min TElI(u,))
ΣΣT,
T
(5)
|
llll
17
where v) = TT E [T = u, 1, 2). and T is the transport plan, interpreting the amount of mass shifted from Fh Fz.
23


Reviews:
[{"review_id": "Cdd368hrdL", "reviewer": "Reviewer_AEU3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## strength:\n\n+ This paper conducts experiments to validate SAM on multiple domains.\n\n+ The idea is straight-forward\n\n+ This paper is easy-to-follow\n\n## weakness:\n\n- This paper delivers too many terminologies that are either incorrect, inaccurate or vague, which clearly do not meet the standard of a topic-tier journal. For example:\n\n(1) Natural domain adaptation and medical domain adaptation for SAM may be not correct in many cases. SAM can do zero-shot inference on these domains, without adaptation. \n\n(2) Since in the left of Fig.1 medical domain adaptation is already claimed. Then, why do you claim patient domain adaptation in the right? They are overlapped and are both in medical domain, right?\n\nThe proposed method lacks either theoretical insight or technical innovation. \n\n- The selection of the part number is realized by the proposed distribution-guided retrieval approach, which is highly empirical and varies from dataset to dataset, and from domain to domain. Therefore, its versablity and feasibility could be limited.\n\n- The cosine similarity is used to highlight the parts that are important to the target image, which is a common and off-the-shelf tool, and does not have much technical innovation.\n\n- Regarding the technical correctness. The cosine similarity does not implement a normalization. Besides, only the average pooling is used. This makes the part selection does not work well in some extreme cases, either over-fit or under-fit. \n\n- More experimental analysis is needed, if other commonly-used distance metrics other than Wasserstein distance can measure the distribution similarity and impact the proposed method. \n\n- The compared state-of-the-art methods are very limited. More recent SAM based methods in 2023-2024 should be involved for comparison. \n\n- The motivation and rationale of Part-aware Prompt Mechanism is unclear. As SAM can automatically generate multiple masks, the claim on the part-aware is incremental. \n\n- The delivery of the methodology is clearly below the standard. Each component of the proposed method should occupy an independent subsection.\n\n- Some failure case and analysis should be made. \n\n- The abstract is vague to deliver the motivation and benefit of both proposed modules.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a foundation model based semantic segmentation method. \n\nSpecifically, on top of the segment anything image encoder, it enables the prompt to perceive a more fine-grained part features. \n\nFurther, a distribution-guided retrieval approach is proposed to determine the optimal number of parts. \n\nExperiments show its effectiveness over the baseline.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Natural domain adaptation and medical domain adaptation for SAM may be not correct in many cases. SAM can do zero-shot inference on these domains, without adaptation. The necessaity of both terms needs reconsideration.\n\n- Since in the left of Fig.1 medical domain adaptation is already claimed. Then, why do you claim patient domain adaptation in the right? They are overlapped and are both in medical domain.\n\n- The proposed method lacks either theoretical insight or technical innovation. \n\n- The selection of the part number is realized by the proposed distribution-guided retrieval approach, which is highly empirical and varies from dataset to dataset, and from domain to domain. Therefore, its versablity and feasibility could be limited.\n\n- The cosine similarity is used to highlight the parts that are important to the target image, which is a common and off-the-shelf tool, and does not have much technical innovation.\n\n- Why doess the cosine similarity does not implement a normalization?\n\n- Only the average pooling is used in the part selection. This makes the part selection does not work well in some extreme cases, either over-fit or under-fit. \n\n- More experimental analysis is needed, if other commonly-used distance metrics other than Wasserstein distance can measure the distribution similarity and impact the proposed method. \n\n- The compared state-of-the-art methods are very limited. More recent SAM based methods in 2023-2024 should be involved for comparison. \n\n- Please further clarify the motivation and rationale of Part-aware Prompt Mechanism and its contribtion. \n\n- In Methodology section, each component of the proposed method should occupy an independent subsection.\n\n- Some failure case and analysis should be made. \n\n- Please improve the abstract, so as to clearly deliver the motivation and benefit of both proposed modules.", "broader_impact_concerns": "The reviewer feels there is no potential ethical issues on this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzQTjInEuJ", "reviewer": "Reviewer_6yNP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ New prompt mechanism. This paper proposes a part-aware prompt mechanism to select multiple-point prompts based on part-level features of the one-shot data, and a distribution-guided retrieval approach to determine the optimal number of part-level features for a specific case.\n+ Data-efficient segmentation. The proposed method can be adapted to new patients relying only on oneshot patient-specific data without model fine-tuning.\n+ Good generalization ability. P2SAM can handle both medical and natural image segmentation.\n+ The experiments can verify the effectiveness of the key parts of the proposed method, with visualization of segmentation results provided for a better understanding of the proposed method.\n+ State-of-the-art Performance. P2SAM improves the performance by +8.0% and +2.0% mean Dice score in two different patient-adaptive segmentation applications and achieves a new state-of-the-art result, i.e., 95.7% mIoU on the personalized segmentation benchmark PerSeg.\n\nCons:\n\n1. Please clarify the motivation for formulating patient-adaptive segmentation as an in-context segmentation problem. Why formulate the problem in this way, i.e., what are their connections?\nIn addition, though SAM may suffer from the ambiguity issue (probably due to the scarce prompt points provided), why does it fail to generalize external patients? The relation between the ambiguity/multi-point prompts and the external patients could be better presented.\n\n2. There are many methods adapting SAM to the medical domain, e.g., Medical SAM Adapter (Med-SA) and MedSAM mentioned in Related Work, why not build the P2SAM upon these medical SAM methods, of which the encoder can be better adapted to medical images?\n\n3. Why not also provide negative points in Figure 1 to prevent ambiguity? SAM needs both positive and negative points for better performance. Without negative points, many SAM-based methods may suffer from the same issue.\n\n4. Can P2SAM work well without the one-shot patient-specific prior data but using the interactive points? Please answer this question and (if feasible) show some results for comparison.\n\n5. Please clearly clarify the major differences between the P2SAM and the previous methods in Related Work. The current version only briefly points out the drawbacks of previous methods, but the comparison with P2SAM is absent.\n\n6. In the Segmentation Generalist part of Related Work, please also include some methods for Referring Image Segmentation [A] because P2SAM also uses a reference image pair.\n[A] https://github.com/MarkMoHR/Awesome-Referring-Image-Segmentation\n\n7. In Related Work, it would be better to move SAM-based medical segmentation models to the main body rather than in the Suppl., with the Segmentation Generalist subsection for introducing natural image segmentation only.\n\n8. In Related Work, this paper mentions that previous medical SAM methods rely on clinician-given prompts and may not achieve zero-shot performance. However, I do not think relying on clinician-given prompts can be a disadvantage as the single-shot patient prior may not be available in some clinical scenarios. In this case, clinician-given prompts provide more flexibility. \n\n\n9. What is the number of part-level features on page 2? Is it the number of groups as stated in \"multiple groups in the feature space\"? If yes, please revise the manuscript to make it easier to understand.\n\n10. On page 5, please use equations to describe how to measure the distribution similarity and determine the optimal number $n$.\n\n11. In Tab. 1 and 2, comparing only three state-of-the-art methods seems to be less. Please compare with more state-of-the-art methods, particularly SAM-based methods (please refer to the survey [B]) and referring image segmentation ones [A].\n[B] Zhang et al. Segment anything model for medical image segmentation: Current applications and future directions. Computers in Biology and Medicine,\n2024\n\n\n12. On page 4, whether $n_f$ and $n$ are the same, both denoting the number of foreground features?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a Part-aware Prompted Segment Anything Model (P2SAM) for data-efficient segmentation. P2SAM can be adapted to new patients relying only on one-shot patient-specific data without model fine-tuning. It introduces a part-aware prompt mechanism to select multiple-point prompts based on part-level features of the one-shot data. Such a mechanism is further improved by a distribution-guided retrieval approach to determine the optimal number of part-level features for a specific case. P2SAM can achieve clear performance improvement for patient-adaptive segmentation and generalize well to the natural image domain.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the questions in Cons and revise the manuscript accordingly.", "broader_impact_concerns": "There is no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cnxoRiQ77T", "reviewer": "Reviewer_ToqS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe method itself is elegant, intuitive, and effective. It makes sense that the way to sample the prompts for reference-based segmentation could be conducted in a more fine-grained manner than doing a mask-level matching only. The proposed method essentially decomposes the reference mask into different parts, each part will be matched with the target image feature separately in a more specialized manner. Such a matching mechanism would enable the method to handle different granularity better, which is evident in the experiments on the more general pretrained SAM weights. The additional mechanism to select the optimal parts according to the distribution match is well-motivated, such a mechanism enables the proposed method to dynamically decide the part number in a sample-wise manner.\n\nSufficient experiments have been conducted for natural and medical images, ablation studies have been conducted to validate the important design choices. \n\n\nWeakness:\n\nThe description of the method part can be improved, in equation (1), the author could give a more precise and formal definition on the meaning of \"o\" in the mask selection process. Though it might not be too hard to speculate what it means, such an operation can be explained in a clearer manner.  \n\nAs this method is named \"part-aware\", I understand the motivation is to bridge the gap between different user cases. I would suggest the author further include some part-level segmentation evaluation set, which I believe will further show the wide applicability of the proposed method compared to PerSAM and Matchers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The segment anything model (SAM) is a segmentation foundation model designed for general application. When it comes to specific use cases, adaptations are needed. This paper aims to design a part-aware prompt sampling strategy to adapt SAM for a specific use case with only one-shot data. The proposed method has been validated in:\n1) different domains: both natural and medical settings, \n2) different models: models with or without domain adaptation, SAM and SAM2\n3) different segmentation tasks that require different granularities. \n\nPreviously, personalized segmentation methods have been investigated. Nevertheless, the proposed method designed specialized mechanism so that the adaptation can be conducted in a part-aware manner, which enables the proposed to process the target image in a wider granularity range. In general, I think this is a good paper that meets the standard of TMLR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a method for adaptive image segmentation based on the Segment Anything Model (SAM). Specifically, a part-aware prompt mechanism was proposed to select prompts based on part-level features given the one-shot data. Experiments on medical image segmentation and natural image segmentation tasks show the effectiveness of the proposed method.

This paper received review comments from three expert reviewers. After a few rounds of discussion between the reviewers and the authors, all three reviewers recommended positively, with 1 Accept and 2 Leaning Accept. The reviewers acknowledged that their concerns were addressed by the authors during the discussion and agreed on the publication of this paper. But in the meantime, they also agreed on the limited contributions and novelty of this paper and room for improvement.
Given that the claims and designs were validated and supported by the provided evidence, and the findings in this paper could be of interest to a group of TMLR audience, the AE is happy to recommend an Accept.
Decision: Accept (poster)