Personalize Segment Anything Model with One Shot
Authors: Renrui Zhang
Zhengkai Jiang
Ziyu Guo
Shilin Yan
Junting Pan
Hao Dong
Yu Qiao
Peng Gao
Hongsheng Li
Venue: ICLR 2024 poster (2024)
Keywords: Segment Anything Model (SAM)
one-shot learning
text-to-image generation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Given one-shot data, we propose a training-free method (PerSAM) and a 2-parameter fine-tuning variant (PerSAM-F) to personalize Segment Anything Model (SAM) for segmenting any user-indicated objects.

Abstract: Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful promptable framework, revolutionizing the segmentation field. Despite the generality, customizing SAM for specific visual concepts without man-powered prompting is under-explored, e.g., automatically segmenting your pet dog in numerous images. In this paper, we introduce a training-free Personalization approach for SAM, termed PerSAM. Given only one-shot data, i.e., a single image with a reference mask, we first obtain a positive-negative location prior for the target concept in new images. Then, aided by target visual semantics, we empower SAM for personalized object segmentation via two proposed techniques: target-guided attention and target-semantic prompting. In this way, we can effectively customize the general-purpose SAM for private use without any training. To further alleviate the ambiguity of segmentation scales, we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce a scale-aware fine-tuning to aggregate multi-scale masks, which only tunes 2 parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new dataset, PerSeg, for the evaluation of personalized object segmentation, and also test our methods on various one-shot image and video segmentation benchmarks. Besides, we propose to leverage PerSAM to improve DreamBooth for personalized text-to-image synthesis. By mitigating the disturbance of training-set backgrounds, our approach showcases better target appearance generation and higher fidelity to the input text prompt. Code is released at https://github.com/ZrrSkywalker/Personalize-SAM.

Published as a conference paper at ICLR 2024
PERSONALIZE SEGMENT ANYTHING MODEL WITH ONE SHOT
Renrui Zhang' :2, Zhengkai Jiang Ziyu Guo Shilin Yan Junting Pan'
Hao Dong', Yu Qiao", Peng Gao?, Hongsheng Li
CCUHK MMLab Shanghaii Artificial Intelligence Laboratory
IInstitute of Automation, Chinese Academy of Sciences
*CFCS, School CS, Peking University
$CPII of InnoHK
(renzuizhang, siyuguotelse link...... cuhk. edu edu.hk hsli0ee www.. edu. hk kaikaijiang aakegmail... com {gaopeng, qiaoyu]epjlab. www.lla..rrg..oo. en
ABSTRACT
Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as powerful promptable framework, revolutionizing the segmenta- tion field. Despite the generality, customizing SAM for specific visual concepts without man-pow mow-powered prompting under-explored, e.g., automatically segmenting your pet dog in numerous images. In this paper, we introduce training- free Personalization approach for SAM, termed PerSAM. Given only one-shot data, i..., single image with reference mask, first obtain positive-negative lo- cation prior for the target concept in new images. Then, aided by target visual semantics, empower SAM for personalized object segmentation via pro- posed techniques: target-guided attention and terget-semantic prompting. In this way, can effectively customize the general-purpose SAM for private use without any training. To further alleviate the ambiguity of segmentation scales, we present efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce a scale-aware fine-tuning to aggregate multi-scale masks, which only tunes parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new dataset, PerSeg, for the evaluation of personalized object segmentation, and also test our methods on various one-shot image and video segmentation benchmarks. Besides, propose to leverage PerSAM improve DreamBooth for personalized text-to-image synthesis. By mitigating the disturbance of training-set backgrounds, our approach showcases better target appearance generation and higher fidelity to the input text prompt. Code released www.eeeeeeeeeaaaeeeeeeeeee Ze-SAM.
at https::/ /ii tuub. com/
(1) User provides
One-shot Learning
(3) Personalized Segmentation
Training-free
Fine-turing
PerSAM-F
Image
...in varlous poses or scenes
Figure 1: Personalization of Segment Anything Model. We customize Segment Anything Model (SAM) (Kirillov et al., 2023) for specific visual concepts, e.g., your pet dog. With only one-shot data, we introduce two efficient solutions: a training-free PerSAM, and fine-tuning PerSAM-F.
* Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2024
PerSAM
PerSAM-F
Par smeters
User provides
DreamBooth
PerSAM
10 Seccnds
The hi Lokky bear
"A [V] cat on beach"
...In various poses or scenes
Fine-ture tureture
Thebwdy
'A [V] backpack on table of classroom"
Personalized Segmentation Exam- ples. Our PerSAM (Left) can segment personal objects in any context with favorable perfor- mance, and PerSAM-F (right) further alleviates the ambiguity issue by scale-aware fine-tuning.
Figure 2:
Figure 3: Improving DreamBooth (Ruiz et al., 2022) with PerSAM. By mitigating the distur- bance of backgrounds during training, our ap- proach can help to achieve higher-quality person- alized text-to-image generation.
INTRODUCTION
Foundations models in vision (Li et al., 2022; Zou al., 2023; Wang al., 2022), language (Brown et al., 2020; Touvron et al., 2023; Radford et al., 2019), and multi-modality (Radford et al., 2021; Jia et al., 2021; Li et al., 2023) have gained unprecedented prevalence, attributed to the availability of large-scale datasets and computational resources. They demonstrate extraordinary generalization capacity zero-shot scenarios, and display versatile interactivity incorporating human feedback. Inspired by this, Segment Anything (Kirillov et al., 2023) develops a delicate data engine for collecting 11M image-mask data, and subsequently trains a segmentation foundation model, known as SAM. defines a novel promptable segmentation framework, i.e., taking as input a handcrafted prompt and returning the expected mask, which allows for segmenting any objects in visual contexts.
However, SAM inherently loses the capability to segment specific visual concepts. Imagine intending to crop your lovely pet dog a thick photo album, or find the missing clock from a picture of your bedroom. Utilizing the vanilla SAM would be highly labor-intensive and time-consuming. For each image, you must precisely find the target object within complicated contexts, and then activate SAM with proper prompt for segmentation. Considering this, we ask: Can personalize SAM to automatically segment user-designated visual concepts in a simple and efficient manner?
To this end, introduce PerSAM, wrainingfrree personalization approach for Segment Anything Model. As shown in Figure 1, our method efficiently customizes SAM using only one-shot data, i.e., first
user-provided reference image and rough mask of the personal concept. Specifically, obtain a location confidence map for the target object in the test image by feature similarities, which considers the appearance of every foreground pixel. According to confidence scores, two points are selected as the positive-negative location prior, which are finally encoded as prompt tokens and fed into SAM's decoder for segmentation. Within the decoder, we propose inject visual semantics of the target object to unleash SAM's personalized segmentation power with two techniques:
Target-guided Attention. We guide every token-to-image cross-attention layer in SAM's decoder by the location confidence map. This explicitly compels the prompt tokens to mainly concentrate on foreground target regions for intensive feature aggregation.
Target-semantic Prompting. To explicitly provide SAM with high-level target semantics, we fuse the original prompt tokens with the embedding of the target object, which provides the low-level positional prompt with additional visual cues for personalized segmentation.
With the aforementioned designs, along with cascaded post-refinement, PerSAM exhibits favorable personalized segmentation performance for unique subjects in a variety of poses or scenes. Notably, approach can cope well with scenarios that require segmenting ones, simultancously segmenting several identical objects in the same image, or tracking different objects along video. Nevertheless, shown in Figure 2, there might be occasional failure cases,
object among multiple similar
2

---

Published as a conference paper at ICLR 2024
where the object comprises visually distinct subparts hierarchical structures be segmented, e.g., the hat on top of a teddy bear, or the head of a robot toy. Such ambiguity casts a challenge for PerSAM in determining the appropriate scale of mask as output, since both the local part and the global shape can be regarded as valid masks by SAM
To alleviate this issue, further propose fine-tuning variant of our approach, PerSAM-F. We freeze the entire SAM to preserve its versatile pre-trained knowledge, and only fine-tune 2 parameters within 10 seconds on a single A100 GPU. In detail, we enable SAM to produce several potential segmentation results of different mask scales. To adaptively select the best scale for varying objects, we employ a learnable relative weight for each mask scale. and conduct weighted summation as the final output. By such efficient scale-aware training, PerSAM-F avoids over- werr-fitinng on the one-shot data and exhibits better segmentation accuracy shown in Figure (Right).
Moreover, we observe that our approach can also assist DreamBooth (Ruiz et al., 2022) to better fine-tune diffusion models for personalized text-to-image generation, as shown in Figure 3. Given a few images containing specific visual concept, e.g., your pet cat or backpack, DreamBooth learns to convert these images into an identifier [V] in the word embedding space, which, however, can simultaneously include the background information, e-g., stairs or the forest. This would override the newly prompted backgrounds, and disturb the target appearance generation. Therefore, we propose to leverage PerSAM to segment the target object within training images, and only supervise DreamBooth by the foreground area, enabling text-to-image synthesis with higher quality.
We summarize the contributions of our paper follows:
Personalized Object Segmentation. We first investigate how to customize a general- purpose segmentation model (SAM) into personalized scenarios with minimal expense. To this end, we introduce two efficient and effective methods, along with a new segmentation dataset, PerSeg, for the evaluation of personalized object segmentation.
PerSAM and PerSAM-F. In PerSAM, we propose three Training-free techniques to guide SAM by the high-level semantics of target objects. In PerSAM-F, we design scale-aware fine-tuning with 2 parameters in 10 seconds to well alleviate the mask ambiguity issue.
Our approach achieves competitive results various tasks, including the PerSeg benchmark, one-shot part and semantic segmentation, and video object segmentation. In addition, PerSAM can enhance DreamBooth for better personalized text-to-image synthesis.
2 METHOD
In Section 2.1, we first briefly revisit Segment Anything Model (SAM) (Kirillov et al., 2023), and introduce the task definition for personalized object segmentation. Then, illustrate the methodology of our PerSAM and PerSAM-F Section 2.2 and 2.3, respectively. Finally, we utilize our approach to assist DreamBooth (Ruiz et al., 2022) for better text-to-image generation in Section 2.5, and specifically discuss some application scenarios in Section 2.4.
2.1 PERSONALIZED OBJECT SEGMENTATION
A Revisit of Segment Anything. SAM consists of three components, prompt encoder, image encoder, and a lightweight mask decoder, respectively denoted as Encp, Enc,, and DecM- As a promptable framework, SAM takes as input an image 1, and a set of prompts P, which can be a point, a box, or coarse mask. Specifically, SAM first utilizes Enc, to obtain the input image feature, and adopts Encp to encode the human human-given prompts of length into prompt tokens
(I) where F, E Iilxuxxe and Tp e k with h, denoting the resolution of the image feature map and e denoting the feature dimension. After that, the encoded image and prompts are fed into the decoder DeCM for attention- based feature interaction. SAM constructs the input tokens of the decoder by concatenating several learnable mask tokens Thr as prefixes to the prompt tokens Tp. These mask tokens are responsible for generating the mask output, formulated as
Fi = En), Tp = Encp(P),
M = Dec Fr, Concat(TMr (T.. TT))). where M denotes the final segmentation mask predicted by SAM.
(2)
3

---

Published as a conference paper at ICLR 2024
PerSAM's DerSAMMsDecoder
Similarity
Fr
Target-guided Attention
Target-
(PTTR him1
Image Token Cross Amention
Encode
Prompting
)
(5r Local Confidesce Maps
Test Image
Tolten to lmaage Oroce-ätarntion
Target Local
Features
Aggr egale
Attention Matrix A
Local Reatures
Token Self Ataention
(Till-1
Overall Confidence Map
Aggr agete
Concalt
+ Repeat
M₁ Fi
Encode
Confidence Map S
Tp
5
One-shot Image lR One-shot Mask
Positive Prior
Low-level
High-level
Negative Prior
Positional Prompt
Semantic Prompt
Figure Positive-negative Location Prior. We calculate a location confidence map for the target object in new test image by the appear- ance of all local parts. Then, we select the loca- tion prior as the point prompt for PerSAM.
Figure 5: Target-guided Attention (Left) & Target-semantic Prompting (Right). To in- ject SAM with target semantics, we explicitly guide the cross-attention layers, and propose additional prompting with high-level cues.
Task Definition. Although SAM is generalized enough for any object by prompting, it lacks the ability to automatically segment specific subject instances. Considering this, we define for personalized object segmentation. The user provides only single reference image, and a mask indicating the target visual concept. The given mask can either be an accurate segmentation, rough sketch drawn on-the-fly. Our goal is to customize SAM to segment the designated object within new images or videos, without additional human prompting. For evaluation, we annotate new dataset for personalized segmentation, named PerSeg. The raw images are collected from the works for subject-driven diffusion models (Gal al., 2022; Ruiz al., 2022; Kumari al., 2022), containing various categories of visual concepts in different poses or scenes. In this paper, we propose two efficient solutions for this task, which we specifically illustrate as follows.
task
2.2 TRAINING-FRER PERSAM
Location Confidence Map. Conditioned the user-provee user-provided image Ia and mask MR. PerSAM first obtains a confidence map that indicates the location the target object in the new test image T. As shown in Figure 4, we apply image encoder extract the visual features of both In and T. The encoder can be SAM's frozen backbone or other pre-trained vision models, for which we adopt SAM's s image encoder Ene, by default. We formulate the process as
F₁ (n), =R= = Ency (TR),
where F1, FR E R Then, utilize the reference mask Ma p to crop the features of foreground pixels within the visual concept from FR. resulting in set of n local features as
(T MR Fn.
where Th E lxe and o denotes spatial-wis wise multiplication. After this, we calculate confidence maps for each foreground pixel by the cosine similarity between Tj and test image feature F7
(S y (F,TH T y where St E kxx
(5)
Note that F7 and TV have been pixel-wisely L2-normalized. Each represents the distribution probability for different local part of object in the test image, such the head, the body, the paws of a dog. On top of this, we adopt an average pooling to aggregate all n local maps to obtain the overall confidence map of the target object
S =
Σ St
R
(6)
i=
4

---

Published as a conference paper at ICLR 2024
By incorporating the confidences of every foreground pixel, S can take the visual appearance of different object parts into consideration, and acquire a relatively comprehensive location estimation.
Positive-negative Location Prior.
To provide PerSAM with location prior the test image, we select two points with the highest and lowest confidence values in S, denoted as Ps. and Pt, respectively. The former represents the most likely center position of the target object, while the latter inversely indicates the background. Then, they are regarded as the positive and negative point prompts, and fed into the prompt encoder as
T = Encp
E ER2x
which denote the prompt tokens for SAM's decoder. In this way, SAM would tend segment the contiguous region surrounding the positive point, while discarding the negative one's 's on the image.
Target-guided Attention. Although the gositive-negativy point prompt has been obtained, we further propose more explicit semantic guidance to the cross-attention operation in SAM's decoder, which concentrates the feature aggregation within foreground target regions. As shown in Figure the overall confidence map S in Equation 6 can clearly indicate the rough region of the target visual concept in the test image (hotter colors indicate higher scores). Based on such a property, we utilize to guide the attention map in every token-to-image cross-attention layer of the decoder. Specifically, we denote every attention map after the softmax function as A E Rhx and then modulate its attention distribution by A = softmax (A a softmax(s)).
(8)
where denotes balancing factor. With the attention bias, the mask and prompt tokens compelled to capture more visual semantics associated with the target subject, other than the unimportant background area. This contributes more effective feature aggregation in attention mechanisms, and enhances the final segmentation accuracy of PerSAM in a training-free manner.
Terget- Prompting. The vanilla SAM only receives prompts with low-level positional information, such as the coordinate of a point or a box. To provide SAM's decoder with more high- level cues, we propose to utilize the visual feature of the target concept as an additional high-level semantic prompting. We first obtain the global embedding Tn of the object in the reference image by both average pooling between different local features as
TR=
T ERIX
Then, we elesent-wisely add Tn to all the input tokens of the test image in Equation 2, before feeding them into the decoder block, which is shown in Figure 5 as
T Repcat(Tri + Concat(TM: Tr),
(10)
where 9 denotes the input token guided by target semantics for the decoder Dec and the Repeat operation duplicates the target visual embedding. Aided by the simple token incorporation, PerSAM is not only prompted by lowelevel location points, but also high-level target visual cues.
Cascaded Post-refinement. Via the above techniques, obtain an initial segmentation mask on the test image from SAM's decoder, which however, might include rough edges and isolated background noises. For further refinement, we iteratively feed the mask back into the decoder Deca for a two-step post-processing. In the first step, we prompt the decoder by the currently predicted mask along with the previous positive-negative point prompt. For the second step, we acquire the bounding box enclosing the mask from the first step, and prompt the decoder additionally with this box for more accurate object localization. As we only iterate the lightweight decoder without the large-scale image encoder, the post-proesssing is efficient and only costs extra 2% latency.
2.3 FINE-TUNING OF PERSAM-F
Ambiguity of Segmentation Scales. The training-free free PerSAM can tackle most cases with satisfac- tory segmentation accuracy. However, some target objects contain hierarchical structures, which leads
5

---

Published as a conference paper at ICLR 2024
Test Image
PerSAM
Boothhooth
Random Noisn
O put
"a [V] cat"
Reconstruction Loss
T Theee scales
M2
+
M2
+
M3
PerSAM
W1
W2
1-(
W1
W2
Decouple
User provides ga
1
Output Mask
M
Fine-tune
Backgroual Disturbance
W Weighted
Summation
Freeze
Figure 6: The Scale-aware Fine-tuning in PerSAM-E. To alleviate the scale ambiguity, PerSAM-F adopts two learnable weights for adaptively aggregating three-scale masks.
Figure 7: PerSAM-assisted DreamBooth We utilize PerSAM to decouple the target ob- jects from the background for improving the generation of DreamBooth.
the ambiguity of mask scales. As shown in Figure 6, the teapot on top of platform is comprised of two parts: a lid and a body. If the positive point prompt (denoted by green pentagram) is located at the body, while the negative prompt (denoted by a red pentagram) does not exclude the platform in similar color, PerSAM would be misled for segmentation. Such an issue is also discussed in SAM, where it proposes an alternative to simultancously generate multiple masks of three scales, i.c., the whole, part, and subpart of object. Then, the user must manually select mask out of three, which is effective but consumes extra manpower. In contrast, our personalized task aims to customize SAM for automatic object segmentation without the need for human prompting. This motivates to further develop scale-aware version of PerSAM by parameter-eflicien fine-tuning.
Scale-aware Fine-tuning. For adaptive segmentation with the appropriate scale, we introduce fine-tuning variant, PerSAM-F. Unlike the training-free model only producing one mask, PerS AM-F first follows PerSAM to obtain the location prior, and refers SAM's original solution to output three-scale masks, denoted as M1. M2,, and M3, respectively. On top of this, we adopt two learnable mask weights, W1, W22 and calculate the final mask output by weighted summation as
(11) where Wi, W2 are both initialized as 1/3. To learn the optimal weights, we conduct one-shot fine- tuning on the reference image, and regard the given mask as the ground truth. Note that, we freeze the entire SAM model to preserve its pre-trained knowledge, and only fine-tune the parameters of W1, W2 within 10 seconds single A100 GPU. In this way, our PerSAM-F efficiently learns the scale-aware semantics of objects, and adaptively outputs the best segmentation scale for different concepts, improving the generalization capacity of PerSAM.
M w1 M1 + W2 M2 (1 wi w2) M3;
2.4 PERSAM-ASSISTE DREAMBOOTH
For personalized text-to-image synthesis, DreamBooth (Ruiz et al., 2022) fine-tunes pre-trained diffusion model by the given 3~55 photos of specific object, i.g., a pet cat. It learns to generate the cat referred to by a text prompt, "a [V] cat", and calculates the loss over the entire reconstructed images. This This would inject the redundant background information the training images into the identifier [V]. Therefore, as shown in Figure 7, we introduce our strategy to alleviate the disturbance of backgrounds in DreamBooth. Given an object mask for any of the few-shot images, we leverage our PerSAM to segment all the foreground targets, and discard the gradient back-propagation for pixels belonging to the background area. Then, the Stable Diffusion is only fine-tuned to memorize the visual appearances of the target object. With no supervision imposed on the background, our PerSAM- assisted DreamBooth can not only synthesize the target object with better visual correspondence, but also increase the diversity of the new backgrounds guided by the input text prompt.

---

Published as a conference paper at ICLR 2024
Table Personalized Object Segmentation on the PerSeg Dataset. We compare the overall mloU, bloU, and learnable parameters for different methods (Bar al., 2022; Wang al., 2022; 2023; Zou et al., 2023), along with the mloU for 10 objects PerSeg. ** denotes works concurrent to ours.
Back- pack
Teddy Bear
Duck Toy
Thinn Bird
Red Cartoon
Robot
Method
mIoU
bloU
Param.
Can
Barn
Clock
Cat
Toy
Painter
56.4
42.0
354M
19.1
3.2
42.9
94.1
88.1
93.0
33.3
20.9
98.2
65.0
VP
65.9
25.5
383M
61.2
58.6
59.2
76.6
66.7
79.8
89.9
67.4
81.0
72.4
SEEM
87.1
341M
65.4
82.5
72.4
91.1
94.1
95.2
97.0
95.8
S
94.3
76.5
354M
96.6
63.8
92.6
94.1
94.4
93.7
97.2
92.6
97.3
96.2
PerSAM
89.3
71.7
96.2
38.9
96.2
90.70
95.39
94.6
97.3
93.7
97.0
60.6
PerSAM-F
95.3
77.9
96.7
97.5
96.1
92.3
95.5
95.2
97.3
94.0
97.1
96.7
Table 2: Video Object Segmen- tation on DAVIS 2017 val (Pont- Tuset et al., 2017). We utilize gray color to denote the methods involving in-domain training.
Table 3: One-shot Semantic and Part Segmentation on FSS- 1000 (Li et al., 2020), LVIS-92' (Gupta et al., 2019), PASCAL- Part (Morabia et al., 2020), and PACO-Part (Ramanathan et al., 2023). We report the mloU scores and utilize gray color to denote the methods involving in-domain training.
Method
J&F
One-shot Semantic Seg.
One-shot Part Seg.
Method
FSS-1000
LVIS-92-
PASCAL-Par
PACO-Part
AGSS
67.4
64.9
69.9
AFB-URR
74.6
73.0
76.1
HSNet
17.4
32.4
22.6
VAT
90.3
18.5
33.6
23.5
Painter
34.6
SEEM
58.9
Painter
61.7
10.5
30.4
14.1
SegGPT
75.6
78.6
SegGPT
18.6
PerSAM
66.9
63.4
PerSAM
81.6
15.6
32.5
22.5
PerSAM-F
76.1
73.2
PerSAM-F
86.3
18.4
32.9
22.7
2.5 DISCUSSTON
Can PerSAM segment multiple Different objects? Yes. the user indicates three different objects in the reference image, can also personalize SAM with these three objects, as visualized in Figure 9 for video object segmentation. For new test image, we first adopt the image encoder to extract its visual feature only once. On top of this, we independently calculate the location confidence maps for the three objects, and then prompt the decoder three times to respectively segment them. As decoder (50ms) is more lightweight than the encoder (2s), this brings a marginal time cost.
Comparison with Concurrent Works. Some concurrent works, such as SegGPT (Wang et al., 2023) and SEEM (Zou al., 2023), can also segment objects given one-shot data, i.e., reference image and a mask. However, our goal is to customize an off-the-shelf generalist (SAM) into specialist for private use with good efficiency (training-free or 10-second fine-tuning). In contrast, they target generalization capabilities requiring extensive training data optimizing large-scale parameters. Moreover, due to the efficient tuning characters, PerSAM can be regarded as a general extensible framework, acceptable for any image encoder and promptable segmentation models. Therefore, these excellent works and our approach are trying to solve different problems segmentation fields.
3 EXPERIMENT
We first evaluate approach for personalized segmentation PerSeg in Section 3.1, along with various existing one-shot segmentation benchmarks in Section 3.2. Then, we illustrate the effectiveness of our PerSAM-assisted DreamBooth in Section 3.3. Finally, we conduct several ablation studies to investigate our designs on PerSeg in Section 3.4.
3.1 PERSONALIZED EVALUATION
PerSeg Dataset. To test the personalization capacity, construct new segmentation dataset, termed PerSeg. The raw images are collected from the training data subject-driven diffusion works (Ruiz et al., 2022; Gal et al., 2022; Kumari et al., 2022). PerSeg contains 40 objects of various
7

---

Published as a conference paper at ICLR 2024
The ring clock
The teapst
The backpack cariled NEmaY
The top part
Figure 8: Visualization of PerSAM-F's Im- provement. Our scale-aware fine-tuning can well alleviate the scale ambiguity of PerSAM.
Figure 9: Visualization of Video Object Seg- mentation. Our approach performs well for segmenting multiple objects in a video.
categories in total, including daily necessities, animals, and buildings. In different poses or scenes, each object is associated with 5~7 images and masks, where we fix one image-mask pair as the user-provided one-shot data. The mloU and bloU (Cheng al., 2021a) are adopted for evaluation.
Performance. In Table we observe the fine-tuned PerSAM-F achieves the best results, which effectively enhances PerSAM by +2.7% and +5.9% overall mloU and bloU. We show more vi- sualization of PerSAM-F's improvement in Figure 8. Visual Prompting (VP) (Bar et al., 2022), Painter (Wang al., 2022), SEEM (Zou al., 2023), and SegGPT (Wang et al., 2023) are in-context learners that can also segment objects according to the given one-shot prompt data. As shown, the training-free PerSAM AM
already achieve better performance than Painter, VP, and SEEM with different margins. By the efficient 2-parameter fine-tuning, our PerSAM-F further surpasses the powerful SegGPT by +2.4% and +4.1% overall mloU and bloU. As analyzed in Section 2.5, different from their motivations, our method is specially designed for personalized object segmentation, and exhibits much more efficiency in both time and computational resources.
3.2 EXISTING SEGMENTATION BENCHMARKS
Video Object Segmentation. Given the first-frame image and object masks, our PerSAM and PerSAM-F achieve competitive object segmentation and tracking performance on the validation set of DAVIS 2017 (Pont-Tuset et al., 2017) As shown in Table 2, compared to methods without video training, the training-free PerSAM largely surpasses Painter by +32.3% I J&F score, and our PerSAM-F can achieve +0.5% better performance than SegGPT. Notably. our one-shot fine-tuning approach can outperform methods (Lin et al., 2019; Liang et al., 2020) fully trained by extensive video data. The results fully illustrate strong generalization ability for temporal video data and complex scenarios, which contain multiple similar or occluded objects, as visualized in Figure 9,
One-shot Semantic and Part Segmentation. In Table 3, we evaluate our approach for one-shot image segmentation respectively on four datasets, PSS-1000 (Li et al., 2020), LVIS -2' (Gupta et al., 2019), PASCAL-Part (Morabia et al., 2020), and PACO-Part (Ramanathan et al., 2023), where we follow (Liu al., 2023b) for data pre-processing and evaluation. attains consistently better results than Painter, and performs comparably to SegGPT. For models (Min et al., 2021; Hong et al., 2022) with in-domain training, our approach can achieve higher scores than HSNet. The experiments well demonstrate that, proposed approach is not limited to object-level segmentation, but also works for category-wise and part-wise personalization of SAM.
shown, our PerSAM-F
3.3 PERSAM-ASSISTED DREAMBOOTH
We follow all the hyperparameters in DreamBooth (Ruiz et al., 2022) to fine-tune pre-trained Stable Diffusion (Rombach et al., 2022) for personalized image synthesis. visualize more examples of PerSAM-assisted DreamBooth in Figure 10. For the dog lying sofa, the "jungle" and "snow" by DreamBooth are still the sofa with green and white decorations. Assisted by PerSAM-F, the newly-generated background is totally decoupled with the sofa and well
addition to Figure 3, we grey
8

---

Published as a conference paper at ICLR 2024
User provides
DreahBooth
Asd by
PerSAM
User provides
DreamBoolte
Assited by PerSAM
"A ungle?
with forest background
"A [V] dog snow"
[V] barn with blue sky in the background
Figure 10:
Visualization of PerSAM-guided DreamBooth. The improved DreamBooth (Ruiz et al., 2022) can better preserve the diversity for synthesizing various contexts in new images.
Table 4:
Ablation of Main Com-
Table 5: Ablation of Different Fine-tuning Methods.
Table 6: Ablation of using Box-image as Reference.
ponents in our proposed method.
Variant
mloU
Gain
Method
Param.
mloU
Method
Mask
Box
Positive Prior
69.1
PerSAM
89.32
Painter
56.4
42.0
+ Negative Prior
72.5
+3.4
Prompt Tuning
12K
76.5
VP
65.9
38.1
+ Post-refinement
83.9
+11,4
Adapter
196K
78.3
SEEM
87.1
64.9
+ Guided Attention
85.8
+1.9
LoRA
293K
90.0
SegGPT
94.3
36.0
+ Semantic Prompt
89.3
+3.5
3 Mask Weights
3
92.9
PerSAM
89.3
88.1
Scale Tuning
95.3
+6.0
PerSAM-F
95.3
PerSAM-F
95.3
94.9
corresponds to the textual prompt. For the barn in front of the mountains, our approach also alleviates the background disturbance to correctly generate the "forest" and "blue sky".
3.4 ABLATION STUDY
Main Components. In Table 4, we investigate different components by starting from baseline that only adopts the positive location prior. Then, we add the negative point prompt and cascaded post-refinement, enhancing +3.6% and +11.4% mloU, respectively. On top of that, we introduce the high-level target semantics into SAM's decoder for attention guidance and semantic prompting, The resulting +1.9% and +3.5% improvements fully indicate their significance. Finally, via the efficient scale-aware fine-tuning. PerSAM-F boosts the score by +6.0%, demonstrating superior accuracy.
Different Fine-tuning Methods. Table 5, we experiment with other parameter-efffiiiet fine-tuning (PEFT) methods for PM-F i.e., prompt tuning (Liu et al., 2021), Adapter (Houlsby et al., 2019), and LoRA (Hu et al., 2021). We freeze the entire SAM, and only tune the PEFT modules injected into every transformer block in PerSAM's decoder. As shown, the prompt tuning and Adapter would over-fit the one-shot data and severely degrade the accuracy. Instead, our scale-aware fine-tuning can best improve the performance of PerSAM, while tuning the least learnable parameters.
Using Box-image as Reference. Requiring an accurate mask as one-shot data might too strict for some users. In Table 6, relax the input restrictions to bounding box designating the expected object. For our method, we can regard the box as a prompt and utilize off-the-shelf SAM to generate the one-shot mask. Therefore, the box reference only leads to marginal performance drop in PerSAM and PerSAM-F, but severely influences other methods.
4 CONCLUSION
In this paper, we propose to personalize Segment Anything Model (SAM) for specific visual concepts with only one-shot data. Firstly, we introduce PerSAM, which injects high-level target semantics into SAM with training-free ffre techniques. On top of this, we present a scale-aware fine-tuning variant, PerSAM-F. With only learnable parameters, PerSAM-F effectively alleviates the ambiguity of mask scales and achieves leading performance various benchmarks. Besides, we also verify the efficacy of our approach to assist DrcamBooth in fine- tuning better text-to-image diffusion models. We hope work may expand the applicability of SAM wider range of scenarios.

---

Published as a conference paper at ICLR 2024
5 ACKNOWLEDGEMENT
This work is partially supported by the National Key R&D Program of China (NO.2022ZD0161 100), the National Natural Science Foundation of China (No. 66.66220,72), the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, and General Research Fund of Hong Kong RGC Project 14204021. Hongsheng Li a of CPII under the InnoHK.
REFERENCES
Groundel-semment-anytinngg
https:://itthbb.cm wtwp:/hwww..com/e--eeeeeeeeeeeee
Grounded Segplleddd-eeeemtt-------e Anything. 2023.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2481229548 2017.
Yanqi Bao, Kechen Song, Jie Wang, Liming Huang, Hongwen Dong, and Yunhui Yan. Visible and thermal images fusion architecture for few-shot semantic segmentation. Journal of Visual Communication and Image Representation, 80: 103303, 2021.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005-25017, 2022.
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1801,01 401,,, 2018.
Gary Bradski. The opencv library. Dr. Dobb's Journal: Software Tools for the Professional Programmer, 25011):120-123, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. arXiv preprint 2tt.:23304.13303. 2023.
Liang- Chigh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Paltern Analysis and Machine Intelligence, 40(4):834-845 2017.
Xi Chen, Zhiyan Zhao, Feiwu Yu, Yilei Zhang, and Manni Duan. Conditional diffusion for interactive segmentation. In Proceedings of the IEEE International Conference Computer Vision, pp. 7345-7354, 2021.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions, arXiv preprint arXiv:2205.088 085544, 2022.
Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15334-15333 2021a.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-1777 -1875, 2021b.
Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
10

---

Published as a conference paper at ICLR 2024
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bermt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and paterr recognition, pp. 3213-3223, 2016.
Pedro Cuenca and Sayak Paul. Using lora for efficient stable diffusion fine-tuning, https: :// huggingfacee co/blolll//o.cco log/ lora, January 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint Stti::220.006111 2022.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng. Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.0595: 2024.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.000 00615, 2023.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp.
2019.
Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5108-5115 IEEE, 2017.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905 2023.
Yuying Hao, Yi Liu, Zewu Wu, Lin Han, Yizhou Chen, Guowei Chen, Lutao Chu, Shiyu Tang, Zhiliang Yu, Zeyu Chen, al. Edgeflow: Achieving practical interactive segmentation with edge-guided flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1551-1560, 2021.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pp. 59-75. Springer, 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- King-KKikpattickk and Graham Neubig. Towards unified view of parameter-efficient transfer learning. International Conference on Learning Representations, 2022. URL https: wpwprevieneeeeeeeeeeeeeeee net/ forum
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Lukas Hedegaard, Aman Alok, Juby Jose, and Alexandros Iosifidis. Structured pruning adapters. arXiv preprint arXiv:2211.101555 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2 2021.
11

---

Published as a conference paper at ICLR 2024
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision. pp. 108-126. Springer. 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan. and Sylvain Gelly. Purameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhuu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106. 09685, 2021.
Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al. Segment anything model for medical images? arXiv preprint arXiv:2304. 14660, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 9904-4916 PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Zhengkai Jiang, Yuxi Li, Ceyuan Yang, Peng Gao, Yabiao Wang, Ying Tai, and Chengjie Wang. Pro- totypical contrast adaptation for domain adaptive semantic segmentation. In European Conference on Computer Vision, pp. 36-54. Springer, 2022.
Zhengkai Jiang, Zhangxuan Gu, Jinlong Peng, Hang Zhou, Liang Liu, Yabiao Wang, Ying Tai, Chengjie Wang, and Liqing Zhang. Ste: spatio-temporal contrastive learning for video instance segmentation. In European Conference on Computer Vision Workshops, pp. 539-556. Springer, 2023.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality, arXiv preprint arXiv:2306.0156 2023.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9404-9413, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.026643 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. arXiv preprint urXiv:22/2.04488, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprini arXiv:2 (et::210440899) 2021.
Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang. Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. arXiv preprinl arXiv:221 1.09808, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language ianguage-image pre- training with frozen image encoders and large language models. arXiv preprint arX1v:2301..2257, 2023.
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- Wing Tai, and Chi- Chi-Keung Tang. Fss-1000: A 1000-class dataset for sho-shot segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1869-28778 2020.
12

---

Published as a conference paper at ICLR 2024
Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention-guiin guoiddd unified network for panoptic segmentation. In Proceedings of the IEEE Confer- on Computer Vision and Pattern Recognition, pp. 7026-7035, 2019.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. Advances in Neural Information Processing Systems, 33: 2020.
Huaijia Lin, Xiaojuan Qi. and Jiaya Jia. Agss-vos: Attention guided single-shot video object segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. arXiv preprint arXiv:2004.03829, 2020.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pp. 388 404. Springer, 2022.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Kegin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.0755 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- tr-irriinig for open-set object detection. arXiv preprint arXiv:2303.05499 2023a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint ar arXhv:2110.07603 2021.
Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint a3X3v.2305.13310, 2023b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13-23, 2019.
Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304. 12306. 2023.
13

---

Published as a conference paper at ICLR 2024
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference vision (3DV), pp. 563-571. leee, 2016.
Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5941-6952 2021.
Keval Morabia, Jatin Arora, and Tara Vijaykumar. Attention-batt joint detection of object and semantic part. arXiv preprini arXiv:2007.0244 2020.
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 622-631, 2019.
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.087744 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 193, 2023.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23641-23651, 2023.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.0022. 2020.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sornine-Horuung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675. 2017.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint (it::2004000599, 2021.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. arXiv preprint arXiv:2307.01197, 2023.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 7141-7151, 2023.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in Neural information processing systems, 30, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695.. 2022.
14

---

Published as a conference paper at ICLR 2024
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image -maag diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, and Nanning Zheng. Rethinking learnable tree filter for generic feature transform. Advances in Neural 2020b.
Information Processing Systems, 33:3991-4002, 2020bb.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-languaag tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In European Conference on Computer Vision, pp. 282-298. Springer, 2020a.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050-1065 2020b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training- free name-only transfer of vision-language models. arXiv preprint arXiv:2211.16199 2022.
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. Advances in Neural information processing systems, 33:17721-17732, 2020.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint 4rXiv:2212.02499, 2022.
Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang. Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint -rXXi::3304.32244 2023.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12099 2021.
Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao. Learning exommety-disentangee representation for complementary understanding of 3d object point cloud. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3056-3064, 2021.
Qianxiong Xu, Wenting Zhao, Guosheng Lin, and Cheng Long. Self-calibrated cross attention network for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 655-665, 2023.
Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. Advances in Neural Information Processing Systems, 35:4203 4217, 2022.
Jinyu Yang. Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang. and Feng Zheng. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.. 1968, 2023.
15

---

Published as a conference paper at ICLR 2024
Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023a.
Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph net- works with connection attentions for region-based one-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9587-9595, 2019.
Jian-We Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot segmentation. Advances in Neural Information Processing Systems, 35:6575-6588 2022.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiiient fine-tuning, arXiv preprint arXiv:2303. 205.:22300. 2023b.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling, arXiv preprint arXiv:2111.0322 2021.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. alaperrdapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023c.
Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. arXiv preprint arXiv:2303.02151 2023d.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306. 12156, 2023.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881-6890, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao. and Yong Jae Lee, Segment everything everywhere all at once. arXiv preprint arXiv:2304.067. 2023.
16

---

Published as a conference paper at ICLR 2024
A OVERVIEW
Section B:
Related work.
Section C: Experimental details and visualization.
Section D: Additional experiments and analysis.
Section E: Additional discussion.
B RELATED WORK
Foundation Models. With powerful generalization capacity, pre-trained foundation models can be adapted for various downstream scenarios and attain promising performance. In natural language processing, BERT (Devlin et al., 2018; Lu et al., 2019), GPT series (Brown et al., 2020; OpenAI, al., 2023)
2023; Radford & Narasimhan, 2018; Radford et al., 2019), and LLaMA (Touvron have demonstrated remarkable in-context learning abilitics, and can be extended to multi-modal scenarios (Zhang al., 2023c; Gao et al., 2024; Lin et al., 2023; Han et al., 2023; Guo al., 2023; Zhang et al., 2024). Similarly, CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which conduct contrastive learning on image-text pairs, exhibit exceptional accuracy in zero-shot visual recognition. Painter (Wang et al., 2022) introduces a vision model that unifies network architectures and in-context prompts accomplish diverse vision tasks, without downstream fine- tuning. CaFo (Zhang et al., 2023d) cascades different foundation models and collaborates their pre-trained knowledge for robust low-data image classification. SAM (Kirillov et al., 2023) presents foundation model for image segmentation, which is pre-trained by 1 billion masks and conducts prompt-based segmentation. There are some concurrent works extending SAM for high-quality segmentation (Ke et al., 2023), faster inference speed (Zhao et al., 2023; Zhang et al., 2023a), all- purpose matching (Liu et al., 2023b), 3D reconstruction (Cen et al., 2023), object tracking (Yang et al., 2023), medical (Ma & Wang, 2023; Huang et al., 2023) image processing. From another perspective, we propose to personalize the segmentation foundation model, i.., SAM, for specific visual concepts, which adapts a generalist into specialist with only one shot. Our method can also assist the personalization of text-to-image foundation models, i.e., Stable Diffusion (Rombach et al., 2022) and Imagen (Saharia et al., 2022), which improves the generation quality by segmenting the foreground target objects from the background disturbance.
Large Models in Segmentation. As a fundamental task in computer vision, segmentation (Long et al., 2015; Jiang et al., 2022; Zhao et al., 2017; Xu et al., 2021; Jiang et al., 2023; Lin et al., 2022) requires a pixel-level comprehension of a image. Various segmentation-relltee tasks have been explored, such semantic segmentation, classifying each pixel into predefined set of classes (Badri- narayanan et al., 2017; Chen et al., 2017; Zheng et al., 2021; Cheng et al., 2022; Xie et al., 2021; Song al., 2020b); instance segmentation, focusing on the identification of individual object instances (He et al., 2017; Wang et al., 2020; Tian et al., 2020a); panoptic segmentation, assigning both class labels and instance identification (Kirillov et al., 2019; Li al., 2019); and interactive segmentation, involving human intervention for refinement (Hao et al., 2021; Chen et al., 2021). Recently, inspired by language foundation models (Zhang et al., 2023e; Brown et al., 2020), several concurrent works have proposed large-scale vision models for image segmentation. They are pre-trained by extensive mask data and exhibit strong generalization capabilities on numerous image distributions. Segment Anything Model (SAM) (Kirillov et al., 2023) utilizes data engine with model-ine moodel-in-the-loe -n-tthe-ooop anno- tation to learn a promptable segmentation framework, which generalizes to downstream scenarios in zero-shot manner. Painter (Wang et al., 2022) and SegGPT (Wang et al., 2023) introduce robust in-context learning paradigm and can segment any images by a given image-mas prompt. SEEM (Zou al., 2023) further presents general segmentation model prompted by multi-modal references, e.g., language and audio, incorporating versatile semantic knowledge. In this study, we introduce new task termed personalized object segmentation, and annotate new dataset PerSeg for evaluation. Instead of developing large segmentation models, our goal is to personalize them to segment user-provided objects any poses or scenes. We propose two approaches, PerSAM and PerSAM-F, which efficiently customize SAM for personalized segmentation.
Parameter-efffcient Fine-tuning. Directly tuning the entire foundation models on downstream tasks can be computationally expensive and memory-intensive, posing challenges for resource-
17

---

Published as a conference paper at ICLR 2024
Table 7:
Personalized Object Segmentation on the PerSeg Dataset. We report the mloU scores of 30 objects in addition to the 10 objects in Table 1. denotes works concurrent to ours.
Tortaise Plushy
Round Bird
Colorful Sneaker
Colorful Тeapot
Method
Dog
Dog2
D
Dog4
Dog5
Dog6
Puinter (Wang et ul., 2022)
80.41
73.77
46,98
22.39
82.03
76.16
55.31
39.83
0.00
13.69
VP (Bar al. 2022)
6.80
12.26
23.84
20.61
21.44
32.05
24.42
34,09
30.32
34,89
SEEM* (Zou et al., 2023)
71.04
35.35
67.02
81.87
75.02
72.99
78.75
38.74
20,08
44,44
SogGPT (Wang et al., 2023)
73.82
65.07
61.11
81.66
82.94
76.44
77.85
82.89
72,24
80.44
PerSAM
96,79
95.66
88.85
95.22
97.10
94.66
93.06
96,79
94.48
96.27
PerSAM-F
96.81
95.79
88.67
95.18
97.22
94.85
97.09
96.85
95.13
84.41
Method
Dog7
Dog8
Candle
Funcy
Sloth Plushie
Poop
Rc
Shiny Sneaker
Wolf Plushie
Wooden
Boot
Emoji
Car
Pot
Painter (Wang al., 2022)
40,97
57.15
24.36
49.06
45.78
23.42
23.69
0.00
38.97
$7.61
VP (Bar al., 2022)
17.67
12.24
12.71
39.13
29.31
37.55
29.98
30.88
28.86
34.30
SEEM (Zou et al., 2023)
63.77
70.34
26.99
34.90
81.46
45.55
34.94
82.30
76.27
74,81
SogGPT (Wang et al., 2023)
66.20
82.21
81.60
76.06
80.54
81.32
79.26
85.26
72.48
78.00
PerSAM
93.69
95.34
74.16
95.87
96.37
96.01
39.30
97.00
94.34
PerSAM-F
93.77
95.61
96.75
95.96
96.64
96.43
96.12
96.87
94.32
97.43
Method
Table
Teapot
Chair
Elephant
Duck Toy
Monster Toy
Pack
Dog
Hear Plushie
Berry
Bowl
Statue
Cat
Painter (Wang et al., 2022)
16.92
7.00
$0.09
40.80
29.24
34.80
40.73
81.30
45.98
19.96
VP (Bar al., 2022)
16.00
10,00
27.20
22.01
52.14
30.92
22,80
23.95
11.32
27.54
SEEM* (Zou 2023)
30.15
12.30
66.15
46.64
89,92
41.49
66.83
61.27
38.29
24.27
SegGPT (Wang et al., 2023)
81,95
89.89
78.97
80,38
84,48
83.33
77.53
75,54
73.00
76.54
PerSAM PerSAM-F
94.68
40.02
92.22
96.05
97.31
93.75
95.85
89.28
91.81
95.42
94.66
96.93
92.14
96.07
97.31
94.21
95.76
95.32
91.27
95.46
constrained applications. To address this issue, recent works have focused on developing parameter- efficient methods (Sung et al., 2022; He et al., 2022; Rebuffi et al., 2017; 7; Qin & Eisner, 2021) to freeze the weights of foundation models and append small-scale modules for fine-tuning. Prompt tuning (Lester et al., 2021: : Zhou et al., 2022; Jia et al., 2022; Liu et al., 2021) suggests using learnable soft prompts alongside frozen models to perform specific downstream tasks, achieving more competitive performance with scale and robust domain transfer compared to full model tuning. Low-Rank Adaption (LoRA) (Hu et al., 2021; Cuenca & Paul, 2023; Zhang et al., 2023b; Hedegaard et al., 2022) injects trainable rank decomposition matrices concurrently to each pre-trained weight, which significantly reduces the number of learnable parameters required for downstream tasks. Adapters (Houlsby al., 2019; Pfeiffer et al., 2020; Lin et al., 2020; Chen al., 2022) are designed to be inserted between layers of the original transformer, introducing lightweight MLPs for feature transformation. Different from existing works, we adopt a more efficient adaption method delicately designed for SAM, i.e., the acale- fine-tuning of PerSAM-F with only 2 parameters and 10 seconds. This effectively avoids the wer-fitting issue on one-shot data, and alleviates the ambiguity of segmentation scale with superior performance.
C EXPERIMENTAL DETAILS AND VISUALIZATION
C.I PERSONALIZED EVALUATION
Implementation Details. We adopt a pre-trained SAM (Kirillov al., 2023) with a ViT-H (Doso- vitskiy et al., 2020) backbone as the foundation model, and utilize SAM's 's encoder to calculate the location confidence map. For PerSAM. we apply the target-guided attention and target-semantic prompting to all three blocks in the decoder. The balance factor in Equation 8 is set as 1. For PerSAM-F, we conduct one-shot training for 1,000 epochs with a batch size 1, supervised by the dice loss (Milletari et al., 2016) and focal loss (Lin et al., 2017). We set the initial learning rate as 10 and adopt an AdamW (Loshchilov & Hutter, 2017) optimizer with a cosine scheduler.
Complete Results on the PerSeg Dataset. In Table 7, report the mloU scores of the other 30 objects in the PerSeg dataset, except for the 10 objects in Table (1) of the main paper. As compared, our PerSAM without any training can achieve superior segmentation results Painter (Wang et al.,
18

---

Published as a conference paper at ICLR 2024
2022), Visual Prompting (VP) (Bar et al., 2022), and SEEM (Zou et al., 2023) on most objects. Note that, we here compare the results of SEEM with the Focal-L (Yang et al., 2022) vision backbone, its rest-performing variant. Aided by the 2-parameter fine-tuning, PerSAM-F further performs comparably with SegGPT (Wang et al., 2023), a powerful in-context segmentation framework. Therefore, our approach exhibits a high performance--fficienee trade-off by efficiently customizing the off-the-shelf SAM (Kirillov et al., 2023) for personalized object segmentation.
Visualization. In Figure 11, we visualize the location confidence maps, segmentation results of PerSAM with positive-negative location prior, and the bounding boxes from the cascaded post- refinement As shown, the confidence map (hotter colors indicate higher scores) can clearly indicate the rough region of the target object in the image, which contributes to precise foreground (green pentagram) and background (red pentagram) point prompts selection. The bounding boxes in green also well enclose the targets and prompt SAM's decoder for accurate post-refinement.
C.2 EXISTING SEGMENTATION BENCIMARKS
Implementation Details. For experiments in existing segmentation datasets, we utilize DI- NOv2 (Oquab et al., 2023) as the image encoder to calculate the location confidence map, which produces a more accurate location prior. Note that, the generality and extensibility of our approach enable to apply any vision backbones for location confidence map calculation. For video ob- ject segmentation, different from PerSeg, where one image contains only one object, DAVIS 2017 dataset (Pont-Tuset et al., 2017) requires to personalize SAM to track and segment multiple different objects across the video frames. In PerSAM, we regard the top-2 positive location prior, and additionally utilize the bounding boxes from the last frame to prompt the decoder. This provides more sufficient temporal cues for object tracking and segmentation. In PerSAM-F, we conduct shot fine-tuning on the first frame for 800 epochs with learning rate 4 4-4. As discussed in Section 2.5, for N objects, we only need to run SAM's scage-scale encoder (2s) once to encode the visual feature of the new frame, while running the lightweight decoder for N times to segment different objects, which takes marginal 50.ms. For one-shot semantic segmentation, we evaluate our method on FSS- (Li et al., 2020) following HSNet (Min et al., 2021) and LVIS-92' (Gupta et al., 2019) pre- rrepprocesed by (Liu et al., 2023b). The benchmark contains objects in wide range of semantic categories within various backgrounds. For one-shot part segmentation, utilize the part-level benchmarks of PASCAL VOC (Morabia et al., 2020) and PACO (Ramanathan et al., 2023) built by (Liu et al., 2023b), requiring to segment partial objects with challenging scenarios.
points as the
Visualization. In Figure 12, visualize more results of PerSAM-F for multi-object tracking and segmentation in consecutive frames of the DAVIS 2017 dataset. We utilize different colors to denote different objects, along with the additional prompts for SAM's decoder, including a bounding box from the last frame and its center point. Aided by our techniques and the last-frame temporal cues, PerSAM-F exhibits favorable video segmentation performance and tracking consistency, even for objects occluded by others or objects of the same category with similar appearances. In Figure 13, we also visualize the results of PerSAM-F for one-shot semantic and part segmentation on four datasets. The satisfactory performance illustrates that our approach is not limited to object-level personalization, but also part- and category- wise segmentation with good generalization capability.
C.3 PERSAM-ASSISTED DREAMBOOTH!
Implementation Details. We follow most model hyperparameters and training configurations in DreamBooth (Ruiz et al., 2022), including a 10-6 learning rate and a batch size 1. We generate 200-image regularization dataset by the pre-trained Stable Diffusion (Rombach et al., 2022) using the textual prompt: "photo of a [CLASS]" We fine-tune DreamBooth and our approach both for 1,000 iterations on a single A 100 GPU, and adopt "t@y' as the word identifier [V] for the personal visual concepts. We utilize DDIM (Song al., 2020a) sampling with 100 steps and 10-scale classifier-free guidance for generation.
Quantitative Evaluation. Besides visualization, we also evaluate the PerSAM-assisser Dream- Booth by three quantitative metrics in Table 19. We leverage CLIP (Radford et al., 2021) to calculate
19

---

Published as a conference paper at ICLR 2024
Figure 11: Visualization of Location Confidence Maps and PerSAM's Segmentation Results. We represent the positive (foreground) and negative (background) location prior by green and red pentagrams. The green bounding boxes denote the box prompts in the cascaded post-refinement.
the feature similarity of generated images with textual prompts ('Text-Align) and reference images ("Image-Align') (Kumari et al., 2022), along with KID (Birkkowski al., 2018) (the smaller, the better). 'Text-Align' (Gal et al., 2022) and 'Image-Align (Hessel et al., 2021) indicate the semantic correspondence of the synthesized images with the textual prompt and few-shot reference images, respectively. KID (Binkowski et al., 2018) measures how much the fine-tuned models over-fit the specific visual concepts few-shot images, for which we utilize Stable Diffusion generate 500 images as the validation set. These quantitative results demonstrate our effectiveness in generating better visual correspondence with the target objects and input prompts.
Visualization. In Figure 14, visualize more examples that demonstrate effectiveness to enhance DreamBooth for higher-fidelle personalized synthesis. utilize PerSAM-F to decouple the table and plushy tortoise from their backgrounds in the few-shot images, i.e., the couch and
20

---

Published as a conference paper at ICLR 2024
Figure 12: Visualization of PerSAM-F for Video Object Segmentation on the DAVIS 2017 (Pont- Tuset al., 2017) dataset. We represent different objects in different colors, and visualize the input prompts for SAM's decoder: a positive location prior (pentagram), an enclosing bounding box from the last frame, and its center point (dot).
One- Semontic Segmentation on F55-3308
mayg
inay
trages
Object Sepmentation PASCAL
oeter encrerce
Telt inegs
Neter
tragss
Telt trages
Figure 13: Visualization of PerSAM-F for One-shot Semantic and Part Segmentation on FSS- 1000 (Li et al., 2020) and PASCAL-Part (Morabia et al., 2020) datasets. Our approach exhibits superior generalization capabilities for diverse segmentation scenarios.
carpet. this way, the PerSAM-assisted DreamBooth generates new backgrounds corresponding to the textual prompts of "in the garden", "and orange sofa", "on the grass", and "swimming in a pool". In addition, our approach can boost the appearance generation target objects with high text-image correspondence, while the vanilla DreamBooth might be interfered by textual prompts, e.g.. the orange on the table and the blue on the turtle shell. The experiments fully verify our efficacy for better personalizing text-to-image models.
21

---

Published as a conference paper at ICLR 2024
User provides
DreamBooth
Assisted by
PerSAM
*A [V] dog jungle"
A of a dig)
"A [V] dop
*A [V] barn with forest In the background*
of a barn
[V] backgr
"A [V] table in garden"
pp
"A [V] table and erange safa"
[V] plushy grass"
photo pluthy
'A [V] plushy tortoise swimming in posl"
Figure 14: Visualization PerSAMMassisted DreamBooth. Our approach can alleviate the background disturbance, and boost DreamBooth (Ruiz et al., 2022) for better personalized synthesis.
D ADDITIONAL EXPERIMENTS AND ANALYSIS
D. EVALUATION ON ADDITIONAL BENCHMARKS
COCO-20' (Nguyen & Todorovic, 2019). Constructed from MSCOCO (Lin al., 2014), COCO- 20' divides the diverse 80 classes evenly into 4 folds for one-shot semantic segmentation. We directly test our method on the validation set without specific in-domain training. As shown in Table 8, our PerSAM(-F) achieves favorable segmentation performance over wide range of object categories, comparable to previous in-domain methods, i.e., FPPTarn (Zhang et al., 2022), SCCAN (Xu et al., 2023), and HDMNet (Peng al., 2023).
Tokyo Multi-Specttal-4 (Bao et al., 2021). Sampled from Tokyo Multi Multi- Spectral (Ha et al., 2017), Tokyo Multi-Spectrale contains 16 classes within outdoor city scenes, similar to CityScapes (Cordts
22

---

Published as a conference paper at ICLR 2024
Table 8: One-shot segmentation on COCO-
Table 9: One-shot segmentation on Tokyo
20' (Nguyen & Todorovic, 2019).
Multi-Spectral-4 (Bao et al., 2021).
Method
In-domain Train
mloU
Method
In-domain Train
mloU
FPTrans
47.0
PFENet
14.0
SCCAN
48.2
PGNet
17.5
HDMNet
50.0
V-TFSS
V
26.1
PerSAM
47.9
PerSAM
18.4
PerSAM-F
50.6
PerSAM-F
25.6
Table 10: Comparison with two text-guided models: OVSeg (Liang et al., 2023) and Grounded-SAM (gro, 2023).
Method
Prompt
PerSeg
COCO-20*
OVSeg
Category Name
76.5
37.8
Grounded-SAM
Category Name
93.2
51.3
PerSAM
One-shot Data
89.3
47.9
PerSAM-F
One-shot Data
95.3
50.6
Table 11:
Running Efficiency com- pared to SAM (Kirillov al., 2023).
Table 12: Comparison with SAM-PT (Rajič et al., 2023) on DAVIS 2017 (Pont-Tuset et al., 2017).
Method
Propagation
J&F
Method
FPSt
Memory (MB)
SAM-PT
Point Tracking
76.6
SAM
2.16
5731
PerSAM
Feature Matching
66.9
PerSAM
2.08
5788
PerSAM
+Point Tracking
68.2
PerSAM-F
1.98
5832
PerSAM-F
Feature Matching
76.1
PerSAM-F
+Point Tracking
77.2
et al., 2016). Different from existing methods, we only take as input the RGB images without the paired thermal data, and do not conduct in-domain training. As shown in Table 9, our approach still exhibits good generalization capacity in street scenarios, compared to the specialist models: PFENet (Tian et al., 2020b), PGNet (Zhang et al., 2019), and V-TESS (Bao et al., 2021).
D.2 COMPARISON TO ADDITIONAL METHODS
Text-guided Segmenters. Recently, open-world segmentation models guided by text prompts have driven increasing attention. To compare our approach with them, we select two popular methods: OVSeg (Liang et al., 2023) and Grounded-SAM (gro, 2023). OVSeg leverages MaskFormer (Cheng et al., 2021b) to first generate class-agnostic mask proposals, and then adopts fine-tuned CLIP for zero-shot classification. Grounded-SAN utilizes powerful eext-guided detector, Grounding DINO (Liu et al., 2023a), generate object bounding boxes, and then utilize them to prompt SAM for segmentation. Instead of giving a one-shot reference, we directly prompt them by the category name of the target object for text-guided segmentation, e.g., "cat", "dog". "chair", As shown in Table 10, our PerSAM-F consistently achieves competitive results on two different datasets: PerSeg and COCO-20$ This indicates that, utilizing PerSAM with class-agnostic one-shot reference is par with recognizing the category and then segmenting it with text-guided methods.
SAM-PT (Rajič et al., 2023). Although both our PerSAM(-P and the concurrent SAM-PT are developed based on SAM, approach can be generalized most one-shot segmentation tasks peasonaaizze/viddo//semattt weeppartteggmtttitiiteee segmentation). while SAM-PT specifically aims at video object segmentation. One key difference between approach and SAM-PT is how to locate and associate
23

---

Published as a conference paper at ICLR 2024
Table 13: Few-shot segmentation on the PerSeg dataset.
Table 14: Few-shot segmentation on
FSS-1000 (Li et al., 2020) benchmark.
Method
Shot
mloU
bloU
Method
Shot
mloU
SegGPT
1-shot
94.3
76.5
SegGPT
1-shot
85.6
SegGPT
3-shot
96.7
78.4
SegGPT
5-shot
89.3
PerSAM
1-shot
89.3
71.7
PerSAM
1-shot
81.6
PerSAM
3-shot
90.2
73.6
PerSAM
5-shot
82.3
PerSAM-F
1-shot
95.3
77.9
PerSAM-F
1-shot
86.3
PerSAM-F
3-shot
97.4
79.1
PerSAM-F
5-shot
89.8
Table 15: Different pre-trained encoders for obtaining the positive-negative location prior.
Method
Encoder
DAVIS 2017
FSS-1000
LVIS-92'
PASCAL-Part
PACO-Part
Painter
34.6
61.7
10.5
30.4
14.1
SegGPT
75.6
85.6
18.6
PerSAM
SAM
62.8
74.9
12.9
31.3
21.2
PerSAM
DINOv2
66.9
81.6
15.6
32.5
22.5
PerSAM-F
SAM
73.4
79.4
16.2
32.0
21.3
PerSAM-F
DINOv2
76.1
86.3
18.4
32,9
22.7
objects from the previous to the current frame, i.c., propagating the location prompt for SAM across frames. In detail, our PerSAM(-F) simply calculates location confidence map by feature matching, while SAM-PT relies on an external point tracking network, PIPS (Harley et al., 2022). As shown in Table 12, DAVIS 2017 dataset (Pont-Tuset al., 2017), SAM-PT performs slightly better than the original PerSAM-F. However, inspired by SAM-PT, we can also incorporate its point tracking strategy (the PIPS tracker) with PerSAM((F) propagate the nosiaive-negative point prompt, which effectively enhances the segmentation performance. This demonstrates the flexible extensibility of our approach for applying more advanced trackers in a andg-and-llay way.
D.3 FEW-SHOT SEGMENTATION BY PERSAM
Our approach is not limited one-shot segmentation, and accept few-shot references for improved results. As an example, given 3-shot references, we independently calculate 3 location confidence maps for the image, and adopt pixel-wise max pooling to obtain the overall location estimation. For PerSAM-F, we regard all 3-shot data as the training set to conduct the scale-aware fine-tuning
We respectively conduct experiments for 3-shot segmentation on PerSeg dataset and 5-shot segmenta- tion on FSS-FO00 dataset (Li et al., 2020). The results are respectively shown in Tables 13 and 14. By providing more visual semantics in few-shot data, both our training- ffee PerSAM and the fine-tuned PerSAM-F can be further enhanced.
D.4 ABLATION STUDY
Different Pre-trained Encoders. For video object segmentation in Table 2 and other one-shot segmentation in Table 3, we adopt the DINOv2 (Oquab al., 2023) encoder to obtain the positive- negative location prior by default. In Table 15, we show the results by using SAM's original image encoder. As DINOv2 is particularly pre-trained by large-scale contrastive data, it produces more discriminative image features than SAM's encoder. This contributes negative location prior for better segmentation results, especially on the challenging FSS-1000 dataset (Li et al., 2020). Despite this, with SAM's original encoder, our PerSAM-F and the training- free PerSAM still obtain better segmentation accuracy than Painter (Wang et al., 2022) or SEEM (Zou al., 2023) different datasets, demonstrating the effectiveness of approach.
more precise positive-
24

---

Published as a conference paper at ICLR 2024
Table 16: Different Image Encoders of SAM for PerSAM and PerSAM-F.
Table 17: Robustness to Mask Reference. We resize the reference mask by 'erode' and 'dilate' functions in OpenCV (Bradski, 2000).
Method
Encoder
mloU
bloU
ViT-B
63.98
49.30
PerSAM
ViT-L
86.61
69.86
Method
Shrink
Shrink
Enlarge
Enlarge
VIT-H
89.32
71.67
SegGPT
80.39
81.79
83,22
76.43
ViT-B
87.24
69.36
PerSAM
78.48
81.10
89.32
88,92
PerSAM-F
ViT-L
92.24
75.36
PerSAM-F
85.16
88.28
83.19
81.19
VIT-H
95.33
77.92
Image Encoders of SAM. By default, we adopt a pre-trained ViT-H (Dosovitskiy et al., 2020) in SAM the image encoder for PerSAM and PerSAM-F. In Table 16, we investigate the performance of other vision backbones for our models, i.c., VIT-B and ViT-L. As shown, stronger image encoders lead to higher segmentation mIoU and bloU scores. When using ViT-B as the encoder, the accuracy of training-for frree PerSAM is largely harmed, due to weaker feature encoding ability, while the one-shot training of PerSAM-F effectively mitigate the gap by +23.26% mloU and +20.06% bloU scores, which demonstrates the significance of our fine-tuning on top of a weak training-free baseline.
Robustness to the Quality of Mask Reference. For more robust interactivity with humans, we explore how our approach performs if the given mask is of low quality. In Table 17, respectively shrink and enlarge the area of the reference mask, and compare the results on PerSeg dataset. When the mask is smaller than the target object (shrink), PerSAM-F, aided by one-shot fine-tuning, exhibits the best robustness. In this case, the target embedding cannot incorporate complete visual appearances from the reference image, which largely harms the training-free techniques in PerSAM. When the mask becomes larger (enlarge), the oversize mask would mislead the scale-aware training of PerSAM- In contrast, despite some background noises, the target embedding can include all the visual semantics of objects, which, thereby, brings little influence to PerSAM.
E DISCUSSION
E. WHAT'S S THE ADDITIONAL RUNNING SPEED/MEMORT COMPARED TO SAM?
We test the additional running consumption of PerSAM and PerSAM-F on single NVIDIA A100 GPU with batch size As shown in Table 11, our PerSAM and PerSAM-F bring marginal latency and GPU memory consumption over SAM, indicating superior running efficiency.
E.2 How TO DIFFERENTIATE SIMILAR OBJECTS IN VIDEO OBJECT SEGMENTATION?
For video object segmentation, our approach tries to accurately locate the target object among similar ones by the following three aspects.
Discriminative Features from the Encoder. Due to large-scale pre-training, the SAM's image encoder, or the more powerful DINOv2, can already produce discriminative visual features for different similar objects, which fundamental the calculation of location confidence map.
Comprehensive Location Confidence Map. We calculate set of confidence maps for all fore- ground pixels within the target object, such as the head, the body, or the paws of a dog, and then aggregate them to obtain overall location estimation. This strategy can comprehensively consider the slight differences in any local parts between similar objects.
Temporal Cues between Adjacent Frames. To better leverage the temporal consistency along the video, we prompt SAM's decoder additionally with the object bounding box from the last frame. As different objects have different trajectories, such temporal constraints can better differentiate similar objects by spatial locations.
25

---

Published as a conference paper at ICLR 2024
One-shot Segmentation in Outdoor Street Scenes
One-shot Data
Test Images
Figure 15: One-shot segmentation of PerSAM-F in outdoor street scenes.
As visualized in Figures 12, our method can precisely segment the dancing man front of a crowd (the 2nd row) and differentiate different fishes within a group (the last row).
E.3 CAN PERSAM ALSO WORK ON SELF-DRIVING SCENARIOS?
Yes. In most cases, model can segment the designated cars with distinctive appearances in dense traffic. As visualized in Figure 15, for the user-provided target (c.g., a red car, a truck, and a bus), our PerSAM-F can well locate and segment them under severe occlusion surrounded by similar cars.
E.4 FAILURE CASES OF PERSAM-F
After solving the scale ambiguity issue, the three types of failure cases of PerSAM-F are shown in Figure 16: (a) different people with the same clothes, indicating our approach is not very sensitive to fine-grained human faces; (b) the key appearance of the target object is occluded by in test images (the red chest of the bird), indicating that we still need to improve our robustness when there is too large appearance change in test images; (c) discontinuous objects that SAM cannot tackle, for which we can replace SAM with stronger segmentation foundation model for assistance.
E.5 CAN PERSAM SEGMENT MULTIPLE IDENTICAL OBJECTS IN AN IMAGE?
Yes. As shown in Figure 17, given the one-shot image of reference cat, the test image contains two similar cats that expected to be both segmented, propose simple strategies for PerSAM:
26

---

Published as a conference paper at ICLR 2024
Failure Cases of PerSAM-F
One-shot Data
Test Images
(c)
(b)
adidas
adidas
adidas
Figure 16: Three types of failure cases of PerSAM-F.
Table 18: Statistic of Location Confi- dence Scores for Different Objects in the PerSeg dataset.
Table 19: DreamBooth Assisted by PerSAM with quantitative results. We adopt CLIP (Radford et al., 2021) calculate the image-text and -image similarity.
Similarity Statisties
0.30
Difference
Same
0.25
Probability
Method
Text-Align
Alige
KID (x 10
0.20
0.15
DreamBooth
0.812
0.793
29.7
PerSAM
0.830
0.814
29.2
0.10
PerSAM-F
0.834
0.818
28.9
0.05
0.00
0.0
0.2
0.6 Cosine Similarity
0.8 1.0
Iterative Masking. For two similar cats, we first calculate the location confidence map S1, and utilize PerSAM segment of the cats, denoting the obtained mask prediction M,. Then, reweigh the confidence map Sj by assigning zeros to the area within Mj. We denote the masked confidence map as S2. After this, we enable PerSAM's decoder to subsequently segment the second cat and acquire M2. In this way, our approach can iteratively mask the already segmented objects and segment all the expected similar objects, until there is no target in the image.
Confidence Thresholding. How stop the iteration when there is no other expected object in the image? We introduce thresholding strategy for adaptive control. As shown by the statistics in Table 18. we count the confidence scores of the positive location prior (the maximum score on the confidence map) for two groups of objects in the PerSeg dataset: 'Same' and 'Different', where we utilize DINOv2 (Oquab et al., 2023) as the image encoder. 'Same' utilizes the same object for reference and test, just like the normal evaluation. 'Different" utilizes one object for reference, but tests on all other 39 objects. We observe the scores 'Same" are almost all larger than 0.5, while those in 'Different" are lower than 0.4. Therefore, the iterative segmentation based on the confidence map with a threshold 0.45, which can well discriminate different objects categories for most cases, e.g., segmenting all the cats dogs in the image shown in Figure 18. this way, for a test image, if the maximum score in the confidence map lower than 0.45, there is target object in the image and would stop the iteration.
adopt simple thresholding strategy to stop
27

---

Published as a conference paper at ICLR 2024
By Encoder
By Deceder
Re Ar,
Decoder
Reweigh.s by
Thresheld by scores
pr der
Twxt teage
Castideate May
M, oftiw Foretor
Godferron Vop
Me forond
Endigg
Object
No Object
Figure 17: Segmenting Multiple Similar Objects in Image. We adopt strategies for PerSAM to simultaneously segment multiple similar objects: iterative masking and confidence thresholding. We denote the positive and negative location prior by green and red pentagrams, respectively.
User provides
Confidence
of Object
Confidence
Two Ubjects
Image
One Mask
User provides
Confidence Map
of One Object
Confidence Map
My Two Objects
One Image
One Mask
Segmenting Objects of the Same Category. Besides specific visual concepts, our approach can also be personalized by category, dog, with confidence thresholding strategy.
Figure 18:
E.6 Is PERSAM-F GENERALIZED ONLY TO A SPECIFIC OBJECT?
Our PerSAM-F can not only be personalized by specific object, but also generalize to certain category with the same amount of parameters. As visualized in Figure 13, given a reference cone/armour/crrcodilee in FSS-1000 dataset (Li et al., 2020), our PerSAM-F can well segment other similar conew//rmoons/crroodiisss in test images. This because objects of the same category can contain similar hierarchical structures, the learned scale weights of PerSAM-F by one sample also be applicable to different objects within the same category. In contrast. for different categories, one needs to fine-tune two sets of scale weights respectively fit their scale information.
28

---

Published as a conference paper at ICLR 2024
Figure 19: Visualization of the Enlarged PerSeg Dataset generated by a fine-tuned Dream- Booth (Ruiz et al., 2022). We show the examples of four objects with three different text prompts: 'A photo of an [OBJECT] in swimming pool/in Times Square/in front of Eiffel Tower.'
Table 20: Personalized Object Segmentation on the Enlarged PerSeg Dataset with 5x largaer in size. We compare the overall mloU and bloU for different methods (Bar et al., 2022; Wang et al., 2022: 2023; Zou et al., 2023).
Method
Painter
SEEM
SegGPT
PerSAM
PerSAM-F
mIoU bloU
43.6
82.8
87.8
85.9
89.6
37.5
51.3
69.7
66.2
72.4
E.7 WILL PERSAM BE CONSTRAINED BY SAM'S LIMITED SEMANTICS BY TRAINING?
Yes, due to SAM's inherent class-agnostic training. the visual features extracted by SAM's encoder contain limited category- semantics. This might constrain the category-level discriminative capability for complex multi-object scenes. Observing this limitation, we locate the target object among other objects in test images entirely by feature matching, i.e., the location confidence map. Such a matching strategy only considers the appearance- class-agnostic similarity, without category semantics. To this end, we can leverage other semantically rich image encoders, e-g., CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023), for PerSAM(-F) to improve the multi-object performance. We conduct an ablation study of different image encoders on DAVIS 2017 dataset (Pont-Tuset et al., 2017) for video object segmentation, which contains multiple similar objects within video. As shown in Table 15, applying CLIP and DINOv2 with more sufficient semantic knowledge can improve the results of PerSAM-F for more challenging multi-object segmentation.
E.8 CAN PERSAM HELP DREAMBOOTH ACHIEVE BETTER MULTI-OBJECT CUSTOMIZAAIOON?
Yes. Similar to oingee-obecct personalization, we only calculate the loss within foreground regions for DreamBooth (Ruiz et al., 2022) with multi-object training samples. As visualized in Figure 20, show the improvement for two-object customization assisted by our PerSAM. The backgrounds within images generated by DreamBooth severely disturbed by those within few-shot training images, while the PerSAM-assisted DreamBooth can accurately synthesize new backgrounds according to the input language prompts.
E.9 SCALING PERSEG DATASET
Although our newly constructed PerSeg dataset contains different objects in various contexts, it is relatively small in scale compared to existing segmentation benchmarks. For a more robust evaluation, we enlarge the PerSeg dataset (40 objects with 5~7 images per object) to 30 images per object, 5x larger in scale. We leverage the existing few-shot images to fine-tune DreamBooth (Ruiz et al., 2022) respectively for each object, and then generate new images with diverse backgrounds or poses (swimming pool, Times Square, Eiffel Tower, etc. ...). including richer data examples as shown in Figure 19. We report the segmentation results in Table 20, the scale-aware fine-tuned PerSAM-F still achieves the best performance, and the training- free PerSAM can also surpass Painter and SEEM, demonstrating the superior robustness of our approach.
29

---

Published as a conference paper at ICLR 2024
Dream Booth
Assisted by PerSAM
"A photo of a [V;] cat and [Vg] backpack the classroom."
User Provides
"A photo a [V]] cat and [V2] backpack in the swimming pool."
photo of a cat
*A photo [V,] and [V,] backpack the beach." "
A photo of backpack
Figure 20: Multi-object generation of PerSA.Massistted DreamBooth (Ruiz et al., 2022).
E.10 ANY OTHER APPLICATIONS FOR PERSAM?
For CLIP-based (Radford et al., 2021) few-shot image classification, a series of works (Zhang et al., 2021; 2023d; Udandarao et al., 2022) extract the visual features few-shot images by CLIP, and cache them category prototypes for downstream adaption of CLIP. However, such prototypes contain the visual noises the backgrounds that disturb the category semantics. Therefore, our PerSAM is helpful in segmenting the objects of the same category in few-shot images, and enables the CLIP-based methods to cache only foreground informative features. For 3D reconstruction by NeRF (Mildenhall al., 2021), existing approaches can only lift the objects, which are annotated with multi-view masks, into 3D space. Considering that the multi-view annotation is labor-intensive our approach provides a solution for NeRF to lift any object in a scene, simply by prompting SAM to segment the object in one view. On top of that, PerSAM can be personalized to generate the masks all multi-view images, allowing for efficient and flexible 3D reconstruction. We leave these applications as future works.
30


Reviews:
[{"review_id": "5SFIujlmBv", "reviewer": "Reviewer_XA1U", "summary": "The paper presents a lightweight method to leverage the Segment Anything Model (SAM), to perform single shot image segmentation. SAM is a powerful image segmentation framework, however it is becomes challenging to use it to perform personalized image segmentation by manually tuning its input prompts. In the approach proposed in the paper, the authors present Personalized approach for SAM (PerSeg) which is training free and can be used to segment a particular object (class) across images given only a single example image along with its binary segmentation mask. To do this, the authors present two approaches: (1) target-guided attention, and (ii) target-semantic prompting. Additionally, to alleviate the problem of objects being present in different scales the paper presents a simple fine-tuning technique that keeps SAM frozen, to combine information from multi-scale masks and improve the final segmentation outputs. \n\nA new dataset PerSeg is also introduced to test the proposed method, and finally the paper also shows how PerSAM can be used to improve DreamBooth [1], for the task of custom text-to-image synthesis. \n\nThe paper quantitatively and qualitatively demonstrates the efficacy of the proposed method across multiple datasets, and achieves competitive performance as compared to existing methods.\n\n[1] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022", "questions": "How comparable is this work or results against methods that do zero-shot or text guided segmentation like DenseCLIP [1]? And why / why not does it make sense to compare against such methods?\n\n[1]. DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting, Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu, https://arxiv.org/abs/2112.01518", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem of single-shot image segmentation is an important problem to solve. This has many downstream utilities in real-world applications ranging from design to healthcare. And the paper introduces a simple but effective technique to solve this by leveraging the powerful Segment Anything Module (SAM) [1]. \n\nThe introduced method is called Personalization approach for SAM (PerSAM), and it takes as input a single example image of the desired object we want to segment, and its corresponding segmentation mask. This is then used to segment out the given object across multiple images automatically. To do this PerSeg involves 2 approachs:\n(1) target-guided attention\n(2) target semantic prompting\n\nAdditionally to handle the object occurring in different scales, the paper introduces a lightweight fine-tuning technique that keeps the SAM model frozen, and aggregates information across multiple scales for finer segmentation.\n\nA new testing dataset is also introduced called PerSeg. \n\nThe work also shows the utility of the proposed method for improving text-to-image synthesis [2].\n\nAnd finally the authors show qualitative and quantitative results for the proposed method, and it performs competitively as compared to existing approaches.\n\nOverall a nicely written paper, with a simple idea and good results.\n\n\n[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023\n[2] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022", "weaknesses": "Overall it is a nicely written paper, with good results.\n\nHowever, it is somewhat lacking in it's quantitative evaluation. The choice of evaluation datasets is limited.\n\nIt would be worthwhile to also see the performance of the proposed method for one-shot segmentation on additional (more challenging) datasets like- MS-COCO, AED20K, CityScapes to also compare with more powerful existing state of the art models.\n\nAlso the comparison is lacking. It would be nice to compare against methods that do zero-shot or text guided segmentation like DenseCLIP [1].\n\n[1]. DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting, Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu, https://arxiv.org/abs/2112.01518", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u064BYcFlG", "reviewer": "Reviewer_Fp1Y", "summary": "The paper proposes a training-free Personalisation approach for SAM. For PerSAM, it uses a reference mask for obtaining the positive-negative location prior and then enable SAM for segmentation by: 1) target-guided attention and 2) target-semantic prompting. To reduce the ambiguity of segmentation scales, PerSAM-F is further proposed to fine-tuning SAM. PerSeg dataset is also constructed to evaluate the personalisation segmentation performance. The paper is also evaluated on the one-shot image and video segmentation benchmarks.", "questions": "Can the paper provide more failure cases visualisation for PerSAM? I am generally positive about this paper. If my concerns in the weakness section can be addressed, I will consider to further upgrade my rating.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organised with clear motivation and easy to understand. The illustration and visualisation figures are well presented.\n\n2. PerSAM is training-free and computationally efficient, where the ablation experiment for PerSAM in Table 4, 5 and 6 are extensive.\n\n3. The paper demonstrates good performance not only on the constructed PerSeg benchmark, but also on many image/video segmentation benchmarks.", "weaknesses": "1. In the appendix, the author mentioned using dinov2 features. Can the authors also provide the results in Table 2 and 3 by using the default image encoder features of SAM?\n\n2. What is the running speed/ memory consumption of PerSAM comparing to SAM?\n\n3. In Table 2, can the author provide performance comparison to SAM-PT [a]? [a] is a related work in adapting SAM for video object segmentation.\n\n[a] SAM-PT: Extending SAM to zero-shot video segmentation with point-based tracking. arXiv, 2023.\n\n4. On the video object segmentation benchmark, how dose PerSAM differentiate objects with similar appearances? Can PerSAM also work on self-driving scenario with many similar vehicles/cars in the dense traffic?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1RXddyEvWI", "reviewer": "Reviewer_inm4", "summary": "This paper proposes PerSAM, a novel training-free method, to customize the general-purpose SAM for personalized object segmentation by using a single image with a reference mask. Additionally, the paper introduces PerSAM-F, an efficient variant that enhances performance by tuning just two parameters within 10 seconds. The effectiveness of the proposed method is demonstrated by comprehensive experiments and ablation studies.", "questions": "* Can PerSAM be extended to do few-shot segmentation with more reference masks for achieving better performance? In the real scenario, utilizing 5-shot or 10-shot examples does not significantly increase the cost compared to 1-shot, but it does offer more precise visual information.\n\n* Is the fine-tuned PerSAM-F generalized only to a specific object? Can PerSAM generalize to different objects using the same parameters? What about using different fine-tuning methods such as LoRA for multi-objects generalization performance?\n\n* Can PerSAM help DreamBooth achieve more complex multi-object customization?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper is well-written and easy to understand.\n\n* This paper first studies an interesting task of customizing a general-purpose segmentation model for personalized scenarios. And the paper presents a highly effective method to address this task.\n\n* The method is simple and easy to follow. The proposed PerSAM can guide SAM to segment target objects by three effective training-free techniques.  By tuning 2 parameters within 10 seconds, PerSAM-F efficiently alleviates the mask ambiguity issue and improves the performance.\n\n* This paper has comprehensive discussions and experiments and shows good performances on various tasks.", "weaknesses": "The feature semantics of SAM might be limited due to SAM's class-agnostic training. While PerSAM and PerSAM-F demonstrate promising performance in personalized object segmentation, their effectiveness may be constrained by SAM's feature semantics in scenarios involving multiple different objects. This may require additional training to enable better transfer of SAM's features to downstream tasks. Alternatively, introducing other representations with stronger semantics, such as CLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)