SLiMe: Segment Like Me
Authors: Aliasghar Khani
Saeid Asgari
Aditya Sanghi
Ali Mahdavi Amiri
Ghassan Hamarneh
Venue: ICLR 2024 poster (2024)
Keywords: one-shot segmentation
computer vision
text-to-image models
stable diffusion
cross attention
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: a one-shot image segmentation method capable of segmenting at various levels of granularity

Abstract: Significant strides have been made using large vision-language models, like Stable Diffusion (SD), for a variety of downstream tasks, including image generation, image editing, and 3D shape generation. Inspired by these advancements, we explore leveraging these vision-language models for segmenting images at any desired granularity using as few as one annotated sample. We propose SLiMe, which frames this problem as an optimization task. Specifically, given a single image and its segmentation mask, we first extract our novel “weighted accumulated self-attention map” along with cross-attention map from the SD prior. Then, using these extracted maps, the text embeddings of SD are optimized to highlight the segmented region in these attention maps, which in turn can be used to derive new segmentation results. Moreover, leveraging additional training data when available, i.e. few-shot, improves the performance of SLiMe. We performed comprehensive experiments examining various design factors and showed that SLiMe outperforms other existing one-shot and few-shot segmentation methods.

Published as a conference paper at ICLR 2024
SLIME: SEGMENT LIKE ME
Aliasghar Khani Saeid Asgari Taghanaki Aditya Sanghi', Ali Mahdavi Amiri?,
Ghassan Hamarneh
1 Autodesk Research
2 School of Computing Science, Simon Fraser University
SLiMe
Figure 1: SLiMe. Using just one user-annotated image with various granularity (as shown in the leftmost column), SLiMe learns segment different unseen images in accordance with the same granularity (as depicted in the other columns).
ABSTRACT
Significant advancements have been recently made using Stable Diffusion (SD), for variety of downstream tasks, e.g., image generation and editing. This moti- vates us to investigate SD's ss capability for image segmentation at any desired gran- ularity by using few only one annotated sample, which has remained largely an open challenge. In this paper, we propose SLiMe, a segmentation method, which frames this problem as one-shot optimization task. Given a single image and its segmentation mask, we propose to first extract our novel weighted accu- mulated self-attention map along with cross-attention map from text-conditioned SD. Then, we optimize text embeddings to highlight areas in these attention maps corresponding segmentation mask foregrounds. Once optimized, the text em- beddings can be used to segment unseen images. Moreover, leveraging additional annotated data when available, i.e., few-shot, improves SLiMe's performance. Through broad experiments, SLiMe outperforms existing one- and few-shot segmentation methods. The source code of the project is publicly available.
examined various design factors and showed that
INTRODUCTION
Image segmentation a multifaceted problem, with solutions existing at various levels granular- ity. For instance, in applications like expression recognition facial alignment, segmenting images of faces into basic regions like nose and eyes might suffice. However, in visual effects applications, more detailed segments such eye bags, forehead, and chin necessary for tasks like wrinkle removal. Moreover, from the perspective of an user, a straightforward and effective approach to guide segmentation method determining what segment and the desired level of detail across

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
Figure 2: Our proposed weighted accumulated self-attention maps' sample results. Employing cross-attention naïvely without the self-attention for segmentation leads to inaccurate and noisy output (a and c). Using self-attention map along with cross-attentin map to create map enhances the segmentation (b and d).
a broad set of images by providing only one or a few segmented examples for the method to use for training. Meanwhile, the user should not need to curate large dataset with segmentation masks, train a large segmentation model, or encode elaborate and specific properties of target objects into the model. As result, customizable segmentation method that can adapt to different levels of granularity, using a few annotated samples, and provide users with the ability to intuitively define and refine the target segmentation according their specific requiremente is of high importance.
Recent research has tackled the lack of segmentation data by delving into zero-shot, textual descrip- tion based segmentation, and few-shot learning. DiffSeg (Tian et al., 2023) is a zero-shot segmen- tation method based on SD, which segments everything in the image. However, DiffSeg cannot be used to segment a specific object or part in the test images given a train sample, because its segmentation is not controllable in terms of which object to segment and segmentation granularity. Peekaboo (Burgert et al. 2022) is another work, which uses textual description for segmentation. To this end, given an image and textual description of the target object to be segmented, they use Stable Diffusion and its loss function to optimize a randomly initialized segmentation mask to reach the desired mask. Nevertheless, it cannot be used to segmentation of test images given train images, because the textual description of the target object in each image is unique and is not transferrable. Another promising method is ReGAN (Tritrong al. 2021). ReGAN first trains GAN (Goodfel-) low et al., 2014) on the data of specific class they aim to segment. Following this, they generate data by this GAN and the user manually annotates the generated data. Then both the generated data's features from the GAN and the annotations are utilized to train a segmentation model. In contrast, SegDDPM (Baranchuk al. (02I)) extracts features from pre-trained diffusion model (DM) and trains an ensemble of MIPs for segmentation using few labeled data. Both excel in segmentation with 10-50 examples but struggle with extremely limited samples. Furthermore, these models re- quire training on data specific to each category. For instance, to segment horses, it is necessary to collect a large dataset of horse images, a task that can be inherently cumbersome.
Whereas, SegGPT (Wang et al. 2023) employs one-shot learning, training color-randomized segmentation data which includes both instance and part-level masks. During inference, it segments only one region in target image using reference image and its binary segmentation mask. While SegGPT is effective, it demands a significant amount of annotated segmentation data for initial training, keeping the challenge of training effectively with single annotation still unaddressed.
In this paper, we propose Segment Like Me (SLiMe), which segments any object/part from the same category based given image and its segmentation mask with an arbitrary granularity level in a one-shot manner, avoiding the need for extensive annotated segmentation data or training a generative model like GAN for a specific class (see Figure [ and Figure for some examples). For this purpose, we leverage the rich knowledge of existing large-scale pre-trained vision/lane sssionn/annuage model. Stable Diffusion (SD) (Rombach et al. 2022a). Recent studies like (Hertz et al., (0022) have shown that the cross-attention maps of models like SD highlight different regions of the image when the corresponding text changes. This property has been utilized to modify generated images (Hertz et al., (022) and to achieve image correspondence (Hedlin et al., (023). Expanding on this idea, we present two key insights. First, the multifaceted segmentation problem can be framed as one-shot optimization task where we fine-tune the text embeddings of SD to capture semantic details such as segmented regions guided by a reference image and its segmentation mask, where each text embedding corresponds to distinct segmented region. Second, we observed that using standalone cross-attention maps lead to imprecise segmentations, as depicted in Figure 2 To rectify this, we propose novel weighted accumulated self (WAS)-attention map (see Section 4) This
2

---

Published as a conference paper at ICLR 2024
attention map incorporates crucial semantic boundary information and employs higher-resolution self-attention maps, ensuring enhanced segmentation accuracy.
Based on these insights, SLiMe uses single image and its segmentation mask to fine-tune SD's text embeddings through cross- and WAS-attention maps. These refined embeddings emphasize segmented regions within these attention maps, and are used to segment real-world images during inference, mirroring the granularity of the segmented region from the image used for optimization. Through various quantitative and qualitative experiments, we highlight the efficacy of our approach. SLiMe, even when reliant on just one or a handful of examples, proves to be better or comparable to supervised counterparts demanding extensive training. Furthermore, despite not being trained on a specific category. SLiMe outperforms other few-shot techniques on average and on most parts, across almost all the datasets. For instance, we outperform ReGAN (Tritrong et al., 2021 by nearly 10% and SegDDPM (Baranchuk et al. (2021) by approximately 2% in 10-sample setting. Additionally, in 1 -sammle context, we exceed SegGPT by around 12% and SegDDPM by nearly 11%.
2 RELATED WORK
Semantic Part Segmentation. In computer vision, semantic segmentation, wherein class label is assigned to each pixel in an image, is an important task with several applications such as scene pars- ing, autonomous systems, medical imaging. image editing, environmental monitoring, and video
analysis (Sohail et al.] 2022; He et al. 2016: Chen et al. 201 Zhao et al., 2017: He et al.) 2017]] Chen et al., 2017b] Sandler et al. 2018 Chen et al. all]]2018 20188 A more fine-graine derivative of seman- tic segmentation is semantic part segmentation, which endeavors to delineate individual components of objects rather than segmenting the entirety of the objects. Algorithms tailored for semantic part segmentation find applications in subsequent tasks such as pose estimation (Zhuang et al.) [021), activity analysis (Wang& & Yuille, 2015), object re-identification (Cheng et a (2016). autonomous driving and robot navigation (Li et al. 2023). Despite notable advancements in this domain (Li ct al. 2023 2022). predominant challenge faced by these studies remains the substantial need for annotated data, resource that is often difficult to procure. Hence, to address these challenges, search has pivoted towards exploring alternative inductive biases and supervision forms. However, limitation of such methodologies is their reliance on manually curated information specifie the object whose parts they aim to segment. For example, authors of (Wang & Yuille (2015) integrate inductive biases by harnessing edge, appearance, and semantic part cues for enhanced part segmen- tation. Compared to these approaches, our method only necessitates a single segmentation mask and doesn' 't rely on ad-hoc inductive biases, instead leveraging the knowledge embedded in SD.
Few-shot Semantic Part Segmentation. One approach to reduce the need for annotated data is to frame the problem within the few-shot part segmentation framework. There is a large body of work on few-shot semantic segmentation (Catalano & Matteuccil 2023) Xiong et al. [022 Johnander iet al. 2022 Zhang et al. 2022 Li et al. (022)) however, they mostly focus on the object- (not part-) level. recent paper, ReGAN (Tritrong et al. 2021), proposed a few-shot method for part seg- mentation. To achieve this, the researchers leveraged large pre- pre-trained GAN, extracting features from it and subsequently training a segmentation model using these features and their associated annotations. While this approach enables the creation of semantic part segmentation model with limited annotated data, it suffers from a drawback. Specifically, to train a model to segment parts of a particular object category, first a GAN is required to be trained from scratch on data from the same category. For instance, segmenting parts of a human face would necessitate a GAN trained on generating human face images. Thus, even though the method requires minimal annotated data, it demands a substantial amount of images from the relevant category. Following that, a few images, which are generated by the GAN, need to be manually annotated to be used for training the segmen- tation model. Afterward, a multitude of images should be generated by the GAN and segmented by the trained segmentation model. Finally, all the annotated data and pseudo-segmented data used for training segmentation model from scratch. Instead, we leverage pre- rre-trained DMs that are trained on large general datasets, eliminating the need to curate category-speciin datasets.
Diffusion models for semantic part segmentation. DMs (Sohl-Dickstein al. 2015) are class of generative models that have recently gained significant attention because of their ability to generate high-quality samples. DMs have been used for discriminative tasks such segmentation, as shown
3

---

Published as a conference paper at ICLR 2024
in SegDDPM (Baranchuk et al. 2021). Given few annotated images, they use internal features of DM, to train several MLP modules, for semantic part segmentation. Compared to SegDDPM, we utilize the semantic knowledge of text-conditioned SD, and just optimize the text embeddings. This way, we have to optimize fewer parameters for the segmentation task, which makes it possible to optimize using just one segmentation sample.
SD (Rombach et al. 2022a has been used for several downstream tasks such as generating faithful
images (Chefer et )023), inpainting, outpainting (Rombach al. 2022a), generating 3D shapes using text (ang 2022), and editing images guided by a text prompt (Brooks et al. (023). J. In addition to these, large body of work fine-tune SD or use its cross-attention modules to perform interesting tasks. For instance, (Gal et al.) 2022) fine-tunes SD's text embeddings to add a new object or style to its image generation space. Another example, (Hertz et al. 2022) uses SD's cross-attention modules to impose more control over the generation process. Moreover, in third instance, authors of (Mokady et al. 2023) edit a real image using SD's cross-attention modules. SD's cross-attention maps have been used for image correspondence by (Hedlin et al., 2023 Lastly, recent paper (Patashnik et al. 2023), uses SD's s self-attention and cross-attention modules for object level shape variations. Although these papers explore the applicability of SD in different tasks, its utilization in semantic part segmentation not fully explored. Therefor, in this work, we take advantage of SD's self-attention and cross-attention modules and fine-tune its text embeddings through these attention mechanisms to perform semantic part segmentation even with just one annotated image.
3 BACKGROUND
Latent Diffusion Model (LDM). One category of generative models are LDMs, which model the data distribution by efficiently compressing it into the latent space of an autoencoder and utilizing DM to model this latent space. An appealing feature of LDMs is that their DM, denoted as e(:; can be extended to represent conditional distributions, conditioned on text or category. To train text-conditioned LDM, natural language prompt is tokenized to obtain P. Then P is passed to G(P; Alternatively, is possible
a text encoder ((:: f) to get initializing a tensor of the same size. Afterward, the input image / is encoded to obtain I, and standard Gaussian noise e is added to it with respect to time step / to get I. Finally, the following objective is used to optimize the parameters of both ((.; () and e(.; 9), with the aim of enabling the model to acquire the capability to predict the added noise E:
obtain by randomly
LLM = ET,
- - (:
(1)
In this work, we use text-conditioned SD (Rombach et al.) 2022b). as our LDM, for two reasons. First, SD is conditioned on the text using the cross-attention modules, which have shown to exhibit
rich semantic connections between the text and the image embeddings (Hertz et al. 2022). Second, the internal features of SD are semantically meaningful and preserve the visual structure of the input image, enhancing the interrelation between text and image.
Attention Modules. SD's DM employs a UNet structure, which has two types of attention mod- ules (Vaswani et al. 2017): self-attention and cross-attention. The self-attention module calculates attention across the image embedding, capturing relationships between specific element and other elements within the same image embedding. On the other hand, the cross- atteettono module com- putes relationships between the latent representations of two different modalities, like text and image in the case of text-conditioned SD.
An attention module comprises three components: query, key, and value. It aims to transform the query into an output using the key-value pair. Therefore, given query Q, key K, and value V vectors with the dimension of d, the output O of an attention module is defined as follows:
QKT Vd
0 = Softmax
V.
In the self-attention module, the query, key, and value vectors are derived from the image embedding, while in the cross-attention module, the query vector is derived from the image embedding, and
4

---

Published as a conference paper at ICLR 2024
Cross-atiention
Ground truth mask
Text Embedding
CE Lowi
Attention-
UNet
Extraction
WAS WAS-attention
module
MSE Loss
Image Encoder
MSE Loss
Noise
Predicted Noise
Figure 3: Optimization step. After extracting image embeddings and adding noise, we pass them, along with text embedding obtained either by using text encoder or initialized randomly, through the UNet to obtain cross- and WAS-attention maps. Two losses are then calculated using these maps and the ground truth mask. Additionally, SD's loss is incorporated from comparing the added noise with the UNet's predicted noise.
the key and value vectors are derived from the text embedding. In our scenario, we extract the normalized attention map denoted S Softmax
which is applicable both the self- attention and cross-attention modules, and we note them as Sea E R xd'"""'XX x xw' and Seo E Rd xw' respectively. In this context, H/' and W" represent the height and width of the image embedding and T denotes the total number of text tokens. Sza shows the pairwise similarity of the elements in its imput image embedding. Hence, each element p its imput, is associated with an activation map, highlighting the similar elements p (Patashnik al.] 2023). Moreover, the intensity of the similar elements decrease as we move farther away from p. On the other hand, for each text token, Sca has an activation map, which effectively spotlights elements within the image embedding that align with that token within the model's semantic space. For example, if the model is instructed to generate image of a bear with the text prompt "a bear", the activation map associated with "bear" token within Sea, will emphasize on those elements that correspond to the bear object within the generated image.
QKT va
METHOD
We introduce SLiMe, method that enables us perform segmentation various levels of granular- ity, needing only one image and its segmentation mask. Prior research has demonstrated that SD's cross-attention maps can used in detecting coarse semantic objects during the generation process for more control in generation (Hertz et al. 2022) or finding correspondence between images (Hedlin al. 2023). However, there remains uncertainty regarding the applicability of cross-attention maps for finer-grained segmentation of objects or parts, especially within real-world images. To resolve this, we frame the segmentation problem one-shot optimization task where we extract the cross- attention map and our novel WAS-attention map to fine-tune the text embeddings, enabling each text embedding to grasp semantic information from individual segmented regions (Figure 3 During the inference phase, images. In what follows, we will first delve into the details of the text embedding optimization and then the inference process.
use these optimized embeddings to obtain the segmentation mask for unseen
4.1 OPTIMIZING TEXT EMBEDDING
Given pair of an image (I E "#*xxx xx3 and segmentation mask (M 1,, 111 with K classes, we optimize the text embeddings using three loss terms. The first loss term is cross entropy loss between the cross-attention map and the ground truth mask. The second one, is the Mean Squared Error (MSE) loss between the WAS-attention map and the ground truth mask. These loss terms refine the text embeddings and enable them to learn to emphasize segmented regions within both cross- and WAS-attention maps. Additionally, there is a subsequent SD regularization term ensure that the optimized text embeddings remain within the trained distribution of SD.
To optimize the text embeddings, it is necessary to extract the cross-atte aross-attention and self-attention maps. These maps derived from SD's UNet by initially encoding the training image I into the image embedding, I. Subsequently, a standard Gaussian noise added to this embedding with respect to the time step lope, resulting in I, Next, text prompt is converted sequence of
5

---

Published as a conference paper at ICLR 2024
Antention-Extraction module
UNet
Asa
S WAS
Pk-
Weighted Sum
Predicted
Noisc
I
Average
Resize and Average
Flatten
Keenet Layer
F
Self anlentnnnoon Cross-ateetion Layer
Figure 4: Attention-Extractinn module. To extract WAS-attention map of k text embedding
with respect to an image, we follow these three steps: (1) We feed the hth text embedding (Px) together with the noised embedding the image (I₂) to the UNet. Then calculate A by extracting the cross-attention maps of Pk from several layers, resizing and averaging them. (2) We extract the self-attention maps from several layers and average them (Asa). (3) Finally, we flatten A to
get Fh, and calculate a weighted sum of channels of Asas by weights coming from F&: and call
"Weighted Accumulated Self-attention map" was).. The UNet also produces an output that represents the predicted noise, which is used for calculating the loss of the SD.
text embeddings denoted as P. We then take the first K text embeddings and optimize them. The corresponding text embedding of each class is denoted by Px. essential to note that SD is con- figured handle 77 text tokens. Consequently, classes, which is sufficient for most applications. Finally, P and I, are fed into the UNet to obtain the denoised image embedding I' and extract the cross- and self-attention maps.
method can accommodate up 77 segmentation
SD has multiple cross-attention modules distributed across various layers. We denote the normalized cross-attention map of the 1th layer as {S.a) J E IR M, W xT and average them over different layers, we have empirically observed that this averaging improves the results. However, since Hi and W/ vary across different layers, we resize all {Seo to a consistent size for all the utilized layers. Finally, the attention map employed in our loss function calculated follows:
Aca Average: Auerage((Resize Resize ((caa))))))))))))
where Aca E ER" "R""" XT Average; computes the average across layers, and Resize refers to bilinear interpolation for resizing to dimensions H" W" Figure depicts this procedure. Finally, we compute the cross-entropy loss between the resized ground truth mask M to H" x W" (referred to M' and first K channels in the resized cross-attention map Aca for h (0, 1}. as outlined below:
(4) where CE refers to cross-entropy. Using this loss, we optimize k text embedding such that A highlights the t class's region in the segmentation mask, for k {1, K - 1}. Note that we do not optimize the first text embedding and assign A to the background class, as empirically we have found that optimizing it yields suboptimal performance.
LCe CE (E(A M')).
However, as the resolution of (Seah h we use are lower than the input image, object edges are vague in them. To enhance segmentation quality, we propose WAS-attention map, which integrates both self-attention and cross-attention maps. Besides possessing pairwise similarity between the image embedding's elements, the self-attention map has two additional features that make it suitable to be used for improving the segmentation results. First, the self-attention maps that we use, have higher resolution of feature maps compared utilized cross-attention maps. Second, it shows the boundaries in more detail. Table [ shows the importance of using the WAS-altention map which yields an average improvement 6.0% in terms of mloU over simply using the rros-aatentioon map for generating the segmentation mask. Like the cross- cooss-attntionn maps, we extract self-attention maps from multiple layers and compute their average follows:
Asa = Average, (S.a)))
(5)
where Asa E X and Average, calculates the average across layers. In equation there is need for Resize function the self-attention maps that use, all have the same size.

---

Published as a conference paper at ICLR 2024
Table 1: Ablating the effeet of WAS-attention. These numerical results, underscore the crucial contribution of WAS-attention maps to the quality of SLiMe's outcomes.
Use WAS-Attention Map
Body
Light
Plate
Wheel
Window
BG
Average
77.8 0.2
48.2
2.5
44.1
4.2
63.9 0.1
66.9
0.2
75.3 1: 0.2
62.7 1.3
81.5
1.0
56.8 1.2
5488 2.7
68.3 0.1
70.3 0.9
78.4 + 1.6
68.3 + 1.0
To calculate WAS-attention map, first resize A to match the size of Asa using bilinear inter- polation and call R Consequently, for each element p in R we have a channel in Asa that highlights relevant elements p. Finally, calculate the weighted sum of channels of Asa to obtain Say (WAS-attention map). The weight assigned to each channel is the value of the corre- sponding element of that channel in R (Figure This process can be outlined as follows:
WAs sum sum(flatten(FF.a) @ Asa).
This refinement enhances the boundaries because Aso possesses rich understanding of the semantic region boundaries (see the cross-attention and WAS-attention maps in Figure 3). At the end, we resize Swas to H" x W and calculate the MSE loss this way:
K-1
LMSE
Σ
(Resize(SWas) - M/ l2.
where Mí. is a binary mask coming from the resized ground truth mask M', in which only the pixels of the 4 class are 1.
The last loss we use the SD's loss function (LLOM), which is the MSE loss between the added noise and the predicted noise. We use this loss to prevent the text embeddings from going too far from the understandable space by SD. Finally, our objective to optimize the text embeddings defined as:
= + o&MMSE + BBLDMM
where a and are the coefficients of the loss functions.
4.2 INFERENCE
During inference, our objective is segment unseen images the same level of details the image used during optimization. To achieve this, we begin with the unseen image and encode it into the latent space of SD. Following this, standard Gaussian noise is introduced to the encoded image, with the magnitude determined by the time parameter heat Subsequently, we use the optimized text embeddings along with the encoded image to derive corresponding cross-attention and self-attentio maps from the UNet model. These attention maps, as shown in Figure 4 enable us to obtain WAS- attention maps for each text embedding. Afterward, we select the first WAS-attentiin maps that correspond to K classes. These selected maps are then resized using bilinear interpolation to match the dimensions of the input image and are stacked along the channel dimension. Subsequently, we generate segmentation mask by performing argmax across the channels. It is important to note that this process can repeated for multiple unseen images during inference, without requiring a new optimization. An analysis of the selection of various parameters used in our method is provided in the Appendix A.2
5 EXPERIMENTS
In this section, we demonstrate the superiority of SLiMe in semantic part segmentation. We use mloU to compare our approach against three existing methods: ReGAN (Tritrong et al.) 2021). SegDDPM (Baranchuk et al. 2021), and SegGPT (Wang et al.) 2023) two datasets: PASCAL- Part (Chen et al.] 2014) and CelebAMask-
(Lee et al.] 2020). ReGAN and SegDDPM utilize pre- trained GAN and DDPM models, respectively, training them FFHQ and LSUN-Hors, datasets for face and horse part segmentation. Additionally, ReGAN employs pre-trained GAN from the
LSSU-CC Car dataset for car part segmentation. We present the results for both 10-sample and 1-sample
7

---

Published as a conference paper at ICLR 2024
Table 2: Segmentation results for class car. SLiMe consistently outperforms ReGAN, even though ReGAN utilized generated data alongside 10 annotated data for training. Furthermore, our method exhibits superior performance to SegGPT on average, despite SegGPT being supervised. The first two rows show the supervised methods, for which we use the reported numbers ReGAN. The second two rows show the 10-sample setting and the last two rows, refer to the I-sample scenario. indicates the supervised methods.
Body
Light
Plate
Wheel
Window
Background
Average
CNN
73.4
42.2
41.7
66.3
61.0
67.4
58.7
CNN+CRF
75.4
36.1
35.8
64.3
61.8
68.7
57
ReGAN
75.5
29.3
17.8
57.2
62.4
70.7
52.15
SLiMe
81.5 + 1.0
56.8
1.2
54.8 + 2.7
68.3 + 0.1
70.3 + 0.9
78.4 1.6
68.3 + 1.0
SegGPT
62.7
18.5
25.8
65.8
69.5
77.7
53.3
SLiMe
79.6 + 0.4
37.5 = 5.4
46.5 2.6
65.0 = 1.4
65.6 1.6
75.7
3.1
61.6 0.5
Table 3: Segmentation results for class horse. SLiMe outperforms ReGAN, SegDDPM, and SegGPT on average and most of the parts. The first two rows show the supervised methods, for which we use the reported numbers ReGAN. The middle three rows show the 10-sample setting and the last three rows, are the results of the 1-sample scenario. indicates the supervised methods.
Head
Leg
Nock+Torso
Tail
Background
Average
Apperacceeee CNN+CRF"
47.2
38.2
66.7
55.0
46.8
37.2
76
ReGAN
50.1
49.6
70,5
19.9
81.6
54.3
ScgDDPM
41.0
59.1
69.9
39.3
84.3
58.7
SLiMe
63.8 = 0.7
59.5 + 2.1
68.1 + 4.4
45.4 + 2.4
79.6 + 2.5
63.3 + 2.4
SegGPT*
41.1
49.8
58.6
15.5
36.4
40.3
SegDDPM
12.1
42.4
54.5
32.0
74.1
43.0
SLiMe
61.5 = 1.0
50.3 + 0.7
55.7 = 1.1
40.1 + 2.9
74.4 + 0.6
56.4 + 0.8
settings, utilizing single validation sample for 10-sample experiments of SLiMe. Also, all exper- iments are conducted three times with different initializations, reporting their mean and standard deviation. We conduct experiments for SegDDPM and SegGPT using the custom version of test sets of the above-mentioned datasets, which are based on ReGAN settings, and report their results accordingly. For the remaining methods, we reference the results reported by ReGAN. Note that Re- GAN and SegDDPM are not universally applicable to arbitrary classes, unless a large dataset for the given class is collected and a generative model is trained. However, SLiMe does not require collect- ing large category specific data and training an additional generative model, because of the inherent semantic knowledge embedded in SD (Figure g) Whereas SegGPT requires a large segmentation dataset to be trained initially.
PASCAL-Part. This dataset provides detailed annotations of object parts. For our experiments, we focus on car and horse classes (for more details, please refer to Appendix [B.1. Table 2 presents results for the car class. As there is no available pre-trained model for the car class in SDgDDPM, couldn't 't make a comparison with this model for this category. As evident from Table 2 SLiMe outperforms ReGAN in the 10-sample setting on average and all the part segments by significant margin. Moreover, in the I-sample setting, SLiMe either outperforms SegGPT by a large margin or performs comparably. Likewise, Table 3 displays our results for the horse class, where it is evident that our method, SLiMe, outperforms ReGAN, SegDDPM, and SegGPT on average and for most of the parts. It is worth noting that, even though SPg only requires single segmentation sample for inference, it is fully supervised method and demands large segmentation dataset for training. In contrast, SLiMe is truly one-shot technique, where only single sample is needed for optimization.
CclebAMask-IHE This is a dataset of the facial part segmentation, and we report results on the parts used in ReGAN for comparison (for more details, please consult Appendix B.I). Figure Table 4 showcase our qualitative and quantitative results. In the 1-sample setting, SLiMe outperforms other methods average and for the majority of parts, demonstrating its superiority in I-sample scenario. On the other hand, in the 10-sample setting, except for three parts, our method either performs better comparably other methods. As mentioned earlier, note that SegGPT benefits from training on
8

---

Published as a conference paper at ICLR 2024
Table 4: Segmentation results of CelebAMask-HQ10. Our method consistently outperforms ReGAN, ScgDDPM, and SegGPT in the majority of parts in I-sample setting in the last four rows. Additionally, SLiMe either outperforms or performs comparably ReGAN and SegDDPM in 10-sample setting in the first three rows. is used to denote supervised methods.
Fyehrow
Pye
Mouth
Neck
Nose
Rockgrund
Averae
ScoDDPM
64.2
79.4
88.8
ReGAN
57.8
71.1
76.0
SegGPT"
24
48.8
$2.3
$1.7
82.7
66.7
71.3
73.6
85.7
28.01
57.1 60.
SegDDPM
28.9
46.6
57.3
61.5
72.3
44.0
66.6
69.4
77.5
76.6
SLIMe
$2.6 14
44.2 =: 2.1
57.1 3.6
61.3 à 4.6
809 a: 0.5
7488 a: 2.9
78,9 d: 1.3
77.5 1,8
86.8 =: 0.3
81.6 :: 0.5
08.6 0.3
Figure 5: Segmentation results of camouflaged objects. The larger images are used for optimizing SLiMe, and as the source image for SegGPT. Notably, SLiMe outperforms SegGPT.
a large segmentation dataset. Also, the other two methods employ class-specific pre-trained models. In contrast, SLiMe utilizes model pre-trained general data, equipping it with the ability work across a wide range of categories rather than being limited to specific class.
Additional Results. We also showcase the versatility of our method, which can be optimized an occluded object and infer images without the occlusion, or conversely, be optimized on fully visible object and make predictions on occluded objects. This shows our method's capability to comprehend part and object semantics. Figure [I] illustrates that despite occlusion of the target region caused by the person in the image used for optimization, our method performs well. is also possible to segment occluded objects using a visible reference object (see Figure T2). Moreover, Figure 5 we compare our method against SegGPT (Wang et al.) 2023) using two camouflaged animals, namely crab and lizard. Remarkably, SLiMe achieves precise segmentation of these animals, even in situations where they were challenging to detected with naked eye. This shows that SLiMe learns rich semantic features about the target object that do not fail easily due to the lack of full perception.
CONCLUSION
We proposed SLiMe, a one-shot segmentation method capable of segmenting various objects/parts in various granularity. Through extensive set of experiments and by comparing it state-of-the- art few-shot and supervised image segmentation methods, we showed its superiority. We showed that, although SLiMe does not require training on specific class of objects dataset, it outperforms other methods. On the other hand, SLiMe has some limitations. For example, it may result in noisy segmentations when the target region is tiny. This can be attributed to the fact that the attention maps, which we extract from SD for segmentation mask generation, have smaller size than the input image. To counter this, we employed bilinear interpolation for upscal- ing. Nonetheless, due to scaling, some pixels might be overlooked, leading to the undesired noisy outcomes. For visual examples of this case, please refer to Appendix A. Resolving the mentioned limitation, and making applicable to 3D and videos, would be an interesting future direction.
large segmentation

---

Published as a conference paper at ICLR 2024
REFERENCES
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21 2021.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint arXiv:2211.13222 2022.
Nico Catalano and Mattco Matteucci. Few shot semantic segmentation: a review of methodologies and open challenges. arXiv preprint arXiv:2304.0558 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite:
Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4):1-10, 2023.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysts and machine intelligence, 40(4): 834-848 848, 2017a.
Liang-Chin Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: (.06:::700605887,, 2017b.
Liang-Chie Ching Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1971-1978, 2014.
De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. Person re-identification by multi-channel parts-based enn with improved triplet loss function. In Proceedings of the iEEE conference on computer vision and pattern recognition, pp. 1335 1344, 2016.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.016688 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. arXiv preprint arXiv:230S.15580 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or.
Prompt-to-pree to-mmpp image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
10

---

Published as a conference paper at ICLR 2024
Joakim Johnander, Johan Edstedt, Michael Felsberg, Fahad Shahbaz Khan, and Martin Danelljan. Dense gaussian processes for few-shot segmentation. In European Conference on Computer sion, pp. 217-234. Springer, 2022.
Cheng-Ham Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL
htttps://openrevii net (otee//ooooommimni--dooee
Xiangtai Li, Shilin Xu, Yibo Yang. Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, and Dacheng Tao. Punopticpartfomerr+: A unified and decoupled view for panoptic part seg- mentation. arXiv preprint 2rXiv:2301.00954, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6038-6047 2023.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor and Daniel Cohen-Or. Lo- calizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303. 1306, 2023.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10683 2022a.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-16695, 2022b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiell Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1510-4520. 2018
Ali Sohail, Naeem A Nawaz, Asghar Ali Shah, Saim Rasheed, Sheeba Ilyas, and Muhammad Khur- ram Ehsan. A systematic literature review machine learning and deep learning methods for semantic segmentation. IEEE Access, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaxiang Tang. http:///ggtuubb.ooasssawwweeeee http://shaaakkestttledeaattttiit dreamfusion.
Stable-dreamfusion
Text-to-3d with stable-diffusion, 2022.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. arXiv preprint ar0iv:2308. 12469, 2023.
Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakom. Repurposing gans for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF conference on com- puler vision and pattern recognition, pp. 4475-4485. 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jaany Wang and Alan L Yuille. Semantic part segmentation using compositional model combining shape and appearance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1788-1797, 2015.
11

---

Published as a conference paper at ICLR 2024
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.03288: 2023.
Zhitong Xiong, Haopeng Li, and Xiao Xiang Zhu. Doubly deformable aggregation of covariance matrices for few-shot segmentation. In European Conference on Computer Vision, pp. 133-150. Springer, 2022.
Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot seg- mentation. Advances in Neural Information Processing Systems, 35:6575-6555 6588, 2022.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 2881-28900 2017.
Chungang Zhuang, Zhe Wang, Heng Zhao, and Han Ding. Semantic part segmentation method based 3d object pose estimation with rgb-d images for bin-picking. Robotics and Computer- Integrated Manufacturing, 68: 102086, 2021.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
ADDITIONAL RESULTS
Figure showcase our qualitative and quantitative results
Figure 6: Qualitative face segmentation results. Results of SLiMe optimized with 10 samples.
In Figure 7 we provide comparisons with ReGAN (Tritrong et al. 2021). It is evident that SLiMe exhibits more intricate hair segmentation in the second and third rows, showcasing its ability to capture finer details compared to ReGAN. Additionally, in the second row, the ear segmentation produced by ReGAN appears be noisy by comparison.
Input
SI.Mee
Jigsaw
Color
Figure 7: Qualitative comparison with other methods on CelchAHQ-Mask, Qualitative results of several methods on the 10-sample setting of CelebAHQ-Mask. As you can see, SL.Me captures the details better than ReGAN and other methods (e.g., hairlines in the second row). All the images are taken from TTitronn al. 202T).
In addition to the segmentation results presented for several object categories in the paper, Figure showcases additional 1-sample visual results. These results encompass wide range of objects and provide evidence of SLiMe's capability to perform effectively across various categories.
Another noteworthy feature SLiMe is its generalization capacity, as illustrated in Figure This figure demonstrates, despite being optimized can grasp the concepts of head, body, legs, and tail and effectively apply them to unseen images from various categories. However, Figure Tojllustrates that when optimized a dog and a cow, SLiMe is adept at learning to exclusively segment the dog class unseen images. These two figures, highlight SLiMe's ability acquire either high-level concepts exclusive object classes.
single dog image with segmented parts, SLiMe
image containing both
One more appealing feature of SLIMe is that not only is it able to learn segment from occluded image and segment fully visible objects (Figure [T]. but it can also be optimized on a fully visible object and make predictions on occluded samples. As example, in Figure 12, SLiMe is optimized on a fully visible bear and predicts an accurate segmentation mask for the occluded bears.
A.1.1 FAILURE CASE OF SLiMe
As mentioned in the paper, SLiMe may fail in the cases where the target region tiny. Figure I3 shows examples of this, where the target to be segmented is necklace, which is pretty small.
13

---

Published as a conference paper at ICLR 2024
Figure 8:
Part segmentation results on different objects. SLiMe exhibits strong performance across a wide variety of objects. The images, along with their corresponding annotations used for optimization, displayed on the left.
A.2 ABLATION STUDIES
In this section, we present the results of our ablation studies, which aim to illustrate the impact of each component in model. These experiments were conducted using 10 samples from the class in the PASCAL-Part dataset.
In Table present the results of text prompt ablation experiments demonstrate the robustness of SLiMe to the choice of the initial text prompt. The "part names" row displays results obtained by using specific part names in the prompts. In the row, labeled "" the text prompt is left empty.
14

---

Published as a conference paper at ICLR 2024
Figure 9: Generalizability of SLiMe. SLiMe optimized on dog's parts, can accurately segment corresponding parts of other animals.
Figure 10: SLiMe exclusive segmentation. SLiMe optimized on an image containing both dog and cow seen in the left image pair), can segment only dogs, even in presence of other animals.
Figure 11: Segmentation results of occluded objects. Although SLiMe is optimized using an occluded car's image (the leftmost image), it demonstrates proficiency car segmentation on unseen images (the remaining images the right). Particularly noteworthy is its ability to accurately segment all three cars in the top-right image.
In the final row, we use the term "part" instead of specific part names. By comparing these rows, we observe minimal influence of the initial text prompt on the results.
Moving we conducted experiments to determine the optimal coefficients for our loss functions. As illustrated in the first four rows of Table 6 where a =, the most suitable coefficient for LsD found to be 0.005. Furthermore, when comparing the 3rd and 5th rows, the significance of Luse becomes apparent.
Next, we turn attention to ablation of the parameters trian and best Initially, we vary the range used to sample topt. Table 7 shows that the best range is [5, 100), which introduces a reasonable amount of noise. A larger end value in the range results in very noisy images in some steps, making difficult to optimize the text embeddings. Conversely, a very small end value means that SLiMe does not encounter sufficient range of noisy data for effective optimization.
15

---

Published as a conference paper at ICLR 2024
Figure 12:
Occluded object in inference. SLiMe undergoes initial optimization with a bear image, as depicted in the left image. Subsequently, is put to the test with images featuring occluded portions of the bear. Notably, SLiMe precisely segments these occluded objects.
B
Figure 13: Failure case. The segmentation masks generated by SLiMe, depicted in images (b) and (d), reveal an inherent challenge. Our method encounters difficulty when it comes to accurately segmenting minuscule objects, such the necklace in this image. These tiny objects often diminish size, and at times, even vanish within the cross-attention maps we employ, primarily due to their limited resolution.
After examining topt. ablate the parameter tiet. Selecting appropriate value for frest is crucial, as demonstrated in Table SLiMe performs optimally when we set hest to 100.
Another parameter subjected ablation is the learning rate. Choosing the correct learning rate is essential, as a high value can result in significant deviations from text embeddings that SD can com- prehend. Conversely, low learning rate may not introduce sufficient changes to the embeddings. Our experiments in Jable that the optimal learning rate for our method is 0.1.
Finally, we performed ablation study the choice of layers utilize their cross-attention mod- ules. Based on our experiments in Table TO we determined that the best set of layers to use are the gth th layers.
B IMPLEMENTATION DETAILS
We opted for SD version 2.1 and extracted the cross-attention and self-attention maps from the 8th to t2 and last three layers of the UNet, respectively. For the text prompt, we use sentence where the word "prompt" repeated by the number of parts be segmented.
During optimization, we assigned random value to the time step of SD's noise scheduler, denoted as lopt. for each iteration. This value was selected randomly between 5 and 100, where lopt can in range spanning from 0 to 1000. During inference, consistently set hest to 100.
For optimization, we employed the Adam optimizer with a learning rate of 0.1, optimizing our used weighted cross-entropy loss,
method for 200 epochs with a batch size of 1. Additionally, with each class's weight determined as the ratio of the number of whole pixels in the image to the number of pixels belonging that class within the image. Furthermore, the values for and were set to and 0.005, respectively.
We set H and 4 to be 64. Regarding the CelebAMask-HO.. divided the 512 512 images into 4 patches of size 400. After acquiring their WAS-attention maps from SLiMe, we aggregated them to generate the final WAS-attention map and subsequently derive the segmentation mask.
16

---

Published as a conference paper at ICLR 2024
Table 5: Ablating the text prompt. Minimal effect of the initial text prompt for SLiMe. "part names": using all part names separated with space for text prompt. ("background body light plate wheel window"); ": leaving the text prompt empty; "part": using "part" instead of part names ("part part part part part port"), ); SLiMe is with the second settings.
Text Prompt
Body
Light
Plate
Wheel
Window
BG
Average
"port names"
82.0 + 0.3
55.3 + 2.0
56.1 + 1.1
69.4 + 0.6
69.6 0,9
79.5 1.1
68.7 = 0.4
81.6 + 1.0
56.7 0.4
54.4 +: 3.5
69.6 : 1.3
68.1 0.6
80.2 0,9
68.4 + 1.2
"part"
81.5 + 1.0
56.8 -
54.8 + 2.7
68.3 = 0.1
70.3 + 0,9
78.4 = 1.6
68.3 = 1.0
Table 6: Ablating the loss terms, Comparing the first four rows shows the importance of Lsp- Furthermore, when comparing the last two rows, it underscores the effectiveness of LUSE-
Body
Light
Plate
Wheel
Window
BG
Average
0.0
0.0
0.0
+00+0.00 0.0
0.0 = 0.0
0.0 + 0.0
0.0 0.0
31.8 0.0
5.3 - 0.0
1
0.05
80.1 + 0,4
57.6
46.3 0.2
67.4 1.0
63.0
0.2
65.4 + 0.9
0.005
81.5 + 1.0
56.8 + 1.2
54.8 2.7
68.3 0.1
70.3 + 0,9
1.6
68.3 +: 1.0
0.0
73.7 2.3
39.7
1.2
3..0
55.0
61.3 4.33::000
2.2
55.8 + 1.6
0
0.005
80.6 + 0.5
56.1
= 1.1
57.4 0.4
69.1 + 0.4
66.7 + 1.6 78.0 18.0 1.4 68.0 + 0.3
When constructing the WAS-attention map from the cross-attention and self-attention maps, only considered the corresponding cross-attention map of a token if the maximum value in that map exceeded 0.2. Otherwise, disregarded that token and assigned zeros its corresponding channel in the WAS-attention map.
For optimizing PASCAL-Part classes, applied the following augmentations: Random Hor- izontal Flip, Gaussian Blur, Random Crop, and Random Rotation. For the car class, we set the random crop ratio range to [0.5, 1]. while for the horse class, was adjusted [0.8, 1). Addition- ally, we applied random rotation within the range of [-30, 30] degrees.
Moreover, when optimizing CelebAMask-HOQ we incorporated set augmentations, which encompassed Random Horizontal Flip, Gaussian Blur, Random Crop, and Random Rotation. The random crop ratio was modified fall within the range of [0.6, 1], and random rotation was applied within the range of 1010, 10] degrees.
B. DETAILS THE DATASETS
In this section, we initially present further elaboration on the datasets on which we optimized our method. This includes information about the categories as well as the segmentation labels. Subse- quently, we offer details about the dataset preparation process.
B. PASCAL-PART
Car: Background, Body, Light, Plate, Wheel, and Window.
Horse: Background, Head, Leg, Neck+Torso, and Tail.
B.1.2
CELERAMASK-HO
Face:
Background, Cloth, Ear, Eye, Eyebrow, Face, Hair, Mouth, Neck, and Nose.
B.1. DATASET PREPARATION
PASCAL-Part. For this dataset. follow the procedures of ReGAN (Tritrong et al. 202T]: We start by cropping the images with the bounding boxes provided in the dataset. Afterward, remove those images where their bounding boxes have overlap of more
17

---

Published as a conference paper at ICLR 2024
Table 7: Ablating trian- Our results across different ranges for choosing topt indicate that optimal performance is achieved when Eopt is selected from the range [5, 100].
Range of topt
Body
Light
Plate
Wheel
Window
BG
Average
15 5,20
78.8 1.6
52.6 +: 2.8
53.6 + 1.0
66.6 + 0.2
70.3 + 0.1
78.7 + 1.2
66.8 1: 0.2
5, 100
81.5
1.0
56.8 + 1.2
54.8 2.7
68.3 + 0.1
70.3 0.9
78.4 1 1.6
68.3 1.0
900
80.4 1.1
54.4
1.6
54.3 2.4
67.3 1.3
70.2 0.4
79.9 + 1.7
67.8
Table 8: Ablating hiet. Evident from the table, get the best results when hest 100
hust
Body
Light
Plate
Wheel
Window
BG
Average
80.3 + 0.4
50.8 + 1.8
49.7
4.2
66.5 + 0.8
65.5 + 0.7
78.4 + 1.2
65.2 + 1.4
20
80.1 + 0.8
53.1 + 2.7
53.6
1.9
65.6 2.7
67.3 = 2.1
76.0 + 2.5
65.9 + 0.7
100
81.5 + 1.0
56.8 + 1.2
54.8 2.7
68.3 + 0.1
70.3 + 0.9
78.4 + 1.6
68.3 + 1.0
than 5% with other bounding boxes. Finally, we remove the cropped images smaller than
50 x 50 for the car class and 32 x 32 for the horse class.
CalebAMask-HOO The size of images that we use for this dataset is 512 x 512.
18

---

Published as a conference paper at ICLR 2024
Table 9: Ablating lr. The results of optimizing SLiMe with various learning rates reveal a crucial relationship. When the learning rate too low, SLiMe struggles to learn effectively, resulting in minimal progress. Conversely, when the learning rate is excessively high, the text embeddings deviate significantly from the comprehensible embeddings of SD.
Ir
Body
Light
Plate
Wheel
Window
BG
Average
10.1 + 9.2
3.6 + 6.2
12.4 10.8
11.2 + 11.2
13.2 + 14.5
39.7 + 11.
15 7.4
0.1
81.5 + 1.0
56.8
+ 1.2
54.8 1 2.7
68.3 1 0.1
70.3 1 0.9
78.4 1.6
68.3 + 1.0
0.01
81.5 = 0.0
54.9
0.4
52.8
+ 1.7
68.5 + 0.5
69.8 0.2
79.2
0.6
67.8 0.3
0.001
70.0 0.9
10.5 1.5155 5.1
40.6 0.5
0.7 1 0.5
1 6.1
62.4 1 1.6
33.7 4
Table 10: Ablating layers to use their cross-attention module. The middle layers of SD's UNet exhibit superior semantic understanding compared to the other set of layers.
set of cross attention layers
Body
Light
Plate
Wheel
Window
Average
1s to 7 layers
12.7 9.5
13.8 ::: 12.1
10.7 3.7
29.5 : 14.2
32.1 2.9
34.2 + 4.1
22.2 +: 3.4
8th to 12th layers
81.5 +: 1.0
56.8 1.2
54.8 + 2.7
68.3
70.3 +: 0.9
78.4 +: 1.6
68.3 1.0
13th to 16 layers
76.8 0.8
51.9
56.2
62.6
65.3
68.6 2.6
63.6
19


Reviews:
[{"review_id": "EZY6t9ESeu", "reviewer": "Reviewer_kAQy", "summary": "This paper presents SLiMe that allows for the segmentation of various objects or parts at different granularity levels with just one annotated example. The method leverages the knowledge embedded in pre-trained vision-language models and uses weighted accumulated self-attention maps and cross-attention maps to optimize text embeddings. The optimized embeddings then assist in segmenting unseen images, demonstrating the method's effectiveness even with minimal annotated data. The paper includes extensive experiments showing that SLiMe outperforms existing one- and few-shot segmentation methods.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: SLiMe introduces a unique one-shot optimization strategy for image segmentation, which is useful when the available data is limited.\n\n2: The proposed method demonstrates superior performance over existing one- and few-shot segmentation methods in various tests, indicating its practical applicability and effectiveness.\n\n3: The paper showcases the method's versatility by successfully applying it to different objects and granularity levels, emphasizing its broad applicability.", "weaknesses": "My main concerns focus on the **text prompt**.\n\n1: The introduction of the text prompt is quite abrupt. In the Introduction, SLiMe is described as requiring only an image and a corresponding mask to achieve segmentation of any granularity. However, immediately after, the author talks about fine-tuning text embeddings. What is the definition of 'text' in this task? How are text embeddings obtained? And do different granularities correspond to the same text? The author is encouraged to provide further clarification.\n\n2: The role of \"text prompt\" in the method. The authors claim that \"our novel WAS-attention map to fine-tune the text embeddings, enabling each text embedding to grasp semantic information from individual segmented regions\". However, I haven't found evidence from the main text to illustrate the correspondence between \"text embedding\" and \"individual segmented regions\", especially in the arbitary granularity situation, which is one of the most important parts of this paper. Moreover, how to construct the text is also ignored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doB5ePyED8", "reviewer": "Reviewer_Fjuo", "summary": "This paper proposes to retarget the Stable Diffusion model (SD) for few-shot semantic segmentation. Instead of taking the in-context learning or data generation approaches, this paper proposes a new pipeline which optimizes the text embedding in SD on the input image to \"find\" the text embedding to correspond to the segmented region. In addition to the cross-attention map of the optimized text token, it proposes a new self-attention map fusion module to regress the ground truth mask with a higher resolution. The proposed method achieves SOTA result on the benchmarks used in one previous work. The ablation studies shows the effectiveness of the model design.", "questions": "- For the learned text embedding, are they interpretable? e.g. one can use the data-driven approach to find text tokens whose embedding are closest to the optimized ones.\n\n- How is the performance on other benchmark datasets like ADE-Bedroom-30 (used by segDDPM) and FSS-1000 (used in segGPT)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel and interesting idea. The idea of retargeting the feature representation in a pretrained generative model for few-shot semantic segmentation is not new. But it's novel to exploit the text branch in the text-based image generation model (i.e. Stable Diffusion). Instead of training the adaptor model to map between the generative features to the semantic masks which may overfit on input image features, the proposed method aims to find the text embedding which may be more generalizable.\n\n- The proposed method significantly outperforms SOTA result on the benchmarks used in the ReGAN paper. The ablation studies shows the effectiveness of the model design.\n\n- The paper provides enough details in the appendix for reproduction and the open-sourced code is straightforward to follow.", "weaknesses": "- Unconvincing importance of WAS attention: Figure 1 shows a good intuition that we need WAS attention to refine the object boundary. However, In Table 5, the contribution of the WAS attention doesn't seem to be significant. For the without WAS attention results (fig. 2a, 2c), will they be much improved by using GrabCut or other methods for segmentation refinement post-processing?\n\n- Lack of comparison on benchmark datasets like ADE-Bedroom-30 (used by segDDPM) and FSS-1000 (used in segGPT). The current quantitative results are only on horse/car/face datasets used in ReGAN. The result can be more solid with evaluation on diverse types of objects/parts.\n\n- Figure 3 is a confusing. Currently, it seems like the predicted noise is from the cross-attention map and the WAS-attention map. It'll be better to put the Unet from SD in the box. Then from this Unet, one output is the predicted noise from the original SD and the other output is fed to the attention-extraction module which selects layers and combines attention maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bMp00kG2EX", "reviewer": "Reviewer_VRf7", "summary": "This work proposes a one-shot segmentation approach using the stable diffusion model. They pose the problem as one-shot optimization to perform object segmentation at different granularity levels conditioned on a single segmentation map. They take advantage of the self / cross-attention layer in the diffusion model to optimize the text embeddings for the given semantic segmentation task. They evaluate the proposed method on two public datasets and show its outperformance against the recent related work.", "questions": "1. Is there any specific reason behind not including the SegDDPM results in Tab. 1?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea of optimizing the text embedding based on the attention maps for semantic segmentation is interesting and novel.\n\n2. The proposed method is shown to be advantageous in two datasets both quantitatively and qualitatively.\n\n3. The paper is well-written and easy to follow. The theoretical background is explained well.\n\n4. The limitations are discussed.\n\n5. The method is well-ablated for the different components.", "weaknesses": "1. The proposed method seems to be adapted for the segmentation task based on the work by Hedlin et al for unsupervised semantic correspondence.\n\n2. Related works which are not cited:\n[a] Burgert, Ryan, et al. \"Peekaboo: Text to image diffusion models are zero-shot segmentors.\" arXiv preprint arXiv:2211.13224 (2022).\n[b] Tian, Junjiao, et al. \"Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion.\" arXiv preprint arXiv:2308.12469 (2023).\n\n3. The number of works that are compared against is limited.\n\nMinor:\n1. Making the best result in the ablation study tables bold would improve the readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "deTlRptpNU", "reviewer": "Reviewer_16rP", "summary": "This paper examines the problem of class-specific semantic part segmentation, under a one/few-shot data setting.\n\nThe method proposed by the paper is to leverage recent advances and findings in diffusion models, specifically in stable diffusion the cross-attention modules capturing relevant spatial regions and being used for semantic correspondence.\n\nLearning is done by fine-tuning the text embedding from a stable diffusion model on an image and a corresponding part segmentation mask.  The embedding is optimized under a loss with three components, encouraging the cross-attention map, and introduced weighted accumulated self-attention map, to match the ground-truth segmentation, while not straying too far from the original stable diffusion loss.\n\nExperimental validation is done on car, horse, and face part segmentation, with comparable to favorable results when compared against several recent state-of-the-art baselines.", "questions": "see weaknesses above, in particular the issue raised in the second bullet", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- a nice framework for tackling the problem of one/few-shot semantic part segmentation, having clear benefits over existing proposed approaches in terms of amount of additional supervised annotations required for training, while maintaining similar performance\n\n- in general, paper details are clearly presented, including a thorough supplementary appendix", "weaknesses": "- one of the stated contributions is the introduced weighted accumulated self-attention map.  This seems important enough that the incremental contribution of this component to the overall performance should perhaps be added to the main text rather than deferred to the appendix.  \n\n- further, within the appendix, Table 9 shows an improvement from adding WAS-attention, from 62.7 on average to 68.3.  I'm a little confused, then, on how this differs from Table 5, as I would have though the last row, setting $\\alpha=0$, would also correspond to dropping WAS-attention, and here the average performance is 68.0\n\n- lastly, on initial read, I was unsure of how text/text prompt was being used within the proposed method.  This was clarified by the first ablation study in A.2, but perhaps a sentence mention of this in the main text would also be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)