Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval
Authors: Dvir Samuel
Rami Ben-Ari
Matan Levy
Nir Darshan
Gal Chechik
Venue: NeurIPS 2024 poster (2024)
Keywords: Tex-to-image diffusion model
instance retrieval
Primary area: diffusion_based_models
TL;DR: We present a new approach for personalized tasks such as segmentation or retrieval without training using text-to-image diffusion models

Abstract: Personalized retrieval and segmentation aim to locate specific instances within a dataset based on an input image and a short description of the reference instance. While supervised methods are effective, they require extensive labeled data for training. Recently, self-supervised foundation models have been introduced to these tasks showing comparable results to supervised methods. However, a significant flaw in these models is evident: they struggle to locate a desired instance when other instances within the same class are presented. In this paper, we explore text-to-image diffusion models for these tasks. Specifically, we propose a novel approach called PDM for Personalized Diffusion Features Matching, that leverages intermediate features of pre-trained text-to-image models for personalization tasks without any additional training. PDM demonstrates superior performance on popular retrieval and segmentation benchmarks, outperforming even supervised methods. We also highlight notable shortcomings in current instance and segmentation datasets and propose new benchmarks for these tasks.

Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval
Dvir !.2u
Rami Ben-Ari
Matan Levy
Nir Darshan 2
Gal Chechik 1.4
Bar-Ilan University, Israel AriginAT, Israel
The Hebrew University of Jerusalem, Israel
*NVIDIA Research, Israel
Input Image
Target Image
DINOv2
PerSAM
Ours
Figure 1: Personalized segmentation task involves segmenting specific reference object in scene. Our method is capable to accurately identify the specific reference instance in the target image, when other objects from the same class present. While other methods capture visually semantically similar objects, our method can successfully extract the identical instance, by using personalized feature map and fusing semantic and appearance cues. Red and green indicate incorrect and correct segmentations respectively.
Abstract
Personalized retrieval and segmentation aim to locate specific instances within a dataset based input image and short description of the reference instance. While supervised methods are effective, they require extensive labeled data for training. Recently, self-supervised foundation models have been introduced these tasks showing comparable results to supervised methods. However, sig- nificant flaw in these models is evident: they struggle to locate desired instance when other instances within the same class are presented. In this paper, we explore text-to-image diffusion models for these tasks. Specifically, we propose a novel approach called PDM for Personalized Diffusion Features Matching, that lever- ages intermediate features of pre-trained text-to-image models for personalization tasks without any additional training. PDM demonstrates superior performance on popular retrieval and segmentation benchmarks, outperforming even super-
"Correspondence to: Dvir Samuel <dvirsamuel @ @gmail.com>..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

vised methods. We also highlight notable shortcomings in current instance and segmentation datasets and propose new benchmarks for these tasks.
1 Introduction
Personalized retrieval and segmentation focus on identifying specific instances within a dataset. When provided with an imput image featuring a particular instance (such as your beloved cat) and brief description ("A cat"), the objective is to locate and segment this exact instance throughout large collection of images. Personalized methods are useful in various applications, including instance search [34], product identification [6,41], and landmark recognition [47]. Furthermore, personalized segmentation can be applied to video tracking [45], automatic labeling [43], and image editing [5, 8].
While supervised methods can be effective for these tasks, they require an extensive amount of labeled training data. Recently, a self-supervised foundation model was proposed [45] to address this task. This model uses the SAM encoder [ [14] or DINOv2 [23] foundation model to extract spatial features from given reference instance. These features are then used to localize the object instance in the target image. While effective when a single instance appear in the target image, both DINOv2 and SAM fall short when multiple instances within the same object class are presented in the image. This is illustrated in Figure showing failure cases of D2 and SAM in localizing the correct dog or van (see first and second row). They also fail when two similar objects from different semantic classes are presented (wrongly segmenting the dog instead of the cat.)
In this paper, we propose explore text-to-image diffusion models for these tasks. Text-to-image foundation models have achieved remarkable success in generating new and unique images from text prompts [7, 30,30,11,, 31,33J. These models have the capability to generate an infinite array of objects and instances, each exhibiting unique appearances and structures. Consequently. it is reasonable to hypothesize that properties of generated objects are encoded within the intermediate features of the diffusion model during generation. Recent studies [1, 11.37,40 40] show zero-shot capabilities to create subtle changes in generated instances by manipulating the intermediate activation of the diffusion layers, during generation. Although effective, using text-to-image diffusion models "out of the box" for instance-related tasks, beyond generation or editing, remains unexplored.
In this paper, we present a new approach, called PDM, Personalized Diffusion Features Matching, for personalized retrieval and segmentation. PDM requires no training or fine-tuning, no prompt optimization, or any additional models. We demonstrate how a specific layer and block contain hidden textural and semantic information. These features are then used for the localization of a reference instance within a given target image. enabling both personalized segmentation and retrieval. PDM builds upon these newly discovered diffusion features, and surpasses other superviedd methods (like DINOv2 [23]. SAM [14] and DIFT [36]) weakly supervised methods (CLIP, OpenCLIP) and even supervised methods on personalized instance retrieval and segmentation tasks.
We also address significant limitations in traditional benchmarks for retrieval and segmentation. Current benchmarks often feature images with single, distinct object or multiple objects from different categories, allowing semantic-based methods to achieve high accuracy. To overcome these deficiencies, we construct new benchmarks based on newly published video tracking and segmentation dataset [4]. This dataset includes videos with multiple instances from the same category (e.g. two dogs playing or a group of people talking). Our method significantly outperforms all baselines on this new dataset, highlighting its ability to accurately handle multiple similar instances and demonstrating its superior capability in personalized retrieval and segmentation.
2 Related Work
Exploring pre-trained diffusion features. Text-to-image diffusion models [7,30,31, 31,33] have demonstrated tate-oof-hhe-art performance for image generation tasks. With its superior generation ability, recent studies started investigating the internal representation of diffusion models. DIFT [36] and Fuse [44] showed that extracting features from the ResNet layers of the denoising module provides a semantic correspondence between two objects which can also be used for image editing propagation. Plug-and-Play [40] suggested to extract features from self-attention layers of reference image, during the image generation process, while incorporated with text prompt. This approach

---

showcased that output images can retain the structure of the reference image while embodying the appearance described in the text prompt. Posss-ImageAAtemeee layers in self-attention layers correspond to the structure and the appearance of generated images. Their findings enabled the generation of images that blend the structure from one image with the appearance from another. ConsiStory [37] recently suggested injecting the self-attention features of an instance from a pre-generated image into the generation process of other images to ensure consistent reproduction of the same instance across images. DiffSeg [38] introduced method using self-attention maps for zero-shot image segmentation. They aggregate attention maps from multiple self-attention layers during image generation and merge them iteratively produce stack of object proposals. Segmentation maps are then obtained by applying Non-Maximum Suppression over the merged maps. In contrast to these studies, in this paper, we explore using internal features of pre-trained diffusion models for instance related tasks.
[1] further showed that sub-
Personalized Segmentation: PerSAM [45] introduced the use of SAM [45] for personalized image segmentation. They employed the SAM [14] encoder (or DINOv2 [23]) for the representation of the reference and target images, which are then used to calculate confidence map localizing the user's reference instance in the target image. Finally, it predicts positive and negative points on the target image to be used prompts for SAM. Additionally, they proposed a new benchmark, called PerSeg, for personalized image segmentation. includes 40 objects across various categories, each associated with 5-7 images, and is evaluated using mIoU and bloU metries.
Content-based instance retrieval can be seen a variant of personalized retrieval where images contain only a single instance. Recent supervised methods, GSS [21] and HP [2] proposed Graph Networks for effective retrieval. SuperGlobal [34] proposed a memory-effiiiet image retrieval method, that specifically focuses on the global feature extraction while in the re- ranking stage, they update the global features the query and top-ranked images by only considering feature refinement with small set of images, thus being very efficient. Recently, also self-supervised models [9, 1123,,66 show comparable performance to supervised methods on retrieval tasks. These techniques achieve impressive results in zero-shot scenarios however, they often necessitate model fine-tuning to achieve optimal performance. In this study, we investigate text-to-inn text-to-image diffusion models, which belong to the category of self-supervised models, for zero-shot personalized retrieval and segmentation tasks. Our findings show that diffusion features supress features from other self-supervised foundation models.
Instance Retrieval:
Semantic-level Feature Matching. Recent works have focused on improving semantic-leve feature matching in various tasks. SIGMA [19] introduces semantic-complete graph matching for Domain Adaptive Object Detection, addressing within-class variance through node-to-node matching. Light- Glue [20] enhances local feature matching efficiency with deep network adaptive to image difficulty, making ideal for latency-sensitive tasks. In this paper, we propose Personalized Diffusion Features Matching (PDM), which leverages intermediate features from pre-trained text-to- tmage diffusion models for personalized retrieval and segmentation without additional training.
Method
In this section, describe our approach to leverage pre-trained diffusion models for personalized retrieval and segmentation. We begin by defining these tasks and then delve into identifying features that encompass both semantic and appearance aspects. Lastly, demonstrate the application of these features in personalized instance retrieval and segmentation.
3.1 Personalized Retrieval and Segmentation.
In personalized retrieval and segmentation, the user supplies a single reference image, and a mask indicating the reference instance [45] or the class name of the instance [10]. This work focuses on the case where only class names are provided, For personalized retrieval, the goal is to retrieve images from database that contains the exact instance specified in the reference image. personalized segmentation, the objective is to segment the specified instance in new images and videos.
3

---

(a)
(b)
PCA visualization Queries from attention Block
Cross Attention Maps C = "dag"
30
50
4
Figure 2: (a) PCA visualization of Q features obtained from the first self-attention block in the last layer of the U-Net module, various diffusion timesteps. Objects with similar textures and colors have similar features. The dog's s color in / is similar to the colors of both the dog and the in 12, indicating textural similarity. Additionally, the localization is sharper larger timesteps. (b) Visualization of the cross-attention map J CT for a given prompt "dog". Note the higher region correlation (brighter colors) corresponding the dog, while overlooking the the bottom image.
3.2 Are instance features even encoded in a pre-trained text-to-image model?
Pre-trained text-to-image -mage models generate endless variety of objects and instances, each with unique visual characteristics and structures. Recent methods have demonstrated that specific changes in the activations of self and cross-attention activations of the diffusion layer influence the appearance of specific instances in the generated image. These methods typically modify all activations across all denoising timestamps to affect the generated image. This indicates that instance features are indeed encoded within these models. One can propose to use all diffusion activations during generation and aggregate them for downstream tasks. However, using all features extracted from diffusion layers is memory -intenrvv and computationally demanding. It also raises the challenge of merging all these features coherently.
We aim to identify a single layer at a unique timestamp where both the semantics and appearance (texture) of reference instance encoded. We first briefly explain how extract features from Stable Diffusion [31] J, a pre-trained text-to-image model. The architecture of Stable Diffusion consists of VAE encoder and VAE decoder that facilitates the conversion between the pixel and latent spaces, and denoising U-Net module that operates in the latent space. We refer the reader to Appendix A, for preliminary on the internal structure of the denoising U-Net layer. We first encode input image / into the latent space of a VAE using an encoder to produce a latent code 2g. Next, we employ diffusion inversion method [24,35], to compute the latent code 2 at the time step with the class name embedding as inputs. We then run denoising step at timestamp t to extract activations (features) from the denoising U-Net.
Previous studies [36, 40,44) observed that outputs of earlier layers from the U-Net decoder capture coarse yet consistent semantic correspondences, while deeper layers capture more low-level details and high-frequency information. Based on these observations, and in contrast to previous work, we conducted more thorough analysis of features extracted from all blocks of the last U-Net layer, examining their role across different timestamps. Interestingly, we consistently found that appearance features are encoded in the queries (Q*A) and keys (KC matrices of the self-attention (SA) block. This is illustrated in Figure 2(a), where we perform Principal Component Analysis (PCA) on features extracted for a pair of images, at various timestamps. It shows that S features of the dog I are similar (same color and texture) to those of the middle dog and cat in Iz. indicating that textural features are encoded in these layers (similar results are observed for SAA features).
We therefore define appearance features of an image to be the average tensor of QSA and KSAA features with dimensions h x d extracted from the self-attention (SA) block, at the last layer L of timestamp t:
1 (L
+ K (ALL)) E TRhxxxd
2
Here, and w represent spatial resolutions of features extracted from layer L,, while d denotes the feature dimension.

---

Diffusion
Appearance Map
4
Features Extraction
Mf
JPAu
Eq.5
Combined Map
"dog"
Eq.7
Semantic Map
Diffusion
C
Eq.
Features
Extraction
An overview of our Personalized Diffusion Features Matching approach. PDM combines semantic and appearance features for zero-shot personalized retrieval and segmentation. We first extract features from the reference, I,- and target I, images. Appearance similarity is determined by dot product of cropped foreground features from the reference feature map, J.A. and the target feature map F&A (Eq. 5) Semantic similarity calculated the product between class name token and the target semantic feature map F to create Semantic Map (Eq. 6). The final similarity map SDF combines both maps by average pooling. Note, that while the appearance and semantic maps attend on two dogs, their fusion yields a single and correct result.
Figure 3:
For semantic similarity, [16] observed that cross-attention maps establish the relationship between the textual imput prompt and patch/pixel-wise image features, effectively allowing coarse semantic segmentation map that highlights areas of potential object localization. This further illustrated in Figure 2(b), where the
attention of the word "dog" with both images results in attention map highlighting the location where dogs can be found. This cross-attention map is calculated by fusion of the spatial feature map and the token embedding, after projection. Therefore, define the semantic features to be the projected spatial features of the cross-attention (CA) block:
JS 2CALu CA(L) Ruxxxxd
3.3 Personalized Diffusion Features Matching
We now describe our method for combining semantic and appearance features to address personalized retrieval and segmentation tasks in a zero-shot manner, without training or fine-tumn fine-tuning. We call our approach PDM for Personalized Diffusion Features Matching. See Figure 3 for illustration.
Let J.A. FS and J,A, FS denote the appearance and semantic features extracted for the reference image I, and target image I, respectively. Next we define our appearance and semantic similarity functions.
Appearance Similarity: We start by localizing objects in the target image that have similar visual features the reference instance in 1r. To this end, make use of C E 1xxd as the projected token vector the class name, extracted from the cross-attention block CA(L)(same block as FFS)
We first use the cross-attention map between spatial image features and C to obtain reference mask M,. Specifically:
C Vd
M, L(softmazt
T) g
This mask is used to crop relevant appearance features of the instance from the feature map F.A. which will later be used for searching within target images. The masked appearance feature map thus defined as:
JAM M, o
Il is the indicator function and is a threshold, resulting eventually in binary mask, with foreground features (discarding zeroed-out tokens). Note that denotes spatial-wise multiplication. This approach leverages the U-Net's ability to preserve spatial information in its latent codes and
5

---

ROxford
DAVIS
Peršeg
PerM (Ours)
Figure 4: Examples of personalized retrieval and segmentation benchmarks. Current benchmarks mostly show one single instance in an image multiple instances from different object classes. Our benchmark for both retrieval and segmentation introduces a realistic and challenging case where multiple instances from the same object class are in the image, e.g. two dogs or multiple cars.
features during the diffusion process. Next, we compute map for the appearance similarity score between the reference and target image by simply applying dot product between the corresponding masked reference feature map and target feature map, followed by average pooling:
S
AM (1) F(
(5)
where S Rhxx and FAM(() refers the feature map in J.A J.A (i).
Semantic Similarity: Here we would like to localize all objects that have the same semantic category as the reference instance. To achieve this, we make use of the semantics encoded in the input class name and calculate a score map between and F... Specifically, we compute:
S F CT
(6)
The overall diffusion feature (DF) score map combining both semantic (conceptual) and appearance (textural) features is then
1 SDF =
2 (S4 + +S$) E R hxar
(7)
Using diffusion features for personalized retrieval and segmentation. For personalized retrieval, we rank the target images, using global score, obtained from the average of SDF, indicating the matching score between target (candidate) and the reference (query) image. For personalized segmentation, we propose two variations: (1) The score map SDF upsampled to the size of the target image, using binary threshold. We then segment all pixels that are above that threshold. (2) Following [45], we select the point with the highest confidence value in SDF as positive prompt for the position of the target object, and use it to segment the object with SAM [14].
4 Evaluation Datasets for Personalized Retrieval and Segmentation
For the evaluation of PDM, we adopted traditional instance retrieval and one-shot segmentation benchmarks, where we also used the provided class names. While preparing these benchmarks, we discovered that most existing instance retrieval and one-shot segmentation benchmarks predominantly showcase only a single instance per object class. For instance, widely used instance retrieval benchmarks such as RParis [28] and ROxford [28], focus on single landmarks their images, with categories typically representing only one possible instance. Similarly, image and video segmentation benchmarks such as the popular Davis [27] dataset and PerSeg [45] mainly comprise either a single instance multiple instances from diverse object classes, each exhibiting distinct visual and semantic characteristics. This is illustrated Figure 4. These trends make relatively straightforward for semantic-based methods to accurately retrieve or segment instances, as there are often hard negative instances (objects from the same category but different instance) within or across images. Consequently, comparing instance-based based features with current methods such benchmarks often yields comparable results, failing to highlight the strengths of instance-basee methods.
To establish clear distinction between semantic-basee and instance-level methods, we introduce two new benchmarks: Personalized Multi- Retrieval (PerMIR) and Personalized Multi- Instance Segmentation (PerMIS). Our proposed benchmarks are constructed using the recently

---

introduced BURST dataset [4], which serves for Object Recognition, Segmentation, and Tracking in Video. This dataset contains videos with pixel-precise segmentation masks for all unique object tracks spanning different object classes. As the dataset encompasses both single-instance and multi-instance videos, we focus on videos containing at least one hard negative instance per video. Specifically, we select videos with a minimum of two instances belonging to the same object class. We then filter out frames that do not contain these instances. This filtering process results in 150 videos across 16 object classes, with an average of 3.1 instances per frame. Detailed statistics can be found in Appendix C. Finally, for the personalized instance retrieval (PerMIR), we randomly chose three frames from each video, designating one the query frame and the remaining two the database (gallery) frames. For the personalized image segmentation task (PerMII-image) we randomly pick three frames from every video, assigning one the query frame and the others for evaluation. Ground-trute masks used for cropping the instance from query images and are also used for segmentation evaluation. We further evaluate the task of video label propagation. For this task use the first frame of video as the reference image and the subsequent frames for evaluation. We intend to make our generated datasets publicly available for future work.
5 Experiments
We evaluate PDM across three main tasks: (1) Personalized image and video segmentation, (2) Personalized retrieval and (3) Video label propagation where a single video frame is given with object segmentation and the aim is propagate labels (masks) across video frames, leveraging the information provided by previous frames. The ablation study can be found in Appendix B
Implementation details. The main bottleneck of PDM is the real image inversion process, where the image is converted to its noise latent representation for subsequent feature extraction. Using SoTA inversion technique by [24] with Vanilla StableDiffusion, takes about 5 seconds for each image a single A100. This is due to the requirement of 50 inversion steps. In order to mitigate this, we integrated [24] into SDXL-turbo. variant of stable diffusion requiring only 4 inversion steps. This decreases the inversion time to 0.5 seconds per image. Therefore, for all our experiments, features were extracted from SDXL-turbo at the last. U-Net layer the first timestep & = 4. Furthermore, all images were resized to 512 x 512 for proper image inversion. We set т, the threshold for M, to be 0.7 for all our experiments.
5.1 Personalized Image Segmentation
Datasets. We conducted experiments across two personalized (one-shot) image segmentation bench- marks. We first evaluate PDM the PerSeg [45] dataset, which comprises 40 objects spanning diverse categories such as daily necessities, animals, and buildings. Each object is represented by 5-7 images and masks, capturing different poses or scenes. Additionally, we assessed our method's performance on the PerMIS- Image benchmark (Section 4).
Baselines. We evaluate our method by contrasting it with different self-supervised foundation models: (1) DINOv2 [23], (2) PerSAM [45], (3) DIFT [36] and DiffSeg [38]. Additionally, we benchmark it against -upA-super techniques trained specifically for image segmentation, namely SEEM [48] and SegGPT [42].
Following [23,36], we report mIOU and bIOU metrics over all benchmarks. Segmentation with PDM is done by upsampling SDF to image size. Segmentation with DINOv2 and DIFT is done using features as similarity function. Specifically, nearest neighbors are found between the query features and target gallery features. No training is involved. We additionally report results with SAM integration, as proposed by [45] (see 3). Here, features are utilized to derive positive point, followed by segmentation using SAM.
Evaluation protocol.
Results. Table la presents the results of our experiments in personalized image segmentation. Our approach, denoted as ours, outperforms supervised methods trained specifically for image segmen- tation. Additionally. our method achieves superior performance compared to other self-supervised models, including DINOv2 [23], DIFT [36], and PerSAM [45]. We also demonstrate a significant im- provement in performance by applying PerSAM with our method, called PerSAM(PDM), surpassing both benchmarks by a considerable margin. Figure 5(a) provides qualitative segmentation results showing that our method reliably identifies the reference instance despite substantial variations in
7

---

Table 1: Benchmark (a) Personalized Segmentation (b) Video Label Propagation. Our method
shows the best performance on all benchmarks and achieves a notable balance between J and F, indicating effectiveness capturing both region and contour details.
(a)
(b)
Perosnalized Image Segmentation
Video Label Propogation
PerSeg
PerMIS
DAVIS
PerMIS (Video)
(Image)
Model
mloU
bloU
mloU
bloU
J&F
F
J&F
J
F
SEEM [48]
87.1
55.7
14.3
35.8
SegGPT [42]
94.3
76.5
18.7
39.5
MAST [15]
65.1
61.7
69.2
SFC [12]
71.2
74.0
73.2
70.2
76.3
DINOv2 [23]
68.7
27.6
20.2
41.9
71.4
67.9
74.9
5.4
62.5
68.6
DIFT [36]
63.2
26.9
21.9
43.1
70.0
67.4
78.6
69.7
67.3
71.8
DiffSeg [38]
38.6
37.9
7.9
6.4
PerSAM(SAMM [45]
95.3
77.9
16.5
38.3
76.1
74.9
79.7
64.0
61.8
67.1
PDM (ours)
95,4
79.8
42.3
86.8
75.8
72.9
80.1
75.1
72.1
78.0
PerSAM(PDM) (ours)
97.4
81.9
49.7
89.3
78.0
75.1
81.9
76.5
73.5
79.4
(a)
(b)
Input Image
Target Image
DINOv2
PerSAM
Ours
Input Image
OpenClip
DINOv2
Ours
"horse"
"van"
"man"
"bus"
Figure 5: Qualitative Comparison: (a) Personalized Segmentation: Red and green indicate incorrect and correct segmentation, respectively. Our method accurately recognizes the reference instance despite significant variations (view angle, pose, scale), while other methods often capture false positives from the same category. (b) Image Retrieval: Top-1 retrieved image is shown for each method. Note how our model identifies images containing the same instance, despite their small size and large variations. Other methods tend to capture only semantic similarity. Retrieval images have been zoomed in and cropped for clarity.
the target image, whereas other methods frequently capture false positives within the same category. Additional qualitative results Appendix F
5.2 Video Label Propagation
Datasets. We further conducted experiments across two temporal one-shot image segmentation benchmarks. We conducted evaluations on the DAVIS17 dataset [27]. This dataset comprises 150 video sequences, with object masks provided for all frames. Furthermore, evaluated our method's performance on the PerMIS-Viaee benchmark (Section 4).
Evaluation protocol. Following [36,45 45), we used the first frame image and the corresponding object masks as the user-provided query data. We also follow them and report region-based similarity J (the Jaccard Index, measuring the overlap between the predicted and ground truth regions), contour-based accuracy F (evaluating the accuracy the predicted contour compared to the ground truth contour) and J & as evaluation metrics.

---

Table 2: Personalized Retrieval: Mean Average Precision (mAP) on various benchmarks comparing PDM with state-of offte-of-thearnd art self-supervised, weakly supervised, and supervised methods. While our method yields superior performance, other methods leveraging our PDM features also yield performance boost.
ROxford Medium Hard Medium Iard
RParis
PerMIR
Methods
Self & Weakly Supervised
MAE [1]]
11.7
2.2
19.9
4.7
BBTT [46]
39.0
12.7
70.7
47.0
DINOv2 [28]
75.1
54.0
92.7
83.5
29.7
CLIP [29]
28.5
7.0
66.7
41.0
20,9
OpenClip [13]
50.7
19.7
79.2
60.2
26.7
GLIP
31.2
BLIP [17]
33.3
SLIP [22]
35,9
PDM (ours)
77.2
58.3
93.4
84.7
73.0
OpenClip PDM (ours)
70.1
57.7
90.1
82.0
69.9
DINOv2 + PDM (ours)
80.4
62.1
93.6
85.1
70.8
Supervised
GSS [2]]
80.6
64.7
93.4
85.3
HP [2]
85.7
70.3
83.3
SuperGlobal [34]
90.9
80.2
93.9
86.7
33.5
GSS PDM (ours)
89.3
84.8
62.0
SuperGlobal PDM (ours)
91.2
80.3
94.0
86.8
69.1
Compared methods. We compare our approach with various self-supervised foundation models: (1) DINOv2 [23], (2) PerSAM [45] and (3) DIFT [36] and DiftSeg [38]. We also compare with SoTA supervised methods that were trained on the task of video segmentation. Namely, MAST [15] and SFC [12].
Results. Table presents the results of our experiments the video label propagation task. Our method demonstrates competitive performance the DAVIS [27] dataset and superior results on PerMIS benchmark. Our method achieves a notable balance between and F, indicating its effectiveness in capturing both region and contour details. Improvement in PerSAM(PDM) demonstrated that our PDM can boost results also for other methods.
5.3 Personalized Retrieval
Datasets. We conduct experiments across various retrieval benchmarks, including both single- instance and ROxford and RParis datasets 26] with revised annotations [28]. These datasets consist of 4,993 and 6,322 images, each featuring a single instance. Evaluation involves 70 query images per dataset, categorized into Easy, Medium, and Hard tasks based retrieval complexity, with focus primarily on the more challenging Medium and Hard tasks. Instance masks are obtained from [3]. We further evaluate model on the PerMIR benchmark (Section 4).
datasets. Initially, we assess our model's performance on the widely-used
Baselines. We compare our approach with state-of offtt-rrrttttt models, including self-supervised founda- tion models: MAE [11]. SEER, and DINOv2 [23]; seakly-supervised foundation models: CLIP [29] and OpenClip [13]; and fully supervised methods: GSS [21], HP [2], and SuperGlobal [34]. Both self-supervised foundation models and weakly supervised foundation models were evaluated without further training or fine-tuning. We showcase results utilizing PDM both independently and as a re-ranking technique built upon various frozen pre-trained models (used for global feature retrieval). We denote this combination methods, in Table by the name of the pre- pre-trained model + PDM. We follow [34] and apply re-ranking the top 400 global features with the highest scores from the pre-trained model.
Evaluation Protocol. Following [23, 34], we report the mean average precision (mAP) for all methods. all experiments, we used code and parameters provided by the authors of the compared methods.

---

Results. Table 2 presents the Mean Average Precision (mAP) across all benchmarks, highlighting the retrieval performance of PDM. Our method consistently outperforms all self-supervised and weakly supervised foundation methods and achieves comparable results to supervised methods. Notably, surpasses DINOv2 [23] on the ROxford-hard dataset by +4.3% and by +-33% on the PerMIR benchmark. Additionally, using PDM for reranking, we achieve better performance than SoTA- supervised methods, on the RParis and ROxford benchmarks. The results on the PerMIR benchmark underscore the inherent challenges faced by current methods in handling multi-instance samples. In contrast, our method demonstrates the robustness and effectively retrieves the correct samples, highlighting the efficacy of features derived from pre-trained diffusion models for instance- based retrieval tasks. Figure 5(b) provides qualitative retrieval results showing that our model successfully identifies images containing the same instance. while other methods primarily capture semantic similarity. See Appendix F for additional qualitative results.
Summary and Limitation
In this paper, we introduce zero-shot approach for utilizing pre-trained Stable Diffusion (SD) features for personalized retrieval and segmentation tasks. We also review existing benchmarks for these tasks and propose a new benchmark to better evaluate performance. Our method showcases SoTA performance in three different personalization tasks. Nevertheless, it requires image inversion for feature extraction and therefore may depend on the success of image reconstruction quality.
References
[1] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or. Cross-image attention for zero-shot appearance transfer, 2024. 2,
[2] Guoyuan An, Yuchi Huo, and Sung Eui Yoon. Hypergraph propagation and community selection for objects retrieval. In NewrlPS, 2021. 3,9
[3] Guoyuan An, Woo Jac Kim, Saclyne Yang, Rong Li, Yuchi Huo, and Sung-Fui Yoon. Towards content- based pixel retrieval in revisited oxford and paris. ICCV, 2023. 9
[4] Ali Athar, Jonathon Luiten, Paul Voigtlaender. Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: A benchmark for unifying object recognition, segmentation and tracking video. In
WACV, 2023. 2, 7, 14
[5] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. Proceddings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[6] Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang. Products- 10k: large- scale product recognition dataset, 2020.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:22/1.011 1.01324, 2022. 2
[8] Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models:
A survey, 2024. 2
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties Self-Supervised Vision Transformers, ICCV, pages 9630-9640. IEEE,
2021. 3
[10] Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicom, fluffy": Personalizing frozen vision languaeee representations, 2022, 3
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR, 2022. 3,
[12] Yingdong Hu, Renhao Wang, Kaifeng Zhang, and Yang Gao. Semantic-aware fineinrained correspondence.
arXiv, 2022. 8,9 9
[13] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, et al. Openclip. If you use this software please cite it as below, 2021.
[14] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything.
arXiv:2304.00 0264402633,,..0.330 2023. 2, 3,
[15] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A eeemory-augmentee self- superviee tracker, 2020. 8,
[16] Jiachen Lei. Qinglong Wang. Peng Cheng. Zhongjie Ba, Zhan Qin, Zhibo Wang, Zhenguang Liu, and Kui Ren. Masked diffusion models are fast distribution learners, 2023.
10

---

[17] Junnan Li. Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping Language-Ingn Image Pre-training with Frozen Image Encoders and Large Language Models. CoRR, abs/2301. 2023. 9
[18] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training, 2022. 9
[19] Wuyang Ii, Xinyu Liu, and Yixuan Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection, In CVPR, 2022. 3
[20] Philipp Lindenherger, Paul-Fdouard Sarlin, and Marc Pollefeys. Lightglue: Local feature matching light speed, 2023. 3
[21] Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai. Junwei Ma, and Satya Krishna
Gorti. Guided similarity separation for image retrieval. In NeurIPS, 2019. 3,9
[22] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-
image pre-training arXiv preprint arXiv:21 122.200 2021. 9
[23] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang-Wact Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. CoRR, abs/2304.07199.
2023. 2, 3, 7, 8,9, 9, 10
[24] Zhihong Pan, Riccardo Gherardi, Xiufeng Xic, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In ICCV, 2023. 4,
[25] James Philbin, Ondrej Chum, Michael Isard. Josef Sivic, and Andrew Zisserman. Object retricval with large vocabularies and fast spatial matching. In CVPR, 2007. 9
[26] James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Lost quantization: Improving particular object retrieval in large scale image databases. In CVPR, 2008. 9
[27] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Horkknng and Luc Van Gool. The 2017 davis challenge on video object segmentation, 2018. 6, 8, 9
[28] Filip Radenovic, Ahmet Iscen. Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In CVPR. IRFF, 2018. 6,
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning
transferable visual models from natural language supervision. In ICMI., 2021. 9
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional
image generation with clip latents. arXiv preprint (rXiiv2204.06123. 2022. 2
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution
image synthesis with latent diffusion models. In CVPR, 2022. 2, 4, 13
[32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. 13
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Scyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic texi-to-
Kamyar
image diffusion models with deep language understanding. NeurlPS, 2022. 2
[34] Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui. André Araujo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In JCCV, 2023. 2, 3.
[35] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021, 4 [36] Luming Tang. Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent
correspondence from image diffusion. In NeurlPS, 2023. 2, 4, 7, 8,
[37] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training- free consistent text-to-image generation, 2024. 3,3
[38] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar González-Franco, Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion. CoRR, abs/2308. 12469, 2023.
3. 7, 8.
[39] Narck Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. CVPR, 2022. 14
[40] Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven mage-to-immag translation. CVPR, 2023. 2, 4
[41] Shuang Wang and Shuqiang Jiang. Instre: A new benchmark for instance-level object retrieval and recognition. ACM Trans. Multimedia Comput. Commun. Appl., 2015. 2
[42] Xinlong Wang, Xiaosong Zhang, Yue Cao. Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context, 2023. 7,8
[43] Xiaohao Xu, Jinglu Wang, Xiang Ming. and Yan Lu. Towards robust video object segmentation with adaptive object calibration. In ACM, 2022. 2
11

---

[44] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-HSuue Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. 2023. 2,4 4
[45] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize Segment Anything Model with One Shot. ICLR, 2024. 2, 3,
9
[46]
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. 2022. 3,9
[47] Sijie Zhu, Linjie Yang. Chen Chen, Mubarak Shah, Xiaohui Shen, Heng Wang. R2former: Unified retrieval and reranking transformer for place recognition. In CVPR, 2023. 2
[48] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li. Linjie Li. Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Lee. Segment everything everywhere once, 2023. 7.
12

---

Appendix
A Perliminaries
Denoising module of text-to-image diffusion model. We start by describing the different layers that compose the denoising module of a Bext-to-Image diffusion model. Latent Diffusion Model [31], applies the diffusion pro- cess in the latent space of a pre-trained image autoencoder, This model adopts U-Net [32] architecture conditioned on the guiding prompt P. The U-Net is composed of sev- eral layers where each consists of three types of blocks: (1) a residual block, (2) a self-attention block, and (3) a cross-attention block as illustrated in Figure 1. At each timestep of the denoising process, the noised latent code fed input to the U-net. The residual block convolves image features, 2, to produce intermediate features ((x)). In the self-attention block, (2+)) projected into "queries" Q. "keys" K and "values" V. For each query vector Ji,j. representing patch, located at the spatial location (t,j) of Q, the self-attention map then given by:
U-Net Layer
Resnet
Attn Blocks
Cross Attn
Block
Blocks
prompt
Fig. S 1: Single block of a U-Net layer ( Stable Diffusion [31].
A(a,) = softmar bi K.T
vd )
(8)
The last block, the cross-attention block, facilitates interaction between the spatial image features extracted from the self-attention block and the token embeddings of the text prompt P. The process is similar to that in the self-attention layer, but here, Q is derived from the spatial features of the previous self attention layer, while IC and are projected from the token embeddings of the prompt.
B Ablation Study
In this section, we ablate key components of our method.
Personalized Retrieval. (1) Object Mask Instead of Class Name: In this scenario, we considered the case where the class name not provided, but an object mask available. We tested this configuration on PerMIS, resulting in mIOU of 45.0% compared to the original 42.3% when using the class name. The bIOU was 89.2% compared to the original 86.8% when using the class name. This shows that using object mask leads to improved segmentation performance, indicating its potential as a valuable alternative when class names are not available. (2) Appearance vs. Semantic Maps: We examined the individual contributions of the Appearance and Semantic maps to the final similarity map. For this experiment, we used each map independently as the final similarity map, ignoring the other. When using only the Appearance Map, we achieved mIOU of 30.2%, compared to 94.9% when using only the Semantic Map. Both results are significantly lower than our original mIOU of 42.3% when using both maps and averaging them. These findings underscore the necessity of integrating both maps to achieve optimal performance in the final similarity map, and eventually in personalized matching.
Personalized Segmentation. (1) Object mask instead of class name: Here we explore our approach when the input image not accompanied by class name but rather by a precise segmentation mask of the personalized object. During inversion, the prompt is set to be an empty string. The segmentation mask is used to distinguish the personalized object's features from the input image instead of cross attention map. We tested this configuration on PerMIR, resulting in mAP of 76.2 compared the original 73.0 when using the class name. This illustrates the strong capabilities of the semantic map obtained using the cross-attention layer. (2) Appearance vs Semantic maps: Here we examine the individual contributions the Appearance and Semantic maps to the final similarity map DD calculated in our method. For this experiment, we use each map independently as the final similarity map SDF, ignoring the other (instead of averaging them, as explained Section 3, (q) When using only the Appearance Map, we achieve mAP of 42.3, compared to 32.9 when using only
13

---

Table S I: Performance comparison across diffusion models. We report segmentation performance (mloU, bloU), feature extraction run time per image, and mean PSNR for the inversion-recoonstrciion quality.
Diffusion Model
PerSeg bloU
PerMIS
Feature Extraction
Mean
mIoU
mIoU
bloU
Run Time (s)
PSNR
SDXL-turbo
95.4
79.8
42.3
86.8
0.5
24.1
SDXL
97.0
80.9
44.8
87.7
5
25,9
SDv2.1
95.9
80.1
43.7
87.1
5
25.8
the Semantic Map. Both results are significantly lower than our original mAP of 73.0 when using both maps and averaging them. These findings underscore the necessity of integrating both maps to achieve optimal performance in the final similarity map S
C PerMIR and PerMIS Statisties
In this section, we describe the statistics of our newly introduced benchmark, Personalized Multi- Instance Retrieval (PerMIR). Following our image extraction process from the BURST dataset (detailed in Section 4), each video results in three different images of the personalized object, with each image containing an average of 3.1 different objects. We randomly select one image to serve as the query, while the other two are labeled as positive instances in the gallery. This process yields total of 150 queries and a gallery comprising 450 images. The object distribution among the 150 query images as follows: 51 persons, 52 cars, 10 animals, food items, and other objects (e.g. cup, drawer, tennis racket, slippers).
Random frame selection was done once during dataset preparation ensure fair comparisons among all methods. We manually inspected the frames for quality and diversity, finding them acceptable and adequate given the BURST [4] dataset's 's quality and video length. We thus further quantified frames quality and diversity. Using the CLIP model, we found an average cosine similarity of 0.17 17 between frames, indicating low similarity (compared 0.31 for adjacent frames) and thus high diversity. For quality, the mean SSIM between dataset frames and random ImageNet subset was 13.2 (compared to 11.8 for ImageNet samples, higher values indicate better quality).
D Performance Across Diffusion Models
To evaluate how Personalized Diffusion Model (PDM) performance and quality vary with different diffusion models, we conducted experiments using three models: SDXL-turbo, SDXL, and SDv2.1. These experiments were performed on two personalized image segmentation datasets: PerSeg and PerMIS. For each diffuusin model, we report segmentation performance in terms mloU and bloU, as well as the feature extraction run time per image and the mean PSNR of the twersion-reconstruutiio process.
Table 1 summarizes the results. It shows that while SDXL and SDv2.1 provide better performance in both mloU and bloU compared to Stroo, their inversion-reconstruudd time significantly longer, these models require more inversion steps. Specifically, the reconstruction time for SDXL and SDv2.1 is 10 times slower than SDXIL-turb. Nevertheless, these models yield higher PSNR values, indicating better wwwersion-reconstriccc
quality.
As indicated by the results, PDM features can be found for other diffusion models like SDXL and SDv2.1, which yield better segmentation performance (higher mloU and bloU values) and improved reconstruction quality (higher PSNR) at the cost of longer inversion times. These findings further confirm the robustness of PDM features across different diffusion models.
This paper focused on bNeed diffusion models because they are currently the most widely-used text-to-image models. We are optimistic that similar features can be found in other diffusion models, for the following reasons. First, recent studies [39] identified structural and appearance features in vision transformer- bassed models. Second, it not hard to find anstance-fcature in several Unet
14

---

Table 2: Comparison of performance between feature averaging and weighted averaging for combining appearance and semantic features on the ROxford Hard and PerMIR datasets.
Method
ROxford-Hard PerMIR
Feature Averaging (in the paper)
53.2
71.2
Weighted Averaging
58.4
76.9
Input Image
OpenCLIP
DINOv2
Ours
"taxi"
"toy"
Fig. 2: Qualitative examples for personalized retrieval: DINOv2 exhibits improved mastance-based characteristics compared to OpenCLIP. However, unlike other methods that attend to the color or texture, our (PDM) method leverage both semantic and appearance cues to successfully identify instances, even under substantial variations.
diffusion models as illustrated above. Thus, we assume that other diffusion models (such as DiT's will also exhibit comparable or better instance features.
E
Combining Appearance and Semantic Features
In the main paper, feature averaging was used to combine appearance and semantic features in order avoid training or hyperparameter tuning on labeled data. We conducted a further analysis using weighted combination of semantic and appearance features, optimized on training set explore more complex fusion methods.
For this, the PerMIR and ROxford-Haee Hard datasets were split into 20% training and 80% test sets, and the weighted fusion parameters were optimized on the training sets. The results are summarized in Table 2. The weighted combination of features led to improvements in performance compared to simple averaging, with gains of 5.2% on the ROxford-Hard dataset and 5.7% on the PerMIR dataset.
These findings suggest that, when a training set is available, weighted fusion can significantly enhance performance. This opens up the potential for further exploration of more sophisticated, learnable fusion methods in future work.
These results highlight the benefits of weighted fusion for combining appearance and semantic features, and future work will investigate more advanced techniques that dynamically adapt to the data.
15

---

Input Image
Target Image
DIFT
"van"
Fig. 3: Qualitative examples for personalized segmentation: Rows 1,2 show cases where existing similar objects in the scene often distract previous features, while our proposed PDM successfully identifies and segments the correct instance. Note the successful segmentation of the small blanket (row 3) and substantially occluded drink (row 4).
Additional Qualititive Results
We provide additional qualitative results for personalized retrieval and personalized segmentation. Figure S3 shows segmentation results on PerMIS and Figure S2 shows top-1 retrieved image of different methods PerMIR.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we clearly mention our contributions both and provide specific paragraph for the contributions, while mentioning the generalization capabilities.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss our assumptions and limitations in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We present and prove the derivation for our NR based inversion.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We mention all the details required to reproduce our results, including models and prree-pprameters. As we use public datasets/benchmark,, all datasets used cited.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification: Not currently. We use public datasets, so the data used available. We are working on formal approval to publicly release the code, upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/ /nipsnips...... cc/
public/ guides Code Subomi ss wesnooooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss, wwtpcccpppccccccccccccccc.ccc.cc wwwpc//ggissseeeeeeee CodeeooSSui Submi ssi onoolo icyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We mention all the necessary information for testing and follow the previous benchmarks. We build upon pre-trained method for our approach.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We follow all previous evaluation protocols for each benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We use standard GPUs
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttes:///nnuuate http://eerrrp.eeeeeeeeeeee cc httppp/lc//tttcccccceeeeeeeeoeee
Answer: [Yes]
Justification: Image retrieval and segmentation were widely studied before. We are not aware of any ethical considerations here
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: shortly discuss this issue in our summary Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks. Our new benchmark proposal is based on an existing, publicly available dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use publicly available code resources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thoode com/ dat asets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We cite all the benchmarks and code repositories used.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "ctS91c1MnK", "reviewer": "Reviewer_8kmZ", "summary": "This paper introduces a novel approach called Personalized Diffusion Features Matching (PDM) for personalized retrieval and segmentation tasks. Most current self-supervised and supervised methods struggle to accurately identify specific instances within a dataset when multiple instances from the same class are present. So the proposed PDM method leverages intermediate features from pre-trained text-to-image diffusion models. PDM combines semantic and appearance cues to accurately locate and segment specific instances within target images.", "questions": "* what is the performance difference if PDM uses SDXL-turbo vs non-turbo? SDXL-turbo was notorious for not able to reproduce good quality details such as small human faces or limbs.\n\n* how does PDM performance vary wrt to the strength of the underlying diffusion model?\n\n* in table 2, the authors show openCLIP+PDM and DINOv2+PDM. what happens if you compute OpenCLIP+DINOv2+PDM?", "limitations": "yes the authors adequately address the limitations.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* its an innovate use of pretrained model (already available foundation models ) in a zero shot setting. This is beneficial for people without a lot of compute and good for environment.\n\n* the proposed method combines both appearance and semantic similarity, which is a well motivated design", "weaknesses": "* the authors used features from stable diffusion, which is text guided image generation model. its not that fair to compare to DINOv2 or SAM since those models were not trained with text supervision.\n\n* the method depends on diffusion inversion, which is quite slower than other segmentation methods. the authors used sdxl-turbo to mitigate the latency but its unclear the impact on quality\n\n* the method seems only applicable to Unet diffusion models. its unclear if it works on DiTs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful feedback. We believe we address all your concerns below. \n\n**Q1: It is not fair to compare PDM to Dinov2 or SAM since those models were not trained with text supervision.**  \n**A1:** We thank the reviewer for the comment. The paper compared two types of baselines: (1) Baselines trained ***with*** text supervision (CLIP and OpenCLIP; See Table 2, Figure 5b and Figure S2) and (2) baselines without supervision, to be consistent with the literature [1,2,3]. \nFollowing this comment, we add comparisons with more methods trained with text supervision. Specifically, we evaluated personalized retrieval with features from  BLIP2 [4], GLIP [5], and SLIP [6], on the PerMIR dataset. The table below shows that PDM outperforms all text-image models by a large margin. We will add this experiment and additional details in our revised manuscript/suppl. \n| Model                  | mAP  |\n|------------------------|------|\n| CLIP (in the paper)         | 20.9 |\n| OpenCLIP (in the paper)| 26.7 |\n| GLIP [5]                         | 31.2 |\n| BLIP [4]                         | 33.3 |\n| SLIP [6]                         | 35.9 |\n| **PDM (ours)**             | **73.0** |\n\n\n**Q2: How does PDM performance and quality vary with the underlying diffusion model?**  \n**A2:** Thank you for the insightful question. Indeed, it is not clear a-priori that similar features with comparable quality can be found in other diffusion models. To answer this question, we repeated the feature finding process in two diffusion models: SDXL, and SDv2.1. \nWe then ran new experiments for the Personalized Image Segmentation task using two datasets, PerSeg [7] and PerMIS. The table below reports segmentation performance using three diffusion models: (1) SDXL-turbo (results taken from the paper), (2) SDXL, and (3) SDv2.1. We also report the run time for feature extraction per image and the mean PSNR of images to measure inversion-reconstruction quality. \n\nThe results show that one can find PDM features for other diffusion models (SDv2.1 and SDXL) that produce better results than SDXL-turbo in terms of PSNR, mIoU and bIoU. Their inversion reconstruction time is 10x slower because they require more inversion steps. We will report and discuss these results in the final version. \n\n\n| | PerSeg | | PerMIS | | | |\n|---|---|---|---|---|---|---|\n| | mIoU | bIoU | mIoU | bIoU | feature extraction run time / image (sec) | mean PSNR |\n| SDXL-turbo (in the paper) | 95.4 | 79.8 | 42.3 | 86.8 | 0.5| 24.1 |\n| SDXL | 97.0 | 80.9 | 44.8 | 87.7 | 5| 25.9 |\n| SDV2.1 | 95.9 | 80.1 | 43.7 | 87.1 | 5| 25.8 |\n\n\n\n**Q3: The method seems only applicable to Unet diffusion models.**  \n**A3:** The paper focused on UNet-based diffusion models because they are currently the most widely-used text-to-image models.  We are optimistic that similar features can be found in other diffusion models, for the following reasons. First, recent studies [8] identified structural and appearance features in vision transformer-based models. Second, it was not hard to find instance-features in several Unet diffusion models. Thus, we assume that other diffusion models (such as DiTs) will also exhibit comparable or better instance features.\nWe appreciate the reviewer's insightful question, as it highlights a promising avenue for future research.\n\n\n**Q4: What happens if you compute OpenCLIP+DINOv2+PDM?**  \n**A4:** Following this question, we tested a combination of DINOv2 and PDM. Naturally, there are many ways to combine several features, and here we followed the protocol from [9,10]. We first used OpenCLIP to select the 400 most similar images to each query image. Then, we selected 100 images with highest DINOv2 scores, and then ranked them using PDM. We tested the combined method for personalized retrieval in two datasets: PerMIR and ROxford-Hard. \nIn PerMIR, this method achieved a mAP of 70.2%, compared to 69.9% for OpenCLIP + PDM and 70.8% for Dinov2 + PDM. In ROxford-Hard, it achieved a mAP of 58.3%, compared to 57.7% for OpenCLIP + PDM and 62.1% for Dinov2 + PDM. This approach shows a slight improvement over OpenCLIP + PDM but underperforms compared to Dinov2 + PDM, likely due to OpenCLIP's less effective retrieval capabilities.\n\n\n[1] Tang et al. (2023) “Emergent Correspondence from Image Diffusion”.  \n[2] Lue et al. (2023) “Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence”.  \n[3] Zhang et al. (2023) “A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence”.  \n[4] Li et al. (2023) “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”.  \n[5] Li et al. (2022) “Grounded Language-Image Pre-training”.  \n[6] Mu et al. (2022) “SLIP: Self-supervision meets Language-Image Pre-training”.  \n[7] Zhang et al. (2023) “Personalize Segment Anything Model with One Shot”.  \n[8] Tumanyan et al. (2022) “Splicing ViT Features for Semantic Appearance Transfer”.  \n[9] Shao et al. (2023) “Global Features are All You Need for Image Retrieval and Reranking”  \n[10] Zhu et al. (2023) “R2Former: Unified Retrieval and Reranking Transformer for Place Recognition”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fLBlISVtk", "reviewer": "Reviewer_JGDB", "summary": "This paper explores the use of text-to-image diffusion models for personalized retrieval and segmentation tasks. The authors introduce a novel method called PDM (Personalized Features Diffusion Matching), which leverages intermediate features from pre-trained text-to-image models for personalization tasks without requiring additional training. PDM demonstrates superior performance across multiple benchmarks, surpassing many supervised methods. Additionally, the paper identifies deficiencies in existing datasets and introduces a new benchmark dataset that effectively addresses the challenges presented by traditional datasets in scenarios involving multiple instances of the same class. This new dataset focuses on complex scenes containing multiple instances of the same class, further validating PDM's efficiency and accuracy in precisely handling and locating multiple similar instances.", "questions": "1.Given your mention that the method's performance might depend on the success of image reconstruction quality, I am curious about how you have managed to enhance image reconstruction quality to achieve such impressive results with your approach. \n2.Have you considered more complex ways of combining appearance and semantic features? If so, why did these approaches not yield better results, and what are your thoughts on this?\n3.Why opt for random selection of video frames without addressing potential quality issues in the generated data? How do you manage this randomness to ensure fairness and consistency across different methods tested on this data?", "limitations": "The authors briefly discuss the limitations of their method in the conclusion but do not provide any viable future solutions. Could the authors possibly offer some feasible ideas or directions for addressing these limitations in future work?", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper introduces a novel challenge in personalized retrieval and segmentation tasks involving complex scenes with multiple instances of the same class. In response to this scenario, the authors have constructed a new benchmark dataset and devised a novel method—PDM (Personalized Features Diffusion Matching), which ingeniously combines appearance and semantic features to address the issue of multiple instances within the same category. This approach is quite innovative.\n\n2.The overall structure of the paper is logically organized and easy to understand, and the figures greatly assist in comprehension. Overall, the paper is excellently written.", "weaknesses": "1.The authors simply take the average of appearance and semantic features as the overall diffusion features without considering a weighted fusion of these two features. This approach may lead to errors in segmentation or retrieval on some datasets, as it fails to balance the contribution between different features effectively.\n\n2.There may be some issues with the process of generating the personalized multi-instance retrieval and segmentation dataset, such as the randomness in selecting video frames. While this approach improves the efficiency of data generation, it could mislead model training and evaluation if the chosen frames lack sufficient feature diversity or have quality issues.\n\n3.Although the paper conducts extensive experiments in personalized retrieval and segmentation tasks and briefly supplements with ablation study results of key components in the appendix, and qualitatively analyzes the differences from other methods, ensuring comprehensive experimentation, there are still shortcomings. The ablation experiments are only conducted for multi-instance retrieval, which does not prove the importance of key components in multi-instance segmentation. It would be beneficial to include experiments addressing this aspect as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We are encouraged that you find our approach innovative, the challenge novel and our paper to be excellently written. We address your comments below.  \n\n**Q1: Have you considered more complex ways of combining appearance and semantic features?**  \n**A1:** We thank the reviewer for their suggestion.  In the paper, we chose feature averaging to avoid training or hyperparameter tuning on labeled data. Following the reviewer's suggestion, we tested a weighted combination of semantic and appearance features: $wF_{appearance} + (1-w)F_{semantic}$.  We split the PerMIR and ROxford-Hard [1] datasets into training (20%) and test sets (80%), and optimized $w$ on the training sets. The table below shows that when a training set is available, weighted fusion improves results. We also plan to explore learnable fusion methods with advanced techniques in future work. We will include this analysis in the final version.\n\n\n|               | ROxford-Hard | PerMIR |\n|---------------|--------------|--------|\n| Avg (in the paper)           | 53.2         | 71.2   |\n| Wegithed Avg  | **58.4**         | **76.9**   |\n\n\n**Q2: In the proposed multi-instance benchmarks, selected frames may lack feature diversity or suffer from quality issues. How do you ensure fairness and consistency across methods despite randomness?**  \n**A2:** We appreciate the reviewer’s comment. Random frame selection was done once during dataset preparation to ensure fair comparisons among all methods. We manually inspected the frames for quality and diversity, finding them acceptable and adequate given the BURST [2] dataset’s quality and video length. In response to this comment, we further quantified frames quality and diversity. Using the CLIP model, we found an average cosine similarity of 0.17 between frames, indicating low similarity (compared to 0.31 for adjacent frames) and thus high diversity. For quality, the mean SSIM between dataset frames and a random ImageNet subset was 13.2  (compared to 11.8  for ImageNet samples, higher values indicate better quality). We will clarify and include this analysis in the final version.\n\n**Q3: Missing ablation study for Personalized Segmentation.**  \n**A3:** Following this comment, we now added the following ablation study for personalized segmentation:  \n*(1) Object Mask Instead of Class Name:* In this scenario, we considered the case where the class name is not provided, but an object mask is available. We tested this configuration on PerMIS, resulting in a mIOU of 45.0% compared to the original 42.3% when using the class name. The bIOU was 89.2% compared to the original 86.8% when using the class name. This shows that using an object mask leads to improved segmentation performance, indicating its potential as a valuable alternative when class names are not available.  \n\n*(2) Appearance vs. Semantic Maps:* We examined the individual contributions of the Appearance and Semantic maps to the final similarity map. For this experiment, we used each map independently as the final similarity map, ignoring the other. When using only the Appearance Map, we achieved a mIOU of 30.2%, compared to 24.9% when using only the Semantic Map. Both results are significantly lower than our original mIOU of 42.3% when using both maps and averaging them. These findings underscore the necessity of integrating both maps to achieve optimal performance in the final similarity map, and eventually in personalized matching.  \nWe thank the reviewer for this feedback. We will make sure to add this ablation study to the final version.\n\n**Q4: How did you manage to enhance image reconstruction quality?**  \n**A4:** We simply build on recent advances in inversion with diffusion models [3, 4]. For SDXL-turbo, using [3] for inversion results in a PSNR of 24.1, close to the upper bound of 26.3 set by the diffusion VAE. This high PSNR indicates good reconstruction quality and hence high-quality features.  \n\n**Q5: Could the authors suggest ways to address the limitations in future work?**  \n**A5:** Image inversion is the primary limitation of our method, and thus it falls into a different line of work. We did not focus on developing new inversion techniques within the scope of this study. Note that our method is agnostic to the inversion approach, allowing us to adopt faster and more accurate methods developed by the community in the future.  \n\n[1] Radenovic et al. (2018) “Revisiting oxford and paris: Large-scale image retrieval benchmarking”.   \n[2] Athar et al. (2023) “Burst: A benchmark for unifying object recognition, segmentation and tracking in video”.  \n[3] Pan et al. (2023) “Effective real image editing with accelerated iterative diffusion inversion”.  \n[4] Garibi et al. (2024) “ReNoise: Real Image Inversion Through Iterative Noising”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UG8UUzrTYp", "reviewer": "Reviewer_CWZR", "summary": "The paper introduces Personalized Diffusion Features Matching (PDM), a novel zero-shot approach that utilizes pre-trained text-to-image diffusion models for personalized image retrieval and segmentation without requiring additional training. PDM extracts and fuses semantic and appearance features from an off-the-shelled diffusion model to accurately identify and segment unique instances, even when multiple similar objects are present. It demonstrates superior performance on various benchmarks, outperforming both self-supervised and supervised methods. The authors also address limitations in current datasets by proposing new benchmarks that include challenging scenarios with multiple instances from the same category. Despite its reliance on image inversion, which may affect performance, PDM offers a significant advancement in the field of personalized instance retrieval and segmentation.", "questions": "1. It is suggested that some works related to semantic-level feature matching and their applications [1,2] be added to the related work section.\n\n2. (Just constructive suggestions for future works). I am curious whether a similar or more profound phenomenon about attention will appear in DiT [3] and stable video diffusion [4]. Can the method be extended to these more advanced architectures [3,4]?\n\n\n[1] Lindenberger, P., Sarlin, P. E., & Pollefeys, M. (2023). Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 17627-17638).\n\n[2] Li, W., Liu, X., & Yuan, Y. (2022). Sigma: Semantic-complete graph matching for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5291-5300).\n\n[3] Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205).\n\n[4] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., ... & Rombach, R. (2023). Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized, clearly written, and easy to follow, especially for the motivation clarification.\n2. The experimental observation and justification of the effect of self-attention and cross-attention in the diffusion model are insightful.\n3. The proposed method is technically sound and well-motivated.\n4. The contribution is sound, consisting of new observations, methods, and more practical benchmark settings.\n5. The proposed method is training-free and seems to have great transferability.", "weaknesses": "I didn't find a major weakness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach insightful, technically sound with great transferability and our paper to be well motivated and easy to follow. We kindly address your comments below.\n\n**Q1: Cite relevant work.**  \n**A1:** Thank you for your feedback. We will make sure to cite these papers in the final version.   \n\n**Q2: Can the method be extended to DiT and stable video diffusion?**  \n**A2:** We thank the reviewer for their suggestion. We are optimistic that PDM can be applied to DiT and video diffusion models. First, recent studies [1] identified structural and appearance features in vision transformer-based models. Second, it was not hard to find instance-features in several Unet models, SDXL and SDv2.1. Thus, we assume that other diffusion models will also exhibit comparable instance features. We appreciate the reviewer's insightful question, and we leave this for future work.    \n\n[1] Tumanyan et al. (2022) “Splicing ViT Features for Semantic Appearance Transfer”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xqtz6gHgro", "reviewer": "Reviewer_Axpb", "summary": "The authors propose a novel approach without any additional training for Personal retrieval and segmentation tasks, which focus on identifying specific instances within a dataset. The existing methods struggle to locate a desired instance when other instances within the same class are presented. They propose to utilize the features from a pre-trained text-to-image diffusion model and design a Personal Features Diffusion Matching method. The proposed method outperforms on both existing datasets and the new dataset which is proposed in this paper.", "questions": "Please refer to the *Weakness* part.\nAdditional comments:\n1. The authors need to clearly define the form of the task and formally present the settings for differernt scenarios.\n2. The Figure 3 is hard to understand. Why \"dog\" is not fed into the diffusion features extraction in I_t branch? As F^s_t is features from the cross attention, what information is used to calculate the cross attention with I_t?\n3. The authors should explain why using the features from diffusion models and provide theoretical justification. They should figure out the differences between the diffusion models and the other models (e.g. common resnet, ViT, Vision Mamba,  Transformer+KAN, ..., and the other variants, why not choose other SoTA foundation models). Why they can present the textural and semantic information?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The idea of introducing the features from the text-to-image diffusion models to handle the personal retrieval and segmentation problem is interesting.\n+ The authors propose a new approach called PDM which requires no training or fine-tuning, no prompt optimization, or any additional models.\n+ The authors explore the hidden textural and semantic information in the layers and blocks in the diffusion model.\n+ The authors construct a new benchmark includes videos with multiple instances from the same category.", "weaknesses": "- The logical flow before the Method Section of this paper needs to be adjusted. It is hard to understand what the authors want to do and the problem they want to address, although the utilization of features from diffustion models and the PDM itself is somewhat clear.\n- The form of the task in the this paper is not clear. \n\t\"... a dataset based on an input image and a short description of the reference instance.\" in Line.1-2\n\t\"this work focuses on the case where only class names are provided.\" Line.102-103\n\t... Line.103-105\n\tAs observed in the figures, the input information includes reference images, target images, and the class name (some of them are optional)\n- I think the SUPERVISED models for personal retrieval and segmentation, which need a large amount of annotated data, and the SELF-/WEAKLY-SUPERVISED foundation models are not in the same aspect. The authors should rearrange the related works and the motivation of their work.\n- The authors should list their contributions to help the readers better understand thier work.\n- The visualized results in both the main text and appendix do not fully capture the problem the authors aim to address.  Few of them present the situation where a desired instance appears together with other instances of the same class.\n- There is a lack of sufficient theoretical justification for why the authors choose diffusion models and why the features of diffusion models can achieve the capabilities that the authors expect. Only using \"These models have the capability to generate an infinite array of objects and instances, each exhibiting unique appearances and structures. Line.33-34\" and the two examples in Figure 2 is not enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach novel and the ides of our method interesting. We address your comments below.\n\n**Q1: The logical flow before the method section needs to be adjusted. The authors should rearrange the related works and the motivation of their work. The authors should list their contributions.**  \n**A1:** We appreciate the reviewer's valuable feedback and their contribution to improving our paper. We will make sure to rearrange the related works and the motivation of our work to improve the logical flow leading up to the method section. We will make a clear distinction between supervised and self/weakly supervised foundation models in the text and tables. We will also explicitly list our contributions to highlight the key aspects and innovations of our work.  \n\n**Q2: It is hard to understand what the authors want to do and the problem they want to address.**  \n**A2:** This paper addresses Personalized Retrieval and Personalized Segmentation, where a user provides a reference image and either a mask or the object's class name  (lines 1-2, 101-102 in the paper). Personalized retrieval aims to find database images with the exact object from the reference image, while personalized segmentation seeks to segment the specified object in new images and videos. We demonstrate that features from pre-trained diffusion models achieve high zero-shot performance on these tasks without training or fine-tuning (please refer to Section 3.1 for details). Lastly, we further point to major drawbacks with current personalized retrieval and segmentation benchmarks and offer two new alternative benchmarks. We show that our approach (PDM) suppresses all supervised and self-supervised methods on all benchmarks. We will make sure that these issues are better emphasized in the revised version. \n\n**Q3: What is the “short description” mentioned in the paper?**  \n**A3:** The “short description” is  the “class name” (as we use in lines 1-2, 101-102, 124, 157). We will make sure to clarify this in the final version.\n\n**Q4: Visualized results do not present the situation where a desired instance appears together with other instances of the same class.**  \n**A4:** Due to space constraints, we cropped the images in the figures to focus on the relevant objects among other similar objects. We will include more examples without cropping and enlarge the images in the final version.\n\n**Q5: Why using diffusion features and not other models?**    \n**A5:** The fundamental difference between diffusion models and the other models listed by the reviewer is that they are generative, in the sense that they focus on p(image|text). The diffusion training objective (i.e., coarse-to-fine reconstruction loss) requires the model to *generate* informative features for every object. This is different from other, non-generative models, some of which use image-level contrastive learning objectives (e.g. CLIP), and others use image-level discriminative objectives (Resnet, ViT).  Such image-level discriminative or contrastive objectives often remove information about specific instances and objects [4]. Furthermore, current diffusion models contain a self-attention component which controls how some parts of the image depend on other parts. This allows the model to generate coherent objects and makes the model contain an explicit representation of object instances.  For additional details, please refer to recent papers on diffusion features [1,2,3]. We will clarify and add this discussion to the final version. \n\n**Q6: Figure 3 is hard to understand. Why \"dog\" is not fed into the target branch?**    \n**A6:** We thank the reviewer for their feedback. We will make sure to simplify the figure in the final version. This figure illustrates the flow of our algorithm and aligns with the algorithm described in Section 3. Colors are used to differentiate between \"Appearance\" (green) and \"Semantic\" (yellow) features. In this figure, a reference image of a dog labeled \"dog\" and a target image are provided. Both images and the class name undergo feature extraction (see L155-156).  The reference and target branches follow the same process, making it irrelevant which branch receives the class name.\n\n**Q7: What information is used to calculate the cross attention with the target image?**   \n**A7:** Semantic similarity is calculated as the cross attention between class name token $C$ and the semantic feature map $F^{S}_{t}$ as detailed in lines 170-172 and Eq. 6.\n\n\n[1] Tang et al. (2023) “Emergent Correspondence from Image Diffusion”.  \n[2] Lue et al. (2023) “Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence”.  \n[3] Zhang et al. (2023) “A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and ACs,  
We were happy to see that reviewers found our approach **“novel”**, **“innovative”** (**All**), **“well-motivated"** (**R1, R3**) and recognized its potential as a **“significant advancement in the field of personalized instance retrieval and segmentation”** (**R3**). Additionally, they acknowledged our tasks and benchmarks as **“novel”** (**R1,R3**), **“practical”**, and **“challenging”** (**R1, R2, R3**), and found our method to be **“insightful”**, **“easy-to-follow”** and **“excellently written”**, with **“superior performance”** (**R2, R3**) compared to current SoTA approaches.  
We have addressed the reviewers' concerns in our rebuttal and are open to further discussion.  
Your input has been instrumental in improving our paper.

Meta-review: The reviewers agree on the contribution, but had some differences on whether the limitations in computation time and potential coverage would limit the approach.  This was discussed in the rebuttal process and did not seem problematic enough to reject the paper.  The authors are encouraged to clarify the extent of these limitations and other reviewer comments.
Decision: Accept (poster)