Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval

Authors: dvir  samuel
rami  ben-ari
matan  levy
nir  darshan
gal  chechik
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 16:50:27

Abstract: Personalized retrieval and segmentation aim to locate specific instances within a dataset based on an input image and a short description of the reference instance. While supervised methods are effective, they require extensive labeled data for training. Recently, self-supervised foundation models have been introduced to these tasks showing comparable results to supervised methods. However, a significant flaw in these models is evident: they struggle to locate a desired instance when other instances within the same class are presented. In this paper, we explore text-to-image diffusion models for these tasks. Specifically, we propose a novel approach called PDM for Personalized Diffusion Features Matching, that leverages intermediate features of pre-trained text-to-image models for personalization tasks without any additional training. PDM demonstrates superior performance on popular retrieval and segmentation benchmarks, outperforming even supervised methods. We also highlight notable shortcomings in current instance and segmentation datasets and propose new benchmarks for these tasks.

Main paper: 
 introduction: 
 Personalized retrieval and segmentation focus on identifying specific instances within a dataset. When provided with an input image featuring a particular instance (such as your beloved cat) and a brief description ("A cat"), the objective is to locate and segment this exact instance throughout a large collection of images. Personalized methods are useful in various applications, including instance search [34] , product identification [6, 41] , and landmark recognition [47] . Furthermore, personalized segmentation can be applied to video tracking [45] , automatic labeling [43] , and image editing [5, 8] .While supervised methods can be effective for these tasks, they require an extensive amount of labeled training data. Recently, a self-supervised foundation model was proposed [45] to address this task. This model uses the SAM encoder [14] or DINOv2 [23] foundation model to extract spatial features from a given reference instance. These features are then used to localize the object instance in the target image. While effective when a single instance appear in the target image, both DINOv2 and SAM fall short when multiple instances within the same object class are presented in the image. This is illustrated in Figure 1 showing failure cases of DinoV2 and SAM in localizing the correct dog or van (see first and second row). They also fail when two similar objects from different semantic classes are presented (wrongly segmenting the dog instead of the cat.)In this paper, we propose to explore text-to-image diffusion models for these tasks. Text-to-image foundation models have achieved remarkable success in generating new and unique images from text prompts [7, 30, 31, 33] . These models have the capability to generate an infinite array of objects and instances, each exhibiting unique appearances and structures. Consequently, it is reasonable to hypothesize that properties of generated objects are encoded within the intermediate features of the diffusion model during generation. Recent studies [1, 37, 40] show zero-shot capabilities to create subtle changes in generated instances by manipulating the intermediate activation of the diffusion layers, during generation. Although effective, using text-to-image diffusion models "out of the box" for instance-related tasks, beyond generation or editing, remains unexplored.In this paper, we present a new approach, called PDM, Personalized Diffusion Features Matching, for personalized retrieval and segmentation. PDM requires no training or fine-tuning, no prompt optimization, or any additional models. We demonstrate how a specific layer and block contain hidden textural and semantic information. These features are then used for the localization of a reference instance within a given target image, enabling both personalized segmentation and retrieval. PDM builds upon these newly discovered diffusion features, and surpasses other self-supervised methods (like DINOv2 [23] , SAM [14] and DIFT [36] ) weakly supervised methods (CLIP, OpenCLIP) and even supervised methods on personalized instance retrieval and segmentation tasks.We also address significant limitations in traditional benchmarks for retrieval and segmentation. Current benchmarks often feature images with a single, distinct object or multiple objects from different categories, allowing semantic-based methods to achieve high accuracy. To overcome these deficiencies, we construct new benchmarks based on a newly published video tracking and segmentation dataset [4] . This dataset includes videos with multiple instances from the same category (e.g. two dogs playing or a group of people talking). Our method significantly outperforms all baselines on this new dataset, highlighting its ability to accurately handle multiple similar instances and demonstrating its superior capability in personalized retrieval and segmentation. 
related work: 
 Exploring pre-trained diffusion features. Text-to-image diffusion models [7, 30, 31, 33] have demonstrated state-of-the-art performance for image generation tasks. With its superior generation ability, recent studies started investigating the internal representation of diffusion models. DIFT [36] and Fuse [44] showed that extracting features from the ResNet layers of the denoising module provides a semantic correspondence between two objects which can also be used for image editing propagation. Plug-and-Play [40] suggested to extract features from self-attention layers of a reference image, during the image generation process, while incorporated with a text prompt. This approach showcased that output images can retain the structure of the reference image while embodying the appearance described in the text prompt. Cross-Image-Attention [1] further showed that sublayers in self-attention layers correspond to the structure and the appearance of generated images. Their findings enabled the generation of images that blend the structure from one image with the appearance from another. ConsiStory [37] recently suggested injecting the self-attention features of an instance from a pre-generated image into the generation process of other images to ensure consistent reproduction of the same instance across images. DiffSeg [38] introduced a method using self-attention maps for zero-shot image segmentation. They aggregate attention maps from multiple self-attention layers during image generation and merge them iteratively to produce a stack of object proposals. Segmentation maps are then obtained by applying Non-Maximum Suppression over the merged maps. In contrast to these studies, in this paper, we explore using internal features of pre-trained diffusion models for instance related tasks.Personalized Segmentation: PerSAM [45] introduced the use of SAM [45] for personalized image segmentation. They employed the SAM [14] encoder (or DINOv2 [23] ) for the representation of the reference and target images, which are then used to calculate a confidence map localizing the user's reference instance in the target image. Finally, it predicts positive and negative points on the target image to be used as prompts for SAM. Additionally, they proposed a new benchmark, called PerSeg, for personalized image segmentation. It includes 40 objects across various categories, each associated with 5-7 images, and is evaluated using mIoU and bIoU metrics.Instance Retrieval: Content-based instance retrieval can be seen as a variant of personalized retrieval where images contain only a single instance. Recent supervised methods, GSS [21] and HP [2] proposed Graph Networks for effective retrieval. SuperGlobal [34] proposed a memory-efficient image retrieval method, that specifically focuses on the global feature extraction while in the reranking stage, they update the global features of the query and top-ranked images by only considering feature refinement with a small set of images, thus being very efficient. Recently, also self-supervised models [9, 11, 23, 46] show comparable performance to supervised methods on retrieval tasks. These techniques achieve impressive results in zero-shot scenarios however, they often necessitate model fine-tuning to achieve optimal performance. In this study, we investigate text-to-image diffusion models, which belong to the category of self-supervised models, for zero-shot personalized retrieval and segmentation tasks. Our findings show that diffusion features supress features from other self-supervised foundation models.Semantic-level Feature Matching. Recent works have focused on improving semantic-level feature matching in various tasks. SIGMA [19] introduces semantic-complete graph matching for Domain Adaptive Object Detection, addressing within-class variance through node-to-node matching. Light-Glue [20] enhances local feature matching efficiency with a deep network adaptive to image difficulty, making it ideal for latency-sensitive tasks. In this paper, we propose Personalized Diffusion Features Matching (PDM), which leverages intermediate features from pre-trained text-to-image diffusion models for personalized retrieval and segmentation without additional training.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Yes, we discuss our assumptions and limitations in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we describe our approach to leverage pre-trained diffusion models for personalized retrieval and segmentation. We begin by defining these tasks and then delve into identifying features that encompass both semantic and appearance aspects. Lastly, we demonstrate the application of these features in personalized instance retrieval and segmentation.In personalized retrieval and segmentation, the user supplies a single reference image, and a mask indicating the reference instance [45] or the class name of the instance [10] . This work focuses on the case where only class names are provided. For personalized retrieval, the goal is to retrieve images from a database that contains the exact instance specified in the reference image. In personalized segmentation, the objective is to segment the specified instance in new images and videos. Pre-trained text-to-image models can generate an endless variety of objects and instances, each with unique visual characteristics and structures. Recent methods have demonstrated that specific changes in the activations of self and cross-attention activations of the diffusion layer can influence the appearance of specific instances in the generated image. These methods typically modify all activations across all denoising timestamps to affect the generated image. This indicates that instance features are indeed encoded within these models. One can propose to use all diffusion activations during generation and aggregate them for downstream tasks. However, using all features extracted from diffusion layers is memory-intensive and computationally demanding. It also raises the challenge of merging all these features coherently.We aim to identify a single layer at a unique timestamp where both the semantics and appearance (texture) of a reference instance are encoded. We first briefly explain how we extract features from Stable Diffusion [31] , a pre-trained text-to-image model. The architecture of Stable Diffusion consists of a VAE encoder and a VAE decoder that facilitates the conversion between the pixel and latent spaces, and a denoising U-Net module that operates in the latent space. We refer the reader to Appendix A, for preliminary on the internal structure of the denoising U-Net layer. We first encode input image I into the latent space of a VAE using an encoder to produce a latent code z 0 . Next, we employ a diffusion inversion method [24, 35] , to compute the latent code z t at the time step t with the class name embedding as inputs. We then run denoising step at timestamp t to extract activations (features) from the denoising U-Net.Previous studies [36, 40, 44] observed that outputs of earlier layers from the U-Net decoder capture coarse yet consistent semantic correspondences, while deeper layers capture more low-level details and high-frequency information. Based on these observations, and in contrast to previous work, we conducted a more thorough analysis of features extracted from all blocks of the last U-Net layer, examining their role across different timestamps. Interestingly, we consistently found that appearance features are encoded in the queries (Q SA ) and keys (K SA ) matrices of the self-attention (SA) block. This is illustrated in Figure 2(a) , where we perform Principal Component Analysis (PCA) on features extracted for a pair of images, at various timestamps. It shows that Q SA features of the dog in I 1 are similar (same color and texture) to those of the middle dog and cat in I 2 , indicating that textural features are encoded in these layers (similar results are observed for K SA features).We therefore define appearance features of an image to be the average tensor of Q SA and K SA features with dimensions h × w × d extracted from the self-attention (SA) block, at the last layer L of timestamp t:EQUATIONHere, h and w represent spatial resolutions of features extracted from layer L, while d denotes the feature dimension.Figure 3 : An overview of our Personalized Diffusion Features Matching approach. PDM combines semantic and appearance features for zero-shot personalized retrieval and segmentation. We first extract features from the reference, I r and target I t images. Appearance similarity is determined by dot product of cropped foreground features from the reference feature map, F AM r and the target feature map F A t (Eq. 5) . Semantic similarity is calculated as the product between class name token C and the target semantic feature map F S t to create a Semantic Map (Eq. 6). The final similarity map S DF combines both maps by average pooling. Note, that while the appearance and semantic maps attend on two dogs, their fusion yields a single and correct result.For semantic similarity, [16] observed that cross-attention maps establish the relationship between the textual input prompt and patch/pixel-wise image features, effectively allowing a coarse semantic segmentation map that highlights areas of potential object localization. This is further illustrated in Figure 2 (b), where the cross attention of the word "dog" with both images results in an attention map highlighting the location where dogs can be found. This cross-attention map is calculated by fusion of the spatial feature map and the token embedding, after projection. Therefore, we define the semantic features to be the projected spatial features of the cross-attention (CA) block:F S = Q CA(L) t ∈ R h×w×d .(2)We now describe our method for combining semantic and appearance features to address personalized retrieval and segmentation tasks in a zero-shot manner, without training or fine-tuning. We call our approach PDM for Personalized Diffusion Features Matching. See Figure 3 for illustration.Let F A r , F S r and F A t , F S t denote the appearance and semantic features extracted for the reference image I r and target image I t respectively. Next we define our appearance and semantic similarity functions.Appearance Similarity: We start by localizing objects in the target image that have similar visual features as the reference instance in I r . To this end, we make use of C ∈ R 1×d as the projected token vector of the class name, extracted from the cross-attention block CA(L)(same block as F S ).We first use the cross-attention map between spatial image features and C to obtain a reference mask M r . Specifically:EQUATION)This mask is used to crop relevant appearance features of the instance from the feature map F A r , which will later be used for searching within target images. The masked appearance feature map is thus defined as:EQUATIONI is the indicator function and τ is a threshold, resulting eventually in a binary mask, with n foreground features (discarding zeroed-out tokens). Note that • denotes spatial-wise multiplication. This approach leverages the U-Net's ability to preserve spatial information in its latent codes and features during the diffusion process. Next, we compute a map for the appearance similarity score between the reference and target image by simply applying a dot product between the corresponding masked reference feature map and target feature map, followed by average pooling:EQUATION)where S A ∈ R h×w and F AM r (i) refers to the feature map i in F AM r (i).Semantic Similarity: Here we would like to localize all objects that have the same semantic category as the reference instance. To achieve this, we make use of the semantics encoded in the input class name and calculate a score map between C and F S t . Specifically, we compute:EQUATIONThe overall diffusion feature (DF) score map combining both semantic (conceptual) and appearance (textural) features is thenEQUATION)Using diffusion features for personalized retrieval and segmentation. For personalized retrieval, we rank the target images, using a global score, obtained from the average of S DF , indicating the matching score between a target (candidate) and the reference (query) image. For personalized segmentation, we propose two variations: (1) The score map S DF is upsampled to the size of the target image, using a binary threshold. We then segment all pixels that are above that threshold. (2) Following [45] , we select the point with the highest confidence value in S DF as positive prompt for the position of the target object, and use it to segment the object with SAM [14] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Image retrieval and segmentation were widely studied before. We are not aware of any ethical considerations here Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
evaluation datasets for personalized retrieval and segmentation: 
 For the evaluation of PDM, we adopted traditional instance retrieval and one-shot segmentation benchmarks, where we also used the provided class names. While preparing these benchmarks, we discovered that most existing instance retrieval and one-shot segmentation benchmarks predominantly showcase only a single instance per object class. For instance, widely used instance retrieval benchmarks such as RParis [28] and ROxford [28] , focus on single landmarks in their images, with categories typically representing only one possible instance. Similarly, image and video segmentation benchmarks such as the popular Davis [27] dataset and PerSeg [45] mainly comprise either a single instance or multiple instances from diverse object classes, each exhibiting distinct visual and semantic characteristics. This is illustrated in Figure 4 . These trends make it relatively straightforward for semantic-based methods to accurately retrieve or segment instances, as there are often no hard negative instances (objects from the same category but different instance) within or across images. Consequently, comparing instance-based features with current methods on such benchmarks often yields comparable results, failing to highlight the strengths of instance-based methods.To establish a clear distinction between semantic-based and instance-level methods, we introduce two new benchmarks: Personalized Multi-Instance Retrieval (PerMIR) and Personalized Multi-Instance Segmentation (PerMIS). Our proposed benchmarks are constructed using the recently introduced BURST dataset [4] , which serves for Object Recognition, Segmentation, and Tracking in Video. This dataset contains videos with pixel-precise segmentation masks for all unique object tracks spanning different object classes. As the dataset encompasses both single-instance and multi-instance videos, we focus on videos containing at least one hard negative instance per video. Specifically, we select videos with a minimum of two instances belonging to the same object class. We then filter out frames that do not contain these instances. This filtering process results in 150 videos across 16 object classes, with an average of 3.1 instances per frame. Detailed statistics can be found in Appendix C. Finally, for the personalized instance retrieval (PerMIR), we randomly chose three frames from each video, designating one as the query frame and the remaining two as the database (gallery) frames.For the personalized image segmentation task (PerMIS-image) we randomly pick three frames from every video, assigning one as the query frame and the others for evaluation. Ground-truth masks are used for cropping the instance from query images and are also used for segmentation evaluation. We further evaluate on the task of video label propagation. For this task we use the first frame of a video as the reference image and the subsequent frames for evaluation. We intend to make our generated datasets publicly available for future work. 
experiments: 
 We evaluate PDM across three main tasks: (1) Personalized image and video segmentation, (2) Personalized retrieval and (3) Video label propagation where a single video frame is given with object segmentation and the aim is to propagate labels (masks) across video frames, leveraging the information provided by previous frames. The ablation study can be found in Appendix B Implementation details. The main bottleneck of PDM is the real image inversion process, where the image is converted to its noise latent representation for subsequent feature extraction. Using SoTA inversion technique by [24] with Vanilla StableDiffusion, takes about 5 seconds for each image on a single A100. This is due to the requirement of 50 inversion steps. In order to mitigate this, we integrated [24] into SDXL-turbo, a variant of stable diffusion requiring only 4 inversion steps. This decreases the inversion time to 0.5 seconds per image. Therefore, for all our experiments, features were extracted from SDXL-turbo at the last U-Net layer at the first timestep t = 4. Furthermore, all images were resized to 512 x 512 for proper image inversion. We set τ , the threshold for M r to be 0.7 for all our experiments.Datasets. We conducted experiments across two personalized (one-shot) image segmentation benchmarks. We first evaluate PDM on the PerSeg [45] dataset, which comprises 40 objects spanning diverse categories such as daily necessities, animals, and buildings. Each object is represented by 5-7 images and masks, capturing different poses or scenes. Additionally, we assessed our method's performance on the PerMIS-Image benchmark (Section 4).Baselines. We evaluate our method by contrasting it with different self-supervised foundation models:(1) DINOv2 [23] , (2) PerSAM [45] , (3) DIFT [36] and DiffSeg [38] . Additionally, we benchmark it against SoTA-supervised techniques trained specifically for image segmentation, namely SEEM [48] and SegGPT [42] .Evaluation protocol. Following [23, 36] , we report mIOU and bIOU metrics over all benchmarks. Segmentation with PDM is done by upsampling S DF to image size. Segmentation with DINOv2 and DIFT is done using features as a similarity function. Specifically, nearest neighbors are found between the query features and target gallery features. No training is involved. We additionally report results with SAM integration, as proposed by [45] (see 3). Here, features are utilized to derive a positive point, followed by segmentation using SAM.Results. Table 1a presents the results of our experiments in personalized image segmentation. Our approach, denoted as ours, outperforms supervised methods trained specifically for image segmentation. Additionally, our method achieves superior performance compared to other self-supervised models, including DINOv2 [23] , DIFT [36] , and PerSAM [45] . We also demonstrate a significant improvement in performance by applying PerSAM with our method, called PerSAM(PDM), surpassing both benchmarks by a considerable margin. Figure 5 (a) provides qualitative segmentation results showing that our method reliably identifies the reference instance despite substantial variations in the target image, whereas other methods frequently capture false positives within the same category. Additional qualitative results in Appendix FDatasets. We further conducted experiments across two temporal one-shot image segmentation benchmarks. We conducted evaluations on the DAVIS17 dataset [27] . This dataset comprises 150 video sequences, with object masks provided for all frames. Furthermore, we evaluated our method's performance on the PerMIS-Video benchmark (Section 4).Evaluation protocol. Following [36, 45] , we used the first frame image and the corresponding object masks as the user-provided query data. We also follow them and report region-based similarity J (the Jaccard Index, measuring the overlap between the predicted and ground truth regions), contour-based accuracy F (evaluating the accuracy of the predicted contour compared to the ground truth contour) and J &F as evaluation metrics. Compared methods. We compare our approach with various self-supervised foundation models: (1) DINOv2 [23] , (2) PerSAM [45] and (3) DIFT [36] and DiffSeg [38] . We also compare with SoTA supervised methods that were trained on the task of video segmentation. Namely, MAST [15] and SFC [12] .Results. Table 1b presents the results of our experiments in the video label propagation task. Our method demonstrates competitive performance on the DAVIS [27] dataset and superior results on PerMIS benchmark. Our method achieves a notable balance between J and F , indicating its effectiveness in capturing both region and contour details. Improvement in PerSAM(PDM) demonstrated that our PDM can boost results also for other methods.Datasets. We conduct experiments across various retrieval benchmarks, including both singleinstance and multi-instance datasets. Initially, we assess our model's performance on the widely-used ROxford and RParis datasets [25, 26] with revised annotations [28] . These datasets consist of 4,993 and 6,322 images, each featuring a single instance. Evaluation involves 70 query images per dataset, categorized into Easy, Medium, and Hard tasks based on retrieval complexity, with our focus primarily on the more challenging Medium and Hard tasks. Instance masks are obtained from [3] .We further evaluate our model on the PerMIR benchmark (Section 4).Baselines. We compare our approach with state-of-the-art models, including self-supervised foundation models: MAE [11] , SEER, and DINOv2 [23] ; weakly-supervised foundation models: CLIP [29] and OpenClip [13] ; and fully supervised methods: GSS [21] , HP [2] , and SuperGlobal [34] . Both self-supervised foundation models and weakly supervised foundation models were evaluated without further training or fine-tuning. We showcase results utilizing PDM both independently and as a re-ranking technique built upon various frozen pre-trained models (used for global feature retrieval).We denote this combination of methods, in Table 2 by the name of the pre-trained model + PDM.We follow [34] and apply re-ranking on the top 400 global features with the highest scores from the pre-trained model.Evaluation Protocol. Following [23, 34] , we report the mean average precision (mAP) for all methods. In all experiments, we used code and parameters provided by the authors of the compared methods.Results. Table 2 presents the Mean Average Precision (mAP) across all benchmarks, highlighting the retrieval performance of PDM. Our method consistently outperforms all self-supervised and weakly supervised foundation methods and achieves comparable results to supervised methods. Notably, it surpasses DINOv2 [23] on the ROxford-hard dataset by +4.3% and by +43% on the PerMIR benchmark. Additionally, using PDM for reranking, we achieve better performance than SoTAsupervised methods, on the RParis and ROxford benchmarks. The results on the PerMIR benchmark underscore the inherent challenges faced by current methods in handling multi-instance samples. In contrast, our method demonstrates the robustness and effectively retrieves the correct samples, highlighting the efficacy of features derived from pre-trained diffusion models for instance-based retrieval tasks. Figure 5 (b) provides qualitative retrieval results showing that our model successfully identifies images containing the same instance, while other methods primarily capture semantic similarity. See Appendix F for additional qualitative results. 
summary and limitation: 
 In this paper, we introduce a zero-shot approach for utilizing pre-trained Stable Diffusion (SD) features for personalized retrieval and segmentation tasks. We also review existing benchmarks for these tasks and propose a new benchmark to better evaluate performance. Our method showcases SoTA performance in three different personalization tasks. Nevertheless, it requires image inversion for feature extraction and therefore may depend on the success of image reconstruction quality. When using only the Appearance Map, we achieve a mAP of 42.3, compared to 32.9 when using only the Semantic Map. Both results are significantly lower than our original mAP of 73.0 when using both maps and averaging them. These findings underscore the necessity of integrating both maps to achieve optimal performance in the final similarity map S DF .In this section, we describe the statistics of our newly introduced benchmark, Personalized Multi-Instance Retrieval (PerMIR). Following our image extraction process from the BURST dataset (detailed in Section 4), each video results in three different images of the personalized object, with each image containing an average of 3.1 different objects. We randomly select one image to serve as the query, while the other two are labeled as positive instances in the gallery. This process yields a total of 150 queries and a gallery comprising 450 images. The object distribution among the 150 query images is as follows: 51 persons, 52 cars, 10 animals, 4 food items, and 33 other objects (e.g. cup, drawer, tennis racket, slippers).Random frame selection was done once during dataset preparation to ensure fair comparisons among all methods. We manually inspected the frames for quality and diversity, finding them acceptable and adequate given the BURST [2] dataset's quality and video length. In response to this comment, we further quantified frames quality and diversity. Using the CLIP model, we found an average cosine similarity of 0.17 between frames, indicating low similarity (compared to 0.31 for adjacent frames) and thus high diversity. For quality, the mean SSIM between dataset frames and a random ImageNet subset was 13.2 (compared to 11.8 for ImageNet samples, higher values indicate better quality).To evaluate how Personalized Diffusion Model (PDM) performance and quality vary with different diffusion models, we conducted experiments using three models: SDXL-turbo, SDXL, and SDv2.1. These experiments were performed on two personalized image segmentation datasets: PerSeg and PerMIS. For each diffusion model, we report segmentation performance in terms mIoU and bIoU, as well as the feature extraction run time per image and the mean PSNR of the inversion-reconstruction process.Table 1 summarizes the results. It shows that while SDXL and SDv2.1 provide better performance in both mIoU and bIoU compared to SDXL-turbo, their inversion-reconstruction time is significantly longer, as these models require more inversion steps. Specifically, the reconstruction time for SDXL and SDv2.1 is 10 times slower than SDXL-turbo. Nevertheless, these models yield higher PSNR values, indicating better inversion-reconstruction quality.As indicated by the results, PDM features can be found for other diffusion models like SDXL and SDv2.1, which yield better segmentation performance (higher mIoU and bIoU values) and improved reconstruction quality (higher PSNR) at the cost of longer inversion times. These findings further confirm the robustness of PDM features across different diffusion models, and we will expand on these results in the final version of the paper.This paper focused on UNet-based diffusion models because they are currently the most widely-used text-to-image models. We are optimistic that similar features can be found in other diffusion models, for the following reasons. First, recent studies [39] identified structural and appearance features in vision transformer-based models. Second, it was not hard to find instance-features in several Unet diffusion models as illustrated above above. Thus, we assume that other diffusion models (such as DiTs) will also exhibit comparable or better instance features.In the main paper, feature averaging was used to combine appearance and semantic features in order to avoid training or hyperparameter tuning on labeled data. We conducted a further analysis using a weighted combination of semantic and appearance features, optimized on a training set to explore more complex fusion methods.For this, the PerMIR and ROxford-Hard datasets were split into 20% training and 80% test sets, and the weighted fusion parameters were optimized on the training sets. The results are summarized in Table 2 . The weighted combination of features led to improvements in performance compared to simple averaging, with gains of 5.2% on the ROxford-Hard dataset and 5.7% on the PerMIR dataset.These findings suggest that, when a training set is available, weighted fusion can significantly enhance performance. This opens up the potential for further exploration of more sophisticated, learnable fusion methods in future work.These results highlight the benefits of weighted fusion for combining appearance and semantic features, and future work will investigate more advanced techniques that dynamically adapt to the data. We provide additional qualitative results for personalized retrieval and personalized segmentation.Figure 3 shows segmentation results on PerMIS and Figure 2 shows top-1 retrieved image of different methods on PerMIR.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, we clearly mention our contributions on both and provide a specific paragraph for the contributions, while mentioning the generalization capabilities. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We shortly discuss this issue in our summary Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks. Our new benchmark proposal is based on an existing, publicly available dataset.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use publicly available code resources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 Figure 1 : Personalized segmentation task involves segmenting a specific reference object in a new scene. Our method is capable to accurately identify the specific reference instance in the target image, even when other objects from the same class are present. While other methods capture visually or semantically similar objects, our method can successfully extract the identical instance, by using a new personalized feature map and fusing semantic and appearance cues. Red and green indicate incorrect and correct segmentations respectively.Personalized retrieval and segmentation aim to locate specific instances within a dataset based on an input image and a short description of the reference instance. While supervised methods are effective, they require extensive labeled data for training. Recently, self-supervised foundation models have been introduced to these tasks showing comparable results to supervised methods. However, a significant flaw in these models is evident: they struggle to locate a desired instance when other instances within the same class are presented. In this paper, we explore text-to-image diffusion models for these tasks. Specifically, we propose a novel approach called PDM for Personalized Diffusion Features Matching, that leverages intermediate features of pre-trained text-to-image models for personalization tasks without any additional training. PDM demonstrates superior performance on popular retrieval and segmentation benchmarks, outperforming even super- 
 
 Appendix:  


Decision: The reviewers agree on the contribution, but had some differences on whether the limitations in computation time and potential coverage would limit the approach.  This was discussed in the rebuttal process and did not seem problematic enough to reject the paper.  The authors are encouraged to clarify the extent of these limitations and other reviewer comments.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach called Personalized Diffusion Features Matching (PDM) for personalized retrieval and segmentation tasks. Most current self-supervised and supervised methods struggle to accurately identify specific instances within a dataset when multiple instances from the same class are present. So the proposed PDM method leverages intermediate features from pre-trained text-to-image diffusion models. PDM combines semantic and appearance cues to accurately locate and segment specific instances within target images.
strength_weakness: strength_weakness: * the authors used features from stable diffusion, which is text guided image generation model. its not that fair to compare to DINOv2 or SAM since those models were not trained with text supervision.

* the method depends on diffusion inversion, which is quite slower than other segmentation methods. the authors used sdxl-turbo to mitigate the latency but its unclear the impact on quality

* the method seems only applicable to Unet diffusion models. its unclear if it works on DiTs
questions: questions: * what is the performance difference if PDM uses SDXL-turbo vs non-turbo? SDXL-turbo was notorious for not able to reproduce good quality details such as small human faces or limbs.

* how does PDM performance vary wrt to the strength of the underlying diffusion model?

* in table 2, the authors show openCLIP+PDM and DINOv2+PDM. what happens if you compute OpenCLIP+DINOv2+PDM?
limitations: limitations: yes the authors adequately address the limitations.
paper_summary: paper_summary: This paper explores the use of text-to-image diffusion models for personalized retrieval and segmentation tasks. The authors introduce a novel method called PDM (Personalized Features Diffusion Matching), which leverages intermediate features from pre-trained text-to-image models for personalization tasks without requiring additional training. PDM demonstrates superior performance across multiple benchmarks, surpassing many supervised methods. Additionally, the paper identifies deficiencies in existing datasets and introduces a new benchmark dataset that effectively addresses the challenges presented by traditional datasets in scenarios involving multiple instances of the same class. This new dataset focuses on complex scenes containing multiple instances of the same class, further validating PDM's efficiency and accuracy in precisely handling and locating multiple similar instances.
strength_weakness: strength_weakness: 1.The authors simply take the average of appearance and semantic features as the overall diffusion features without considering a weighted fusion of these two features. This approach may lead to errors in segmentation or retrieval on some datasets, as it fails to balance the contribution between different features effectively.

2.There may be some issues with the process of generating the personalized multi-instance retrieval and segmentation dataset, such as the randomness in selecting video frames. While this approach improves the efficiency of data generation, it could mislead model training and evaluation if the chosen frames lack sufficient feature diversity or have quality issues.

3.Although the paper conducts extensive experiments in personalized retrieval and segmentation tasks and briefly supplements with ablation study results of key components in the appendix, and qualitatively analyzes the differences from other methods, ensuring comprehensive experimentation, there are still shortcomings. The ablation experiments are only conducted for multi-instance retrieval, which does not prove the importance of key components in multi-instance segmentation. It would be beneficial to include experiments addressing this aspect as well.
questions: questions: 1.Given your mention that the method's performance might depend on the success of image reconstruction quality, I am curious about how you have managed to enhance image reconstruction quality to achieve such impressive results with your approach. 
2.Have you considered more complex ways of combining appearance and semantic features? If so, why did these approaches not yield better results, and what are your thoughts on this?
3.Why opt for random selection of video frames without addressing potential quality issues in the generated data? How do you manage this randomness to ensure fairness and consistency across different methods tested on this data?
limitations: limitations: The authors briefly discuss the limitations of their method in the conclusion but do not provide any viable future solutions. Could the authors possibly offer some feasible ideas or directions for addressing these limitations in future work?
paper_summary: paper_summary: The paper introduces Personalized Diffusion Features Matching (PDM), a novel zero-shot approach that utilizes pre-trained text-to-image diffusion models for personalized image retrieval and segmentation without requiring additional training. PDM extracts and fuses semantic and appearance features from an off-the-shelled diffusion model to accurately identify and segment unique instances, even when multiple similar objects are present. It demonstrates superior performance on various benchmarks, outperforming both self-supervised and supervised methods. The authors also address limitations in current datasets by proposing new benchmarks that include challenging scenarios with multiple instances from the same category. Despite its reliance on image inversion, which may affect performance, PDM offers a significant advancement in the field of personalized instance retrieval and segmentation.
strength_weakness: strength_weakness: I didn't find a major weakness
questions: questions: 1. It is suggested that some works related to semantic-level feature matching and their applications [1,2] be added to the related work section.

2. (Just constructive suggestions for future works). I am curious whether a similar or more profound phenomenon about attention will appear in DiT [3] and stable video diffusion [4]. Can the method be extended to these more advanced architectures [3,4]?


[1] Lindenberger, P., Sarlin, P. E., & Pollefeys, M. (2023). Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 17627-17638).

[2] Li, W., Liu, X., & Yuan, Y. (2022). Sigma: Semantic-complete graph matching for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5291-5300).

[3] Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205).

[4] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., ... & Rombach, R. (2023). Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127.
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose a novel approach without any additional training for Personal retrieval and segmentation tasks, which focus on identifying specific instances within a dataset. The existing methods struggle to locate a desired instance when other instances within the same class are presented. They propose to utilize the features from a pre-trained text-to-image diffusion model and design a Personal Features Diffusion Matching method. The proposed method outperforms on both existing datasets and the new dataset which is proposed in this paper.
strength_weakness: strength_weakness: - The logical flow before the Method Section of this paper needs to be adjusted. It is hard to understand what the authors want to do and the problem they want to address, although the utilization of features from diffustion models and the PDM itself is somewhat clear.
- The form of the task in the this paper is not clear. 
	"... a dataset based on an input image and a short description of the reference instance." in Line.1-2
	"this work focuses on the case where only class names are provided." Line.102-103
	... Line.103-105
	As observed in the figures, the input information includes reference images, target images, and the class name (some of them are optional)
- I think the SUPERVISED models for personal retrieval and segmentation, which need a large amount of annotated data, and the SELF-/WEAKLY-SUPERVISED foundation models are not in the same aspect. The authors should rearrange the related works and the motivation of their work.
- The authors should list their contributions to help the readers better understand thier work.
- The visualized results in both the main text and appendix do not fully capture the problem the authors aim to address.  Few of them present the situation where a desired instance appears together with other instances of the same class.
- There is a lack of sufficient theoretical justification for why the authors choose diffusion models and why the features of diffusion models can achieve the capabilities that the authors expect. Only using "These models have the capability to generate an infinite array of objects and instances, each exhibiting unique appearances and structures. Line.33-34" and the two examples in Figure 2 is not enough.
questions: questions: Please refer to the *Weakness* part.
Additional comments:
1. The authors need to clearly define the form of the task and formally present the settings for differernt scenarios.
2. The Figure 3 is hard to understand. Why "dog" is not fed into the diffusion features extraction in I_t branch? As F^s_t is features from the cross attention, what information is used to calculate the cross attention with I_t?
3. The authors should explain why using the features from diffusion models and provide theoretical justification. They should figure out the differences between the diffusion models and the other models (e.g. common resnet, ViT, Vision Mamba,  Transformer+KAN, ..., and the other variants, why not choose other SoTA foundation models). Why they can present the textural and semantic information?
limitations: limitations: NA

Comments: title: Official Comment by Reviewer JGDB
comment: Dear reviewer, Thank you for the positive and insightful feedbacks
Dear reviewer, Thank you for the positive and insightful feedbacks
Thank you for your feedback. 
Regarding the robustness of diffusion features, our experiments show they are effective across a wide range of concepts, outperforming ***both supervised and self-supervised methods on various benchmarks***. What tools would the reviewer suggest to further validate the robustness? 
We believe that as better and faster diffusion models are released, PDM can be applied to achieve more robust features and thus better performance. Introducing PDM will indeed allow further research in this direction.
As for the inference speed, it is indeed a limitation as noted in our paper. We currently achieve feature extraction in half a second, making it applicable to a variety of tasks. As inversion methods improve, our approach will also speed up.
Thank you for the detailed explanation. After reading the authors' response, most of my concerns have been resolved. The motivation and proposed method in this work are interesting. The presentation could be further improved to some extent. However,  after reading the comments from other reviewers, I have the same problems as Reviewer 8kmZ. PDM has a weakness in terms of inference speed and is limited by the robustness of the used diffusion model, especially as generative models struggle to generate certain types of objects effectively.
Thank you for reading our response and for your feedback. We believe that it should have addressed your main concerns. If there are any remaining (or new) issues, we would love to get a chance to address them.
Thanks for your detailed response. I would keep my score by a comprehensive consideration of the paper's contribution and  experimental results.
Thanks for the clarification. I have read all the responses. The proposed method can perform better after fine-tuning, which shows more significant potential in benchmark performance. Considering the new observations, clear motivations, methodology designs, and dataset contributions, I would like to keep my original score because this can be seen as a great contribution to the prior work DIFT that explores diffusion features.
thank you for the detailed rebuttal