Personalized Representation from Personalized Generation
Authors: Shobhita Sundaram
Julia Chae
Yonglong Tian
Sara Beery
Phillip Isola
Venue: ICLR 2025 Poster (2025)
Keywords: Synthetic data
personalization
diffusion models
data augmentation
representation learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We explore whether and how synthetic data can be used to train personalized visual representations.

Abstract: Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.

Published as a conference paper at ICLR 2025
PERSONALIZED REPRESENTATION FROM PERSONAL- IZED GENERATION
Shobhita Sundaram 1+f
Julia Chac
Yonglong Tian
Sara Beery
Phillip Isola
MIT
Pretrained Representation Space
Personalized Representation Space
Personal Instance Data
Train
Personalized Generated Data
Other Data (Generated)
Figure 1: Learning personalized representations from limited real data. In this paper explore whether and how synthetic data can be used to train a personalized representation. Given a few real images of an instance, generate novel images and contrastively fine-tune a general-purpon pretrained model to learn personalized representation, useful for diverse downstream tasks.
ABSTRACT
Modern vision models excel general purpose downstream tasks. It is unclear, however, how they may used for personalized vision tasks, which are both fine- grained and data-scarce. Recent works have successfully applied synthetic data general-purpose representation learning, while advances in Text-to-Image (T2I) diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore potential connection between these idcas, and formalize the challenge of using personalized synthetic data learn person alized representationss which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose contrastive learning approach that makes creative use of image genera- tors. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition segmentation, and analyze charac- teristics of image generation approaches that are key to this gain. Our project page:
rep. github.
INTRODUCTION
Representation learning in computer vision seeks to learn general-purpose purpose encodings for objects or semantic concepts that may be flexibly applied to downstream tasks such as recognition and semantic segmentation. recent years we have seen a surge of interest in personalized vision where user
'Equal contribution.
Work partially done student researcher Google.
I Work done at Google. Co-supervised, order by coin flip.

---

Published as a conference paper at ICLR 2025
easily develop customized models for objects of their personal interest, e.g., model capable of detecting their pet dog in personally-collectedd images (Zhang et al.] 2023 Cohen et al. 2022 Nitzan et al. 2022) Among other benefits, personalized systems can keep data private; preferably these models are trained locally, without needing to share user data to a centralized repository, or access other users' data. The personalized setting has two critical challenges. First, it is data-scarce: Curated data collection is time-consuming and expensive; a user would ideally need only provide few examples of their object to obtain a personalized model. Second, it can be extremely fine- grained; e.g., recognizing an individual dog as opposed to recognizing the category "dog".
While modern vision models have proven successful for ceneral-purposs tasks, adapting their rep- resentations to fine- tne-grained problems with scarce labeled data remains challenging (Zhang et al.) 2024 Radford et al., 202T Cohen et al. 2022 Sttevens et al. 2023). As shown in Figure | we con- trast geeggeeeengrrgeee purpose representations with the notion of a personalized representation: a specialized representation space that encodes the knowledge about an instance of interest needed for a variety of downstream personalized tasks. In this paper, we ask: Is it possible to learn a personalized representation from only a few real images of a single instance?
Works such TTia et al. 2023b) have shown that, when intelligently paired with contrastive objec- tives, synthetic data can enable learning strong general-pune purpose visual representations. Other works have investigated personalized generation (Gal et al. 2023 Ruiz al. 2022), but do not extend to representation learning. Our work targets the combination of these ideas: can personalized gener- ation provide effective synthetic data for training personalized representations? We explore what makes for useful generative data augmentation for personalized representation learning and how to best learn from that data. We evaluate our learned representations for four downstream tasks: clas- sification, retrieval, detection, and segmentation, and find that performance universally improves.
In summary, our contributions are the following:
Personalized representations trained with synthetic data, using only three real examples of an instance, significantly outperform pretrained counterparts across datasets, back- bones, and downstream tasks.
We introduce new mechanisms for evaluating personalized representations, including PODS - Personal Object Discrimination Suite - a new dataset of 100 personal objects under specific distribution shifts, and reformulations of existing Instance-level datasets.
Leveraging additional resources can significantly improve personalized representations. While pretrained T2I models are key to achieving the best performance, comparable results can be obtained with fewer computational resources.
Different generators introduce unique biases/imitaaiios that affect representations.
2 RELATED WORKS
Personalized visual generation. Early efforts to personalize generated images attempted edit specific people or styles given user inputs with Generative Adversarial Networks (GANs) (Bau
al. 2019 Roich et al., 2021: Alaluf al. 2021 Dinh al.
2022 Nitzan al., 2022) Recent
efforts focus on Text-to- Image (T2I) diffusion models, usually learning unique identifier for a
target object given few images. Textual Inversion (Gal et al. 2023) freezes a pretrained generative model then learns a unique and personal text token for the object, which can be conditioned on for generation. NeTI (Alaluf et al. 2022)) enhances expressivity and editability by learning different token embeddings for each diffusion timestep and U-Net layer. DreamBooth (Ruiz et al. (022) fine- tunes the entire T2I model to produce more accurate images the target concept. CustomDiffusion (Kumari et al. 2022) instead fine-tunes subset of model weights, and enables joint training over multiple concepts. Follow-up works to these have sought to improve the efficiency and accuracy of personalized generations (Ruiz et al. 2023 Arar al. 2023 Wei al. 2023 Han et al., 2023
Guan et al. 2025), e.g., finetuning-free personalization methods that reduce computational cost (Shi
al. 2024
Chen et al. 2024 Huang al. 2024 Ma al. 2024).
Personalized recognition and representations. Personalized vision involves tailoring vision models user-specific concepts and preferences. PerSAM (Zhang et al. 2023) extends the
2

---

Published as a conference paper at ICLR 2025
Regment-Anything Model (Kirillov al., 2023a to segment user-specified objects with a few
ample images and masks. Porsonalization has also been explored for image captioning (Wang et al.
2023 Chunseong Park et al.] 2017 Park et al. 2018), pose estimation (guyyen et al. 2024b). and
image retrieval via textual inversion: finding a mapping of images to text tokens (Saito et al. 2023
Karthik et al., [023 Baldrati et al., 2023] Cohen et al. 2022: Yeh et al. 2023). Among the textual inversion works, PALAVRA (Cohen et al., (0022) enables personalization for both global and dense
vision tasks but relies on large-scale captioned data for the inversion process. Our approach requires
only few images, without annotations, from the user. Concurrent works have also applied person alization to vision-language models for VQA and object recognition (Nguyen al.) 2024a Alaluf et al. 2024). In contrast, we personalize general-purpose vision backbones using self-supervised framework over generated data, achieving strong performance across both image-level and dense tasks without large-scale data. Recent concurrent work has also proposed PDM, which applies in- termediate features of T2I models to personalized retrieval and segmentation Samuel et al. (2024).
Re-Identification. Personalized recognition is closely related to re-identification, in which model is tasked with recognizing objects (Sun et al.] 2004) or faces (Turk & Pentland| 1991) of the same identity. Early works in Re-ID explored metric learning on hand-crafted features (Ojala eet al., 2002) Gray & Tao 2008 Zhao et al., 2017); later methods learned deep metrics with su-
pervised/unsupervised signals (He et al., 2021 Taigman et al.) 2014 Schroff et al., 2015). Recent
metric learning works use large curated datasets to train on thousands of unique instances of a cer-
tain category (typically humans (Zheng et al., 2015 Yadav & Vishwakarma 2024) or vehicles (Liu et al. 2016; Amiri et al.] 2024)). While our work involves training features with contrastive losses, we focus on personalizing pre-trained features for a single instance with a few images.
Training on synthetic data. Training on synthetic data has been extensively investigated to tackle issues like privacy preservation, data imbalance, and data scarcity (Sakshaug & Raghunathan 2010
Tanaka & Aranha 2019 Khan et al. 2019; Jahanian et al. 2021] Tucker et al.] (020). Diffusion
models (Sohl-Dickstein et al. 2015 Ho et al. 2020) have further unleashed such potential in zero-
shot settings (He et al. 2022b), few-shot settings (He et al. 2022b; Trabucco al. 2023 Lin al.,
2023), out-of-distribution scenarios (Sariyildiz et al. [023] Bansal & Grover [2023] Jung et al.,
2024). 4. and supervised classification (Yeo et al., [024]] Kupyn & Rupprecht] (2024). These works
note the importance of the classifier-free guidance scale (Sariyildiz et al.) [ Tian et al. 2023b) and p selection (Lei et al.] a (023), and propose post-processing filtering (He et al. (2022)) when using off-the-shelf T21 models. Alternatively. (Azizi et al., (202)) and (Yuan et al. 2023) fine-tune diffusion models on ImageNet and show improved classification performance when supplementing real with synthetic data. Similarly, (Zhou et al. 2023 Trabucco et al., 2023) invert training images as conditions for generating new synthetic images. Other studies address data-imbalance (Shin et al.) 2023). domain shifts (Yuan et al. 2022). scaling synthetic data (Fan et al. 2023). and applications to various tasks, including segmentation (Wu et (023), general representation learning
Tian al. 2023b a). and CL.IP training (Hammoud et al. 2024).
METHODS
This paper tackles two questions: how to achieve personalized visual representation by leveraging generative models, and what factors are essential to producing highly effective training data. In Sec- tion 3.1 we formalize the personalized representation task. Our three-stage method then illustrated in Figure 2 We prepare a personalized generator from a few target instance images (Section (32 and produce synthetic personalized data (Section 3.3). We then train a personalized representation on the generated data with a contrastive objective (Section B.4). Lastly, we consider scenarios with additional annotations and data, and how to incorporate them to enhance personalization.
3.1 FORMALIZING THE PERSONALIZED REPRESENTATION CHALLENGE.
We assume access small dataset of real images D of specific object c, and the generic category Cpr of the object. We use generative model (x( to synthesize a novel dataset Dş of images of c and train personalized representation by adapting general purpose vision encoder fa-
We assume we are only provided images of a (which we also denote as an instance) for training our personalized representation. We evaluate on global and local downstream tasks. Note that
3

---

Published as a conference paper at ICLR 2025
evaluate instance performance (e.g., one v. all classification, detection, etc). This differs from many previous works that focus on generating synthetic data for closed-set k-way classification (Shin
et al. 2023 He et al. 2022b).
3.2
PERSONALIZED SYNTHETIC DATA GENERATION
We generate personalized data from Da using Stable Dif- fusion 1.5, T2I model, our generator go. We adapt go using DreamBooth (Ruiz et al. 2022) to generate novel
Personalized Training
Real mages (-33
images of when conditioned on an identifier token.
A T2I diffusion model go generates images given an ini- tial noise latent e (0, 1) and conditioning text em- bedding = (v) where Tw is text encoder, and y is usev-ddovdded prompt. Given ground-truth image and the text embedding Gpr of the generic semantic category Cpr, DreamBooth fine-tunes go using the loss:
g
Ear se (90 + o.c, 0) xx12]
ge
+Xwylllo(aee + ovr''
1p)
where Tpr is an image synthesized with the pre-trained generator conditioned ables og, ot, and wy relate to the noise schedule and sam- pling quality. The first loss term is reconstruction loss on æ, and the second term is a prior preservation loss on Tpr. The two loss terms are weighted by X. Following standard implementations, we also fine-tune r., with the same loss. For further details, refer (Ruiz et al. 2022.
Stage
Stage
Stoge 3
Gpr. is the timestep, and vari-
Figure 2: Personalized Representa- tion Training Pipeline. Our three-stage training method: 1) Generative Model Training 2) Synthetic Data Generation 3) Contrastive LoRA Fine-Tuning.
While there are several alternative methods for personalized generation (Gal et al. 2023 Alalui al. 2023), we focus on DreamBooth, which has been shown maintain highest fideiity fine details (Alaluf et al. 2023).
3.3 CONTROLLING GENERATED DATASET ATTRIBUTES
Prior work has observed that fidelity to the target subject and diversity of generated data are both important factors (Sariyildiz et al., 2023). T21 models offer several mechanisms of injecting diver- sity into generated outputs, allowing us to explore the relationship between these attributes and the quality of learned personalized representations.
Classifier-Free Guidance (CFG). A common way of injecting diversity for diffusion models is modifying the CFG (Ho & Salimans 2022) at inference, which controls how strongly the generation adheres to the conditioning prompt. We experiment with CFG E (4.0, 5.0, 7.5).
LLM-generated captions. As seen in (He al.) 2022b) and (Dunlap et al.) 2023) off-the-shelf
Large Language Models such as T5 (Raffel et al., 2023) can be leveraged to generate text-prompts for each object. Following prior works, ensuring that they introduce rich context descriptions in addition to describing the target object. For example, if the object a shirt, an LLM-generated prompt could be "a shirt on a coat hook". "ook",,e or "a person wearing shirt at a street market", Details in A.4.1
generate image captions with GPT-4 (OpenAl 2023),
3.4 REPRESENTATION LEARNING FROM SYNTHETIC DATA.
Given (DR. Ds) of instance c, we personalize fe via fine-tuning. Critical to representation learning is having both positive and negative examples. We obtain positives from Dz. We generate negatives D& by prompting the pretrained go (Stable Diffusion 1.5) with the generic object category: "a
photo of Cpr
Given (x, x .o.. IN) where x E Dr. x+ E Ds. x; E D3 for = O,...,N, we extract fo features concatenation of the CI.S token and average-pooled final-layer patch-embeddings. We
4

---

Published as a conference paper at ICLR 2025
then finetune fe using the infoNCE loss,
exp(sim(Xo x x+)/+) ))/7)
LIfONCE log
Ei=1 exp(sim(Xp. x,)/T)
This loss pushes together the representations of real and synthetic images of c, and pushes apart rep- resentations of с and other instances. We also experiment with alternate contrastivennon-cntraass losses in the Appendix (C.I) We fine-tune via Low-Rank Adaptation (LoRA), which is more parameter-efficien than full fine-tuning (Hu et al. 2021)
3.5 ALTERNATIVES TO DREAMBOOTH
Given the computational cost of DreamBooth, we also explore alternative datasets:
Real data baseline. A simple baseline is to contrastively fine-tune using only the available real data DR as positives. Here, we still use large pool of negatives.
Comparisons enabled by extra resources. With so few real images, there may be benefit in expending effort upfront to collect further labels and data. A user might annotate their images, download internet-available data, even capture more images of the target object. We describe possible cases below.
Segmentation masks: We consider collecting segmentation masks of DR. This allows for sim- plc, cheap generative model: Cut--nddPaste Here the generator samples independently from fore- grounds containing the target object (carved from DR) and generic backgrounds from T2I model (details in A.4.3). With masks, we can also improve DreamBooth generations by enabling masked DreamBooth training and filtering. Fine-tuning 9o can be affected by signals such as shared back- grounds. To minimize such overfitting,
mask out the gradients for background pixels during DreamBooth training, as (Zhang et al. 2023). This leads to more diverse generations with better prompt adherence. We also masks filter generated datasets. Using perceptual metric (Ful al. 2023) and P et al. (2023) we predict a mask for the generated image and measure the similarity masked training images, filtering out those below threshold. Details in A.4.2
Interbet-availlalee real data: user may download open-source real datasets; these can provide a source of real negatives and real backgrounds for Cut/Paste, avoiding the computational cost of image generation and enabling comparison to real-only approaches.
Extra real positives: A user may expand DR by physically collecting extra real target images. This also provides an expanded set of images for Cut/Paste and DreamBooth generation.
Experimental analysis. In Section 5.2 we explore if there are still benefits to using generated data versus computationally cheaper alternatives these extra annotations or real data are available.
4 EXPERIMENTS
4.1 DATASETS
Evaluating our personalized representations necessitates instance-level datasets with multiple tasks, across various real-world scenarios. To satisfy this criteria, we reformulate two existing datasets DeepFashion2 (Ge al. 2019) (focused shirts) and DogFaceNet (Mougeot al., 2019) (focused on dogs) and introduce a new dataset, PODS (Personal Object Discrimination Suite). PODS features common personal and household objects, enabling instance-level evaluation across classification, retrieval, detection, and segmentation tasks. To assess robustness and generalization, DF2 and Dogs provide in-the-wild test images, and DF2 and PODS include test sets designed with distribution shifts. All datasets split such that for each object there exactly 3 training images and at least 3 test images. We summarize our datasets and procedures below; for additional details and sample images, refer to Section of the Appendix.
DeepFashion2 (DF2) large-scale fashion dataset with 873K Commercial-Consume clothes pairs for instance-level retrieval, detection, and segmentation. We use the Consumer-to-Shen Clothes retrieval benchmark, which matches gallery images clothing items to in-the-wild con- sumer images, thus encoding train-test distribution shift. Out of 13 clothing categories, select
5

---

Published as a conference paper at ICLR 2025
PODS Dataset
Generated Images
Test ID
Test OOD
Dream Personalized
Train
Distractors
Bath
Negatives
Mug
Shoe
Bottle
Tote
Screw-
driver
Figure 3: (left) Examples of instances from our new PODS dataset. We showcase one example instance from each of the five object categories, displaying images from both the training and var- ious test splits. We dim the surrounding scene, highlighting the instance of interest. This masking technique is not applied to our dataset images or during training. (right) We show example gen- erated images from Dreambooth (LLM, cfg 5), which we use as positives in our representation learning finetuning.
the shirts category as our focus. We subselect set of 169 shirts, after filtering out categories which lack sufficient numbers of gallery images.
DogFaceNet (Dogs) dog identification dataset, containing 8600 images of 209 dogs. Dog- FaceNet includes multiple unique dogs of the same breed, making the dataset more challenging. We subselect 80 dogs with sufficient numbers images, and split the images into train and test set. To support evaluation of segmentation and detection, we manually annotate the dataset with masks.
Our new dataset: PODS contains 100 unique objects across 5 every-day categories (mugs, screw- drivers, shoes, bags, waterbottles). Each object is captured in four scenes with varying conditions and vantage points. The train set contains images of each object, displayed
canonical pose with full visibility of key identifying features such logos. The test set contains 80-100 images of each object, captured in four scenes: one in-distribution (ID) and three out-of- aut-of-distribution (OOD). We show examples of each type of scene in Figure 3 The IDD scene is taken in the same conditions as the training images. OOD) scenes include one scene with pose variation, one scene with distractor objects, and one scene with both variations. All OOD scenes are against differing backgrounds from the ID scenes.
The dataset supports evaluation across tasks: classification, retrieval, detection, segmentation. Each test image is associated with the target instance label. From each test scene, 3 randomly- selected images are additionally annotated with the bounding box and segmentation mask of the displayed object. Masks are manually annotated using TORAS (Kar et al. (02T) and SAM (Kirillov) al. 2023b); detection bounding boxes are extracted from the masks. We expect the PODS dataset to be a meaningful benchmark for personalized representation and instance-leve detection research, and a valuable resource for the personalized generation community.
4.2 TRAINING
We fine-tune a vision backbone fo on sets of (x,, 24 To, .....N) where the anchor is drawn from the 3 real positive images, the positive x4 is drawn from the pool of synthetic positives, and z; for i O,...,N .N are drawn from the synthetic negatives. We apply the following data augmentations to all images: random rotations, horizontal flips, and resized crops. We experiment with state-of-the of-the-art backbones: DINOv2-ViT VIT B/14 (Oquab et al. 2023), CLIP- ViT B/16, (Radford et al. 2021), and MAE-VIT B/16 (He al. 2022a).
Each dataset is randomly divided class-wise into a validation set (30 classes), and test set (varying size). We train model per class and report the mean performance over test classes. Using the

---

Published as a conference paper at ICLR 2025
Global Task Inference
Local Task Inference
Reference Set
Test Images
Masks
Dense
Binarize
Training Images
(F- 10-
= fo(tt) Mi
Tast
Confidance Map
Mar Aug. Score
Canf
Image
F= fg(3)
Figure 4: Inference Pipelines. We visualize the global (classification, retrieval) and local (detection, segmentation) evaluation pipelines. Global inference uses cosine similarity between CLS embed- dings, while local inference extracts patch features with spatial information.
validation set we sweep over key training parameters: synthetic positives, # anchor- anchorr-psitive pairs, and the choice of loss function. Based on this sweep LoRA finetune with the infoNCE loss for 2 epochs over 4500 anchor-positive pairs, drawn from 450 synthetic positives and 1000 synthetic negatives. For validation results and training details, refer the Appendix B.
C.T
4.3 EVALUATION
We evaluate personalized representations across all tasks that require global understanding, and the ability to localize with respect to the target object. Due to the few-shot nature of our task, evaluate representations directly, without training task-specific heads. We summarize evalu- ations below and in Figure 4
Classification. For particular instance c, given test image ă, frozen encoder fa. and training images xf E Dn we compute the maximum cosine similarity between the CLS tokens of fo(T) and fo(x"); this taken as the prediction confidence. Samples with confidence above some threshold taken positives. Thus report the Area under the Precision Recall Curve (PR-AUC), which a threshold -ree metric.
Retrieval. We use test set the "query" set, and Dn the "retrieval" set. We compute the cosine similarity between the CLS token of (ā) and those of the images D We score the resultant ranking with the NDCG metric (Jeunen et al.) 2024).
Segmentation. We compute the average cosine similarity between the patch embeddings of fo and those of fo(=²) to generate a local confidence map, where xf are masked to the target, follow- ing the procedure of (Zhang al. 2023). We then apply binarization directly the confidence map using Otsu's 's thresholding method (Otsu et al. 1975) and upscale to the image dimensions to yield segmentation prediction. We report the standard mask AP metric (Deng et al. 2024) and Fl scores, given the high imbalance between positives and negatives in the test sets.
Detection. We apply the same procedure segmentation, and extract bounding box prediction by drawing a box around the predicted mask. We obtain a confidence score for the box by averaging the confidence map within the box region. We report the standard AP metric and the F1 score.
For each task, we compare the performance of our learned personalized representations to pre- trained models. Note that we do not train prediction heads, due the lack of real training data in our setting - rather, we use these evaluations to probe what our personalized features learn about the target object, compared pretrained features.
RESULTS AND DISCUSSION
PERSONALIZED REPRESENTATIINN IMPROVE OVER PRETRAINED REPRESENTATIONS
We LoRA-tune three backbones (DINOv2, CLIP, MAE) and evaluate the personalized representa- tions on four tasks. We sweep over synthetic datasets with different levels of diversity by varying the CFG and usage of I.L.M-generteed prompts described in Section 3.3 and select the best for each

---

Published as a conference paper at ICLR 2025
Classification
Retrieval
Detection
Segmentation
PODS
DF2
Dogs
PODS
DF2
Dogs
PODS
DF2
Dogs
PODS
DF2
Dogs
DINOV2
28.1
14.4
83.1
69.6
36.3
89.4
11.0
5.3
12.2
12.9
4.7
12.5
DINOV2-P (DB)
48.0
35.7
81.6
79.6
64.1
94.6
12.6
9.8
17.4 A
13.8
9.3
17.2
CLIP
26.7
12.7
36.4
61.4
34.7
58.0
0.1
2.9
6.1
0.3
2.9
7.3
CLBP-P(DB)
47.4 + 26.7
65.0
71.7
51.0
80,9 +
0.8
4.8
9.5
1.6 f
4.9 +
10.9 +
MAE
8.7
11.3
34.6
25.8
33.6
0.2
1.4
0.4
1.2
0.7
MAE-P (DB)
17.0 +
12.2 29.9
30.7
23.7
42.7
0.4
1,5
1.54
0.5 +
1.2
0.8 +
Table 1: Performance of personalized v. pretrained representations across backbones, tasks, and datasets. We compare personalized and pretrained backbones, assuming access to only 3 real images and T2I model with no extra data/annotations. For each backbone we report results for the best- best-performing synthetic dataset (chosen using the validation set), averaged over 3 seeds. Personal- ized representations (-P) largely outperform pretrained representations across all tasks. Full results across synthetic datasets, and error over seeds, are in the Appendix CC.2
Classification
Retrieval
Defection
Segmentation
Real Backgrounds
PODS
DF2
Dogg
PODS
DF2
Dogo
PODS
DF2
Digg
PODS
DF2
Drgg
Method
Real Negs
32.4
15.0
Real Tmgs
v
34.4
28.5
81.5
60.6
51.7
92.5
11.7
8.0
14.7
13.2
6.9
15.0
Cut/Paste
X
x
60.4
48.5
83.3
82.0
68.3
93.7
14.6
13.1
16.2
18.4
11.9
16.2
v
58.7
47.3
87.8
77.6
66.0
95.8
15.3
12.9
13.0
19.8
11.2
14.0
Masked DB
17.2
55.6
43.3
84.8
68.4
94.1
16.2
13.7
12.7
19.9
11.2
13.6
Comhined
X
x
61.9
51.6
84.5
83.9
71.3
94.6
14.8
14.8
17.3
19.4
13.6
17.3
4
v
61.6
49.6
88.4
80.2
70.4
96.2
16.9
14.6
13.4
21.5
12.7
14.6
Table 2: Comparisons across data augmentation methods. We compare DINOv2-P trained with different augmentation strategies, including those requiring extra annotations/data. Training with synthetic data improves performance significantly combined Masked DreamBooth + Cut/Paste performs best in cases. Significant boosts are also achievable more cheaply when incorporating internet-available real data with Cut/Paste.
training with the limited real-image dataset;
backbone based validation performance. In Table I we compare the performance of pretrained and personalized backbones on DF2, Dogs, and PODS, using the best synthetic dataset for each backbone. Full results are in the Appendix (C.2)
Personalized models (-P in Table T boost performance 32/36 cases. We observe improvements often substantial in nearly every combination of backbone and task besides MAE retrieval. For example, averaged across datasets, DINOv2 detection improves by 48%, DINOv2 retrieval by 32%, and CLIP classification by 89% relative to pretrained models. Across all three datasets, personalized models boost performance both for global tasks requiring semantic understanding, and dense tasks requiring localization. We visualize dense prediction maps for multi-object images in Figure 3 and the Appendix (D.I), showing that personalized patch features better localize the target object. We also visualize challenging success and failure cases in the Appendix (.2).
5.2
WHAT MAKES FOR THE BEST TRAINING DATA?
In the previous section, we show that personalizing representations significantly boosts personalized task performance. Here, compare various data generation approaches (Section 3.5). some lever- aging additional resources, to investigate tradcoffs between computational cost and performance.
Results are shown in Table 2 with the runtime for each generation method in Table 3 (Appendix A.4.4). We first explore personalizing models only with the existing real training images in DR (Real Imgs). We then examine incorporating segmentation masks of DR. This enables cheap baseline method, Cut-and-Pastte (CP), and improvements to DreamBooth via masked training and filtering (Masked DB). We also test sampling from combined pool of CP and Masked DB images
8

---

Published as a conference paper at ICLR 2025
Input
Pretrained
Personalized
Input
Pretrained
Personalized
Figure 5: Qualitative Results. Each triplet shows the test image (left), dense prediction maps for pretrained DINOv2 (center), and personalized (right). Prediction maps are computed via patchwise embedding similarity between the test and localized train images following Figure 4 Personalized representations distinctly localize the target instance, unlike pretrained embeddings. For visualiza- tion only, the personalized instance highlighted in the test images but this is not applied during training or inference.
predictions pose
DB predictions similarity
Cut Paste
Masked DreamBooth
these object?
Different
Different
Different
Different
A Rist
X
Same X
Figure 7: DreamBooth Cut and Paste Model Fail- ures. We show object pairs where DB-personalized and CP- CP-personalized models differ most in predictions.
Figure 8: (left) CP limitations include pose and realism. (right) Masked DB struggles with fine-grained details.
(Combined). All synthetic data pools are each 450 images for fair comparison. We also ablate the use of generated negatives and generated CP backgrounds by using open-source images real alternatives; this enables comparison to cheap methods that use only real images.
Synthetic data methods outperform real-image-only methods, with the Combined pool performing best across all datasets/tasks, Performance improves significantly from incorporating masks; results in Table 2 out-perform DINOv2-P results in Table [ obtained without masks. Incorporating real negatives and real Cut/Paste backgrounds does not consistently improve performance over their generative counterparts. Comparing performance (Table and runtime (Table 3) reveals a tradeoff between the two. The high performance of Combined indicates that learned models provide valuable knowledge, However, sampling CP with real backgrounds performs similarly to Masked DB alone, showing that strong performance can still be achieved with efficient alternative.
Scaling
Scaling real positives. We collect extra training images for 25 PODS instances (5 per category) with different backgrounds, poses.
foscaiictton Accuracy
3.75
3.73
and lighting. In Figure [ we compare training DINOv2-P on Dn, and on synthetic Ds generated from Dn using the Combined method. Performance increases as 'Dnl increases, saturating at [DRl = 15, likely due to limitations the diversity of the addi- tional real data. Expanding diversity further could improve scaling but requires significant manual effort. Synthetic augmentation re- mains effective
0.33
3.00
Number Real Images
[DRl scales (27% gain with real images, 8% gain with 20). As generative models improve, the ability to com- plement real datasets is expected to grow.
Figure 6: Real and synthetic data scaling curve for sub- set of PODS.

---

Published as a conference paper at ICLR 2025
5.3 HOW DO DIFFERENT DATASETS AFFECT REPRESENTATIONS?
As seen in T Masked DB and CP achieve similar performance. Here, we show that they exhibit in high-confidence predictions of
distinct strengths and limitations. We analyze divergence DB- and CP-trained representations, revealing consistent failure patterns. DB-trained models excel pose generalization but often confuse visually similar instances. Conversely, CP models are more robust to distractors but falter when encountering unfamiliar poses. We show examples in Figure 7 This trend is also quantitatively shown in Figure 9 In the PODS Distractors split, CP models outperform DB models by 7%, whereas in the Pose split, DB models surpass CP models by 6.4%.
In Figure we trace these attributes of learned representations to biases/limitations in DB and CP generations. The main DB limitation is difficulty in preserving fine-grained object details, resulting images that only loosely resemble the target characteristics, even with filtering. These inaceu- racies likely propagate to the learned representation, compromising its fine- lnnegrainnd discriminative ability. Conversely, CP maintains perfect object fidelity but restricts pose variability, potentially leading to pose overfitting.
We also study the impact of fidelity and diversity in Figure 12 (Appendix C.4). Without filtering, DB datasets lie at extremes (high-diversity, low-fidelity or
Classification (PR-AUC)
Imge
0.9
Cut/Paste
0.8
Masked DB
vice versa). Both the Masked DB and CP datasets
achieve a better balance, and thus higher performance.
0.6
0.5
5.4 APPLICATIONS
0.3
Our thresholding evaluation for dense tasks allows di- rect probing of personalized patch features, however does not achieve of-te-of-the-rrt results. We show be easily inte- grated into perSAM practical pipeline - to improve its performance (Zhang al., 2023). Instead of ex- tracting keypoint proposals and confidence scores with pretrained DINOv2, we use DINOv2-P. Segmentation performance improves from 5.8% Fl score to 10.9 % on DF2, 20.9% 26.2% on Dogs, and 20.8% to 24.7% on PODS. Full details/results are in the Appendix (C.3
0.2
Pose
Distractors
Both
PODS Test Split
that personalized representations
Figure 9: DB, CP, and Combined per- formance on PODS test splits. DB out- performs CP on the Pose split, and un- derperforms it on the Distractors split, whereas Combined performs best on both. These results support our qualitative obser- vations.
LIMITATIONS AND CONCLUSIONS
6.1 LIMITATIONS
Our pipeline has potential for high computational cost (Table 3) from T21 models. We defer this limitation to future research in T2I efficiency. A cost-conscious user may choose instead to use cheaper method such as Cut/Paste with worse/comparable performance. There are also cases where the performance boost from T2I models may be worth the cost (e.g., personalized medicine, re-ID for animals, identifying items for low-vision people (Morrison et al. (023)). Furthermore, by training on data generated by T2I models we may inherit their biases and limitations (see (see).3)
6.2 CONCLUSIONS
We leverage generative models to adapt general-purpose representation spaces to personalized ones, using very few real examples of single instance. We quantitatively and qualitatively show that personalized representations consistently boost performance across downstream tasks. Moreover, also study computationally cheaper alternatives that leverage additional resources, and show that combining different generation methods may enable further improvements. We release our new dataset, PODS, and splits and annotations for existing instance-level datasets, offering comprehensive benchmarks for future work.
In the future, generative models will continue to get faster, cheaper, more accurate controllable and potentially less biased. Our work is not limited to existing generation techniques. We are excited by the potential of learning personalized representations in this way, and envision possibility that allows users to have ownership over their own models and data.
10

---

Published as a conference paper at ICLR 2025
CODE OF ETHICS
The authors have read the code of ethics and we acknowledge that we adhere to the code presented.
ETHICS STATEMENT
Below, we discuss potential societal impacts of research in personalized representations.
One potential upplication of such approach, and more broadly of all instance-leve recognition and re-identification work, is surveillance technology, which may infringe on human privacy depending on the use-case and intent. In this paper we focus non-human (object-centric) applications and datasets, as opposed to facial or vehicle re-identification.
By removing the need for real negatives, our approach potentially enables personalized models that do not require people to share their data in a central repository or access others' data during person alization, beyond the prior contained in pretrained T2I model. Users may choose keep personal data siloed (similarly to federated learning settings), while still benefiting from personalized models. Removing the need for external data reduces data storage/access requirements for users.
The power and water usage by the GPUs needed for fine-tuning and T2I inference. Al has become an increasingly large portion of global power use, water use, and carbon emissions Luccioni et al. (2023); ineffi- cient methods exacerbate this effect. Users prioritizing cost over optimal performance may choose computationally cheaper alternatives that do not require T2I models (Section A.4.4).
Computational cost (discussed in Section 6.1 adds another important dimension:
Our work includes reformulations two existing datasets and a novel dataset. Privacy and con- sent measures for all three datasets detailed in Section A.3 ensuring transparency and ethical compliance.
REPRODUCIBILITY
Our website (ttppsr/pprrooallleete......hppp
and Github repository (https:
7/github. com/ hatmm/sauddrramm1111 wttpa//rwwmmm/ppeoaaaaaaeiaaeeee (issoonnlieeddrpp) contain the source code for work, including the necessary metadata to reproduce results, such as the LLM-gennrated captions used for dataset synthesis. The codebase features end-to-end pipelines for data generation, per- sonalized representation training, and inference/evaluaaion. In the appendix, we provide detailed hyperparameters and outlines for data generation (DreamBooth finetuning. Cut and Paste), repre- sentation training (parameters for LoRA and other model training hyperparameters), and our evalu- ation pipelines. We also present extensive ablations and full results in the appendix for transparency, justifying each design choice so that other researchers can replicate our findings. Finally, we release our dataset, PODS, and the reformulated DeepFashion2 and Neg datasets.
ACKNOWLEDGMENTS
This material based on work that is partially funded by an unrestricted gift from Google through the MIT-Google Program for Computing Innovation, and by the Defense Science and Technology Agency, Singapore. This work was also supported by the Al and Biodiversity Change (ABC) Global Climate Center, which is funded by the US National Science Foundation under Award No. 2330423 and Natural Sciences and Engineering Research Council of Canada under Award No. 585136. S.S. is supported by NSF GRFP fellowship. J.C. is supported by NSERC PGS-D fellowship.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano. Hyperstyle: Stylegan inversion with hypernetorkks for real image editing. In CVPR, 2021.
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. neural space-time representa- tion for text-to-image personalization. TOG, 2023.
Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvim: Personalizing vims for user-specific queries. arXiv preprini arXiv:2403. 14599, 2024.
Ali Amiri, Aydin Kaya, and Ali Seydi Keceli. A comprehensive survey deep-learning-aase vehicle re-identification: Models, data sets and challenges. arXiv preprint urXiv:2401.10643, 2024.
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain-agnostic tuning-encoder for fast personalization of text-to-image models. In SIGGRAPH Asia, 2023.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08866 2023.
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15338-15340 -53477, 2023.
Hritik Bansal and Aditya Grover. Leaving reality to imagination: Robust Robust classification via generated datasets. arXiv preprint arXiv:2302.02503. 2023.
David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolci Zhou, Jun- Yan Zhu, and An- tonio Torralba. Semantic photo manipulation with generative image prior. TOG, 2019.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6502-6602, 2024.
Cese Chunseong Park, Byeongchang Kim, and Gunhee Kim. Attend to you: Personalized image captioning with context sequence memory networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 895-903, 2017.
Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. In ECCV, 2022.
Xueqing Deng, Qihang Yu, Peng Wang, Xiaohui Shen, and Liang-Chier Chen. Coconut: Modern- izing coco segmentation. In Proceedings of the IEEC/CVF Conference Computer Vision and Pattern Recognition, pp. 111663218877 2024.
Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetoork. In CVPR, 2022.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E. Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based augmentation, 2023,
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. arXiv preprint arXiv:2312.04500 2023.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing eox-tto-image generation using textual inversion. In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Yuying Ge, Ruimao Zhang, Lingyun Wu, Xiaogang Wang, Xiaoou Tang. and Ping Luo. A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. CVPR, 2019.
Douglas Gray and Hai Tao. Viewpoint invariant pedestrian recognition with ensemble of localized features. In ECCV, 2008.
Shanyan Guan, Yanhao Ge. Ying Tai, Jian Yang. Wei Li, and Mingyu You. Hybridbooth: Hybrid prompt inversion for efficient subject-driven generation. In European Conference on Computer Vision, pp. 403 419. Springer, 2025.
Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.01832, 2024.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris N. Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In ICCV, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022a.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiao- juan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022b.
Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In ICCV, 2021.
Jonathan Ho and Tim Salimans. arXiv:2207.12596 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXi::210600668) 2021.
Qihan Huang, Siming Fu, Jinlong Liu, Hao Jiang, Yipeng Yu, and Jie Song. Resolving multi-condition confusion for finetuning-free personalized image generation. arXiv preprint arXiv:2409.17920 2024.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint arXi::2106.05258, 2021.
Olivier Jeunen, Ivan Potapov, and Aleksei Ustimenko. On (normalised) discounted cumulative gain as an off-policy evaluation metric for top-n recommendation. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp.
2024.
Kyuheon Jung, Yongdeuk Seo, Seongwoo Cho, Jaeyoung Kim, Hyun-seok Min, and Sungchul Choi. Dalda: Data augmentation leveraging diffusion model and Ilm with adaptive guidance scaling. arXiv preprint arXiv:2409.1694. 2024.
Amlan Kar, Seung Wook Kim, Marko Boben, Jun Gao, Tianxing Li, Huan Ling, Zian Wang, and cs. toronto. Sanja Fidler. Toronto annotation suite. aatee:///adeeess.....ooooootoetd 2021.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata, Vision-by-language for training- free compositional image retrieval. arXiv preprint arXhv:2310.09291, 2023.
Shahroz Khan, Erkan Gunpinar, Masaki Moriguchi, and Hiromasa Suzuki. Evolving psycho- physical distance metric for generative design exploration of diverse shapes. Journal of Mechan- ical Design, 2019.
13

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, 2023a.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023b.
Nupur Kumari, Bin Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept tomization of text-to-image diffusion. In CVPR, 2022.
Orest Kupyn and Christian Rupprecht, Dataset enhancement with instance-level augmentations. arXiv preprint arXiv:2406.08229, 2024.
Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint uuXiv:2307.88520 2023.
Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few- shot object detection. In CVPR, 2023.
Hongye Liu, Yonghong Tian, Yaowei Yang, Lu Pang, and Tiejun Huang. Deep relative distance learning: Tell the difference between similar vehicles. In CVPR, 2016.
Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? arXiv preprint arXiv:2311.16663 2023.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Cecily Morrison, Martin Grayson, Rita Faia Marques, Daniela Massiceti, Camilla Longden, Linda Wen, and Edward Cutrell. Understanding personalized accessibility through teachable ai: De- signing and evaluating find my things for people who are blind or low vision. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1-12, 2023.
Guillaume Mougeot, Dewei Li, and Shuai Jia. deep learning approach for dog face verification and recognition. In PRICAI 2019: Trends in Artificial Intelligence, 2019.
Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo'Ilava: Your personalized language and vision assistant. arXiv preprint arXiv:2406. 094400 2024a.
Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Yinlin Hu, Renaud Marlet, Math- ieu Salzmann, and Vincent Lepetit. Nope: Novel object pose estimation from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17923-17932, 2024b.
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. TOG, 2022.
Timo Ojala, Matti Pietikainen, and Topi Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. TPAMI, 2002.
OpenAl Gpt-4 technical report. arXiv preprint
2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv:2304.07/91 2023.
14

---

Published as a conference paper at ICLR 2025
Nobuyuki Otsu et al. A threshold selection method from gray-level histograms. Automatica, 11 (285-296):23-222 27, 1975.
Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. Towards personalized image cap- tioning via multimodal memory networks. IEEE transactions on pattern analysis and machine intelligence, 41 (1499999-11112 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. TOG, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetorrss for fast personaliza- tion of text-to-image models. ArXiv, 2023.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19305- 19314, 2023.
Joseph W. Sakshaug and Trivellore E. Raghunathan. Synthetic data for small area estimation. In Privacy in Statisticll Databases, 2010.
Dvir Samuel, Rami Ben-Ari, Matan Levy, Nir Darshan, and Gal Chechik. Where's waldo: Diffusion features for personalized segmentation and retrieval. NeurIPS, 2024.
Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR, 2023.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image gen- eration without test-time finetuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8543-8552, 2024.
Joonghyuk Shin, Minguk Kang, and Jaesik Park. Fill-up: Balancing long-tailed data with generative models. arXiv preprint arXh::2306.07200 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Y. Berger-Wolf, Wei-Lun Chao, and Yu Su. Bioclip: vision foundation model for the tree of life. arXiv preprint arXiv:2311. 1.1803,,.88. 2023.
Carlos C Sun, Glenn S Am, Ravi Prakash Ramachandran, and Stephen G Ritchie. Vehicle reidenti- fication using multidetector fusion. IEEE TITS, 2004.
Yaniv Taigman, Ming Yang, Mare Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014.
15

---

Published as a conference paper at ICLR 2025
Fabio Henrique Kiyoiti Dos Santos Tanaka and Claus Aranha. Data augmentation using gans. arXiv preprint 0rXiv:1904.09133. 2019.
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. arXiv preprint arXiv:2312.17742, 2023a.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306. 00984, 2023b.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmen- tation with diffusion models. arXiv preprint arXi::2302.0744. 2023.
Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. npj Digital Medicine, 2020.
Matthew A Turk and Alex P Pentland. Face recognition using eigenfaces. In CVPR, 1991.
Xuan Wang, Guanhong Wang, Wenhao Chai, Jiayu Zhou, and Gaoang Wang. User-aware prefix- tuning is a good learner for personalized image captioning. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 384 395. Springer, 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In ICCV, 2023.
Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Syn- thesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303. 1681, 2023.
Ankit Yadav and Dinesh Kumar Vishwakarma. Deep learning algorithms for person identification: sate-of-the-ar and research challenges. Multimedia Tools and Applications, 83 00:222005-20555 2024.
Chun-Hsiao Yeh, Bryan Russell, Josef Sivic, Fabian Caba Heilbron, and Simon Jenni. Meta- personalizing vision- language models to find named instances in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19123-19132. 2023.
Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmaeil Akhoondi, and Amir Zamir. Controlled training data generation with diffusion models. arXiv preprint arXiv:2403.. 15309, 2024.
Jianhao Yuan, Francesco Pinto, Adam Davies, Aarushi Gupta, and Philip Torr. Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations. arXiv preprint 2rXiv: 1237, 2022.
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. arXiv preprint a0X10:2310.0040,, 2023.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023.
Yunhua Zhang, Hazel Doughty, and Cees GM Snoek. Low-resource vision challenges for foundation models. arXiv preprint arXiv:2401.04710 2024.
Haiyu Zhao, Maoqing Tian, Shuyang Sun, Jing Shao, Junjie Yan, Shuai Yi, Xiaogang Wang, and Xiaoou Tang. Spindle net: Person re-identification with human body region guided feature de- composition and fusion. In CVPR, 2017.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training thin air: Improve image classification with generated data. arXiv preprint arXiv:2305.1533 2023.
16

---

Published as a conference paper at ICLR 2025
A DATASETS
A.1 DF2 AND DOGS REFORMULATION
We reformulate datasets: DeepFashion2 and DogFaceNer.
DeepFashion2. In this section, we identify the training split that we subselected from DeepFash- ion2. DeepFashion2 is a large-scale retrieval dataset with public trainvvaliattiontess split. Each split contains its own query/customer and gallery set. We sample our training set from the gallery of the validation split, and we sample our test set from the consumer set of the validation split. Each image in validation split has six-digit identifier and annotation file containing the infor- mation. We organize the data into instance categories; we define the training and testing images for an instance as gallery/consumer images depicting the same clothing item of the same style.
Below we provide metadata for our subselected dataset.
Clothing Item Category: Short-Sleeve Tops, category id
Unique Instances Selected: 169
Total # Training Images: 507
# of Training Images per Instance: 3
Total # Test Images: 1271
Range of Test Images Instance: [4. 24]
Dogs. Our Dogs dataset reformulates the DogFaceNet large split from the datasets released with DogFaceNet for dog re- re-identification studies. Since the original dataset was published with instance- level splits, we perform our own splitting of the dataset fit our personalized learning setting. Due to the nature of data collection (images of dogs collected from sequential footage), we had to pay careful attention to the possibility of data poisoning between the train and test set. The procedure followed for data splitting is follows:
1. Filtered DogFaceNet dog classes to keep classes with above 10 images per instance
Performed random train-test split for every instance, keeping 3 images for train rest for test
3. Manually inspected every instance in the dataset to remove data poisoning. This entailed looking through the training and data and making sure that no test images were from the same sequential footage as the train data. When such images were discovered, they were removed from the test set.
4. After data-cleanup, removed instances with less then 4 remaining test images.
The above procedure resulted in 80 total dog classes. Below is the metadata of our subselected dataset.
Unique Instances Selected: 80
Total # Training Images: 240
# of Training Images per Instance: 3
Total # Test Images: 1218
Range of Test Images per Instance: [6, 38]
Dataset Examples. In Fig. 10 and [1 we show examples training and test images for the Dogs and DF2 datasets. Notably, the test sets cover wide range of diverse in in-the-wild scenarios, including different contexts, positions, backgrounds, camera angles, occlusions, and lightings.
Licenses for existing assets
The datasets
released under the following licenses:
DeepFashion2: MIT DogFaceNet Dataset: Non-Commercial, Research-Only
17

---

Published as a conference paper at ICLR 2025
DF2 Dataset
Train
Test
ORIGINALS
Figure 10: DF2 train/test examples. Training images are of models wearing clothes, and test images are from consumers. Images and classes are randomly sampled.
DOGS Dataset
Train
Test
Figure 11: Dogs train/test images. Images and classes are randomly sampled.
18

---

Published as a conference paper at ICLR 2025
A.2
PODS
A.2.1
PODS OVERVIEW
Unique Instances: 100
Total # Training Images: 300
# of Training Images per Instance: 3
Total Test Images: 10888
Range of Test Images per Instance: [71, 201]
Total # Test Images with Dense Annotations: 1200
Range of Test Images per Instance with Dense Annotations: [12, 12]
A.2.2 PODS CREATION
PODS includes images 100 objects; 20 each from five categories: mugs, screwdrivers, tote bags, shoes, and water bottles, We choose these categories to cover range of personal, everyday objects, both rigid and deformable.
Scenes.
Every object is captured in 4 scenes. We describe each scene through 4 attributes: the background of the scene, the pose of the target object, the presence of distractor objects, and the visibility of the object's key identifying features (such mug's logo).
Below a detailed description of each scene.
Train/In-distribution:
Background:
The object is against plain office background.
Pose:
The object is located flat surface, upright in its "canonical pose".
Distractors:
No similarly-sized distractor objects are nearby, in the clear fore-
ground.
Identifying features: Key identifying features (i.e. logo, handles, etc) clearly visible.
Distractors:
Background: The object is against different background from the Train scene.
Pose: Object is upright in its "canonical" pose.
Distractors: Object is surrounded by 2-5 distractor objects of varying sizes, located in both the foreground and background. These can act as potential occluders.
Iddetiffynng features: Key identifying features may not fully visible (for instance, mug may be occluded by distractors such that the logo only partially visible.)
Pose:
Background: The object against different background from the Train scene.
Pose: Object is in a different pose from the training scene.
Distractors: No similarly-sized distractor objects nearby, or in the clear foreground.
Identifying features: Key identifying features may not be fully visible (for instance, mug may be turned so that its logo is only partially visible.)
Both:
Background: The object is against different background from the Train scene.
Pose: Object is in different pose from the training scene.
Distractors: Object surrounded by 2-5 distractor objects of varying sizes, located in both the foreground and background. These can act as potential occluders.
Identifying features: Key identifying features may not be fully visible (for instance, mug may be turned occluded so that its logo is only partially visible.)
For the final (hardest) scene, attempt capture images that mimic expected in-the-wild settings for each category. For instance, "Both" scenes for shoes are captured outdoors, with sports equip- ment as distractors. Similarly mugs captured drying rack, with other wares distractors.
19

---

Published as a conference paper at ICLR 2025
Capture. We capture all images iPhone 15 Pro. For each scene, use the PolyCam app to capture a video. The app automatically extracts approximately 20 frames to be exported. We capture each video in three 120-180 degree views, each at a different vantage point: Level with the object, above the object (camera looking down), below the object (camera looking up). Note that this results in images where the object is not centered, occluded by distractor objects, out of focus in the background; these are useful as hard positives.
Splits and annotation. For each object, we manually inspect the Training scene and extract three training images, taken at level with the target object, and roughly equally spaced throughout the camera trajectory. The rest of the Training scene images are relegated the test set, and serve as the in-distribution split. Images from the other three scenes serve as the three out-ts-distributiin (OOD) splits.
We record unique identifier for each object, and label every test image with the identifier of the object in that image. These image-level labels are used for classification and re-
Annotation.
trieval.
For each object, we randomly choose images from each test scene to annotate with segmentation masks and bounding boxes. Thus there are 12 images total per object with dense annotations. To notate with masks, we first use Grounding-SAM Kiriiil et al. (2023a) to generate mask proposals. We then manually inspect every image. For images with incorrect generated masks, we manually annotate using TORAS (Kar et al. 2021).
Kirillov
A.3 PRIVACY AND CONSENT
PODS. Our PODS dataset is published under the MIT License. The majority of objects belong to the authors, and for additional objects, obtained explicit consent from their owners. We ensured that no with human subjects.
attributes are present and that the dataset contains only object images,
Dogs. The DogFaceNet dataset is under the MIT License, allowing modification and redistribu- tion. Our reformulation follows this license with proper attribution.
DF2. For DeepFashion2, we comply with the dataset's terms of use by using it solely for non- commercial research purposes without reproducing or redistributing the data. Our reformulation made public only mapping to the original dataset, ensuring that we do not share the data directly.
A.4 SYNTHETIC DATA GENERATION
A.4.1 LLM-GENERATER CAPTIONS
We generated 100 instance-relevant prompts for image generation, which we used to generate im- ages. Here we present 15 examples for each dataset. The full caption set is in our Github repository.
PODS Dataset
Mugs
<newl> mug on wooden desk
<newl> mug cozy living room
<newl> mug on a windowsill
Bottles
1. A <newl> bottle pienic table
A <newl> bottle in a backpack pocket
<newl> bottle on a yoga mat
Screwdrivers
1. A <newl> screwdriver in toolbox
2. A <newl> screwdriver wooden workbench
20

---

Published as a conference paper at ICLR 2025
3. A <newl> screwdriver in carpenter's tool belt Totes (Bags)
<newl> bag car trunk
<newl> bag on a park bench
<newl> > bag in shopping cart
4. A <newl> bag on a library shelf
5. A <newl> bag in gym locker
6. A <newl> bag on wooden table
Shoes
<newl> shoe in the rain
<newl> shoe on sandy beach
<newl> shoe in a gym locker
DF2 Dataset
1. A person wearing <newl> shirt at a park
2. A <newl> shirt on mannequin
3. A person wearing <newl> shirt at party
4. A <newl> shirt on a clothesline
5. A person wearing <newl> shirt at a concert
6. A <newl> shirt on chair
7. A person wearing <newl> shirt at café
8. A <newl> shirt on a laundry basket
9. A person wearing <newl> shirt at a stadium
10. A <newl> shirt hook
11. A person wearing <newl> > shirt at bus stop
12. A <newl> shirt on a drying rack
A person wearing <newl> shirt at gym
14. A <newl> shirt shelf
15. A person wearing <newl> shirt at picnic
Dogs Dataset
1. A <newl> dog in the park
2. A <newl> dog at the vet
3. A <newl> dog car
4. A <new1> dog the groomer
5. A <newl> dog on a walk
6. A <newl> dog in the snow
7. A <newl> dog the lake
8. A <newl> dog in the backyard
9. A <newl> dog at the <newl> dog park
10. A <newl> dog in a sweater
11. A <newl> dog in bed
12. A <newl> dog at the farm
13. A <newl> dog in the woods
14. A <newl> dog in kennel
15. A <newl> dog barbecue
21

---

Published as a conference paper at ICLR 2025
A.4.2 MASKED DREAMBOOTH FILTERING
We apply automatic filtering the Masked DreamBooth pipeline as an additional data-processing step that we can take when masks are available, to ensure high-quality generated data. We use the masks extract a bounding box for the object of interest the training image, and embed it using DreamSim (Fu et al. 2023), perceptual similarity metric. Similarly, at every test-image prediction, we apply perSAM to generate test-mask, and embed the masked test image with DreamSim. A cosine similarity score is computed between the train masked embedding and the test masked em- bedding, and an empirically chosen threshold is used to filter out the data below a certain threshold value. For DF2 and PODS, the threshold was 0.6 and for Dogs it was 0.55.
A.4.3 CUT AND PASTE
To generate cut and paste images, we first require small subset of training images and their coffe- sponding masks, which we use to extract the foreground object. For the background, we generate 600 unique background scenes following the same set of LLM- -oompppppts used to generate the diverse DreamBooth images. To use them for background generation, we removed the "<newll specifi- cation from every prompt: e.g. "photoo of <newl>
the beach" becomes "photo of a beach" and addressed proposition inconsistencies afterwards. We then randomly resized the masked foreground image scale between 0.3 and 1.3 times the original image size, and pasted it onto the background-generaan images at randomly selected location within the image.
A.4.4 RUNTIMES
We report the wall-clock runtimes of synthetic data generation methods, using single NVIDIA A100 GPU, in Table Per-Image generation times are taken as an average over 50 generations. Per- Per-Dataset times indicate the time to generate 450-image dataset with a batch size of 1. We do not take into account the time to download open-source datasets (e.g., real backgrounds for Cut/Paste), as this is highly context dependent.
Method
Fine-tuning (min)
Generation per-Image (sec)
Generation per- Datasel (min)
Total (min)
DreamBooth (no filtering)
3.8
0.98
7.35
11.15
DreamBooth (w/ filtering)
3.8
1,83
13.7-152.5
17.5-156.3
Cui/Paste (real BG)
0.06
0.45
0.45
Cuv/Paste (generated BG)
7.8
7.8
Table Runtime for synthetic data generation. We report the wall-clock runtimes for generating 450-image synthetic dataset, broken down by stage. The fastest method is Cut/Paste with real backgrounds, which does not require T2I model.
B METHODS
B.1 TRAINING
We use the following hyperparameters to LoRA fine-tune each backbone:
Learning rate: 0.0003
Batch size: 16
LoRA rank: 16
LoRA alpha: 0.5
LoRA dropout: 0.3
B.2 EVALUATION
Classification:
We take all the available test examples for each instance of interest as our test set. We evaluate each trained personalized embedding space in one-vs-all binary classification setting
22

---

Published as a conference paper at ICLR 2025
with respect to the rest of our test data, using standard few-shot learning setup. Given test image we use a frozen vision encoder, f. to obtain embeddings of ž. We then compute the maximum cosine similarity between f()) and any real image in DR. and take this the prediction confidence. Samples with a confidence above some threshold are taken as positives; our evaluation metric is thus the Area under the Precision-Recall Curve, which is threshold-free metric. The performance for a dataset is computed as an average over the PR-AUC for each learned embedding space. We compare the performance between our learned personalized embedding spaces and non-personalized models (i.e. pretrained DINOv2).
Retrieval: We take text images as query set and all available reference training images as a retrieval set. Given test image E we use frozen vision encoder, f. to obtain embeddings of ă. We then compute the maximum cosine similarity between (() and every image in the retrieval set, DR. and take this as the prediction ranking for the retrieval task. We score the resultant ranking with the standard NDCG metric, as it considers both relevant and position of all retrieved items, unlike Fl and MRR metries.
To be able to perform localization, we use the encoder's patch embeddings. First, obtain target local features by computing the masked embedding of training image. We then compute the cosine similarity between the target local features and the patch embeddings of the target test image to generate local confidence map, where high confidence regions correspond to localization probability of the object. We then apply binarization directly to the local confidence map with Otsu's thresholding method, and upscale to the image dimensions to yield segmentation prediction. We score this prediction by taking the aggregate local confidence map values in the predicted mask region. We evaluate our segmentation task using the standard mask AP metric, and also report fl scores given the fact that our test sets are highly imbalanced (there are significantly more negatives than objects of interest in the test set).
Segmentation:
For detection, we apply the same pipeline as segmentation obtain a local confidence map and a binarized mask. We extract the bounding box prediction from the prediction by drawing box around the boundaries segmentation mask. We score this prediction by taking the aggregate local confidence map values in the predicted bounding box region. We evaluate our detection task using the standard AP metric, and also report fl score.
Detection:
C ADDITIONAL EXPERIMENTS
C.1 HYPERPARAMETER AND LOSS FUNCTION ABLATIONS
We conduct ablations the key training parameters of our method: # unchor-positive pairs, total generated positives, and choice of loss function. We do the validation of DF2, using the Masked DreamBooth dataset (without filtering); these were chosen arbitrarily, and intended to be representative of trends that we might expect across other datasets.
# Synthetic Imgs
# Anchor-Pou Pos Pairs
Classification
Retrieval
DINOv2
12.0
35.9
300
300
27.2
51.2
DINOv2-P
300
600
35,9
58.7
300
1500
38.0
61.1
3000
38.1
62.1
6000
38.5
61.7
300
15000
37.7
61.3
300
30000
36.9
60.0
Table 4: Ablation on the number of generated positives.
Number of anchor-positii pairs. Given fixed number of synthetic and real images, we potentially sample many combinations of anchors and positives for contrastive learning. We thus sweep over different ratios of generated positives sampled anchor-positive pairs. We fix the size
23

---

Published as a conference paper at ICLR 2025
of Ds. the pool of generated positives, to 300 (arbitrarily chosen) and sample increasing numbers of anchor-pe anchor-positive pairs from this pool for training. We find that performance on the DF2 validation set plateaus 1:10 ratio, and subsequently use this ratio in all of our main experiments. Results are shown Table 4
# Synthetic Imgs
Anchor-Pes Pos Pairs
Classification
Retrieval
DINOv2
12.0
35.9
30
12.8
36.0
DINOv2-P
30
300
28.6
35.9
150
1500
36.8
60.8
38.1
62.1
450
4500
38.7
61.9
600
6000
38.3
60.8
Table 5: Ablation on the number of anchor-positive pairs.
Number of generated positives. We ablate the size of Ds, the pool of generated positíves. To isolate the effect of the positive pool size, we the ratio between the size of DD and number of suechor-posttive pairs that sampled from (DR, R. Ds). We fix this ratio to 1:10, this found to be best in the previous ablation. We find that performance plateaus at 450 generated positives and thus use 450 the size of Ds in all of our main experiments. Results shown in Table
Loss function. We evaluate DINOv2-P trained on the Masked DreahBooth dataset (CFG 5) with three contrastive loss functions, and one non-contrastive loss. For the Cross-Entropy experiment, we add linear layer with a sigmoid that projects the output feature vector to a single prediction scalar indicating the target object, O) for any negative). We evaluate on the validation set of DF2 and find that InfoNCE leads to the best results, and that contrastive losses overall perform better than Cross-Entropy. Results are shown in 6
Loss Function
Classification
Retrieval
DINOv2
12.0
35.9
InfoNCE
36.5
63.3
InfoNCE Multi-Positive)
27.5
28.0
DINOv2-P
Hinge
29.4
48.4
Cross-Entropy
24.9
37.0
Table 6: Loss function ablation.
C.2 FULL EVALUATIONS ACROSS SYNTHETIC DATASETS
We ablate components of our method that contribute to the diversity of generated datasets, in par- ticular the CFG parameter, and the use of LLM-generaen prompts. In Tables 7 we show results across all tested synthetic datasets, backbones, and downstream tasks. We highlight the synthetic datasets that lead to the best performance for each backbone (as determined by average performance across the PODS/DF2/Dogs validation sets). For these, we report the minimum/maximum perfor- mance across three seeds, with the averages shown in Table [ Notably, LLM-generated prompts significantly improve performance on global tasks, however have little impact on dense task perfor- mance.
24

---

Published as a conference paper at ICLR 2025
Classification
Retrieval
PODS
DF2
Dogs
PODS
DF2
Dogs
Model
CFG
LLM
DINOv2
28.1
14.4
83.1
69.6
36.3
89.4
43.1
32.7
80.2
72.9
59.3
92.3
5
X
42,9
32.6
80.4
72.7
58.8
92.2
DINOv2-P
7.5
X
41.5
32.0
81.4
71.6
57.9
92.4
47.4
35.7
80.7
79.8
63.4
94.3
5
(48.0, 48.1)
(35.1, 36.1)
(81.2, 8 81.8
(79.3, 80.0)
(64.0, 64.4)
(94.3, 94.9)
7.5
48.5
36.4
82.8
79.7
62.9
94.4
CLIP
26.7
12.7
36.4
61.4
34.7
58.0
43.4
25.0
57.6
60.3
46.9
43.0
25.0
57.9
72.5
7.5
40.3
24.4
58.2
59.1
45.5
72.7
CLIP-P
46.9
26.8
64.2
72.2
51.2
80.5
(47.1, 47.7)
(26.0, 27.3)
(64.4, 66.0)
(71.3, 71.9)
(50.4, 51.8)
(80.5, 81.6)
48.7
25.3
65.3
49.2
81.0
MAE
8.7
5.2
11.3
34.6
25.8
33.6
4
15.0
14.0
25.1
28.0
24.4
35.8
5
14.9
14.0
26.2
27.9
23.9
36.3
7,5
14.1
13.5
27.0
27.4
23.6
37.0
MAE-P
4
(16.6. 17.5)
(12.0, 12.5)
(29.7. 30.1)
(30.6, 30.8)
(23.7.
(42.5.42.9)
5
17.1
12.3
30.4
30.4
23.0
43.1
7.5
16.7
11.2
32.3
29.8
21.9
43.8
Table 7: Performance of personalized v. pretrained representations on global tasks. We re- port results for all generated synthetic datasets, ablating both CFG and the of LL.M-generated prompts. The best dataset for each backbone (selected using validation performance) is highlighted yellow, with the min/max performance over seeds reported.
Detection (mAP)
Detection (FI)
Seginentation (HAP)
Segmentotion (FT)
PUBS
P0DS
PORSS
DINO
11.0
5.3
12.2
11.1
6.6
13.9
12.9
4.7
12.5
14.5
53
15.6
12.9
8.8
16.0
12.1
10.7
17.9
14.4
80
15.8
150
9.4
19.9
DINDX2
12.1
17.3
10.7
11.3
19.5
13.
9.3
17.2
13.5
10.4
20,8
(124,
(97..9))
13.0
10.4
CLIP
3.5
7.8
0.5
5.0
7.8
1.3
4.7
9.0
0.7
5.0
9.7
7.7
0.2
4.7
7.
9.6
4.7
0.9
4.5
9.01
U.5
4.8
9.3
1.7
4.4
10.3
0.9
4.7
11.0
0.8
9.8
B.4
5.1
9.8
1.6
4.8
10.8
0.9
5.1
11.5
1.6
044
1.5
1.2
U.4
2.1
1.9
015
1.2
0.5
0,5
1.5
0.N
1.2
2.0
1.2
0.3
2.1
1.9
0.4
1.2
0.6
0.5
1.6
0.9
(35
(15.110
m..00.0
(85.0%)
(17.13)
(03.09
(00
0.4
22
17
0.
75 0.5
177
1.6
0.5
23
33
066
12
03
0.6
16
1.0
Table 8: Performance of personalized v. pretrained representations on dense tasks. We re- port results for all generated synthetic datasets, ablating both CFG and the use of LLM--genratte prompts. The best dataset for each backbone (selected using validation performance) is highlighted in yellow, with the min/max performance over seeds reported.
25

---

Published as a conference paper at ICLR 2025
C.3 EVALUATION ON PERSAM
Here, we show that our personalized representations can be plugged into state-of-the-art pipelines for personal tasks. We experiment with PerSAM ZZhng et al. (2023). In the PerSAM method, an image encoder is used to extract patch features of one or more training images, which are compared the patch features generate keypoint proposals to prompt SAM, and guide the attention map of the SAM decoder (refer to Zhang et al. (023) for details). We evaluate PerSAM with DINOv2, and DINOv2-P as the image encoder, across DF2, Dogs, and PODS. Our results are shown in Table [ we find that DINOv2-P improves personalized segmentation performance over DINOv2 for all datasets. We use DINOv2-P trained with the best-performing dataset for the DINOv2 backbone (see Tables
test image extract a confidence map. The confidence map is then used to
Segmentation (mAP)
Segmentation (F1)
DINOv2
17.1
20.8
PODS
DINOv2-P
22.3
24.7+
DINOv2
4.8
5.8
DF2
DINOv2-P
10.0 +
10.9 +
DINOv2
16.6
20.9
Dogs
DINOv2-P
21.4 A
26.2 +
Table 9: Application to PerSAM
C.4 DIVERSITY AND FIDELITY ANALYSIS
vorsiyy Colored Accuracy
Diversity by Accurccy
Divers by Accuracy.
2D hach
Diveraly
Diveraty
Diveralyy
Figure 12: Diversity and fidelity plot colored by accuracy across PODS, DF2 and Dogs synthetic datasets. Note that the fidelity metric may be influenced by the background features appearing in the cropped image, resulting in a reduced fidelity score for some samples.
To better understand the characteristics of our generated datasets, we perform diversity and fidelity analysis in Figure 12 We measure fidelity by computing the DreamSim cosine similarity between synthetic images and the mean embedding of the reference real images, both cropped around the object of interest. To measure diversity, we compute the pairwise similarity between DreamSim features of the synthetic images in each dataset.
Notably, optimal accuracy achieved when both fidelity and diversity are sufficiently balanced too much fidelity at the expense of diversity, too much diversity with low fidelity. both lead to de- graded performance. Maintaining appropriate levels of both results in the best performance, and this be achieved in our work through Masked DreamBooth or Cut and Paste generative methods.
D QUALITATIVE RESULTS
D.1 DENSE PREDICTION VISUALIZATIONS
In Figures 13 15 we visualize dense predictions with personalized patch features for randomly sam- pled images/classes.
26

---

Published as a conference paper at ICLR 2025
Figure 13: DF2 Dense Predictions Images and classes are randomly sampled
Figure 14: Dogs Dense Predictions Images and classes randomly sampled
Caltech
Figure 15: PODS Dense Predictions Images and classes are randomly sampled
D.2 CHALLENGING EXAMPLES
We visualize examples of hard negatives and hard positives from the Dogs dataset to better under- stand the capabilities of personalized representations. For given query image x of a target object, we identify hard positives as the positives with the lowest DINOv2 cosine similarity to æ (often the dog in very different setting or position), and hard negatives the negatives with the highest
27

---

Published as a conference paper at ICLR 2025
DINOv2 cosine similarity to (often different dogs of the same breed). We denote DINOv2-P as successful on a hard negative (positive) if has a lower (higher) cosine similarity to than DINOv2.
We show randomly selected examples of hard positives/negatives in the Dogs dataset in Figures 16 and T7 respectively. DINOv2-P is typically successful on hard positives; the cosine similarity in cases with significant differences (lighting, pose, etc) from the query image. However, we also identify several cases in which the cosine similarity between query images and hard negatives also increases. This is failure case that may be induced by noisy positives in the synthetic training dataset, leading the personalized representation associate the target object with spurious features.
between positive pairs nearly always increases,
Hard Positives the Dogs Dataset
Fastiiz
Postho
Poetive
Poshiva
O DINDx2 1.85
6LE7 BONGU
DIMONE-P: D Ma
DIVOVE 6.95
Tro
Pesttve
Teatis
Poative
DIRONG 0.58
RONS O 2.78/
8-45 DINOUGH Tb/
6.72 DevOrGAt'
DIMOXIAY 9.85/
Piee
Postiss
Tostins
NONAH 3330/
U.IDeE-P 0.837
DONOlea 9.85/
DEMONG-R
Figure 16: Dogs Hard Pusitives. Given images of a target dog (leftmost of each row) we identify the positive test images with the lowest DINOv2 similarity to the query. DINOv2-P cosine similarity typically increases, even for cases with significant differences setting, camera angle, occlusion, etc.
D.3 DREAMBOOTH GENERATED DATA
We present qualitative examples for the following datasets:
PODS: DreamBooth without LLM (Figure 18 PODS DreamBooth with LLM (Figure
PODS Masked DreamBooth with LLM Filtering (Figure 19), PODS negatives (Figure
21}.
DF2: DreamBooth without LLM
DF2
DreamBooth with LLM 23
DF2 Masked
DreamBooth with LLM Filtering 24
DF2 negatives 25
Dogs: DreamBooth without LLM (Figure 266 DreamBooth with LLM (Figure 27
Masked DreamBooth with LLM + Filtering (Figure 28 and synthetic negatives (Figure
29
28

---

Published as a conference paper at ICLR 2025
Hard Negatives the Dogs Dataset
Mogariee
Nogaree
Nogstse
Negatke
Neeathv
SINDU de Blsyy
BRIGNAHH 2
Mayyr
Mogutae
Regptie
Keeathe
DENONO
DEMOND A9
DIMOUD: 0/9
CENONSLO O3
DENONG: 60
DINONG: 366
D NOVG 9.65
NOV. 3.67
Figure 17: Dogs Hard Negatives. Given images of a target dog (leftmost of each row) we identify the negative test images with the highest DINOv2 similarity to the query. In some cases DINOv2-P cosine similarity decreases (top row) however we also identify failure cases where cosine similarity increases (second/third rows).
29

---

Published as a conference paper at ICLR 2025
PODS Generated Images
Figure 18: PODS Generated Images DreamBooth without LLM prompting. Images and
classes are randomly sampled
Figure 19: PODS Generated Images Dream Booth with LLM prompting. Images and classes are randomly sampled
30

---

Published as a conference paper at ICLR 2025
Figure 20: PODS Generated Images DreamBooth with LLM, Masking and Filtering. Images and classes are randomly sampled
Figure 21: PODS Generated Negatives. Images and classes are randomly sampled. Each row are sampled negatives for each object category.
31

---

Published as a conference paper at ICLR 2025
DeepFashion2 Generated Images
Figure 22: DF2 Generated Images - DreamBooth without LLM Images and classes are randomly sampled
Figure 23: DF2 Generated Images DreamBooth with LLM Images and classes are randomly sampled
32

---

Published as a conference paper at ICLR 2025
Figure 24: DF2 Generated Images DreamBooth with LLM, Masking and Filtering. Images and classes randomly sampled
Figure 25: DF2 Generated Negatives. Images and classes are randomly sampled
33

---

Published as a conference paper at ICLR 2025
Dogs Generated Images
Figure 26: Dogs Generated Images DreamBooth without LLM prompting, Images and classes are randomly sampled
Figure 27: Dogs Generated Images - DreamBooth with LLM prompting, Images and classes are randomly sampled
34

---

Published as a conference paper at ICLR 2025
Figure 28: Dogs Generated Images DreamBooth with LLM, Masking and Filtering. Images and classes are randomly sampled
Figure 29: Dogs Generated Negatives. Images and classes are randomly sampled
35


Reviews:
[{"review_id": "wqR5ZVbFoL", "reviewer": "Reviewer_CCxb", "summary": "The paper aims to learn personalized representations given a small number of images of an object. This involves using T2I models to generate additional data followed by contrastive learning. A new dataset, PODS, was also introduced to allow for evaluations of these personalized representations under distribution shifts. Supportive results were shown for several tasks and datasets.", "questions": "- What is the runtime for the methods in Tab. 2? And what is the breakdown e.g., how much time take for each stage?\n- Methods for synthetic data generation e.g., for supervised training, tend to include a filtering step to ensure that the generations are faithful to the prompt. Was any filtering needed and how was it done?\n- What are some of the potential complications from extending the method to multiple instances?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper was well written and easy to follow.\n- The overall idea of having a personalized vision backbone that can work well on several downstream tasks is interesting.\n- The additional experiments on useful synthetic data was insightful.", "weaknesses": "- Additional baselines or comparisons.\n    - For example, generating additional data with augmentations instead of using T2I models is a cheap baseline. If this is done in real-aug (Tab. 2) and it is comparable to the results of Tab. 1, why does it seem to perform worse than no personalization?\n    - It may also be useful to compare against using real data only to get an upper bound of the method.\n    - It would be interesting to see if, with more real images, cut/paste would outperform Masked DB. I.e., if we can tradeoff sampling more images for a faster runtime.\n- The method seems to be computationally expensive as it involves several stages of finetuning and generation. It may be useful to include the runtimes of each method beside the results in e.g., Tab 2.\n- The method learns a personalized representation for a single instance, a more realistic scenario probably involves several personalized instances e.g., one in each object category in PODS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1MsmpIYjj3", "reviewer": "Reviewer_H1To", "summary": "This paper investigates using personalized synthetic images, alongside a few real images, to develop personalized visual representations that enhance performance across multiple downstream tasks, including classification, retrieval, detection, and segmentation. \n\nThe authors propose a novel contrastive learning framework where a personalized representation is trained using only three real positive samples without any real negatives. To support the evaluation of this approach, they introduce the Personal Object Discrimination Suite (PODS) dataset, specifically crafted for testing under conditions such as pose variation and background distractors, along with reformulations of DeepFashion2 and DogFaceNet datasets for the same purpose. By leveraging generative models, particularly DreamBooth with masked training, they demonstrate the effectiveness of synthetic data for learning representations of specific instances. This personalized approach shows promising improvements over general-purpose pretrained representations.", "questions": "In addition to the questions in the \"weaknesses\" section:\n\n1. In Table 1, are the results obtained with or without access to segmentation masks? This point is somewhat unclear—could the authors clarify if this information is noted elsewhere in the paper?\n\n2. For tasks such as detection and segmentation, where the performance improvement was less pronounced, do the authors have plans to refine the pipeline for further gains in these areas? If so, could they share any specific strategies or potential modifications they are considering?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a creative combination of generative models and contrastive learning to address the challenge of instance-specific visual representation with minimal real data. \n\nThe experimental framework is thorough, covering classification, retrieval, detection, and segmentation tasks across three datasets (DeepFashion2, DogFaceNet, and the newly introduced PODS). The results consistently highlight the advantage of personalized representations over pre-trained ones, demonstrating the robustness of the approach. The author also shows that the method could be integrated into existing pipelines.\n\nThe paper is well-organized and has clear explanations. Multiple figures, including pipelines and qualitative results, aid in understanding. Writing is accessible without unnecessary complexity, making the proposed approach and findings straightforward.", "weaknesses": "1. It looks unnecessary to exclude real negatives in the proposed setting of personalized representation learning. Unlike real positives, real negatives might be easily obtained directly from open-source data. The paper relies on generated negatives produced by the generative model, which can be computationally costly. Alternatively, obtaining real negatives from readily available online sources might be a more efficient solution. Could the authors provide additional insights into why they excluded real negatives and whether this impacts performance? Are there any existing works exploring the setting of including real negatives?\n\n2. Figure 1 is somewhat unclear and could be improved to more explicitly illustrate the flow and connection between real and synthetic data in the proposed framework. Additionally, it is not mentioned in the main texts. More explanation may also help.\n\n3. The term \"real-augmentation\" is mentioned but not clearly defined in the context of this paper, which could lead to confusion regarding its role and implementation. By line 473, it seems that the real-aug is just using the real images without any augmentation to train the backbone models. If so, \"real images\" might be a better name than \"real-aug.\"\n\n4. The method of real augmentation on the PODS dataset shows degraded performance compared to pre-trained representations in retrieval, detection, and segmentation tasks, especially in retrieval (combining information from Table 1 and Table 2). However, this phenomenon does not appear in the other two datasets. This anomaly is not fully explored, and further insight into the underlying cause would help readers understand any limitations in generalizing the approach across different datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DHQqgk6hFM", "reviewer": "Reviewer_FvPe", "summary": "This paper addresses the challenge of learning personalized representations. The authors utilize a generative model to augment the dataset for the target personalized object, followed by fine-tuning a pretrained model using contrastive learning. This approach aims to develop a model capable of handling downstream tasks related to the specific object. Several downstream tasks were evaluated, and the proposed method demonstrated an average improvement across these tasks.", "questions": "1. Are there any real-world scenarios where the benefits of personalized representation outweigh the high training costs? \n\n2. In certain challenging tasks, like distinguishing between dog faces of the same breed, it might be helpful to present more difficult or even failure cases to showcase the capabilities of the proposed method. \n\n3. Why does the proposed method sometimes lead to a decline in final performance for certain tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper explores a novel and intriguing problem: learning personalized representations, which could be valuable for downstream tasks related to target objects. The setting is innovative and promising. \n\n2. The proposed method is both simple and effective, as the authors employ an image customization technique to augment the dataset and address the challenge of limited training data. \n\n3. Overall, the paper is well-structured and easy to read.", "weaknesses": "1. While the problem being studied is intriguing, the overall approach is relatively basic. The concepts of using image generation to augment the dataset and contrastive learning are not novel. \n\n2. Additionally, incorporating image generation could significantly increase the cost of the method. \n\n3. Moreover, in certain tasks, the method results in a decline in performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2BjARg5gKK", "reviewer": "Reviewer_2ShX", "summary": "The paper explores the use of synthetic data to train personalized visual representations, addressing data scarcity challenges in fine-grained, instance-specific vision tasks. The authors propose a three-stage pipeline involving synthetic image generation, contrastive learning, and fine-tuning of a general-purpose pretrained model using only three real examples of an object. They introduce an evaluation suite, including two reformulated datasets (DeepFashion2 and DogFaceNet) and a novel PODS dataset, to evaluate the effectiveness of their approach across tasks such as classification, retrieval, detection, and segmentation. Results suggest that the proposed method outperforms pretrained models on these tasks and provides an analysis of the impact of generative models (e.g., DreamBooth) on representation quality. However, this work lacks certain methodological and experimental rigor, as discussed in the critique below.", "questions": "Given the reliance on DreamBooth, which is computationally intensive, have the authors considered alternative  generative models  to achieve similar outcomes? \nHow does the computational cost compare to simpler data augmentation methods, and what trade-offs are involved in model performance?\nHas the study examined the potential biases and limitations introduced by synthetic data in the personalized learning setting, especially in comparison to real data? Could the authors provide any quantitative result to show how the synthetic data representations compare with those learned from real data, particularly in terms of generalization?\nThe proposed approach has clear applications in sensitive areas, such as surveillance or individual tracking, raising potential privacy concerns. How do the authors propose to mitigate these risks,?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents an interesting approach to personalized visual representation learning using synthetic data, offering a novel perspective on addressing data scarcity in fine-grained instance-specific tasks. \nThe study showcases the potential of synthetic data in personalized representation learning, especially for data-limited tasks, indicating that personalized generative models could contribute to a broader field of personalized AI applications.\nThe PODS dataset is a valuable addition to the field, providing a benchmark for evaluating the performance of personalized visual models across multiple tasks. Although limited, it may be useful in future research for controlled comparisons in low-data personalized settings.", "weaknesses": "High Computational Cost: The approach’s dependency on DreamBooth and other costly generative models for fine-tuning makes it impractical for widespread use. By not considering more computationally feasible alternatives, such as GAN-based models, the method lacks the flexibility needed for real-world adoption. Lack of Baseline Comparisons and Systematic Experimentation: The paper does not provide enough rigorous comparisons against simpler, cheaper baselines that could yield similar or competitive results, such as training with real, small datasets or using augmentation strategies on a limited set of real images. This gap leaves the reader questioning the unique benefit of using complex synthetic data pipelines, particularly in relation to common data-efficient techniques in computer vision.\nLimited Task Scope and Lack of Real-World Validation: Although the paper claims improvements across multiple tasks, the experimental setup is relatively narrow, focusing on a limited range of controlled settings. Given that the proposed approach aims for personalization, testing its robustness in diverse and complex real-world scenarios is essential. Additionally, the observed performance gains in detection and segmentation tasks are relatively small, raising concerns about the method's robustness. Ethical and Privacy Considerations: While the authors briefly address these considerations, they do not offer practical guidelines or safeguards to mitigate potential misuse. Expanding on ethical implications and providing specific measures for responsible deployment would improve the paper’s comprehensiveness. Inadequate Analysis of Data Diversity: Although the authors highlight the role of data diversity in improving representation quality, they fail to conduct a thorough, quantitative analysis of the impact of diversity on representation learning. The paper lacks a clear quantification of diversity’s effect on model performance, and only discusses a few configuration parameters, which does not sufficiently explore the potential of synthetic data augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9zUaZSROS5", "reviewer": "Reviewer_eQii", "summary": "This paper addresses the challenge of using modern vision models for personalized vision tasks, which are often fine-grained and limited in data availability. The authors propose a method that combines insights from synthetic data in general representation learning and advances in T2I (text-to-image) diffusion models, leveraging personalized synthetic data to learn representations specific to an object of interest. To support this goal, they introduce an evaluation suite consisting of reformulated datasets and a novel dataset designed for personalized learning. The proposed contrastive learning approach uses image generators to enhance representation learning for diverse downstream tasks, such as recognition and segmentation.", "questions": "- Distillation vs. Personalized Fine-Tuning: Could the authors clarify whether the observed performance gains primarily result from their proposed contrastive method or from pre-existing knowledge in Stable Diffusion 1.5? Would a comparison with a straightforward fine-tuning approach on a large set of synthetic positives help isolate the benefits of the proposed method?\n- Contrastive Learning Setup and Objectives: Regarding the contrastive learning approach, could the authors elaborate on its intended objectives? If the primary aim is to enhance personalized representation capabilities at inference, would adding a baseline using simple LORA or finetuning with I2T+LLM+Stable Diffusion positives be beneficial? Alternatively, if there is a defensive objective to reduce performance for non-target objects, would the authors consider providing results on unrelated objects?\n- Clarification of Typos and Formatting: There are minor typos and formatting inconsistencies throughout the text (e.g., “e.g.” should be formatted as “e.g.,”). Would the authors consider a thorough review to address these to improve readability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Clarity and Structure: The paper is well-organized and clearly written, making it accessible and easy to follow, even for readers who may be less familiar with the technical aspects of personalized representation learning in vision models.\n- Visualization Quality: The visualizations of generated images are well-designed and effectively demonstrate the model’s representation capabilities, enhancing the clarity and impact of the experimental results.\n- Meaningful Personalized Representation: The concept of learning personalized representations for specific objects is a valuable contribution, offering potential applications across various fine-grained and data-scarce tasks where general-purpose representations may fall short.", "weaknesses": "- Effectiveness of contrastive learning: It is unclear if the observed performance gains stem from the proposed method or from the inherent knowledge in Stable Diffusion 1.5. In the appendix, “Table 4: Ablation on the Number of Anchor-Positive Pairs” indicates poor performance when the “# Synthetic Imgs” and “# Anchor-Pos Pairs” are small. Could the authors provide evidence that the improvement in representation learning is not primarily due to the knowledge embedded in Stable Diffusion? A comparison between the proposed contrastive approach and straightforward fine-tuning with the same number of synthetic positives (maybe I2T+LLM+Stable Diffusion) would help clarify this point.\n- Contrastive Learning Setup and Objectives: The purpose and setting of the contrastive learning approach remain ambiguous. If the objective is solely to improve personalized representation at inference, it would be useful to include more baselines to demonstrate the effectiveness of your contrastive learning, such as using simple LORA/finetuning on positives obtained from I2T+LLM+Stable Diffusion. Conversely, if the goal includes a defensive aspect, ensuring that performance declines for objects other than the target, results showing the performance on unrelated objects would provide valuable context.\n- Typos and Formatting:\n  - Line 53: \"e.g. a model\" should be \"e.g., a model\".\n  - Line 59: \"e.g. recognizing\" should be \"e.g, recognizing\".\n  - Line 78: \"i.e. no\" should be \"i.e., no\".\n  - Line 153: \"e.g. segmentation\" should be \"e.g., segmentation\".\n  - Line 162: \"e.g. one\" should be \"e.g., one\".\n  - Line 348: ”query” and ”retrieval” should be formatted as ``query``” and “``retrieval``.\n  - Figure 2: Missing period. \"Pipeline Our\" should be \"Pipeline. Our\".\n  - Figure 5: \"following 4. \" should be \"following Figure 4. \".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)