Personalized Representation from Personalized Generation

Authors: Shobhita Sundaram, Julia Chae, Yonglong Tian, Sara Beery, Phillip Isola
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Synthetic data, personalization, diffusion models, data augmentation, representation learning

Abstract:
Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730110843236

thread_type:
official_review

<thread_object>
creator: reviewer_eqii

<content>
summary:
This paper addresses the challenge of using modern vision models for personalized vision tasks, which are often fine-grained and limited in data availability. The authors propose a method that combines insights from synthetic data in general representation learning and advances in T2I (text-to-image) diffusion models, leveraging personalized synthetic data to learn representations specific to an object of interest. To support this goal, they introduce an evaluation suite consisting of reformulated datasets and a novel dataset designed for personalized learning. The proposed contrastive learning approach uses image generators to enhance representation learning for diverse downstream tasks, such as recognition and segmentation.

strengths:
- Clarity and Structure: The paper is well-organized and clearly written, making it accessible and easy to follow, even for readers who may be less familiar with the technical aspects of personalized representation learning in vision models.
- Visualization Quality: The visualizations of generated images are well-designed and effectively demonstrate the model’s representation capabilities, enhancing the clarity and impact of the experimental results.
- Meaningful Personalized Representation: The concept of learning personalized representations for specific objects is a valuable contribution, offering potential applications across various fine-grained and data-scarce tasks where general-purpose representations may fall short.

weaknesses:
- Effectiveness of contrastive learning: It is unclear if the observed performance gains stem from the proposed method or from the inherent knowledge in Stable Diffusion 1.5. In the appendix, “Table 4: Ablation on the Number of Anchor-Positive Pairs” indicates poor performance when the “# Synthetic Imgs” and “# Anchor-Pos Pairs” are small. Could the authors provide evidence that the improvement in representation learning is not primarily due to the knowledge embedded in Stable Diffusion? A comparison between the proposed contrastive approach and straightforward fine-tuning with the same number of synthetic positives (maybe I2T+LLM+Stable Diffusion) would help clarify this point.
- Contrastive Learning Setup and Objectives: The purpose and setting of the contrastive learning approach remain ambiguous. If the objective is solely to improve personalized representation at inference, it would be useful to include more baselines to demonstrate the effectiveness of your contrastive learning, such as using simple LORA/finetuning on positives obtained from I2T+LLM+Stable Diffusion. Conversely, if the goal includes a defensive aspect, ensuring that performance declines for objects other than the target, results showing the performance on unrelated objects would provide valuable context.
- Typos and Formatting:
  - Line 53: "e.g. a model" should be "e.g., a model".
  - Line 59: "e.g. recognizing" should be "e.g, recognizing".
  - Line 78: "i.e. no" should be "i.e., no".
  - Line 153: "e.g. segmentation" should be "e.g., segmentation".
  - Line 162: "e.g. one" should be "e.g., one".
  - Line 348: ”query” and ”retrieval” should be formatted as ``query``” and “``retrieval``.
  - Figure 2: Missing period. "Pipeline Our" should be "Pipeline. Our".
  - Figure 5: "following 4. " should be "following Figure 4. ".

questions:
- Distillation vs. Personalized Fine-Tuning: Could the authors clarify whether the observed performance gains primarily result from their proposed contrastive method or from pre-existing knowledge in Stable Diffusion 1.5? Would a comparison with a straightforward fine-tuning approach on a large set of synthetic positives help isolate the benefits of the proposed method?
- Contrastive Learning Setup and Objectives: Regarding the contrastive learning approach, could the authors elaborate on its intended objectives? If the primary aim is to enhance personalized representation capabilities at inference, would adding a baseline using simple LORA or finetuning with I2T+LLM+Stable Diffusion positives be beneficial? Alternatively, if there is a defensive objective to reduce performance for non-target objects, would the authors consider providing results on unrelated objects?
- Clarification of Typos and Formatting: There are minor typos and formatting inconsistencies throughout the text (e.g., “e.g.” should be formatted as “e.g.,”). Would the authors consider a thorough review to address these to improve readability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619604538

thread_type:
official_review

<thread_object>
creator: reviewer_2shx

<content>
summary:
The paper explores the use of synthetic data to train personalized visual representations, addressing data scarcity challenges in fine-grained, instance-specific vision tasks. The authors propose a three-stage pipeline involving synthetic image generation, contrastive learning, and fine-tuning of a general-purpose pretrained model using only three real examples of an object. They introduce an evaluation suite, including two reformulated datasets (DeepFashion2 and DogFaceNet) and a novel PODS dataset, to evaluate the effectiveness of their approach across tasks such as classification, retrieval, detection, and segmentation. Results suggest that the proposed method outperforms pretrained models on these tasks and provides an analysis of the impact of generative models (e.g., DreamBooth) on representation quality. However, this work lacks certain methodological and experimental rigor, as discussed in the critique below.

strengths:
The paper presents an interesting approach to personalized visual representation learning using synthetic data, offering a novel perspective on addressing data scarcity in fine-grained instance-specific tasks. 
The study showcases the potential of synthetic data in personalized representation learning, especially for data-limited tasks, indicating that personalized generative models could contribute to a broader field of personalized AI applications.
The PODS dataset is a valuable addition to the field, providing a benchmark for evaluating the performance of personalized visual models across multiple tasks. Although limited, it may be useful in future research for controlled comparisons in low-data personalized settings.

weaknesses:
High Computational Cost: The approach’s dependency on DreamBooth and other costly generative models for fine-tuning makes it impractical for widespread use. By not considering more computationally feasible alternatives, such as GAN-based models, the method lacks the flexibility needed for real-world adoption. Lack of Baseline Comparisons and Systematic Experimentation: The paper does not provide enough rigorous comparisons against simpler, cheaper baselines that could yield similar or competitive results, such as training with real, small datasets or using augmentation strategies on a limited set of real images. This gap leaves the reader questioning the unique benefit of using complex synthetic data pipelines, particularly in relation to common data-efficient techniques in computer vision.
Limited Task Scope and Lack of Real-World Validation: Although the paper claims improvements across multiple tasks, the experimental setup is relatively narrow, focusing on a limited range of controlled settings. Given that the proposed approach aims for personalization, testing its robustness in diverse and complex real-world scenarios is essential. Additionally, the observed performance gains in detection and segmentation tasks are relatively small, raising concerns about the method's robustness. Ethical and Privacy Considerations: While the authors briefly address these considerations, they do not offer practical guidelines or safeguards to mitigate potential misuse. Expanding on ethical implications and providing specific measures for responsible deployment would improve the paper’s comprehensiveness. Inadequate Analysis of Data Diversity: Although the authors highlight the role of data diversity in improving representation quality, they fail to conduct a thorough, quantitative analysis of the impact of diversity on representation learning. The paper lacks a clear quantification of diversity’s effect on model performance, and only discusses a few configuration parameters, which does not sufficiently explore the potential of synthetic data augmentation.

questions:
Given the reliance on DreamBooth, which is computationally intensive, have the authors considered alternative  generative models  to achieve similar outcomes? 
How does the computational cost compare to simpler data augmentation methods, and what trade-offs are involved in model performance?
Has the study examined the potential biases and limitations introduced by synthetic data in the personalized learning setting, especially in comparison to real data? Could the authors provide any quantitative result to show how the synthetic data representations compare with those learned from real data, particularly in terms of generalization?
The proposed approach has clear applications in sensitive areas, such as surveillance or individual tracking, raising potential privacy concerns. How do the authors propose to mitigate these risks,?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645705133

thread_type:
official_review

<thread_object>
creator: reviewer_fvpe

<content>
summary:
This paper addresses the challenge of learning personalized representations. The authors utilize a generative model to augment the dataset for the target personalized object, followed by fine-tuning a pretrained model using contrastive learning. This approach aims to develop a model capable of handling downstream tasks related to the specific object. Several downstream tasks were evaluated, and the proposed method demonstrated an average improvement across these tasks.

strengths:
1. The paper explores a novel and intriguing problem: learning personalized representations, which could be valuable for downstream tasks related to target objects. The setting is innovative and promising. 

2. The proposed method is both simple and effective, as the authors employ an image customization technique to augment the dataset and address the challenge of limited training data. 

3. Overall, the paper is well-structured and easy to read.

weaknesses:
1. While the problem being studied is intriguing, the overall approach is relatively basic. The concepts of using image generation to augment the dataset and contrastive learning are not novel. 

2. Additionally, incorporating image generation could significantly increase the cost of the method. 

3. Moreover, in certain tasks, the method results in a decline in performance.

questions:
1. Are there any real-world scenarios where the benefits of personalized representation outweigh the high training costs? 

2. In certain challenging tasks, like distinguishing between dog faces of the same breed, it might be helpful to present more difficult or even failure cases to showcase the capabilities of the proposed method. 

3. Why does the proposed method sometimes lead to a decline in final performance for certain tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702853260

thread_type:
official_review

<thread_object>
creator: reviewer_h1to

<content>
summary:
This paper investigates using personalized synthetic images, alongside a few real images, to develop personalized visual representations that enhance performance across multiple downstream tasks, including classification, retrieval, detection, and segmentation. 

The authors propose a novel contrastive learning framework where a personalized representation is trained using only three real positive samples without any real negatives. To support the evaluation of this approach, they introduce the Personal Object Discrimination Suite (PODS) dataset, specifically crafted for testing under conditions such as pose variation and background distractors, along with reformulations of DeepFashion2 and DogFaceNet datasets for the same purpose. By leveraging generative models, particularly DreamBooth with masked training, they demonstrate the effectiveness of synthetic data for learning representations of specific instances. This personalized approach shows promising improvements over general-purpose pretrained representations.

strengths:
The paper presents a creative combination of generative models and contrastive learning to address the challenge of instance-specific visual representation with minimal real data. 

The experimental framework is thorough, covering classification, retrieval, detection, and segmentation tasks across three datasets (DeepFashion2, DogFaceNet, and the newly introduced PODS). The results consistently highlight the advantage of personalized representations over pre-trained ones, demonstrating the robustness of the approach. The author also shows that the method could be integrated into existing pipelines.

The paper is well-organized and has clear explanations. Multiple figures, including pipelines and qualitative results, aid in understanding. Writing is accessible without unnecessary complexity, making the proposed approach and findings straightforward.

weaknesses:
1. It looks unnecessary to exclude real negatives in the proposed setting of personalized representation learning. Unlike real positives, real negatives might be easily obtained directly from open-source data. The paper relies on generated negatives produced by the generative model, which can be computationally costly. Alternatively, obtaining real negatives from readily available online sources might be a more efficient solution. Could the authors provide additional insights into why they excluded real negatives and whether this impacts performance? Are there any existing works exploring the setting of including real negatives?

2. Figure 1 is somewhat unclear and could be improved to more explicitly illustrate the flow and connection between real and synthetic data in the proposed framework. Additionally, it is not mentioned in the main texts. More explanation may also help.

3. The term "real-augmentation" is mentioned but not clearly defined in the context of this paper, which could lead to confusion regarding its role and implementation. By line 473, it seems that the real-aug is just using the real images without any augmentation to train the backbone models. If so, "real images" might be a better name than "real-aug."

4. The method of real augmentation on the PODS dataset shows degraded performance compared to pre-trained representations in retrieval, detection, and segmentation tasks, especially in retrieval (combining information from Table 1 and Table 2). However, this phenomenon does not appear in the other two datasets. This anomaly is not fully explored, and further insight into the underlying cause would help readers understand any limitations in generalizing the approach across different datasets.

questions:
In addition to the questions in the "weaknesses" section:

1. In Table 1, are the results obtained with or without access to segmentation masks? This point is somewhat unclear—could the authors clarify if this information is noted elsewhere in the paper?

2. For tasks such as detection and segmentation, where the performance improvement was less pronounced, do the authors have plans to refine the pipeline for further gains in these areas? If so, could they share any specific strategies or potential modifications they are considering?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712315033

thread_type:
official_review

<thread_object>
creator: reviewer_ccxb

<content>
summary:
The paper aims to learn personalized representations given a small number of images of an object. This involves using T2I models to generate additional data followed by contrastive learning. A new dataset, PODS, was also introduced to allow for evaluations of these personalized representations under distribution shifts. Supportive results were shown for several tasks and datasets.

strengths:
- The paper was well written and easy to follow.
- The overall idea of having a personalized vision backbone that can work well on several downstream tasks is interesting.
- The additional experiments on useful synthetic data was insightful.

weaknesses:
- Additional baselines or comparisons.
    - For example, generating additional data with augmentations instead of using T2I models is a cheap baseline. If this is done in real-aug (Tab. 2) and it is comparable to the results of Tab. 1, why does it seem to perform worse than no personalization?
    - It may also be useful to compare against using real data only to get an upper bound of the method.
    - It would be interesting to see if, with more real images, cut/paste would outperform Masked DB. I.e., if we can tradeoff sampling more images for a faster runtime.
- The method seems to be computationally expensive as it involves several stages of finetuning and generation. It may be useful to include the runtimes of each method beside the results in e.g., Tab 2.
- The method learns a personalized representation for a single instance, a more realistic scenario probably involves several personalized instances e.g., one in each object category in PODS.

questions:
- What is the runtime for the methods in Tab. 2? And what is the breakdown e.g., how much time take for each stage?
- Methods for synthetic data generation e.g., for supervised training, tend to include a filtering step to ensure that the generations are faithful to the prompt. Was any filtering needed and how was it done?
- What are some of the potential complications from extending the method to multiple instances?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732204255686

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all of the reviewers for their insightful and helpful feedback. We are glad that they found:
* The paper was well-written [CCxb,H1To,FvPe,eQii]
* The experiments are insightful/thorough [CCxb,H1To]
* The paper studies an interesting, novel problem [CCxb,FvPe,2SHx,eQii]

We highlight key revisions (and answer individual questions in reviewer-specific responses). **Updates are blue in our revised PDF**.

**1. ADDITIONAL COMPARISONS**

**We extend our comparisons/baselines to study the tradeoffs** between training on T2I-generated images – which is computationally costly, but does not require further data/annotation – and cheap alternatives that leverage additional data. 

Segmentation masks enable Cut/Paste (CP) with generated backgrounds, and Masked DreamBooth (Masked DB). In our revision we add two new cases:

**1) Internet-available data:** We use open-source vision datasets as a source of non-customized real data. These can be used as *real negatives* for contrastive learning and *real-backgrounds* for CP. This enables the following comparisons:
* A real-only baseline: (real images from $\mathcal{D}_R$ as positives, real negatives). 
* CP with real backgrounds and real negatives
* Generated (Masked DB) positives with real negatives.

**Results are in Table 2 of our revision.** We find:
* Methods using open-source real images perform similarly to using generated negatives or CP backgrounds.
* Methods using synthetic positives outperform the real-only baseline; Masked DB + CP perform best.
* Strong performance is possible with a compute-efficient alternative (CP with real backgrounds and real negatives).

**2) Scaling real positives:** A user may manually expand $\mathcal{D}_R$ by collecting extra real positives. **In Fig. 6 of our revision we scale $\mathcal{D}_R$** and evaluate training with:
* Real images from the larger $\mathcal{D}_R$
* The best-performing synthetic method (combined CP and Masked DB) using the larger $\mathcal{D}_R$.

We manually collect extra real training images for 25 instances of PODS (5 per category), and show results for real-image training on $|\mathcal{D}_R| \in \\\{3, 5, 10, 15, 20\\\}$. Due to time constraints, we show synthetic results at $|\mathcal{D}_R| \in \\\{3, 20\\\}$. We cannot replicate this experiment on DF2/Dogs due to the limited positives per class.

We observe:
* Real-image training saturates at n=15. While our extra images include new backgrounds/poses/lighting, diversity can still be limited, which we show is key to accuracy gains via **diversity/fidelity analysis in Figure 12**. We aimed to mimic users photographing their objects; thus our training images mainly are single-object images in indoor environments. Better scaling may be possible with significant extra diversity (though this requires intentional, potentially labor-intensive, data collection).   
* Even after real images saturate, synthetic data improves performance. This is likely because the generator (Masked DB + CP) can more readily achieve per-image diversity.

**2. COMPUTATIONAL COST/RUNTIME**

**We report runtimes of each synthetic data method in Table 3** and find a speed v. performance tradeoff. Masked DB + CP performs best, but is also expensive; the real-image baseline is extremely cheap but performs worst. However, performance gains are also achievable without T2I models. CP with real-backgrounds/real-negatives – the fastest method in Table 3 – outperforms the real-image baseline. Thus our results do not rely solely on T2I models, although these help to achieve the best results. 

There are also domains where high performance/robustness is critical, and downstream accuracy gains outweigh computational cost:
* Personalized vision systems are useful for identifying personal items of blind or low-vision people [1]. 
* In ecology, re-identifying specific animals is an open problem; real data collection is often extremely expensive/difficult. 
* Personalization is becoming increasingly useful in medical imaging.

**3. CONTRIBUTION w.r.t COST**

We emphasize that our main contribution – a framework for personalized representations, and exploration of performance benefits – is agnostic to the choice of generator. In future, especially considering the recent pace of progress, generation and fine-tuning will only become more efficient. While we focus on elucidating the *benefits* of personalized representations, we are optimistic that in the future, more efficient techniques can be easily plugged in to further reduce the *cost*. **We have revised our introduction to clarify our contributions.**

**4. CORRECTED RESULTS**

We found a small bug related to image loading, and have updated results to reflect the correction. There is very little change, however we highlight that in Table 1, **personalized backbones outperform pretrained backbones in 33/36 cases.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to address the challenge of using synthetic data to enhance personalized visual representation learning for fine-grained and data-scarce tasks. The authors propose a contrastive learning framework leveraging text-to-image generative models, like DreamBooth, to create synthetic images for instance-specific representation learning. 

This paper received mixed ratings before the rebuttal period. After reading the author's responses, three of the four reviewers upgraded their rating to the positive side or decided to keep their positive ratings. The only one who didn't change their score is Reviewer 2ShX. However, they didn't actively participate in the discussions. Considering all of the above, it is decided to accept the paper. The authors are required to include the discussions in their final version.