GenDataAgent: On-the-fly Dataset Augmentation with Synthetic Data

Authors: Zhiteng Li, Lele Chen, Jerone Andrews, Yunhao Ba, Yulun Zhang, Alice Xiang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: supervised learning, classification, computer vision, synthetic data, generative AI, responsible AI, fairness

Abstract:
We propose a generative agent that augments training datasets with synthetic data for model fine-tuning. Unlike prior work, which uniformly samples synthetic data, our agent iteratively generates relevant samples on-the-fly, aligning with the target distribution. It prioritizes synthetic data that complements difficult training samples, focusing on those with high variance in gradient updates. Experiments across several image classification tasks demonstrate the effectiveness of our approach.

Decision: accept (poster)
Reviewer scores: 3, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730235590986

thread_type:
official_review

<thread_object>
creator: reviewer_nmji

<content>
summary:
This paper investigates how to leverage synthetic data generated by text-to-image generation models, such as Stable Diffusion (SD), to enhance image classification performance in terms of both accuracy and fairness. The authors propose a sampling strategy that prioritizes sample utility, which is determined by the proximity of a sample to the decision boundary, as well as in-distributional data samples, using the VoG score. By integrating a series of reasonable treatment, the proposed method achieves state-of-the-art performance across different image classification benchmarks.

strengths:
The paper is well-written, providing clear motivations and explanations of the proposed method. The introduced metric for valuing synthetic data samples, based on their utility and proximity to the decision boundary, is reasonable. The integration of different techniques, such as perturbing text prompts to enhance diversity and ensuring alignment with the target distribution, appears well-justified and appropriate for the stated problem. The proposed method also demonstrates state-of-the-art performance across all kinds of image classification benchmarks.

weaknesses:
The paper's critical weakness lies in its lack of novelty, as most of the techniques employed, including the VoG metric, are derived from previous works. This aspect positions the paper more as an engineering-heavy work rather than an original contribution. Specifically, the extensively discussed VoG metric appears somewhat subjective, as the choice of using gradients from epochs 10, 20, and 30 to compute the metric lacks proper justification or guidelines for other benchmarks or tasks. Moreover, while the proposed method demonstrates significant improvement over using only real data, it performs only marginally better than other synthetic augmentation methods, such as Real-Fake and Internet Explorer, while also taking much longer GPU hours. In fact, synthetic data augmentation has been extensively studied in the context of image classification, and the observed improvement may merely be a result of overfitting to those small-scale benchmarks and may not generalize to real-world or in-the-wild settings.

questions:
I'm willing to raise the score if the following questions are well-addressed:

1. Can you also include robustness metrics, such as accuracy under common corruptions and perturbations as well as adversarial settings? Is the proposed method still better than the baselines? 
2. Would the proposed method generalize across different Text2Image generation methods, such as SD3 and Kandinsky (does not have to be the two)?
3. Would the proposed method generalize to other discriminative CV tasks such as image retrieval or object detection? 
4. Line 474, the statement that accuracy is highly correlated with the synthetic data volume seems to be contradictory with a concurrent ICLR25 submission (Mousterian: exploring the equivalence of generative and real data augmentation in classification). In that work, they state that synthetic data is more valuable when the quantity is small, yet its value diminishes quickly rather than staying correlated with accuracy. 
5. The discussion on overfitting in line 483 requires more clarity and experimental evidence to support the strong statement made. The current evidence on Pets dataset appears quite weak and insufficient.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626602525

thread_type:
official_review

<thread_object>
creator: reviewer_rpal

<content>
summary:
The paper presents a novel approach for training dataset augmentation in computer vision using a generative agent, GenDataAgent. This method addresses the limitations of previous works that uniformly search across the category space and fail to consider the interaction between synthetic data generation and downstream task training. The proposed agent generates high-quality synthetic data on-the-fly, ensuring alignment with the target training dataset distribution while prioritizing diversity and relevance. Evaluations demonstrate that this approach enhances the generalization performance of downstream models fine-tuned on the augmented datasets.

strengths:
- The introduction of GenDataAgent for on-the-fly synthetic data generation represents a significant advancement in training dataset augmentation, addressing the critical challenges of distribution alignment and sample diversity.

- The method effectively enhances the generalization performance of downstream models by prioritizing the sampling of diverse synthetic data that complements marginal training samples, which is crucial for improving model robustness.

- Unlike prior research, this work does not rely on specific distributional assumptions, making the approach more versatile and applicable to a wider range of datasets, including those with varying class distributions and limited examples.

weaknesses:
- There exists a conflict: tailoring synthetic data generation for specific downstream tasks limits its generality, while using it for general purposes diminishes its specialty. The authors could provide further insights to clarify this issue.

- Is it necessary for synthetic data to be in-distribution? While aligning the distribution of synthetic data with the training data is important, it does not align with the distribution of the test data, particularly in out-of-distribution (OOD) scenarios. This raises the question of whether this paper focuses solely on generalization within in-distribution contexts, potentially limiting its applicability to real-world situations where test data may vary significantly. The authors may evaluate the proposed agent's capability for OOD generalization. 

- Though the on-the-fly generation of relevant data saves the storage cost and improves relevance to the task at hand, how reusable and sustainable is synthetic data?

- For marginal training samples, synthetic data that exhibit higher variance in gradient updates are specifically chosen. Does this imply that these synthetic samples are those that the current model struggles to learn effectively? If so, it suggests that these samples may be more challenging or informative, potentially indicating areas where the model's performance is weak. This focus on high-variance samples could enhance the model's robustness by addressing its limitations and improving its ability to generalize to diverse data distributions. Further clarification on this selection strategy and its implications for model training would be beneficial, e.g., discussing it with the selection criteria in active learning.

- This paper would benefit from a more comprehensive literature review. Specifically, it overlooks some closely related works, such as [a]. I recommend that the authors include a discussion of this work and its connections to their research. This will help to contextualize their contributions within the existing body of knowledge.

[a] A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation. CVPR, 2023.

- The authors claim that maintaining a consistent prompt format for both adapting stable diffusion and generating synthetic data ensures greater alignment between the distribution of the generated data and the target data. However, it is not entirely clear how a consistent prompt format alone guarantees this distribution alignment. I recommend that the authors provide a more detailed explanation of this relationship.

- Additionally, it would be valuable to discuss whether incorporating domain adaptation (DA) techniques, as described in [a], during the model fine-tuning process could lead to further performance improvements. Exploring this potential synergy could enhance the robustness of the findings presented in the manuscript.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661210346

thread_type:
official_review

<thread_object>
creator: reviewer_rgcl

<content>
summary:
The paper introduces the on-the-fly data augmentation method. The method first selects important samples using a marginal score. It then performs caption perturbations to increase the diversity of the data. Finally, outliers are filtered out using the gradients at the initial training iterations. The experiments show that the method effectively improves the accuracy.

strengths:
- The paper is easy to follow. 
- The paper includes clear motivation and a detailed analysis of the proposed framework. 
- The experiments show the benefit of the proposed on-the-fly data augmentation method.

weaknesses:
The proposed method is slow --- it can be up to 350 times slower than using only real data and 2 times slower than the comparing method (Real-Fake). Although the accuracy increases, this can be a major bottleneck.

questions:
Related to the speed issue, I assume the reason is because it's the on-the-fly method. Can the method be adapted/switched between online and offline modes based on its need? Are there other ways to improve the speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730883631812

thread_type:
official_review

<thread_object>
creator: reviewer_ln52

<content>
summary:
The paper presents an algorithm for selecting synthetic data generated from Stable Diffusion, adjusting their contribution, and modifying the generation diversity for better closed-set classification model training as the downstream application. The main hypothesis is around the introduction of downstream training signal to provide feedback to the model generation process to better guide and use generated samples for discriminative model training.

strengths:
The intuition of guiding the sample generation process from signals or feedback from the downstream model training process makes sense. The paper has presented results of both training completely with synthetic data and using synthetic data as augmentations. And they show the proposed method outperforms baselines compared.

weaknesses:
Lack of technical novelty. Though the motivation of the idea makes sense, my biggest concern is on the technical novelty of the proposed algorithm. The proposed design components on ensuring sample diversity, generation distribution alignment with real data distribution, and the marginal sample selection are all coming from existing literature or prior works. 

Lack of sufficient experiment results to validate the effectiveness of the proposed methods and the claims in the contributions. The contribution claims the effectiveness of the proposed method from the three aspects of marginal sample selection, sample diversity and the distribution alignment to real data distribution. However, from the ablation study, it appears that even without these proposed improvements, the classification model trained with synthetic data also show good improvements. And the improvements from the proposed claims are relatively smaller compared to the gains from directly using synthetic samples from SD without these proposed changes, as in Table 3. It then raises the question on the effectiveness of the propose method and the claimed contributions. 

Introduction of additional information from SD. In Table 1 and 2, it shows that the classification performance does increase by adding synthetic samples following the proposed method. However, it is unclear that if the gain is indeed coming from these synthetic augmentation or just distilled information from the diverse datasets which the image generator Stable Diffusion is trained over.

questions:
Can you explain the difference of the proposed marginal sample selection algorithm to Focal loss which also adjusts the sample weights based on the difficulty of the sample indicated by the raw softmax score of the classifier being learnt.

Line 13 of algorithm1, S_f is not defined in section 3.4, pls explain.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732595883992

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers and area chairs for a brief summary

comment:
Dear reviewers and area chairs,

We sincerely thank all reviewers and area chairs for their valuable time and insightful comments.

We are pleased to note that:

1. Reviewer Ln52 acknowledges that the **motivation behind our method makes sense**, both reviewers rGcL and NMJi agree that our approach demonstrates **a clear motivation**, and reviewer rpaL describes our approach as **a significant advancement** in training dataset augmentation.
2. Reviewer rGcL commends the **detailed analysis** of the proposed framework, and reviewer NMJi highlights that our GenDataAgent effectively integrates **a series of reasonable treatments**.
3. All reviewers recognize the **accuracy improvements** achieved by our method, with reviewer rpaL further emphasizing its **enhancement of robustness**.

We have responded to each reviewer individually and would like to summarize our responses here:

1. We clarify the **novelty** of the proposed framework in terms of both purpose and integration. 
2. We demonstrated the **improvement** brought by our method through a scaling ablation study.
3. We addressed concerns about the training overhead and introduced two **lightweight versions for a better accuracy-time trade-off**.
4. We highlighted that our method can operate in **offline mode**, improving efficiency and enabling the reuse of synthetic data.
5. We added a detailed discussion comparing marginal sample selection and active learning.
6. We expanded on **related work, particularly [a]**, which incorporates a domain adaptation (DA) technique.
7. We conducted additional experiments incorporating **robustness metrics**, showing that our method outperforms the state-of-the-art Real-Fake baseline.
8. We clarified that our approach is **generalizable to other TTI methods and CV tasks**.
9. We presented additional results demonstrating that our method **avoids overfitting** compared to Real-Fake.
10. We provided further details in the paper, including the function of SD, the distinction between marginal sample selection and Focal loss, the use of consistent prompts for SD, and the choice of VoG checkpoints.

Again, we extend our gratitude to all reviewers and area chairs!

Best regards,

Authors

[a] A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation. CVPR, 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes the first of its kind method for on-the-fly synthetic data generation for data augmentation tailored to a specific downstream task using stable diffusion models. Specifically, it proposes a DataGenAgent pipeline to create synthetic data that is aligned to the distribution of the downstream AI task, while maintaining diversity and relevance of the data. The proposed method improves accuracy both in the synthetic data only and real + synthetic data training settings versus the state of the art.