Mousterian: exploring the equivalence of generative and real data augmentation in classification

Authors: Haowen Wang, Guowei Zhang, Xiang Zhang, Zeyuan Chen, Haiyang Xu, Dou Hoon Kwark, Zhuowen Tu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: analysis by synthesis, image classification, data generation

Abstract:
In this paper, we address a key question in machine learning: **How effectively can generative data augmentation enhance image classification?** We begin by examining the differences and similarities between real and synthetic data generated by advanced text-to-image models. Through comprehensive experiments, we provide systematic insights into leveraging synthetic data for improved classification performance. Our findings show that: 1). Generative data augmentation by models trained solely on the internal (available training) set can effectively improve classification performance, validating the long-held hypothesis that synthesis enhances analysis by enriching modeling capability.
2). For generative data augmentation by models trained on both internal and external data (e.g. large-scale image-text pairs) separately, the size of equivalent synthetic dataset augmentation can be determined empirically. In addition to being aligned with a common intuition that real data augmentation is always preferred, our empirical formulation also provides a guideline for quantitatively estimating how much larger the size of generative dataset augmentation is, over the real data augmentation, to achieve comparable improvements. Our CIFAR-10 and ImageNet results also demonstrate its impact w.r.t. the size of the baseline training set and the quality of generative models.

Decision: reject
Reviewer scores: 6, 5, 3, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1729024239272

thread_type:
official_review

<thread_object>
creator: reviewer_tbmi

<content>
summary:
This paper explores the effectiveness of generative data augmentation for image classification, focusing on models trained on internal datasets as well as those trained on large-scale external datasets. The authors conduct experiments on CIFAR-10 and ImageNet and find that while generative data augmentation can enhance classification performance, it generally requires significantly more synthetic data to achieve similar effects as real data augmentation. The study provides practical guidelines for quantifying the ratio between generative and real data needed to achieve comparable improvements. The results suggest that high-quality synthetic data can significantly improve performance when real data is limited, though it remains less efficient compared to real data.

strengths:
1. The paper presents a novel investigation into the role of generative data augmentation for image classification, focusing on both internal (training set only) and external (pre-trained on large datasets) generative models. It uniquely addresses the equivalence between synthetic and real data augmentation, providing empirical quantification to match their effects.

2. The paper is well-constructed, featuring rigorous experimentation on widely recognized datasets like CIFAR-10 and ImageNet. The thorough comparison of real vs. synthetic data, using multiple generative models, ensures that the findings are robust and generalizable.

3. The findings offer practical guidelines for leveraging synthetic data when real data is scarce, providing valuable insights into the trade-offs between real and synthetic data. The results have broad implications for data augmentation in machine learning, especially in contexts where data collection is challenging.

weaknesses:
1. The paper's empirical analysis primarily focuses on CIFAR-10 and ImageNet, which limits the generalizability of the findings to more complex or diverse datasets. It would be beneficial to include experiments on additional datasets with varying complexity to validate the proposed approach across different domains.

2. The equivalence formulation between real and synthetic data is presented without sufficient theoretical backing. While the empirical results are insightful, the lack of a theoretical framework makes it difficult to generalize the equivalence relationship beyond the specific datasets and models used in the study.

3. The paper does not deeply address the quality of synthetic data and how it impacts classification performance beyond a certain point. For instance, Figures 1(b) and 1(c) highlight the impact of synthetic data on performance, but a more detailed analysis of the characteristics of synthetic data (e.g., diversity, fidelity) is needed to explain the diminishing returns observed in these figures.

4. The experiments on internal and external generative models are treated as largely separate investigations, without adequately exploring potential synergies. Figure 2 shows equivalence curves for internal and external settings separately, but it would be valuable to examine whether combining internal and external generative models could lead to better data augmentation strategies, especially in settings with limited labeled data.

5. The discussion on the trade-offs between real and synthetic data augmentation could be expanded to consider the computational cost of generating synthetic data. Practical considerations, such as the time and resources required to generate large synthetic datasets compared to collecting real data, are not sufficiently addressed. Figure 5, which illustrates the diminishing returns of synthetic data augmentation, could benefit from a discussion on the associated computational costs to assess the feasibility of the proposed approach in real-world scenarios.

6. The study mainly uses accuracy as the evaluation metric, which may not fully capture the nuances of model performance. For example, Figures 4 and 7 show accuracy improvements with different synthetic data ratios, but incorporating additional metrics, such as robustness to adversarial attacks or generalization to out-of-distribution samples, could provide a more comprehensive evaluation of the benefits and limitations of synthetic data augmentation.

questions:
1. Could the authors provide more theoretical insights or justification for the empirical equivalence between synthetic and real data augmentation? How generalizable is this relationship across different types of datasets and generative models?

2. Have the authors considered combining internal and external generative models to leverage the benefits of both? If so, how would such a hybrid approach affect classification performance, particularly in cases with limited labeled data?

3. Could the authors provide an analysis comparing the computational cost of generating synthetic data to the benefits it provides in terms of performance gains? This would help in assessing the practical feasibility of using generative models for data augmentation in real-world applications.

4. Would the authors consider including other evaluation metrics, such as robustness to adversarial attacks or generalization to out-of-distribution samples, to provide a more holistic assessment of the classifier's performance when trained with synthetic data?

5. Is there an optimal ratio of synthetic to real data that consistently yields the best performance, as observed in Figure 5? If so, how is this ratio affected by the quality of the synthetic data or the complexity of the underlying dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730170569692

thread_type:
official_review

<thread_object>
creator: reviewer_4ssv

<content>
summary:
This paper explores the equivalence of generative and real data augmentation in classification

strengths:
this paper try to figure out how fake data improve classification

weaknesses:
1) the equations are weard. The authors try to give a precise formulation to model the effectiveness of generated data from large models. However, it's well-known that deep networks are too complex to be formulized into explicite equations.

2) Generated data do not provide new information, hence i do not trust this research topic. Although the authors mentioned external large models, the training of large models still need real data.

3) The final classification results are not competive to sota models like CoCa and ViT-e

questions:
1) In euqtion 1, where is the number 15571.2 from?

2) could the authors provide reusults of Percentage correct on the  CIfar10 dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730307845127

thread_type:
official_review

<thread_object>
creator: reviewer_agtc

<content>
summary:
Paper investigate the synthesis data in image classification tasks. The authors compare real and synthetic data, generated by advanced text-to-image models, to assess the impact on classification performance across various scenarios, using datasets like CIFAR-10 and ImageNet. The study provides empirical guidelines for how much synthetic data is needed to match the performance benefits of real data augmentation. Results show that while synthetic data can improve classifier accuracy, especially in data-scarce environments, it requires larger volumes to achieve comparable performance with real data.

strengths:
The paper rigorously quantifies the trade-offs between real and synthetic data, offering empirical formulas to guide the required synthetic-to-real data ratios for effective augmentation. This quantitative approach provides actionable insights for researchers and practitioners on optimizing data augmentation strategies.

By evaluating the impact of both internal (training set only) and external (large-scale pre-trained) synthetic data on well-known datasets (CIFAR-10, ImageNet), the study demonstrates a thorough exploration of generative data's utility.

weaknesses:
The study's research question, while valuable, is already extensively explored within the synthetic data and image classification community. This paper does not introduce new insights or findings beyond what prior studies have already established. The results presented appear to align closely with existing, well-documented conclusions, providing minimal advancement in our understanding of synthetic data's impact on image classification.

1. Numerous papers cover synthetic data for image classification, including studies like arXiv:2304.08821, which also explores synthetic data's utility for classification tasks using modern generative models, arriving at similar conclusions. Additionally, arXiv:2204.08610 provides a comprehensive study of data augmentation across multiple classification datasets, reinforcing comparable outcomes, making the results here somewhat redundant.

2.  results in image classification does not translate well to other domains with higher current relevance, such as large language models (LLMs) or language-related synthetic data. Expanding the scope to more emergent areas—such as synthetic data for LLM training or even diffusion training—could yield broader contributions with wider applicability. 

3 The main results in Table 1 primarily show a basic comparison between data size and performance gain in classification. However, as data size increases, factors like model size and architecture can significantly impact outcomes. Investigating performance variability across different model sizes or architectures, and including training variance across runs, would strengthen the findings and allow for a more nuanced analysis of the impact of synthetic data augmentation on classification performance.

4. it would be better to investigate more under-invesigated topics such as low-resource images, casual relations in image classification. Simply focusing on accuracy is not as meaningful for many research today as we already know "more high quality" data always leads to somewhat better results from all previous studies.

questions:
For quality assessment, FID and IS are very bad scores, since the inception net is old and does not capture enough features. There are many new quality assesment metrics, such as Clip-MMD which uses modern clip model, does author take into consideration of some other scores?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730349459472

thread_type:
official_review

<thread_object>
creator: reviewer_ayqx

<content>
summary:
The paper presents an empirical study to evaluate the effectiveness of generated data in image classification tasks. Specifically, the study reports the accuracy changes when training a classifier with various amounts of generated data added to CIFAR-10, ImageNet10, and ImageNet100 training datasets. The study reveals that (1) generated augmented data can improve classification accuracy, especially when real data is limited, (2) real data are more sample-efficient than synthesized data, and (3) the performance gains diminish when the amount of synthesized data increases.

strengths:
- The analysis is clear and easy to understand.
- The study confirms the findings from previous works.

weaknesses:
I appreciate the effort of the authors to systematically and comprehensively study the impact of using generated augmented data for classification tasks. The experiments are very clear and concise. However, most of the findings (1), (2), and (3) mentioned in the summary section above are generally known and have been reported from previous works [1] on more datasets. The proposed study works out the exact accuracy improvements concerning the syn-real ratios on three datasets. I am afraid this does not demonstrate sufficient novelty and technical contribution.

[1] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, & Xiaojuan Qi (2023). Is Synthetic Data from Generative Models Ready for Image Recognition?. In The Eleventh International Conference on Learning Representations, ICLR 2023.

questions:
Suggestion:

- The authors may consider working out some guidelines for selecting the optimal syn-real ratio for any new datasets. However, this probably needs more experiments on a larger variety of datasets.
- The current manuscript is more like a technical report. It may be more suitable to submit it to a journal related to machine learning applications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495515415

thread_type:
official_review

<thread_object>
creator: reviewer_gl25

<content>
summary:
This paper is an empirical study on the usage of generated data (through generative modelling) for classifier training. The authors introduce a framework in which they analyze the effectiveness on adding generated data in the training. They have two setup: 1) In the first one they train a generative model on their target dataset and investigate how generating additional data improve the classifier's performance. 2) In the second setup, they leverage off-the-shelve large generative models such as StableDiffusion to generate additional data. One of their main takeaway is that when having low amount of training data, then augmenting the training set with generated data improve the performances. However, the gain are less significant when having larger amount of real data. Another main take away is that generated images are much more less "data efficient" than real images, so there is a need to increase significantly the number of generated point.

strengths:
- The paper is well written and easy to follow. I really appreciate the main result highlights.
- The authors performed a in depth empirical analysis of what is the benefit of synthetic data for classifier training.
- The experimental setup on CIFAR10 and ImageNet is convincing. 
- Lot of details about the experimental setup and the implementations.
- Valuable ablation study over the impact of the ratio of synthetic data used.

weaknesses:
- I would be cautious about such strong statement as "synthetic data is less sample efficient than real data". It might just be because of the way those samples are generated or the lack of diversity in the generated sample. As highlighted by the authors, there is a significant domain gap between the generated and real images, so I am not sure that we can say that synthetic data is per say less efficient, but maybe, we could say that current generative models, even if they are able to generate visual appealing images, they are still not able to match the distribution of natural images. And it happens that this failure of current generative model, induces learning a distribution that is indeed less diverse and real than the natural images which make then less useful for sampling relevant data for classifier training.
- This paper provide mostly an empirical study that has indeed value bit I would have expected to see a more fine-grained analysis such as analyzing wether the ratio between real/synthetic data has the same impact across all classes, or if some classes benefits more than others.

questions:
- Do you think that this lack of "efficiency" from synthetic data might due to the fact that those generative models are optimized for quality and not so much for diversity? Any other idea on why this could be happening? 
- I am wondering if you have any suggestions on how to improve the "efficiency" of the synthetic data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535170686

thread_type:
official_review

<thread_object>
creator: reviewer_gjjx

<content>
summary:
The paper examines the impact of generative data augmentation in image classification tasks. It offers a comparison between real and synthetic data, with the latter being produced by advanced text-to-image models, and discusses the benefits of using synthetic data to improve classification performance. The study demonstrates that generative data augmentation can significantly enhance classification performance, especially when real data is scarce, and provides empirical guidelines for determining the appropriate size of synthetic dataset augmentation. The results on CIFAR-10 and ImageNet highlight the influence of the baseline training set size and the quality of generative models on the effectiveness of classification.

strengths:
S1: This study presents a lucid and accessible exploration of the impact of generative data augmentation in the domain of image categorization. It offers a thorough examination of the efficacy of generative data enhancement, not only assessing the performance of generative models across various datasets but also dissecting the interplay between synthetic and real data. The research is bolstered by an extensive array of experimental results, leading to direct and tangible conclusions that enhance the reader's comprehension.

S2: Extensive experimentation substantiates the efficacy of generative data augmentation, particularly under conditions where real data is constrained. The study demonstrates that synthetic data can markedly enhance classification performance, offering valuable insights for practical applications and contributing to the body of knowledge in the field.

S3: The study's methodology is rigorous, employing several renowned datasets and well-established generative models to conduct experiments. This approach ensures the reliability and generalizability of the findings, thereby reinforcing the study's scholarly contribution to the field of image classification and data augmentation.

weaknesses:
W1: There are shortcomings in the ablation studies, as they do not adequately analyze the specific impact of different generative models or the effect of varying dataset sizes. It would be beneficial to compare performance across a wider range of generative model architectures and to systematically analyze how results change with different synthetic dataset sizes. Such comparisons would provide clearer insights into the strengths and limitations of the methods used.

W2 : The paper does not discuss the significant computational resources required for generating high-quality synthetic data. To enhance clarity, I recommend that the authors provide specific details about the computational overhead, including GPU hours, memory requirements, and energy consumption. This information would help readers understand the practical implications of implementing this approach.

W3: Real-world scenarios with variable conditions, such as those involving autonomous vehicles, may not have adequate validation

questions:
1. Why were generative models like Vanilla-DDPM and Stable Diffusion 2 chosen for image generation? How might the choice of these specific models impact the generalizability of the results? Are there alternative models that could also be explored in future work?

2. Can you clarify the rationale behind selecting different image encoders at various stages of the generation process, such as CLIP and DINOv2? How does this choice influence the final results?
</content>
</thread_object>

</thread>


Metareview: