Will the Inclusion of Generated Data Amplify Bias Across Generations in Future Image Classification Models?

Authors: Zeliang Zhang, Xin LIANG, Mingqian Feng, Susan Liang, Chenliang Xu
Primary area: interpretability and explainable AI
Keywords: model bias, image classification

Abstract:
As the demand for high-quality training data escalates, researchers have increasingly turned to generative models to create synthetic data, addressing data scarcity and enabling continuous model improvement. However, reliance on self-generated data introduces a critical question: \textit{Will this practice amplify bias in future models?} While most research has focused on overall performance, the impact on model bias, particularly subgroup bias, remains underexplored. In this work, we investigate the effects of the generated data on image classification tasks, with a specific focus on bias. We develop a practical simulation environment that integrates a self-consuming loop, where the generative model and classification model are trained synergistically. Hundreds of experiments are conducted on Colorized MNIST, CIFAR-20/100, and Hard ImageNet datasets to reveal changes in fairness metrics across generations. In addition, we provide a conjecture to explain the bias dynamics when training models on continuously augmented datasets across generations. Our findings contribute to the ongoing debate on the implications of synthetic data for fairness in real-world applications.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730207529563

thread_type:
official_review

<thread_object>
creator: reviewer_m2ow

<content>
summary:
Great:
The authors investigated an important question on a selected dataset. The future work can be extended to other hierarchical sub-groups and more datasets.
Limited benchmark results are apparent and convincing.

Missing:
It will be helpful to accept the claim with more extensive experiments and analysis.
According to me, citations to the original work/s are missing.
The final section can be elaborated to cover the important findings that support the main problem.

Without additions:
It's a strong acceptance for a poster.
Weak acceptance for the main track.

Explanation:
1. The experiments: If possible, I would like to see the results from other standard datasets or more subgroups for the dataset used.
2. Citations: Some original works in the introduction section and later are missing citations. If the authors think they have cited all the necessary work, please put that in a rebuttal.
3. Conclusion and future work: I think it can be improved or extended to connect with the problem and story.

strengths:
The authors investigated an important question on a selected dataset. The future work can be extended to other hierarchical sub-groups and more datasets.
Limited benchmark results are apparent and convincing

weaknesses:
It will be helpful to accept the claim with more extensive experiments and analysis.
According to me, citations to the original work/s are missing.
The final section can be elaborated to cover the important findings that support the main problem.

questions:
Look at weakness section to know what to improve.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730283885403

thread_type:
official_review

<thread_object>
creator: reviewer_hfuo

<content>
summary:
The paper attempts to analyze whether the inclusion of generated data alleviates the model bias. In the paper, the authors repeatedly train generative models on the generated augmented data and study its impact on multiple fairness metrics.

strengths:
The paper is studying a new problem. In the era of generative AI, more generated content is available on the Internet. Training on the generated data may influence model performance. In this sense, the paper is studying a valid and important problem.

weaknesses:
The paper mostly conducts experiments on three datasets. Among the three datasets, Colourized-MNIST and CIFAR20/100 are very small datasets in terms of the resolutions and the number of data and classes compared to the existing image data. Moreover, the largest model used in the paper is ResNet50, which is a relatively small network when compared to SOTA models like ViT. This raises the concern of whether the observations from the experiments are still valid in realistic scenarios with large datasets and models. 

Some experimental results are hard to understand.
- baseline performance. To my understanding, CIFAR20/100 has a smaller number of classes and should be an easier dataset to classify when compared to CIFAR100. However, the baseline performance of ResNet50 on CIFAR100 is around 80%; on CIFAR20/100 is around 50%. 
- The trends are inconsistent between models. For example, in Fig. 6(a), ResNet50 and LeNet show an opposite trend in Equal Opportunity and Disparate Impact but the same trend in average accuracy and Maximal Disparity; VGG-19 remains unchanged for the tested metrics.
- The trends are inconsistent between datasets. For example, comparing the ResNet50 baseline between Fig 5(a) on CIFAR-20/100 and 6(a) on CIFAR-100. Even though the datasets are similar, the ResNet50 baseline shows a totally different trend for the Equal Opportunity, Disparate Impact, and Maximal Disparity metrics. 
- Similar discrepancies are noted in other sections of the paper. The experimental findings do not appear to explain how the generated data affects model bias in general.

The conclusion is weak. Based on the observations, the authors conjectured that the model bias is affected by multiple factors, including the datasets, models, and data quality across generations. However, the authors did not provide clear experiment evidence or solid theory explaining how these factors influence the model bias.

questions:
The authors reported the FID score for the augmented data from multiple generations in Table 1. It seems that the FID scores for unbiased colorized MNIST show a decreasing trend; biased colorized MNIST is more or less the same; CIFAR-20/100 decreases first and then increases; Hard ImageNet shows a sudden increase from 50.9 to 186.4. Can the author explain the inconsistent changes here? Are there any implications from the observations? Also, it would be better if the author could visualize the generated images from different generations to visually see the changes across generations.

The authors mentioned, “We manually partition the original dataset into multiple subgroups, where subgroups within the same class share similar semantics.” Can the authors explain more clearly how they defined and constructed the subgroups for each dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694786899

thread_type:
official_review

<thread_object>
creator: reviewer_bwfa

<content>
summary:
With the growing prevalence of generative models, the authors raised the concern regarding model bias, particularly in the context of generation and retraining cycles. Then the authors developed a simulation environment and conducted extensive experiments on image classification datasets to reveal how fairness metrics evolve across successive generations.

strengths:
S1. The scenario proposed by the authors is highly relevant, as synthetic data is increasingly shared online and integrated into various domains. More studies on how the synthetic data will affect model training in generations will be beneficial for the research community.

S2. The experiments on the proposed dataset are extensive, e.g., w/ or w/o biased initialization, different base models.

weaknesses:
W1. Lack of experiments on the choice of generative models. Various generative models can differ in behavior, the choice of model likely impacts sample quality and influences the outcomes of subsequent studies.

W2. The motivation of the paper is on future image classification synthetic data plays in it. With foundational models playing a dominant role, integrating settings like synthetic data for transfer learning in classification will enhance the paper better, going beyond the current base case that may lack scalability.

W3. The experiments are mainly targeting the model bias within a self-consuming loop in image classification domain. However, the conclusions/observations are not significant.

questions:
Q1. How many synthetic data are added or ratio p? Is there a rationale behind the choice of p.

Q2. Are there any experiments or preliminary results on tasks with a larger number of classes?

Q3. How is CLIP used in filtering? Specifically, is it based on the similarity score between label texts and images?

Q4. Would different losses lead to varying results?

Q5. Any insights on how could the conclusion generalize to other tasks, or other modality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697190305

thread_type:
official_review

<thread_object>
creator: reviewer_qlxj

<content>
summary:
This paper investigates the bias implications of incorporating generated data from generative models into training downstream tasks. The authors propose an iterative pipeline that repeatedly uses new generators to create additional images that enhance training. Their analysis spans three image datasets of varying complexity, exploring both performance improvements and bias impacts on downstream classification tasks. Bias is examined across different subgroups, covering both single and multiple bias interactions. Through this setup, they observe mixed trends in performance and bias effects across datasets of different complexities and model capacities. The paper concludes with a high-level discussion on potential root causes behind these varied results.

strengths:
- This paper addresses a highly relevant topic by examining the implications of generated data on bias, which is essential for advancing our understanding of the gaps between generated and real data.
- The iterative pipeline for incorporating generated data closely resembles real-world applications. By using datasets of varying complexity and models with different capacities, the study effectively explores different aspects of the problem, enhancing the generalizability of the findings.
- The study provides noteworthy observations with good experimentation support, such as the low correlation between dataset bias and resulting bias effects, the higher susceptibility of pre-trained models to integration bias, and insights into how different factors affect bias across datasets and models.

weaknesses:
- The paper presents mixed findings across datasets and models but does not provide in-depth explanations for these variations. While section 5 includes some discussion on the root causes of observed behaviors, this analysis remains at a high-level and is not well-supported or directly connected to the experiments in earlier sections. The analysis would be more convincing with clearer connections to the results, reinforcing the paper’s claims with evidence from the experiments.
- In table 1, FID fluctuates for Color-MNIST and CIFAR10 after several rounds of data generation, while it increases substantially for HARD-ImageNet starting from the second iteration. This trend suggests a marked difference in data quality for HARD-ImageNet compared to the other datasets. However, the subsequent experiments focus primarily on how generated data impacts downstream performance and bias without addressing how this observed FID trend might influence these results. A discussion on how does the data quality(assesed by FID) could affect interpretations across the three datasets would enhance the clarity of the findings.
- Some methodology details are lacking, making it challenging to fully understand and replicate the study. For example, in section 3.2, there is limited information on the design of the human study, the impact of expert-guided filtering on image quality, and the specific r% used.
- The paper would benefit from some recommendation on the usage of generated data in generative models or downstream tasks with the insights from the experiments.

questions:
1. Can you provide more detail on the human study in section 3.2, and the r% used?
1. What is the impact of image quality from the expert-guided filtering in section 3.2? 
1. How well does the findings generalize to other dataset? For example, section 4.2 showed dataset bias does not amplify model bias. Do authors expect that to hold for other datasets, too?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733075072205

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for your careful review and suggestion on improving  our work:)

comment:
**To all reviewers:**   
We sincerely thank all reviewers for their valuable feedback and for recognizing the merits of our work:  
1. Our studied problem and the proposed framework are novel, important, and practical.  (QLxJ, Bwfa, hfuo, m2oW).  
2. We conduct extensive experiments to answer the studied problem and support our claim. (QLxJ, Bwfa)  

In our revision, we have made the following updates to answer the reviewers's major concerns:  

- **More experiments on ImageNet**:  Details are provided in Appendix A.  
- **Examples of generated images across generations**:  Details are provided in Appendix B.  
- **Detailed expert-guidance filtering**: Details are provided in Appendix C.  
- **Improved Clarity**: We revise both illustration figures and their captions for better clarity.  
- **Typos Corrected**: We fix all identified typographical errors.  

During the discussion, we sincerely appreciate the reviewers' suggestion to further broaden the scope of the studied problem. However, **within the limited scope of a conference paper, we have chosen to focus on the most general and widely used task under a standard setting for concept verification**. To ensure reliable verification, **all results are conducted multiple times to minimize randomness**.

We agree that it would be valuable to explore the impact of generated data on additional tasks, such as transfer learning, domain adaptation, and model robustness. We leave these investigations for future work, aiming to inspire more researchers to delve into this problem and pave the way for **leveraging academic advancements to drive progress in industry practices**.

We hope these updates address your concerns and further strengthen the contributions of our work. Thank you again for your thoughtful reviews and support!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the implications of generated data being used to train generative models, which is a realistic setting given how data is often shared and scraped on the internet. Specific focus is given to how biases can emerge or be exacerbated across subgroups in the data.

Reviewers noted that trends in the data and results were inconsistent across settings, with no clear explanations given as to the mechanisms that could be at play. The datasets used were small scale, focused only on the image modality, and used a small set of model architectures. These choices greatly limit the generality of the study, and its applicability to real world cases (which was the original motivation). Reviewers noted inconsistent results between experiments that were not explained by the authors, and these points were not adequately addressed during the discussion period. For these reasons, I am recommending rejection.