Leveraging Knowledge Distillation to Mitigate Model Collapse

Authors: Ilya Statsenko, Nikita Andriyanov, Oleg Shishkin
Primary area: generative models
Keywords: computer vision, natural language processing, generative models, diffusion, vae, text summarization, model collapse, synthetic data, distillation

Abstract:
Since the amount of data generated by neural networks on the Internet is growing rapidly due to widespread access to corresponding models, it is logical to inquire about the impact of this surge in synthetic data on the training of subsequent models that will utilize it during training.  Previous work has demonstrated a concerning trend: models trained predominantly on synthetic data often experience a decline in performance, which can escalate to a complete loss of the ability to reproduce the initial distribution of real-world data. This phenomenon, now referred to as model collapse, highlights the potential pitfalls of over-reliance on synthetic datasets, which may lack the diversity and complexity inherent in genuine data. To address this issue, we propose a novel method that leverages the well-established technique of knowledge distillation. Our approach aims to mitigate the adverse effects of synthetic data by facilitating a more effective transfer of knowledge from high-performing teacher models to student model. By doing so, we seek to enhance not only the qualitative aspects—such as the richness and variability of the generated outputs—but also the quantitative metrics that gauge model performance. Through extensive experimentation, we demonstrate that our method improves the robustness and generalization capabilities of models trained on synthetic data, for instance, for DDPM enhancement is 68.8%, in terms of the FID metric, contributing to a more sustainable and effective use of synthetic datasets in machine learning applications.

Decision: reject
Reviewer scores: 1, 3, 1, 3

Reviews and discussion:
<thread>
creation date:
1729326973525

thread_type:
official_review

<thread_object>
creator: reviewer_dp5r

<content>
summary:
This paper provides a contribution to addressing the problem of model collapse using synthetic data. The proposed method leverages knowledge distillation to address this problem. Experiments on multiple image genearion models and text generation model are conducted to indicate the effectiveness of the proposed method.

strengths:
1. The proposed method is easy to follow.
2. The structure of the paper is clear.

weaknesses:
1. There is no formal definition about "Model Collapse" indicated in the paper, the author should describe it for both text model and image model in more details. Also I do not agree that the test set loss and ROUGE scores are a good metric for model collapse indication.
2. The adopted datasets for image generation are quite simple. The authors should use more complex datasets.
3. There is no theoretical/empirical analysis about the results and findings, the authors should think about adding these.
4. The proposed method is still worse than $M_0$ after training with longer steps for language models. The authors should analyze this more.
5. The authors have modified the template style, which could be problematic.

questions:
Please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730043105017

thread_type:
official_review

<thread_object>
creator: reviewer_eni5

<content>
summary:
The paper explores the issue of model collapse. The authors propose a solution using knowledge distillation, Experiments across image generation tasks, including Variational Autoencoder (VAE) and Denoising Diffusion Probabilistic Model (DDPM), and text summarization show performance gains.

strengths:
1. Applying KD to mitigate model collapse might be a possible solution.

weaknesses:
This paper’s format does not follow the ICLR requirements. Additionally, the presentation is poor, lacking clear motivation and an introduction to the methodology. Many unnecessary figures that should have been placed in the appendix occupy a large portion of the main text, making the paper resemble an experimental report. Even so, it fails to reach ten pages. This submission appears extremely unprofessional.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472206262

thread_type:
official_review

<thread_object>
creator: reviewer_q7ek

<content>
summary:
This paper presents a method that utilizes knowledge distillation to mitigate the adverse effects of synthetic data by enhancing the transfer of knowledge from high-performing teacher models to student models. Through extensive experiments, they improve the robustness and generalization capabilities of models trained on synthetic data.

strengths:
-	This paper uses knowledge distillation as a solution to address model collapse.
-	This paper conducts experiments on image generation using VAE and DDPM, as well as text summarization using the T5 model.

weaknesses:
-	The template of the article is not officially provided.
-	The models used in this paper are VAE and DDPM. Can more advanced models be used for image generation, and can the resolution of the generated images be improved? This can better prove the generalization of the proposed method.
-	Lack of comparison with existing approaches to mitigate model collapse.

questions:
Is there no existing baseline for comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690293035

thread_type:
official_review

<thread_object>
creator: reviewer_ht3u

<content>
summary:
The authors use a knowledge distillation framework that utilizes a model trained on real data as a teacher to address the issue of model collapse in models trained on synthetic data.

strengths:
- The proposed approach is highly intuitive. (but, too obvious.)

weaknesses:
- Lack of novelty
  - They merely used the conventional knowledge distillation (KD) method. I think they should have compared various KD methods to identify a more suitable approach for resolving model collapse on the same task.
- Lack of references
- Design of experiments
  - Their experimentation is limited to adjusting the hyperparameters of the model.
  - For iterative cases (repeated cycles of data generation and model retraining, as you mentioned in Introduction), I think it is necessary to assess how severe the model collapse becomes and to what extent it can be resolved.

questions:
- (Writing) Please modify sty file for ICLR 2025.
- (Writing) The line style looks unorganized
</content>
</thread_object>

</thread>


Metareview: