A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
Authors: Lianghe Shi
Meng Wu
Huijie Zhang
Zekai Zhang
Molei Tao
Qing Qu
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion model
memorization
generative model
Primary area: deep_learning

Abstract: The widespread use of diffusion models has led to an abundance of AI-generated data, raising concerns about model collapse---a phenomenon in which recursive iterations of training on synthetic data lead to performance degradation. Prior work primarily characterizes this collapse via variance shrinkage or distribution shift, but these perspectives miss practical manifestations of model collapse. This paper identifies a transition from generalization to memorization during model collapse in diffusion models, where models increasingly replicate training data instead of generating novel content during iterative training on synthetic samples. This transition is directly driven by the declining entropy of the synthetic training data produced in each training cycle, which serves as a clear indicator of model degradation. Motivated by this insight, we propose an entropy-based data selection strategy to mitigate the transition from generalization to memorization and alleviate model collapse. Empirical results show that our approach significantly enhances visual quality and diversity in recursive generation, effectively preventing collapse.

A Closer Look at Model Collapse:
From a Generaliaatoon--oMMmmrritttono Perspective
Lianghe Shi" University of Michigan United States 1hshi Thshi@umich @umi th.mich.aa edu
Meng Wu" University of Michigan United States wymond@umi ch. edu
Huijie Zhang University of Michigan United States juijiezh@umic ch edu
Zekai Zhang University of Michigan
Molei Tao
Qing Qu University of Michigan United States qingquBumich edu
Georgia Institute of Technology United States mtao@gatech. edu
United States zzekai @umich edu
Abstract
The widespread use of diffusion models has led to an abundance of Al-generated data, raising concerns about model collapse-a phenomenon which recur- sive iterations of training on synthetic data lead to performance degradation. Prior work primarily characterizes this collapse via variance shrinkage or distri- bution shift, but these perspectives miss practical manifestations of model col- lapse. This paper identifies a transition from generalization to memorization during model collapse in diffusion models, where models increasingly replicate training data instead of generating novel content during iterative training on syn- thetic samples. This transition is directly driven by the declining entropy of the synthetic training data produced in each training cycle, which serves as a clear indicator of model degradation. Motivated by this insight, propose an entropy-based data selection strategy to mitigate the transition from general- ization to memorization and alleviate model collapse. Empirical results show that our approach significantly enhances visual quality and diversity in recur- sive generation, effectively preventing collapse. The source code is available https://githin com/ http:///h/iaaaggoooooooooooooooa
Introduction
As generative models, such as diffusion models, become widely used for image synthesis and video generation, large volume of generated data has emerged the Internet. Since state-of-the-art diffusion models can generate realistic content that even humans cannot easily distinguish, the train- ing datasets for next-generation models will inevitably contain significant proportion of synthetic data. Figure 0 illustrates this self-consuming loop, where at each iteration data generated by the current model is subsequently used train the new model for the next iteration. Unfortunately, sev- eral recent studies have demonstrated that recursively training models on datasets contaminated by Al-generated data leads performance degradation across these self-consuming iterations even when synthetic data comprises only a small fraction of the dataset [I]] This phenomenon, termed model collapse in prior work, poses significant threat the future development and effectiveness of generative models.
"The first authors contribute equally.
2This paper uses "iteration" to denote full cycle of training and generation, rather than gradient update.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Collapse Iteration
D
Training
Sampling
D
Training
Sampling
D3
Dn
M
M2
Roal Data
Selection
Mitigated Iteration
1
Training
Sampling
G1
D2
Training
D
M
M2
Previously
Generated Data
Deprddation
Depadation
Training/Generuted Dataset
Iteration
Iteration
Iteration
Gonorated
Generalization
Memorization
Counterparts (bottom)
Figure 1: High-level depiction of the self-consuming pipeline. Top: Collapse iteration represents the replace paradigm where models are trained solely on synthetic images generated by the previous diffusion model. Middle: In the mitigated iteration, original real data and previously generated data are added to train the next-generation model. Our proposed selection methods construct training subset and can further mitigate collapse. Bottom Right: Evolution of the generated images.
As surveyed by [2]. recent studies have identified various collapse behaviors that depend on the performance metrics employed. A series of papers [G] [i] reveal the model collapse phenomenon through the variance of the learned distribution. They empirically and theoretically show that the model continually loses information in the distribution tail, with variance tending towards 0. Another line of work (5), [-II] investigates the issue from the perspective of population risk or distribution shifts. These studies observe that the generated distribution progressively deviates from the under- lying distribution, causing the model's population risk to increase throughout the recursive process. Numerous studies [ااسК also report that models begin generating hallueinated data. Despite signif- icant theoretical insights regarding variance dynamics, the reduction of variance to negligible levels typically occurs only after extremely large number of iterations. As noted by [2, 2], the collapse progresses at such slow pace that it is rarely practical concern in real-world applications. In con- trast, the visual quality and diversity of generated images deteriorate rapidly. Furthermore, although population risk or distribution shifts offer holistic view of performance degradation, they do not adequately characterize specific collapse behaviors.
Accordingly, this paper conducts an in-depth investigation into the collapse dynamics of diffusion models and identifies tions. Specifically, during early iterations, the model demonstrates strong capability to generate novel images distinct from those the training set, but gradually shifts towards memorization in later iterations, merely replicating training images. This transition significantly reduces the diver- sity generated content and results in higher FID scores. Moreover, directly reproducing images from training datasets may raise copyright concerns []4, (15). Furthermore, we empirically reveal a strong linear correlation between the generalizability of the trained model and the entropy of its training dataset. As iterations progress, the entropy of the data distribution sharply decreases, di- rectly resulting in a decline in the models generalizability, which illustrates a clear transition from generalization to memorization. Motivated by these empirical findings. we propose entropy-based selection methods to construct training subsets from candidate pools. Extensive experimental val- idation demonstrates that proposed methods effectively identify high-entropy subsets, thereby tttiraatioo-o-mmmmmiizzatooooooo transition. Additionally, our approach achieves su-
transition occurring across successive itera-
decelerating the perior image quality and lower FID scores in recursive training loops compared to baseline methods.
Our Contributions.
The contributions of this paper are summarized as follows:
1. We identify the viding novel perspective for studying model collapse and highlighting critical practical issues arising from training synthetic data.
transition within the self-consuming loop, pro-

---

2. We investigate the key factor driving this transition and empirically demonstrate strong correla- tion between the entropy of the training dataset and the generalization capability of the model.
3. Motivated by our empirical findings, we propose entropy-based data selection methods, whose effectiveness is validated through comprehensive experiments across various image datasets.
2 Background
In this work. we focus on the image generation task. Let A' be the d-dimensional image space, X S I2 and let Po be a data distribution over the space X' We use bold letters denote vectors in X. We assume the original training data Dreal = fere Rnal and identically distributed (i.i.d.) according to the underlying distribution Po. i.e., (1() Po-
(No) are generated independently
Diffusion Models. For a given data distribution, diffusion models do not directly learn the proba- bility density function (pdf) of the distribution; instead, they define forward process and reverse process, and learn the score function utilized in the reverse process. Specifically, the forward pro- cess [TG]] progressively adds Gaussian noise to the image, and the conditional distribution of the noisy image is given by: p(x N (are / Vā,ao. (1 - az) where a, is the scale sched- ule, zo is the clean image drawn from Po. and æ; is the noisy image. This forward be described as a stochastic differential equation (SDE) [| dæ = f(x, t)dt + g()) where is the diffusion coef- ficient. and w is standard Brownian motion. This SDE has corresponding reverse SDE as dx = [f t) - V Pr ]] + g(t)dw, where dt represents a negative infinitesimal time step, driving the process from t T to to O. The reverse SDE enables us to gradually convert Gaussian noise to a clean image x ~ Po-
also
I(;;)) R - Rd denotes the vector-valued drift coefficient, g(l)
The score function V log is typically unknown and needs to be estimated using neural network s2 t). The training objective can be formalized as
A(t) v/ (. so(x, t)11
and can be efficiently optimized with score matching methods such as denoising score matching [8].
Self-Cone Saff-Conuumin Loop. Following the standard setting of model collapse [M] B. M7 [1]] we denote the training dataset at the n-th iteration as D,,- Let A(-) denote the training algorithm that takes D, as input and outputs a diffusion model characterized by the distribution Pa. i.e., P. A(Dn). In this work, we train the diffusion model from scratch at each iteration. Subsequently. the diffusion model generates synthetic dataset of size N,, denoted by Gn PNN., which is used in subsequent iterations.
Based on the specific way of constructing training datasets, previous studies two distinct iterative paradigms:
g
22 distinguish
The replaced training dataset. At each iteration, the training dataset consists solely of synthetic data generated by the previous diffusion model, i.e., Dn 9--1- Several studies [HL & 22] refer to this as the "replace" paradigm and have demonstrated that under this setting, the variance collapses to 0 or the population risk diverges to infinity.
The accumulated training dataset. A more realistic paradigm [8 [I] is to maintain access to all previous data, thereby including both real images and all synthetic images generated thus far, i.e., D, 9,) U Dreat- However, continuously increasing the training dataset size quickly demands substantial computational resources. A practical compromise is to subsample fixed- size subset from all candidate images, referred to as the "accumulate saccuullae-suubaanplle -ubsample" paradigm in [=]] Under certain conditions, prior work [+] X] have shown that accumulating real and synthetic data mitigates model degradation, preventing population risk from diverging.
This work focuses on the replace and accumulate-suule subsample paradigms following prior studies of E. a Bj. Please refer to the Appendix for additional related work.
3

---

3 Model Collapses from Generalization to Memorization
In this section, we empirically demonstrate the transition from generalization to memorization that occurs over recursive iterations, and investigate the underlying factors driving this transition. All experiments in this section are conducted on the CIFAR-10 10 dataset [2X] using UNet-based DDPM model [T66], under the replace paradigm, where each model is trained solely on samples generated by the model from the previous iteration. However, in Appendix II we extend the explorations to other datasets and paradigms, where the conclusion remains valid.
Generalization Score. To quantify generalization ability, we adopt the generalization score [|S 24 25]. defined the average distance between each generated image and its nearest training image:
1 9
CS(n) - Dist(Dnn; 9n) =
min k(rr SED.
where x(::)) : TR x TRd -> IR denotes a distance metric between two data points. A higher general- ization score (n) indicates that the model generates novel images rather than replicating training samples.
| measure generalizability as the probability that the similarity between a generated image and its nearest training sample exceeds threshold. [25] assesse memorization via hypoth- esis test. While definitions of generalizability vary across studies, they are fundamentally similar, relying on nearest neighbor distances.
Remark:
Highlight of Observations: The generated data progressively collapses into numerous compact local clusters over model collapse iterations, as evidenced by both the sharp decline in entropy over iterations and visualizations. This localized concentration of data points then facilitates memoriza- tion in subsequent models, reducing their ability to generate novel images. Our claim is supported by the following three findings.
3.1 Finding I: wtwpaliaattiontooMmmoiiiiaaa Transition
A gcneralization-to-mme wwwpratttiontt-mmmoiiii to-memorization transition is revealed by experiments showing that the model ini- tially generates novel images but gradually shifts to reproducing training samples in later iterations. We conduct iterative experiments on the CIFAR-10 benchmark [23] to illustrate this transition. Fig- visualizes generated samples alongside their nearest neighbors in the training set. With relatively large sample size, i.e., 32,768 real samples as the starting training dataset, the model tends generalize first and then memorize. At the early iterations, the diffusion model exhibits strong generalization in early iterations, producing high-quality novel images with little resemblance to training samples. However, its generalization ability deteriorates rapidly over successive iterations, and the model can only copy images from the training dataset after several iterations.
To quantitatively validate the generalization-tomne to-otttoommmmiiiionn transition, tion score (GS) introduced in Equation (I), which measures the similarity between the gen- erated images G; and the corresponding training images D; at each iteration. We follow the protocol of [[55] and construct six nested CIFAR-10 subsets of increasing size: |Di| E (1,024; 2,048; 4,096; 8, 1929; 16,384; 32,768). These subsets span approximately 3% to 65% of the full training set, providing controlled start points that reflect varying degrees of memorization and generalization. As shown in Figure A. GS drops almost exponentially with successive iterations, providing strong empirical evidence for the ceneralizzaiio--o-mmmrriii noticeably slower for larger training subsets, indicating that larger sample sizes preserve general- ization longer and delay the onset of memorization. For the smallest dataset of 1,024 images, the model enters the memorization regime from the first iteration and remains there throughout.
track the generaliza-
transition. The decline
3.2 Finding II: The Entropy of the Training Set Shrinks Sharply over Iterations
We identify entropy as the key evolving factor in the training data that drives the transition from generalization to memorization. Prior work [24]] interprets generalization in diffusion models as
failure to memorize the entire training set. [[S]] further show that diffusion models tend to generalize

---

1024
4096
16384
Iteration Iteration 3 Iteration
Gi
32768
2000
D
Generalization Score
1750
1500
Generalization
Regime
D2 gs
1250
1000
750
500
Memorization Regime
D2 dx
250
2
4
10
Iteration
Figure 2: The generalization-to-nee tommomo-ttommmmiiiiiiio transition. Left: visualization the generated images (Gn) and their nearest neighbors in the training dataset (D,). As the iteration proceeds, the model can only copy images from the training dataset. Right: quantitative results of the generaliza- tion score of models over successive iterations. We use different colors to represent different dataset sizes, A smaller dataset has a larger decaying rate and even falls in the memorization regime at the start [[S]]. We use "iteration" to denote full cycle of training and generation, rather than gradient update.
when trained on large datasets (e.g., >> 24 images in CIFAR-10) 10) and to memorize when trained on small ones (e.g.. 2 images). However, since fix the size the training dataset for every iter- ation, the previous conclusion [[SS] that a larger dataset leads to generalization cannot fully explain the phenomenon observed in Finding I. We hypothesize that although sample size remains constant, the amount of information it contains decreases over time, making it easier for the model to memo- rize. Based on this hypothesis, we adopt the differential entropy quantify the information content and complexity of the continuous image distribution.
Definition 3.1 (Differential Entropy [26]).. Let X be a continuous random variable with probability density function f supported on the set X. The differential entropy H (X) is defined
H = Elg log([(X)))) -- / f(x) log f(x dæ
Estimation. However, the density function f of the image distribution is unknown. We use the fol- lowing Kozachenko-Leomee (KL) estimator proposed in well-known paper [27] empirically estimate ((X) from finite set of i.i.d. samples D drawn from the distribution P:
d
(D,(D) = ((P)) ((7) + log ca +
Σ log (x)) IED
(2)
D
where N R is the digamma function, i.e., the derivative of the logarithm of the gamma function; 7 is any positive integer, Cg denotes the volume the unit ball in the d-dimensional space; and e, (x) k(æ, xx,) represents the 7-nearest neighbor distance, where D. is the 7-th nearest neighbor of æ in the set D. Prior work [28] has shown that the KL estimator is asymptotically unbiased and consistent under broad conditions.
We use the KL estimator with y = 1 to measure the entropy the image dataset at each iteration. As shown in Figure B. the entropy of the generated image dataset- seetasettussddte as the training set in the next iteration consistentsll decreases over iterations. With a fixed dataset size, the only dataset- dependent term in Equation (2) is the sum of nearest-neighbor distances e(x) indicating that samples in D become increasingly concentrated. This suggests the distribution is becoming spiky. Figure further illustrates this trend by projecting high-dimensional images onto the subspace spanned by their top two eigenvectors. The visualization reveals that the generated images form numerous local clusters, while the overall support of the distribution remains relatively stable.
3.3 Finding III: The Correlation between Entropy and Generalization Score
We verify that the generalization score the trained model strongly correlated with the entropy of the training dataset. The similar collapsing trends of entropy and the generalization score motivate
5

---

1024
4096
16384
15
Entropy
1.3
11
10
Reall Images
Real Images
Iteration
21
Iteration
The evolving entropy of the training dataset over iterations. Under the replace paradigm, the training data is the generated data from the last iteration. Middle and Right: 2-D projection of data points onto the first two singular bases of the real dataset. The orange points represent the generated images at the 1-st and 21-st iterations, respectively.
Figure Decreasing entropy and visualizations. Left:
1024
4096
16384
1024
4096
16384
2048
8192
32768
2048
8192
32768
Generalization Score
Generalization Score
10a
10a
10a
10a
10000 11000 12000 13000 14000 15000 16000
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
Estimated Entropy
Estimated Variance
(a) Generalization score versus estimated entropy.
(b) Generalization score versus trace of covariance.
Figure 4: Scatter plots of the generalization score and properties the training dataset, i.c., entropy and variance. Each point denotes one iteration of training in the self-consuming loop. We use different colors to represent the results of different dataset sizes.
deeper investigation into their relationship. In Figure LR we present scatter plot entropy versus generalization score across different dataset sizes and successive iterations, with the y-axis shown on a logarithmic scale. Notably, the entropy the training dataset exhibits a significant linear rela- tionship with the logarithm of the generalization score. The Pearson correlation coefficient is 0.91 with p-value near zero, quantitatively confirming the strength this correlation. Furthermore, the scatter points corresponding line, suggesting the generality of the relationship between entropy and generalization. Specifically, training datasets with higher entropy consistently yield better generalization in the trained model. We also observe that larger datasets typically have higher entropy and result in better generalization, aligning with the conclusion in [US]] regarding the connection between dataset size and generaliza- tion. For comparison, Figure BIG shows the scatter plot of variance versus generalization score. The correlation appears substantially weaker than in Figure A suggesting that variance in the training dataset may not directly influence the generalization performance of the trained model.
different dataset sizes all approximately aligned along a single
Findings IIII collectively indicate that the generated data gradually collapses into compact clusters, as shown by declining entropy. This concentration promotes memorization the model of the next iteration and reduces its ability to generalize. In Appendix D we further extend these explorations the FFHQ dataset and accumulate paradigm, demonstrating more robust relation.
Conclusion.

---

Mitigating Model Collapse with Data Selection Methods
In this section, propose a sample selection method that selects subset of images from the candidate pool to mitigate the model collapse. Motivated by the empirical finding in Section & the selected training images should have high entropy. characterized by large nearest-neighbor distances and greater diversity. This objective can be formalized as the following optimization problem:
(D) DCS, DI=N
max
max
log min <(x, y), wED
DCS, 1
where S denotes the candidate pool. For the accumulate-subsampll setting, S is all the accumulated images so far. In the replace setting, S consists of the images generated by the previous model, with size twice that of the target training set.
Unfortunately, this non-convex max-min problem is hard to solve to a global solution efficiently, Alternatively, approximate the solution through greedy strategy inspired by farthest-point sam- pling, which aims to maximize the nearest-neighbor distance.
Greedy Selection. The procedure iteratively constructs subset D C S of size by adding the farthest point one at a time as follows:
Method I:
1. Initialization: Randomly select an initial point from the dataset S and add it to the set D.
2. Iterative Selection: At each iteration, for every candidate point æ E S \ DD compute the minimum distance from all points currently in D. Select the point with the maximum of these minimum distances and add it to D, i.e., Deelect arg max miDyeD k(æ, (x,y) y).
3. Termination: Repeat the selection process until [D] = N.
The Greedy Selection method can efficiently and effectively extract a subset with a large entropy. However, this greedy method carries risk of over-optimization, which may lead to excessively expanded distribution and progressively increasing variance the selected samples. To mitigate this, also provide the following Threshold Decay Filter, which can control the filtration strength by a decaying threshold.
Methods II: Threshold Decay Filter. The procedure constructs subset D C S of size N by iteratively selecting samples that are sufficiently distant from the current set The algorithm proceeds as follows:
1. Initialization: Set an initial threshold 7 > 0. Randomly select one point from the dataset S and add the set D.
2. fhreshold-based Selection: For each point S D. compute the distance from to all points D. If all distances are greater than the current threshold т, add æ to D.
3. Threshold Decay: If [D] < after complete pass through S DD reduce the threshold by multiplying it with decay factor E (0, 1), and repeat Step 2.
4. Termination: Repeat Steps 2-3 until [D] = N.
Threshold Decay Filter is soft variant Greedy Selection that provides adjustable control over the selection strength. When the initial threshold is set sufficiently high and the decay factor is close to 1, the Threshold Decay Filter behaves similarly to Greedy Selection. Conversely, both the initial threshold and decay factor are set to 0, the filter does not filter out any data point and reduces to the vanilla replace or accumulate- subsamppee paradigm. practice, we first extract image features using DINOv2 [29] model and compute distances in the feature space, i.e., R(xx y) [[h(x:) h(y)ll2. where i(-) denotes the feature extractor.
5
Experiments
We empirically evaluate how our data selection strategies introduced in Section interact with the two self-consuming paradigms: replace and accumulate-subampple In both settings. our method serves as plug-in component for selecting high-quality and diverse training samples from the can- didate pool. We demonstrate that the proposed strategies effectively alleviate memorization and
7

---

Vanilia Paradigm
With Threshald Decay Filter
With Groedy Selection
Peplace
Ascumulace
30.0
27-
25.0
20.0
24
25
17.9
15.
20
itoration
Iteration
Eoration
teration
Figure 5: Generalization Score of the trained model over iterations. We indicate the settings on top of the subfigures. In each subfigure, three different lines are used to represent the vanilla paradigm and variants augmented with the proposed selection methods.
Vanilla Paradigm
With Threshoid Decay Filter
With Greedy Selection
Rop.ace
Replace
70
00
70-
123
60
100
40
20
20
Iteration
Beration
Ileration
Deratton
Figure 6: FID of the generated images over iterations. indicate the settings on top of the subfigures. In each subfigure, three different lines are used to represent the vanilla paradigm and its variants augmented with the proposed selection methods.
reduce FID scores, thereby mitigating model collapse. Additionally, experiments on classifier- guidance (CFG) [50]] generation show that our method effectively mitigates diversity collapse of CFG.
Datasets. We conduct experiments on three widely used image generation benchmarks. CIFAR- 10 [23] consists of 32 32 color images 10 classes. Due to computational constraints, we use subset of 32,768 training images, Our goal is not to achieve state-of-thee FID among large diffu- sion models but rather demonstrate that our method mitigates memorization in the self-consuming loop. As shown in Section BL this subset is sufficient to observe the transition from generalization memorization. We also conduct experiments on subsets FFHQ [S]]], downsampled to 32 32 resolution, and MNIST [32], using 8,192 and 12, 2,,00 training images, respectively.
Model. For CIFAR-10 10 and FFHQ, employ a UNet-based backbone + designed for 32 x 32 RGB images, which predicts noise residuals. The network contains approximately 16M parameters. For MNIST, similar UNet-based architecture adapted for single-channel inputs, with total of 19M parameters. Detailed network configurations are provided in the Appendix.
Implementation. For iterative training and sampling, our implementation is based on the Hugging Face Diffusers codebase [[44] of DDPM. We use mixed-preeision training of FP16 to train the models. We adopt an Adam optimizer with a learning rate of 10-1 and weight decay of 10-6 The batch size is 128. A 1000-step denoising process is used, with all other hyperparameters default values. For Threshold Decay Filter, we use an initial threshold of 60 and a decay rate of 0.95. We show in the Appendix that our method robust wide range of hyperparameters..
their
Evaluation Metrics. We use the generalization score and entropy to evaluate the effectiveness of our method in mitigating memorization. Additionally, adopt the Fréchet Inception Distance (FID) [B5] as metric quantify the distributional divergence between generated images and real images.

---

Vanilia Paradiqm
With Threshald Decay Filter
With Groedy Selection
Rep-ace
Accumulate
Replace
Ascumulete
650
150
140
Entropy
350
620
850
720
600
800
320
550
500
Iterazion
Itertion
Itoration
moratinn
Figure 7: Estimated entropy of the training datasets over iterations. We indicate the settings on top of the subfigures. In each subfigure, three different lines are used to represent the vanilla paradigm and its variants augmented with our selection methods.
FFHO: Accumulate Greedy
FFHO: Accumulate Filter
Accumutate Greedy
Accumulstte Fiter
1.0
Oure
10
Ua Curve
Proportion Too
0.6
0.4
0.4
0.4
0.2
10 11
0.0
0.0
Rerations 10
1er atratiion
Iberations
Iterations
Figure 8: Proportion of the selected images from previous iterations or the real dataset. We use different colors to represent different sources. Particularly, the blue bars denote the proportion of the real images. The red line represents the 1/n curve that indicates the proportion of the real images we evenly select the data subset from all available images (accumulae-ssusaappll) We indicate the settings on top of the subfigures.
Results. To evaluate the efficacy of our selection methods in mitigating memorization within the self-consuming loop, we compare the generalization scores of two vanilla paradigms with those augmented by Greedy Selection or Threshold Decay Filter. As shown in Figure EL Greedy Selec- tion consistently improves generalization scores across all datasets and paradigms and is particularly effective in the accumulate-subsample setting. Importantly, our selection methods mitigate memo- rization without compromising FID performance; in fact, they can even slow down FID degradation. Figure R reports the FID of generated images over successive iterations. Under the accumulate- subsample paradigm, both methods yield notable improvements in FID, with Greedy Selection out- performing Threshold Decay Filter. For example, the vanilla accumulate paradigm reaches an FID of 75.7 iteration 8, whereas Greedy Selection significantly reduces it 44.7. On the FFHQ dataset under the replace paradigm, however, Threshold Decay Filter performs better, suggesting that adaptive selection strength may be beneficial in certain cases.
Analysis for the Improvement. We present the estimated entropy of the training datasets in Fig- ure As shown in the figure, our selection methods effectively increase the entropy of the training data at each iteration, consistent with their design. These more diverse, higher-entropy datasets subsequently enable the next-iteration model to generalize better, as evidenced by the improved generalization scores in Figure [
We further investigate which samples are selected by our methods under the eccumulate-subsampll paradigm, where the model has access to all prior synthetic images and the real images, but is trained on subset of them. Figure shows the proportion of selected samples originating from different sources, with the blue bar indicating the proportion of real images. As illustrated in the figure, both Greedy Selection and Threshold Decay Filter consistently select a significantly higher proportion of real images compared to the 1/n reference curve. For example, on the FFHQ dataset, Greedy Selection selects 65% real images at the 8-th iteration, while vanilla subsampling includes only 12.5%. This outcome arises because the image distribution progressively collapses into compact clusters over iterations, and our selection methods tend to prioritize boundary real images- by maximizing the nearest neighbor distance.

---

301
0 5 3
1
0 0000000
0
3 2
5
6
3.65
3 6 6
3
3
J
0
5 8
2
2
6 0 = 2
1
2
6
33333333300 3 00333333333 333333
3 333 233332 3
33
Entropy
3.85
CPO
fita*
3 5 8
5
6
444 44444444444
44
5.00
7 2
2
/
9
9
5 5 5
5
5 5
5
6
6
7
6
5
6
6
6
6 6 6 6
7
4
7
6
2
2
5
8
8
8 8
1 8
8
8
Peration
2
6
2
9
9 9
9
9
9
9
9
9
(d) Entropy versus itera-
(a) Unconditional
(b) CFG (Scale 2)
(c) CFG With Filtering
tion
Figure 9: Comparisons unconditional, CFG, and CFG augmented with filter method. (a)-(c): Generated samples on MNIST [[2]] at the 8-th iteration under the accumulate paradigm. The FIDs of images in (a)-(c) are 74.4, 66.2, and 22.4 respectively. (d): The estimated entropy of the generated dataset over iterations, which reflects the diversity of the generated images.
Diversity Improvement on Classifier-Free Diffusion Guidance (CFG) [30.. We further validate the effectiveness of our methods in the CFG setting and show that they substantially improve the di- versity of generated images. CFG is widely used conditional generation technique that consistently enhances perceptual quality but often sacrifices diversity [EO, 36, BZ]] Prior work by [B8] identifies the CFG scale as a key factor influencing the rate of model collapse and suggests that setting a moderate scale can help mitigate model collapse. Experimentally, we observe that the CFG genera- tion can indeed generate clearer images than the unconditional baseline, as compared in Figures Da and MR However, the diversity of generated images collapses rapidly even with modest guid- ance scale, with samples within each class soon becoming nearly identical. In contrast, Figures Ma and yd demonstrate that augmenting CFG with our data selection methods significantly improves image diversity and yields substantially lower FID scores compared to the vanilla CFG paradigm. These results demonstrate that our approach effectively mitigates the diversity loss of CFG while preserving its quality advantage throughout the self-consuming loop.
6 Conclusion and Discussion
In this work, we reveal wwwerraliaation-t-mmmmrriizzzzzi transition in diffusion models under recur- sive training, highlighting serious practical concern and offering new perspective model col- lapse. We empirically demonstrate that the entropy of the training data decreases over iterations and strongly correlated with the model's generalizability. Motivated by the findings, we propose an entropy-based data selection strategy that effectively alleviates the gemeraiization----mmmoiiiizzzi transition and improves image quality, thus mitigating model collapse.
Based on this work, we believe many future directions could be further investigated. This paper doesn't 't rigorously explain why the entropy is collapsing. The finite dataset size, training and sam- pling errors, and the model bias could all contribute to the collapsing entropy (or the information loss). We envision that more formal theoretical analysis of collapse dynamics could be developed based on theoretical models such as the mixture of Gaussians [ВУ--П]] Besides, this framework could be extended to the language modality, investigating the discrete diffusion model [ [el Ad- ditionally, a more efficient algorithm is needed, as the current greedy selection method is computa- tionally expensive.
Acknowledgement
LS, MW, HZ, ZZ, and QQ acknowledge funding support from NSF CCF-2212066, NSF CCF. 2212326. NSF IIS 2402950, and ONR N000142512339, and the Google Research Scholar Award. MT is grateful for partial supports by NSF Grants DMS-1847802, DMS-2513699 DOE Grants NA0004261, SC0026274, and Richard Duke Fellowship. We also thank Prof. Peng Wang (Univer- sity of Michigan and University of Macau), Mr. Xiang Li (University of Michigan), and Mr. Xiao Li (University of Michigan) for fruitful discussions and valuable feedback.
10

---

References
[1] Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Strong model collapse. arXiv preprint arXiv:2410.04840 2024.
[2] Rylan Schaeffer. Joshua Kazdan, Alvan Caleb Arulandu, and Sanmi Koyejo. Position: Model collapse does not mean what you think. arXiv preprint arXiv:2503.03150, 2025.
[3] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross J. Anderson, and Yarin
Gal. Al models collapse when trained on recursively generated data. Nature, 631(8022):75- 759, 2024.
[4] Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. Collapse or thrive? perils and promises of synthetic data in self-generating world. arxiv preprint arxv:2410.16713,, 2024.
[5] Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. In ICLR, 2024.
[6] Ananda Theertha Suresh, Andrew Thangaraj, and Aditya Nanda Kishore Khandavally. Rate of model collapse in recursive training. arxiv preprini arxiv:2412.17646, 2024.
[7] Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. In NeurlPS, 2024.
[8] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruy Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David Donoho, and Sanmi Koyejo. model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413 2024.
[9] Elvis Dohmatob, Yunzhen Feng. Arjun Subramonian, and Julia Kempe. Strong model collapse. arxiv preprint arxiv:2410.04880 2024.
[10] Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical under- standings of self-consuming generative models. In ICML,, 2024.
[11] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hos- sein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming gener- ative models go MAD. In ICLR, 2024.
[12] Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse. axxiv preprint arxiv:231 12200, 2023.
[13] Xulu Zhang, Xiaoyong Wei, Jinlin Wu, Jiaxin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. Generating on generated: An approach towards self-evolving diffusion models. arxiv preprint 2ativ:2502.099633. 2025.
[14] Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse C. Cresswell, and Gabriel
A geometric framework for understanding memorization in generative models. arxiv preprint arxiv:2411.00111 2024.
[15] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. In ICML, 2024.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[17] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data dis- tribution. In NewrIPS, 2019.
[18] Pascal Vincent. connection between score matching and denoising autoencoders. Neural Computing, 13:1661-1674, 2011.

---

[19] Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John P. Collomosse, and Richard G. Baraniuk. Self-improving diffusion models with synthetic data. arxiv preprint arxiv:2408. 16333, 2024.
[20] Elvis Dohmatob, Yunzhen Feng, Pu Yang, François Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In ICML, 2024.
[21] Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification. arXiv preprint 2xxx:22006.07515, 2024.
[22] Apratim Dey and David L. Donoho. Universality of the 72/6 pathway in avoiding model collapse. arxiv preprint arxiv:2410.22812 2024.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] TaeHo Yoon, Joo Young Choi, Schyun Kwon, and Ernest K Ryu. Diffusion probabilistic mod- els generalize when they fail to memorize. In workshop structured probabilistic inference { generative modeling, ICML, 2023.
[25] Ahmed M. Alaa, Boris van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In ICML, 2022.
[26] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[27] Leonenko Kozachenko. Sample estimate of the entropy of random vector. Problems of Information Transmission, 23:9, 1987.
[28] Jan Kybic and Ivan Vnucko. Approximate nearest neighbor search for high dimensional entropy estimation for image registration. Signal Processing, 92:1302-1316, 2012.
[29] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024, 2024.
[30] Jonathan Ho and Tim Salimans. a2x55 2022.
Classifier-free diffusion guidance. arxiv preprint
[31] Anoop Jain, Parag Sarda, and Jayant R. Haritsa. Providing diversity in k-nearest neighbor query results. In Honghua Dai, Ramakrishnan Srikant, and Chengqi Zhang, editors, PAKDD, 2004.
[32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE,
1998.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[34] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: Statc-of-the-art diffusion models. https: //gh://.. com/ www. http//uggingfa/e/////// ditfusers, 2022.
[35] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
[36] Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? A fine-grained analysis in simple setting. In NeurIPS, 2024.
12

---

[37] Xiang Li, Rongrong Wang, and Qing Qu Towards understanding the mechanisms of classifier- free guidance. arXiv preprint arXiv:2505. 19210, 2025.
[38] Youngseok Yoon, Dainong Hu, lain Weissburg, Yao Qin, and Haewon Jeong. Model collapse in the self-consuming chain of diffusion finetuning: A novel perspective from quantitative trait modeling. arXiv preprint arXiv:2407. 17493,, 2024.
[39] Xiao Li, Zekai Zhang, Xiang Li, Siyi Chen, Zhihui Zhu, Peng Wang, and Qing Qu Under- standing representation dynamics of diffusion models via low-dimensional modeling, arXiv preprint arXiv:2502.0574 2025.
[40] Peng Wang, Huijie Zhang. Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint 4rXiv:2409.02426, 2024.
[41] Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu Exploring low- dimensional subspaces in diffusion models for controllable image editing. arXiv preprint arXiv:2409.02374, 2024.
[42] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In JCML, 2024.
[43] Kevin Rojas, Yuchen Zhu, Sichen Zhu, Felix X.-F. Ye, and Molei Tao. Diffuse everything: Multimodal diffusion models on arbitrary state spaces. arXiv preprint arXiv:2506.07990 2025.
[44] Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06668, 2025.
[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[46] Sierra Calanda Wyllie, Ilia Shumailov, and Nicolas Papernot. Fairness feedback loops: Train- ing on synthetic data amplifies bias. In ACM FAccT, pages 2113-2147, 2024.
[47] Mohamed EI Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Mérouane Debbah. How bad is training on synthetic data? A statistical analysis of language model collapse. arxiv preprint 0rxxv:2404005099 2024.
[48] Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self-consuming generative models with curated data provably optimize human preferences. In NeurlPS, 2024.
[49] Shi Fu, Yingjie Wang, Yuzhu Chen, Xinmei Tian, and Dacheng Tao. A theoretical perspective: How prevent model collapse in self-consuming training loops. In ICLR, 2025.
[50] Huminhao Zhu Fangyikang Wang, Tianyu Ding, Qing Qu, and Zhihui Zhu. Analyzing and
mitigating model collapse in rectified flow models. arXiv preprint arXiv:2412.08/75. 2024.
[51] Hugo Cui, Cengiz Pehlevan, and Yue M. Lu. A precise asymptotic analysis of learning diffu- sion models: theory and insights. arxiv preprint arxiv:2501.03937, 2025.
[52] Xuekai Zhu, Daixuan Cheng, Hengli Li. Kaiyan Zhang. Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou. How to synthesize text data without model collapse? arxiv preprint arxiv:2412. 2 14689, 2024.
[53] Ian J. Goodfellow, Jean Abudec-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pages 2672-2680, 2014.
[54] Peng Wang. Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409. 02426, 2024.
13

---

[55] Yuqing Wang. Ye He, and Molei Tao. Evaluating the design space of diffusion based generative models, NewrIPS, 2024.
[56] Huijie Zhang, Zijian Huang, Siyi Chen, Jinfan Zhou, Zekai Zhang, Peng Wang, and Qing Qu Understanding generalization in diffusion models via probability flow distance. arXiv preprini arXiv:2505.20123. 2025.
[57] Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In ICLR, 2024.
[58] Jie An, De Wang, Pengsheng Guo, Jiebo Luo, and Alexander Schwing. On inductive biases that enable generalization of diffusion transformers. arXiv preprint arXiv:2410.21273, 2024.
[59] Binxu Wang and John J Vastola. The unreasonable effectiveness of gaussian score approxima- tion for diffusion models and its applications. arXiv preprint 1rXiv:2412.09726, 2024.
[60] Xiang Li, Yixiang Dai, and Qing Qu Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. In NeurlPPS, 2024.
[61] Matthew Niedoba, Berend Zwartsenberg, Kevin Murphy, and Frank Wood. Towards mecha- nistic explanation of diffusion model generalization. arXiv preprint arXiv:2411.19339, 2024.
[62] Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv:2412.2029,, 2024.
[63] Xiangming Gu, Chao Du, Tianyu Pang. Chongxuan Li. Min Lin, and Ye Wang. On memoriza- tion in diffusion models. Transactions on Machine Learning Research, 2025.
[64] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from dif- fusion models. In USENIX Security Symposium, pages 5253-5270, 2023.
[65] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Dif- fusion art or digital forgery? investigating data replication in diffusion models. In CVPR, pages 6048-6058. 2023.
[66] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Un- derstanding and mitigating copying in diffusion models. NeurIPS, 01447733-47800 2023.
[67] Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In ICLR, 2024.
[68] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurIPS, 2022.
[69] Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a statistical theory of data selection under weak supervision. In /CLR, 2024.
[70] Reyhane Askari Hemmat, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro As- tolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano, Improving the scaling laws of synthetic data with deliberate practice. arxiv preprint axiv:2502.155888, 2025.
[71] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 281 2818-2826, 2016.
[72] Sofiane Abbar, Sihem Amer- Yahia, Piotr Indyk. Sepideh Mahabadi, and Kasturi R. Varadara- jan. Diverse near neighbor problem. In SoCG, 2013.
[73] George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony I.. Caterini, J. Eric T. Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In NewrIPS, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state our motivations, empirical findings,
proposed methods, experimental validation, and other contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We carefully discuss the problem setup and assumptions in Section Q While our findings are empirical, exploring their theoretical foundations is an interesting avenue for future investigation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: We don't have theoretical results that need assumptions or proofs. This work empirically focus the generalization-to wtr.rttooo-t-mmmmmmiiiiioo transition in models happened dur- ing model collapse.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details our methods and experiments are provided in Sec- tion B and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We have clearly cited all datasets used in this paper. which are publicly ac- cessible online. Our implementation based on the Hugging Face Diffusers codebase [64]] for DDPM. We will release the code upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (ittpa:) Cttp///ooeSSSaiiia wti.iiisssinpoiccc.o.oom.m..m for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the needed training and test details in Sections Bl and EL
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We present the Pearson correlation coefficient in Section & to provide statisti- cal significance of the correlation between entropy and generalization score.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Our research conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impacts of our paper in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose risks of misuse; rather, our methods help mitigate the risk of model collapse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all datasets, code, and models used, and comply with their licenses and usage terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy lttees//Doarriocccc///ofeerrrerp for what should or should not be described.
21

---

A Related Work
A.1 Model Collapse
As state-of-the-art generative models continue to improve in image quality, Al Al-generated images have become increasingly indistinguishable from real ones and are inevitably incorporated into the training datasets of next-generation models. In fact, [[T]] have shown that the LAION-5B dataset [85], which is used to train Stable Diffusion, indeed contains synthetic data. Unfortunately, recent studies (M. EL D. II [19-2]] demonstrate that model performance deteriorates under such recursive training, potentially leading to model collapse, where the model generates increasingly homoge- neous and meaningless content. The concept of model collapse was first introduced in [G], which also provides a theoretical framework based on Gaussian models. Their analysis shows that if a Gaussian model is recursively estimated using data generated by its predecessor, its variance con- verges to zero collasosnn the distribution into a delta function concentrated at a single point.
Following this important line of work, substantial research has further explored the model collapse phenomenon. [Ki] extend the theoretical analysis from Gaussian to Bernoulli and Poisson models. [ study recursive training under high-dimensional linear and ridge regression settings, providing a linear error rate and proposing optimal regularization to mitigate collapse. [20]] argue that the
conventional scaling laws for foundation models break down when synthetic data is incorporated into the training set. [26]] demonstrate that model biases are amplified through recursive genera- tion and proposes algorithmic reparation techniques to eliminate such biases and negative semantic shifts. | introduce a statistical model that characterizes the collapse process in language models and theoretically estimates the maximum allowable proportion of synthetic data before collapse oc- curs, validated empirically. [48]] show that human preferences can be amplified in an iterative loop: modeling human curation as reward process, the curated distribution P: converges p" that max- imizes the expected reward as - oc. In the work of [48], the reward r(z) is a pointwise function over individual images, whereas our method can be viewed as a curation strategy that maximizes dataset-level entropy at each iteration. Whether the theory in [88] extends to dataset-wise rewards r(S), such as dataset entropy, remains an open question. Nonetheless, their intuition aligns with our results that entropy can be iteratively enhanced compared to vanilla methods. [Я9] provide the first generalization error bounds for Self-Consuming Training Loops (STLs), showing that mitigat- ing model collapse requires maintaining non-negligible portion of real data and ensuring recursive stability. In contrast, our work focuses on a specific collapse phenomenon- the generalization-to- memorization transition- and introduces an entropy- -based data selection algorithm to mitigate this behavior. [50] provide rigorous theoretical analysis for the model collapse in rectified flow models and then propose methods to mitigate it. Finally. [5]] investigate collapse in diffusion models by an- alyzing the training dynamics of a two-layer autoencoder optimized via stochastic gradient descent, showing how network architecture shapes the model collapse.
To mitigate model collapse, several strategies have been proposed in prior work. One common approach is to accumulate all previously generated samples along with real data into the training sef-referred as the accumulate paradigm in our paper. This strategy has been both empirically and theoretically validated. For example, [8] show that the accumulate paradigm prevents divergence of test error under a linear regression setup. [22]] further establish the universality of the error upper bound across a broad class of canonical statistical models. In a related setting, [[]] study an catuuuulatt-uuuaamppae
variant, confirms that the test error plateaus and examines interactions between real and synthetic data. [S]] provide theoretical guarantees for stability in iterative training, assuming the initial model trained on real data is sufficiently accurate and the clean data proportion in each iteration remains high. Despite these findings, recent work [9] present a robust negative result, showing that model collapse persists even when real and synthetic data are mixed. Similarly, [[[]] argue that the accumulate paradigm merely delays, rather than prevents, collapse. Introducing fresh real data in each iteration may be necessary for long-tern robustness. Other complementary approaches include verification [2]]] and re-editing [52]] For instance, [2T]] theoretically underscore the importance of data selection, though it does not propose a specific method. [522] target language models and introduces token-level editing mechanism with theoretical guarantees. Compared to prior work, this paper proposes a novel catropy-based data selection method for diffusion models that improves both generalizability and image quality, thereby mitigating model collapse.
Over the past few years, extensive research has explored model collapse from various perspectives and dimensions. An important study [2] makes thorough survey about different definitions and
22

---

patterns investigated in previous work. A prominent line of work [3-6] focuses on the phenomenon of variance collapse, both empirically and theoretically demonstrating that models progressively lose
information in the distributions tail, with variance tending toward zero. Another series of studies
[5) I--I0J investigates model collapse through the lens of population risk and distributional shift, ob- serving that the generated distribution increasingly diverges from the true data distribution, leading to rising population risk across recursive training cycles. Moreover, several works [ННН report that models begin to generate hallucinated unrealistic data as collapse progresses.
[20 further
suggest that the inclusion of synthetic data alters the scaling laws of model performance. In addi-
tion. [5I]] study the progressive mode collapse [|كا]] in diffusion models, where the number of modes
in the generated distribution gradually decreases. In this paper, we introduce a novel perspective for analyzing model collapse in diffusion models by uncovering a generalization to--ttt--eee to-memorization transition. We show that this transition is closely tied to the entropy of the training dataset, which serves key indicator of model generalizability. Our findings further motivate the development of entropy-baas based data selection strategies that effectively mitigate model collapse.
A.2 Generalization and Memorization
Recent studies [ | have identified two distinct learning regimes in diffusion models, depend- ing on the size of the training dataset and the model's capacity: (I) Memorization regime, when models with sufficient capacity are trained on limited datasets, they tend to memorize the training data; and (2) Generalization regime, as the number of training samples increases, the model be- gins to approximate the underlying data distribution and generate novel samples. To investigate the transition between these regimes, [54]] show that the number of training samples required for the transition from memorization to generalization scales linearly with the intrinsic dimension of the dataset. In addition, the analysis of training and generation accuracies in [55]] provides potential step toward quantifying generalization. [56] propose theoretically grounded and computationally efficient metric, Probability Flow Distance (PFD), to measure the generalization ability of diffusion models. Specifically, PFD quantifies the distance between distributions by comparing their noise- to-data mappings induced by the probability flow ODE. Meanwhile, concurrent work also explores memorization and generalization separately. To understand generalization, studies such as [S7. 58] attribute the generalization to implicit bias introduced by network architectures. Other works study the generalized distribution using Gaussian models [59 600 and patch-wise optimal score functions [B1, 62]. As for memorization, it is investigated in both unconditional and conditional [[33], as well as the text-to-image diffusion models [644] 65). Additionally, methods mitigate memorization in diffusion models have been proposed in [66, 67]]- Distinct from prior work, our study is the first to establish a connection between model collapse and the transition from generalization memo- rization. This connection not only offers novel perspective to understand model collapse but also provides insights mitigate it by mitigating memorization.
A.3 Data Selection
668 [ focus on data pruning techniques, but not necessarily in the context of self-consuming loops.
While both their approaches and ours demonstrate the benefits of data selection, there are several key differences:
Objective and evaluation: [68
primarily study how pruning improves scaling laws, achieving higher accuracy as dataset size varies. In contrast, our work examines how model performance evolves over an iterative training loop with fixed dataset size, focusing on the coterllization-tommmmiiiin
transition.
Task and criteria: Prior works focus classification tasks, where sample selection depends on label information. Our method targets generative modeling, where selection is based on dataset entropy rather than label-driven criteria. Our objectives also differ: prior work emphasizes classification accuracy, while we address model collapse from a generalization perspective, providing different angle of analysis.
Pruning strategy: Methods in [68-ZZ] largely rely on per-sample evaluation, whereas our approach considers global dataset structure and relationships between samples. While this may lead increased computational complexity, it opens up new possibilities for designing pruning criteria beyond per-sample evaluation.
23

---

Entropy definition: In [Z0]], while the authors use generative model to sample images, the goal to improve the classification performance of classifier. And the entropy used in their method is defined in the prediction space of classifier, which is different from the entropy of a dataset measured our work.
B Experimental Details
B.1
Network Structure
For CIFAR-I0 and FFHQ, we use UNet-based backbone, taking RGB images as inputs and pre- dicting noise residuals. Our implementation is based on the Hugging Face Diffusers base code [34]]. The architecture hyperparameters of the neural network are listed as follows:
The numbers of in-channel and out-channel are 3.
The number of groups for group normalization within Transformer blocks is 16.
The number of layers per block
The network contains down-sampling blocks and 6 up-sampling blocks.
The numbers of feature channels for the 6 blocks are 48, 48, 96, 96, 144, 144 respectively,
For MNIST, we use a similar UNet-based backbone, taking single-channel images as inputs and predicting noise residuals. The architecture hyperparameters of the network are listed as follows:
The numbers of in-channel and out-channel are 1.
The number of groups for group normalization within Transformer blocks is 32.
The number of layers per block is 2.
The network contains down-sampling blocks and 4 up-sampling blocks.
The numbers of feature channels for 4 blocks are 64, 128, 256, 512 respectively.
B.2 Implementation Details
In this paper, we use DDPM as our generative method. For efficiency, we use the FP-16 mixing precision train our models, which inherently implemented by the Hugging Face Diffusers code- base. The batch size of training all datasets is set to be 128. A 1000-step denoising process is used as the reverse process. For CIFAR-10, the epoch number set to 500; for FFHQ. the epoch number is set to be 1000. We use the Adam optimizer with a learning rate of 10-4 and a weight decay of 10-6, Other experimental hyperparameters are exactly
default values in the original Hugging
Face Diffusers codebase. We use the DINOv2 model [29] to extract features of images and then calculate the distance between two images in the feature space. We use the Inception model [[]]]] to extract features of images to calculate the FID score. All experiments conducted single NVIDIA A-100 GPU.
B.3 Pseudo-codes for the Algorithms
We present the pseudo-code of the Greedy Selection and Threshold Decay Filter methods in Algo-
rithms [1 and
C Ablation Study
C.1 Training on More Samples
In Section of the main paper, we augment the vanilla replace paradigm with our data selection methods. Specifically, under the replace setting, the vanilla baseline generates N images and trains next-iteration model based solely on these N images from the previous iteration. Instead, data selection methods generates 2N images at each iteration, and select subset of N images from the 2N images for training the next-iteration model. One nature question is: what if we also generate 2N images and then train the next-iteration model on the entire 2N-image dataset without filtering.
24

---

Algorithm Greedy Selection
Require: Dataset S, target size N, distance function :((:))
Ensure: Selected subset 'D of size N
1: Initialize D < {random point from S}
2: while [D] < do
3: for all 8 do
4: Compute d(x) + mingeD k(z, v)
5: end for
6: Jselect + arg ma d(æ:
7: D D--DU --UU (Tislet))
8: end while
9: return D
Algorithm 2 Threshold Decay Filter
Require: Dataset S, target size N, initial threshold decay factor (0, 1), distance
function k(-,-)
Ensure: Selected subset D of size
1: Initialize 'D <- (random point from S}
2: while (D] N do
3: added < false
4: for all x E do
5: Compute dmni (x) +- mingeD k(z,y)
6:
dmmin (x) > t then
7:
D t DU {z}
8:
added +- true
9:
end if
10:
(D = then
11:
return D
12:
end end for
13:
14:
if not added then
end a
No point added decay threshold
16:
end while
18:
return D
Next, show that when training the whole 2.V dataset, the performance (FID score) of the model is between the vanilla replace setting (generating images and training on those N images) and the data selection method (generating 2NV images and training on the selected N N-images subset). The results on CIFAR- are shown in Figure [T]
Several conclusions can be drawn by comparing the results across these settings.
Incorporating more data into the traniig-samplinn recursion indeed mitigate the rate of model collapse. Compared to the vanilla replacement paradigm (i.e., the first line), using 2.V images (second line) yields improved performance. This aligns with prior findings [ [TX that sample size is key factor influencing the collapse rate.
Further augmenting the training data with our selection method leads to even better perfor- mance than training the full set of 2N images. The results validate the effectiveness our selection method, achieving a better FID performance while largely decreasing the training budget.
In fact, there trade-off for the filter ratio. There are two clear extremes:
If the ratio 1, all 2N generated images are used for training. we show Figure I0 still degrades faster than filtering (ratio equals to 1/2)
25

---

Vanilia Paradigm
Thresheld Decay Fllter
Vanilia Paradigm 2N images
Greedy Selection
CIFAR Replace
160
140
120
100
FID
80
60
40
2
3
4
5
Iteration
Figure 10: FID of the generated images iterations. We add additional setting of generating 2N images and training the next model on the entire 2 -immagees data without filtering.
Ratio 0.2
0.4
0.6
0.8
1.0
FID
56.0
44.7
40.0
38.2
49.5
Table I: FID comparison of the models. We use various filter ratios to get the training subsets for those models.
Conversely, if the ratio approaches too few images are selected for training, which detri- mentally starves the model of data.
We then use different filter ratios to get training subsets from the 2N generated images and then train models those filtered datasets. The results in Table II show that intermediate ratio yields the best FID performance.
C.2 Different Decay Rates
The decay rate one important hyperparameter for the Threshold Decay Filter. In this section, use different decay rates and show that the filter is robust to wide range of decay rates.
Figure [ shows the FID scores of generated CIFAR-10 10 images across iterations for different meth- ods. As shown in the figure, the data selection methods consistently outperform the vanilla accumu- late paradigm, indicating strong robustness of the hyperparameter.
D Additional Results
D.1 The GenernliiatonnttoMemmrriaaao Collapse on Accumulate Paradigm
This subsection presents additional results on the accumulate paradigm and FFHQ dataset as sup- plement to the explorations Section Specifically, we show in Figure I2 that the generalization- to memomizzaiion collapse also occurs on the accumulation paradigm. We note that model collapse different definitions previous studies. Here, we clarify that our results show generalization score consistently decreases over early iterations. However, we cannot determine whether the score eventually collapses zero converges lower bound in the accumulate paradigm, as we only have fewer than 10 iterations- cerations-insufficiinn to draw conclusions about convergence.
26

---

Vanilla Paradigm
Threshold Decay (a
With Greedy Selection
With Threshold Decay Filter (a 0.95)
Threshold Decay Filter 0.30)
With Threshold Decay Filter (a 0.98)
CIFAR Accumulate
90
80
70
60
FID
50
30
1
3
4
8
Iteration
Figure 11: FID score of the generated images over iterations. We compare the results for different decay rates on CIFAR-10 dataset.
Iteration 1
D1
G3
Iteration
D3
Iteration 8
Dz g8
Figure 12: The aamneraalzation-tommmmaiaaaa transition on the accumulate paradigm. Similar Figure we visualize the generated images and their nearest neighbors in the training dataset. As the iteration proceeds, the model loses generalization ability, collapsing into the memorization regime.
D.2 The Robust Relationship between Generalization Score and Entropy
In Figure A we plot the results on CIFAR- 10 datasets, demonstrating the dataset ineeinndeeendnnt re- lationship between generalization score and estimated entropy. This subsection further incorporates the results from FFHQ and the accumulate paradigm to show a more general relation.
Specifically, conduct the self-consuming loop for FFHQ with dataset sizes of 8, 192, 16, 384, and 32, 768. Besides, we also include the results from Appendix DJI Combining those, we show more robust relationship in Figure IEL where all the points align around the red dashed line. This result suggests that the entropy the training dataset is a independnn indicator for the generalization score.
D.3 Generated Images Iterations
This subsection provides additional generated images of the trained model across iterations and dataset sizes in grid format. These images generated from the vanilla replace paradigm.
27

---

1024
4096
16384
8192
16384
Accumulate
2048
8192
32768
8192
32768
Generalization Score
103
10a
1.0
1.1
1.2
1.3
1.4
1.5
1.6
Estimated Entropy
Figure 13: Scatter plots the generalization score and estimated entropy of the training data. Each point denotes iteration of training in the self-consuming loop. In addition to the results in Figure Hal we include more results from FFHQ and on the accumulate paradigm. We use different colors denote the loops, different shapes (circles and triangles) represent CIFAR and FFHQ, and solid versus hollow markers to distinguish the replace and accumulate paradigms.
Iteration
Iteration
Iteration
Iteration 4
Iteration
Iteration 6
Figure 14: Generated images of models trained on 1024 CIFAR-10 images over iterations.
Figures B to 1 present the generated images of the model trained CIFAR-10 with various dataset sizes across successive iterations. As shown in Figures I and DS the diffusion models tend to memorize small training datasets throughout the recursive process, exactly copying training images during generation. Because the generated samples are nearly identical to the training data, image quality does not noticeably degrade. However, duplicated generations emerge in later iterations, and diversity declines as the model gradually loses coverage over parts of the original images.
On the contrary, the large dataset of 768 images, the models generate novel images in the beginning. As the model collapses, the quality and diversity of the images gradually degrade over iterations.
28

---

Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Ittration 6
Figure 15: Generated images models trained on 2048 CIFAR-10 images over iterations.
D.4 Generated Image Distribution Becomes Spiky
In the main paper, we adopt the KL estimator of Equation (0) evaluate the entropy given a finite image dataset. With a fixed dataset size, the only dataset-dependent term in Equation (22 is the sum of nearest-neighhor distances =(x) indicating that samples D become increasingly concentrated. This suggests the distribution is becoming spiky. Figure B in the main paper also illustrates this trend by projecting nigh-dimensional images onto the subspace spanned by their top eigenvectors. The visualization reveals that the generated images form numerous local clusters, while the overall support of the distribution remains relatively stable.
To quantitatively measure the spiky degree the empirical image distribution, we further adopt the Mean Nearest Neighbor Distance (MNND) [GT, 72], which removes the data-independent constants and logarithm in the KL estimator:
MNND(D,) Dist(D,, D,) = [Prl ZED,
Σ min d(æ, z).
(3)
sED,
lower MNND suggests tightly clustered and spiky distribution, while a higher distance suggests dispersion and diversity. The KL estimator be related to MNND through
(1 B
MNND(D.),
where Hi represents the estimated entropy of DD with A = 1 and B is a constant offset given fixed dataset size.
We want to clarify the nuance between MNND and variance. A spiky distribution does not imply that all data points are concentrated single small region- this behavior has already been illustrated by the variance collapse behavior [3-5]. Specifically, Figure 20 shows that the variance of the generated dataset only slightly decreases along the successive iterations and is far from complete collapse. On the contrary, MNND decreases almost exponentially and reaches small value after 10 iterations. Thus, the results align with the prior claim [2, [ that the collapse of variance progresses at such slow pace that it is rarely practical concern in real-world applications. In contrast, the collapse in MNND emerges at an early stage, highlighting a critical memorization issue caused by training on synthetic data.
29

---

Iteration
Iteration
Iteration 3
Iteration 4
Iteration 5
Ittraaion 6
Figure 16: Generated images models trained on 4096 CIFAR-10 images over iterations.
Model
A
C
FID
28.0
30.8
27.5
Table FID comparison of three models.
D.5 Self-consuming Loop with Fresh New Data
[T]]] show that incorporating fresh real data can further mitigate model collapse. this section, we conduct experiments verify the effectiveness of our methods in this paradigm. The candi- date data pool in each iteration is jointly composed of three sources: (1) the original real data, (2) synthetic data generated by the previous model, and (3) fresh real data that not used in earlier iterations. This experiment is designed to simulate real-world deployment, where generative models are continuously updated with mixture of prior synthetic data and incoming fresh real data, and the training budget also increases. Concretely. we first train Model A on 32,768 real images, which then generates 10,000 synthetic images. We construct a data poolsimulating an Internet-scale sourceby combining the 32,768 original real images, 10,000 synthetic images, and 10,000 additional fresh real images. From this pool, our entropy-based selection method chooses 40,000 images as the training dataset for Model C. For comparison, Model B is trained on 40,000 randomly selected images from the same pool.
The FID scores of Models A, B, and C are summarized in Table Ø As the results indicate, our entropy-brat based selection method (Model C) enables the model to outperform the baseline (Model A), and far better than random sampling (Model B). This demonstrates that, in a practical scenario with evolving datasets, our approach not only mitigates model collapse but also delivers tangible performance gains.
D.6 Additional Metrie
Since our Greedy Section method is performed on the feature space extracted by the DINO model, also use FDDINO [+ alternate for FID to measure the quality of the generated images. As
30

---

Iteration
Iteration
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Figure 17: Generated images models trained on 8192 CIFAR-10 images over iterations.
show in Figure 21 the Greedy Selection method can also improve the FDDINO metric compared to the vanilla paradigms.
D.7 Additional Results for Section
We provide visualization of the generated FFHQ images with their nearest training neighbors to show the model's generalizability. Figure M shows some images for different training paradigms at 5-th iteration in grid format. With augmentation from the Greedy Selection method, the model generates images that deviate more from the training set compared to the vanilla accumulate paradigm, thereby enhancing its generalization ability.
E
Impact Statement
In this work, we investigate a critical failure mode of diffusion models known as model collapse, which when models are recursively trained synthetic data and gradually lose their gen- eralization ability and generative diversity. As Al-generated data is unintentionally or deliberately incorporated into the training sets of next-generation models, understanding and mitigating model collapse is essential for ensuring long-term model reliability and performance. Our study identi- transition, demonstrates the relation between entropy of the
fies the pnteeaaiiaaiion---mmmmmiiizziz training set and the generalizability of the trained model, and proposes practical solutions to mitigate model collapse through catropy-based data selection.
We believe our findings will contribute to the responsible development and deployment of generative models, especially in scenarios where data source may be mixed partially synthetic. While tech- niques for analyzing collapse may be misused to intentionally degrade generative models through poisoning attack, our intent is solely build more robust, transparent, and self-aware Al systems. We encourage researchers in generative AI to use these results to mitigate model collapse and to build reliable models, even when training data contains Al-generated samples-a scenario that may become increasingly common in the future.
31

---

Iteration 1
Iteration 2
Iteration 3
Iteration
Iteration 5
Iteration 6
Figure 18: Generated images of models trained on 16384 CIFAR-10 10 images over iterations.
Iteration
Iteration
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Figure 19: Generated images of models trained on 32768 CIFAR- 10 images over iterations.
32

---

MNND 1024
MNND 16384
Variance 4096
MNND 4096
Variance 1024
Variance 16384
2500
1.0
2000
Trace of Variance
0.8
MNND
1500
0.6
1000
0.4
500
0.2
0.0
2
8
9
10
Iteration
Figure 20: The MNND and the trace of the covariance matrix over iterations.
Replace Strategies
Venila Replace
with Greedy Selection
1600
1400
ED, DINO
1200
1000
800
2
3
7
Iteration
Accumulated Strategies
Accumrultee
1100
Accurnu ated with Groedy Sclection
1000
FD DID
900
800
700
Iteration
Figure 21: FDDINO comparison vanilla paradigms and our methods.
33

---

Vanilla Accumulate Generalization =29.6
With Greedy Selection (Gereralization soore 37.0)
Figure 22: The visualization of the generated images and their nearest neighbors in the training dataset. Each pair of rows corresponds to one group: the top row shows the generated images, and the bottom row shows their nearest training images.
34


Reviews:
[{"review_id": "cEj7XR1GNr", "reviewer": "Reviewer_dMTS", "summary": "This work studies model collapse of successive generative models trained on the previous version’s output. It shows clear evidence of models increasingly replicate training data instead of generating novel content during iterative training on synthetic samples, and proposed an entropy-based data selection technique that can help mitigate the collapse.", "questions": "- Given that DINOv2 is used to extract image features for the distance computation, the authors could examine the improved DINOv2 evaluation metrics as presented here rather than FID - https://arxiv.org/abs/2306.04675. With the many known issues with FID, FD_DINOv2 would be a better choice for Figure 6.\n- It is interesting that there doesn't seem to be any evidence that supports adding in generated samples helps improve model performance. I understand that in a real use case training on internet data you do not necessarily know what is real or generated and so need automated ways to mitigate collapse, but I wonder if the authors can comment on this. i.e. can the entropy-based data selection result in a better model than naive sampling real images.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- This paper is well written, topical, contains well-designed experiments and reports interesting results.\n\n- The technique introduced to mitigate model collapse has strong supporting evidence of its utility.", "was_revised": true, "final_justification": "My Rating of 5: Accept still holds, as this is a topical ML paper with good experimentation & results. I thank the authors for their additional experiments and clarifications.", "rebuttal": "We thank the reviewer for the detailed and constructive comments.\n\n**Q1:** ***The authors could examine the improved DINOv2 evaluation metrics rather than FID.***\n\n**A1:** We fully agree that incorporating multiple evaluation metrics is important for comprehensively assessing image quality during iterative training. The following tables present the FD_DINOv2 for the generated images over iterations. Our Greedy Selection method yields lower FD_DINOv2 scores in both “replace” and “accumulate” paradigms.\n\n| Algorithm      | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 |  Iter 7 |\n|:-----------:|:----------:|:----------:|:------------:|:----------:|:------------:|:----------:|:------------:|\n| Vanilla Replace     |642.3 | 908.0 | 1128.9 | 1336.3 | 1522.4 | 1686.9 | 1802.3|\n| Replace with Greedy Selection      |642.6 | 820.4 | 911.8 | 973.4 | 1025.9 | 1074.6 | 1128.1|\n\n| Algorithm| Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 |  Iter 7 |\n|:------------------:|:----------:|:----------:|:------------:|:----------:|:------------:|:----------:|:------------:|\n| Vanilla Accumulated     |635.5 | 810.7 | 910.0 | 986.7 | 1043.1 | 1095.8 | 1132.1|\n| Accumulated with Greedy Selection  |638.2 | 792.4 | 822.8 | 838.1 | 847.4 | 855.1 | 859.5|\n\n[1]: Jiralerspong, Marco, et al. \"Feature likelihood divergence: evaluating the generalization of generative models using samples.\" Advances in Neural Information Processing Systems 36 (2023): 33095-33119.\n\n**Q2:** ***It is interesting that there doesn't seem to be any evidence that supports adding in generated samples helps improve model performance. I understand that in a real use case training on internet data you do not necessarily know what is real or generated and so need automated ways to mitigate collapse, but I wonder if the authors can comment on this. i.e. can the entropy-based data selection result in a better model than naive sampling real images.***\n\n**A2:** We thank the reviewer for this insightful question, which allows us to clarify the core premise of our work. It raises two key points:\n1. **Why doesn't adding generated data improve performance in this context?** The reviewer's observation is correct. From a statistical perspective, any real image dataset, $S$, is an i.i.d. sample from the true underlying data distribution, $P$. A generative model, due to practical training and approximation errors, will inevitably learn a slightly shifted distribution, $P′$. Therefore, adding samples from $P′$ to a perfect sample from $P$ will always introduce a distributional shift, leading to a degradation in performance on tasks evaluated against the true distribution, $P$.\n2. **Can our method outperform naively sampling real images?** In an idealized setting, the answer is no. If a practitioner had access to a massive, perfectly labeled pool of purely real images, exclusively using that data would be the optimal strategy.\nHowever, our work addresses the more realistic and challenging scenario central to model collapse: a practitioner has a large, **mixed dataset where real and synthetic images are indistinguishable**. The goal is not to outperform a hypothetical 'pure-real' dataset, but to find the best possible **filtering method** to mitigate the negative effects of the synthetic data within this contaminated pool. Our entropy-based selection method is designed for this specific purpose, and as we show, it demonstrably outperforms other sampling strategies in this practical setting."}, {"review_id": "631JIIoymd", "reviewer": "Reviewer_SEJh", "summary": "This work investigates model collapse in the iterative retraining loop of generative models on their own synthetic data. More precisely, they show that the entropy of the generated data set decreases through the retraining iterations, and that models increasingly memorize their training by generating data in clusters close to the training points. They propose a new method, based on data filtering (and another smoothed out version using a threshold parameter) by iteratively selecting data points based on their distance with others in the training set to ensure higher-entropy in the training set. They empirically show through experiments on image datasets using diffusion models, that this filtering method mitigates model collapse (lower FID divergence) both in the fully-replacement and accumulation scenarios.", "questions": "- You chose in your experiments to filter the synthetic dataset size by half (lines 890 to 895). In appendix C.1 you showed that even when comparing to the baseline using twice more data, you method was slowing down model collapse faster. I was wondering if you studied the impact of the filtering fraction (instead of 1/2 perhaps keep only 1/3 of the data or 2/3) and if there was a trade-off appearing where the optimal fraction should be measurable?\n\n- you measured generalization and quality independently using respectively eq1 for the generalization score and FID for the quality. This reminds me of the FLD score in [1] where the authors design a new score that incorporates both notions in the same metric. I think that tracking such metric could be interesting as it could potentially reveal interesting trade-off during training. \n\n[1]: Jiralerspong, Marco, et al. \"Feature likelihood divergence: evaluating the generalization of generative models using samples.\" Advances in Neural Information Processing Systems 36 (2023): 33095-33119.", "limitations": "An important limitation in my opinion that the authors didn't discuss is the fact even if the proposed method slows down model collapse, I don't think practicionner have any incentive to use it, since it still decreases the performance of the gerative model compared to the initial one. More precisely, I believe that even if the model collapse litterature is interesting to understand the dangers of training on synthetic data, any method that only mitigates it and does not show improvement on the performance of a model is unlikely to be used in practice as it would be better to just keep the original model and not retrain at all. I think it could be valuable to add a discussion on it, or if the author disagree I would be more than happy to engage in a discussion on this topic.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- this work identifies a *new collapse phenomenon*, focused on the entropy of the generated data distribution, and different fromprevious studies which were focused on the variance of the generated distribution. \n- they propose a new method to mitigate the collapse speed by choosing a subset of the generated data points with high entropy. This can potentially be an impactful method in the training of future generation models.\n- the authors performed an interesting ablation study in appendix C, especially C.1 is interesting as it shows that even though this filetring method uses more data than the baseline, this increased sample size does not explain by itself the slow-down in collapse since doubling the sample size in the baseline still degrades faster than with filtering. This is very interesting and was crucial to verify.\n- the paper is clearly written, with good background review and clear mathematical definitions of the main concepts (definition 3.1, equation 2)\n\n**Weaknesses**\n- some related work is missing, especially regarding the impact of data curation (which is used here in the proposed method) on model collapse. For example [1] shows that curating data based on some reward eventually makes the retraining loop collpase to the highest reward region. It is worth discussing this phenomenon here, although a potentially good thing in your framework is that data points are curated collectively (based on the distance to others) and not just using a pointwise reward function which could prevent this from happening. Your method also makes me think of a litterature on data pruning which shows that pruning synthetic data helps learning. See for example, [2,3,5]. I think it would be interesting to draw some paralell between these works and your pruning method which does not directly prune uninformative samples as they advocate, but prune clusters of samples, which could be potentially related. Finally [5] studies how to prevent model collapse from a theoretical view point which could be compared with your method.\n- I am not convinced by the effectiveness of the method proposed here. In particular, the method aims at slowing down model collapse but even with this method, the performance degrade. Hence I don't understand in which scenario practicionners should use this method in the first place? Shouldn't they instead not retrain generative models on synthetic data at all and keep the current trained models that are available? More genrally, I believe that the current litterature on model collapse is interesting but designing methods that only *slow-down* the collapse and do not show any form of imporvement are useless, since they would not be used in any case (practicionner will just keep their current models). \n\n\n[1]: Ferbach, Damien, et al. \"Self-consuming generative models with curated data provably optimize human preferences.\" arXiv preprint arXiv:2407.09499 (2024).\n\n[2]: Sorscher, Ben, et al. \"Beyond neural scaling laws: beating power law scaling via data pruning.\" Advances in Neural Information Processing Systems 35 (2022): 19523-19536.\n\n[3]: Kolossov, Germain, Andrea Montanari, and Pulkit Tandon. \"Towards a statistical theory of data selection under weak supervision.\" arXiv preprint arXiv:2309.14563 (2023).\n\n[4]: Askari-Hemmat, Reyhane, et al. \"Improving the Scaling Laws of Synthetic Data with Deliberate Practice.\" arXiv preprint arXiv:2502.15588 (2025).\n\n[5]:Fu, Shi, et al. \"A theoretical perspective: How to prevent model collapse in self-consuming training loops.\" arXiv preprint arXiv:2502.18865 (2025).", "was_revised": true, "final_justification": "I believe this work should be accepted. \n\nThe contribution is *novel* as the authors identify a new collapse phenomenon related to memorization of generative models which is a very active area of study. They propose an *interesting method* to mitigate it based on selecting high-entropy subsets of synthetic datasets.\n\nThe authors *adressed most of my concerns* on related work and some experimental details. They additionally proposed new interesting experiments in the rebuttal.", "rebuttal": "We appreciate the review for the important suggestions and concerns. Below we provide our response for your comments.\n\n**Q1:** ***Some related work is missing.***\n\n**A1:** We appreciate the important related work mentioned in the review and will incorporate a detailed discussion in the revised paper. Specifically, we plan to add the following paragraphs:\n\n\"[1, 5] study iterative training processes related to our setting. [1] shows that human preferences can be amplified in an iterative loop: modeling human curation as a reward process, the curated distribution $p_t$ converges to $p^*$ that maximizes the expected reward as $t \\to \\infty$. In [1], the reward $r(x)$ is a **pointwise** function over individual images, whereas our method can be viewed as a curation strategy that maximizes **dataset-level entropy** at each iteration. Whether the theory in [1] extends to **dataset-wise** rewards $r(S)$, such as dataset entropy, remains an open question. Nonetheless, their intuition aligns with our results that entropy can be iteratively enhanced compared to vanilla methods.\n\n[5] provides the first generalization error bounds for Self-Consuming Training Loops (STLs), showing that mitigating model collapse requires maintaining a non-negligible portion of real data and ensuring recursive stability. In contrast, our work focuses on a specific collapse phenomenon—the generalization-to-memorization transition—and introduces an entropy-based data selection algorithm to mitigate this behavior.\"\n\n\n\"[2, 3, 4] focus on data pruning techniques, but not necessarily in the context of self-consuming loops. While both their approaches and ours demonstrate the benefits of data selection, there are several key differences:\n1. **Objective and evaluation**: [2, 3, 4] primarily study how pruning improves scaling laws, achieving higher accuracy as dataset size varies. In contrast, our work examines how model performance evolves over an **iterative training loop** with a fixed dataset size, focusing on the generalization-to-memorization transition.\n2. **Task and criteria**: Prior works focus on **classification tasks**, where sample selection depends on label information. Our method targets **generative modeling**, where selection is based on dataset entropy rather than label-driven criteria. Our objectives also differ: prior work emphasizes classification accuracy, while we address **model collapse from a generalization perspective**, providing a different angle of analysis.\n3. **Pruning strategy**: Methods in [2, 3, 4] largely rely on **per-sample evaluation**, whereas our approach considers **global dataset structure** and relationships between samples.  While this may lead to increased computational complexity, it opens up new possibilities for designing pruning criteria beyond per-sample evaluation.\n4. **Entropy definition**: In [4], while the authors use a generative model to sample images, the goal is to improve the classification performance of a classifier. And the entropy used in their method is defined in the prediction space of a classifier, which is different from the entropy of a dataset measured in our work.”\n\n[1]: Ferbach, Damien, et al. \"Self-consuming generative models with curated data provably optimize human preferences.\" arXiv preprint arXiv:2407.09499 (2024).\n\n[2]: Sorscher, Ben, et al. \"Beyond neural scaling laws: beating power law scaling via data pruning.\" Advances in Neural Information Processing Systems 35 (2022): 19523-19536.\n\n\n[3]: Kolossov, Germain, Andrea Montanari, and Pulkit Tandon. \"Towards a statistical theory of data selection under weak supervision.\" arXiv preprint arXiv:2309.14563 (2023).\n\n[4]: Askari-Hemmat, Reyhane, et al. \"Improving the Scaling Laws of Synthetic Data with Deliberate Practice.\" arXiv preprint arXiv:2502.15588 (2025).\n\n[5]:Fu, Shi, et al. \"A theoretical perspective: How to prevent model collapse in self-consuming training loops.\" arXiv preprint arXiv:2502.18865 (2025).\n\n**Q2:** ***I don't understand in which scenario practitioners should use this method in the first place? Shouldn't they instead not retrain generative models on synthetic data at all and keep the current trained models that are available? More generally, I believe that the current literature on model collapse is interesting but designing methods that only slow down the collapse and do not show any form of improvement are useless, since they would not be used in any case (practitioners will just keep their current models).***\n\n**A2:** We thank the reviewer for this insightful question about the practical utility of our method. Our method can be generally applied to improve performance in common, real-world scenarios where generative models are continuously updated with a mixture of previously generated synthetic data and an ongoing stream of **fresh real data**. \n\nWe agree with the reviewer's comment that the ultimate goal is to obtain a better model than the previous one. The academic setup in model collapse, where no new real data is introduced, is useful for isolating the model collapse phenomenon but does not reflect how these models would be trained in practice. To demonstrate the effectiveness of our method in a more realistic setting that includes fresh data and an increasing training budget, we conducted a new experiment and compared three models:\n1. **Model A (Baseline)**: The original model, trained on the initial 32,768 real images, which then generates 10,000 synthetic images.\n2. **Model B (Random Sampling)**: A new model trained on a 40,000-image dataset randomly sampled from the original real images, new fresh data, and the synthetic data.\n3. **Model C (Our Method)**: A new model trained on a 40,000-image dataset selected from the same pool using our **entropy-based method**.\n\nThe performance of these models is shown below: \n| Model        | A | B | C |\n|:------------------:|:----------:|:----------:|:------------:|\n| FID     |   28.0    | 30.8 |   27.5    |\n\nAs the results indicate, our entropy-based selection method (Model C) allows the next-generation model to surpass the baseline (Model A), a feat not achieved by random sampling (Model B). This directly shows that in a practical scenario with evolving datasets, our method does not just slow collapse but enables tangible improvement. We believe this addresses the reviewer's concern about practical usefulness.\n\n**Q3:** ***I was wondering if you studied the impact of the filtering fraction (instead of 1/2 perhaps keep only 1/3 of the data or 2/3) and if there was a trade-off appearing where the optimal fraction should be measurable?***\n\n**A3:** Thank you for this insightful question. The reviewer is correct that this ratio presents a critical trade-off. There are two clear extremes:\n1. If the ratio is 1, all generated images are used for training. As we show in Appendix C, it still degrades faster than filtering (ratio equals to $1/2$).\n2. Conversely, if the ratio approaches 0, too few images are selected for training, which detrimentally starves the model of data.\n\nWhile limited to a single iteration due to time constraints, the following results validate our intuition and demonstrate that an intermediate ratio yields the best performance. To investigate the behavior between these extremes, we will conduct new experiments exploring several intermediate ratios and add a summary of this new analysis to the appendix.\n\n| Ratio        | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |\n|:------------------:|:----------:|:----------:|:------------:|:------------:|:------------:|\n| FID     |   56.0    | 44.7 |   40.0    |   38.2    | 49.5|\n\n**Q4:** ***This reminds me of the FLD score that incorporates both generalization and quality in the same metric.***\n\n**A4:** Thank you for this important suggestion. We use their implementation from GitHub and measure the FLD score of the generated images over iterations. The following table shows the FLD scores of our method with those of the vanilla paradigm on CIFAR-10. But we are not sure whether FLD would be a suitable metric here to measure the collapse trend here. As the table shows, the FLD of the vanilla accumulate-subsample paradigm even decreases along iterations, suggesting that the generalization and quality are improving, **so FLD does not capture the degradation along iterations**. And the differences across iterations are small, indicating that **FLD is not sensitive to the specific generalization-to-memorization transition** studied in our work. \n\nFLD (Accumulate Paradigm)：\n| Iteration        | 1 | 2 | 3 | 4 | 5 | 6 | 7 |\n|:------------------:|:----------:|:----------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| Vanilla Accumulate     | 21.2| 22.5| 22.6|19.7 |19.9 | 19.6| 18.4|\n| With Greedy Selection     | 21.2| 22.6| 22.7| 22.8|22.2 |22.4 |22.3 |"}, {"review_id": "56krPnJetT", "reviewer": "Reviewer_n2Fm", "summary": "If models are trained on the outputs of other models, their performance can decrease in a phenomenon called *model collapse*. This work characterizes model collapse in terms of a transition from generalization to memorization of training data. They find that the entropy of a training dataset is closely related to the generalization score of a model trained on that dataset. Motivated by this finding, they introduce two entropy-based approaches to sub-sampling from real and synthetic datasets, which increase generalization and improve FID scores when models are trained on synthetic and natural data.", "questions": "- The authors examine what happens when we train a model on data contaminated with its own generations. In a realistic setting, internet data contains images generated by many different models--how might your analysis change when considering this fact? \n- There is a deep question here when looking at Figure 2: why do later iterations of models always collapse to training examples from the *original* train dataset, instead of collapsing to some other random set of synthetic faces? To me this does not seem obvious. Is it that the original \"generalized\" models actually do have some degree of memorization, which gets amplified in later iterations? \n- \"projecting high-dimensional images onto the subspace spanned by their top two eigenvectors.\" -> is this just referring to eigenvectors of the dataset's covariance matrix (i.e., PCA)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Figure 2 provides a strong motivation for why the authors decide to look at entropy of training datasets. Despite the fact that dataset size is constant, they find a consistent decrease in generalization scores, suggesting that the dataset itself contains less and less \"information.\" Their empirical evidence seems to support this interpretation. \n- The fact that entropy-aware sampling methods keep generalization high, even in the CIFAR-10 \"replace\" setting, is quite interesting. This suggests that the culprit for model collapse may be decreased entropy of generative outputs. \n- Figure 8 answers a question that I would have asked as a reviewer. The authors interpret it as a good sign that the model over-selects natural images--however, it is also good to see that the algorithm is not *just* filtering out all synthetic images from future runs. \n- It is interesting that this approach can also be used to improve CFG results.\n\nWeaknesses:\nMany weaknesses of this paper simply have to do with a lack of clarity:\n- Figure 2 is confusing due to the word \"iteration.\" Although the authors define it in a footnote on the first page, it would be much clearer to restate this in the Figure 2 caption and in the main body (lines 123-131), and possibly change captions to say \"Model-Training Iteration.\" A quick nod in the caption to why small dataset sizes have much lower generalization scores (even in the first iteration) may help clarify the setup to readers who are skimming. \n- Information on models and datasets used should be included earlier in the paper (to understand Section 3 results better). \n- Figure 4: Again, more details on the setup in the body and caption would be helpful to the reader. Are different points of the same dataset size re-sampled from the underlying training data? Is there a particular reason why points are connected in 4b but not in 4a? \n- Why are DINOv2 features used in Section 4 if they were not needed for Section 3 analyses?\n- It would be interesting to see Figure 2 results in the \"accumulate\" setting, perhaps as an Appendix figure.\n\n\nThe paper could benefit from a little bit more depth, depending on the goal of the authors. If the data sampling method is the central contribution, then how does this approach compare to previous works? If it is the insight that dataset entropy, rather than variance, affects generation, what is the fundamental theoretical difference between these interpretations, and how do your results compare to variance-based sampling? Is this a new observation?", "was_revised": true, "final_justification": "Thank you to the authors for the thorough response. As they have addressed my main questions and concerns on clarity, I will raise my score to 5 (accept).", "rebuttal": "**Q1:** ***Figure 2 is confusing due to the word \"iteration\", although the authors define it in a footnote on the first page.***\n\n**A1:** Thank you for highlighting this. To improve clarity, we will revise the manuscript in two ways. First, we will explicitly define 'iteration' in both the main text and directly in the caption for Figure 2. Second, we will expand the caption to include a brief explanation for why smaller dataset sizes result in lower generalization scores, making the figure easier to interpret.\n\n**Q2:** ***Information on models and datasets used should be included earlier in the paper (to understand Section 3 results better).***\n\n**A2:** Thank you for the important suggestion. In Section 3, we use CIFAR-10 and a UNet-based DDPM model in the experiments. We will provide more details about the models and datasets earlier in Section 3 to improve clarity.\n\n**Q3:** ***Figure 4: Again, more details on the setup in the body and caption would be helpful to the reader. Are different points of the same dataset size re-sampled from the underlying training data? Is there a particular reason why points are connected in 4b but not in 4a?***\n\n**A3:** Thank you for the question regarding Figure 4. This figure plots the generalization score (from Fig. 2) versus the entropy (from Fig. 3) for each of our 54 experiments. We combine the results from Figures 2 and 3 to visualize the relation between entropy and generalization score in Figure 4. Different points of the same dataset size are not just resampled from the underlying data distribution; they denote the results on different iterations of one iterative loop.\n\nThe different visualization styles in 4a and 4b were chosen to best highlight the underlying data structure. Specifically, the points in Figure 4a are not connected because they are strongly collinear; connecting them would result in overlapping lines that hide this key trend. We instead use a single dashed line to clearly illustrate this shared linear relationship. Conversely, in Figure 4b, connecting the points is more informative, as it reveals the distinct trends of each group with different data sizes.\n\n**Q4:** ***Why are DINOv2 features used in Section 4 if they were not needed for Section 3 analyses?***\n\n**A4:** We use DINOv2 for the selection method because it yields a slight improvement in FID performance compared to not using DINOv2, likely because FID is also computed in feature space. And it improves computation efficiency since we can select data points in a low-dimensional latent space. We additionally verified that using DINOv2 does not change the observations presented in the analyses in Section 3.\n\n**Q5:** ***It would be interesting to see Figure 2 results in the \"accumulate\" setting, perhaps as an Appendix figure.***\n\n**A5:** Thanks for the suggestion. We will add the results of the \"accumulate\" setting as a figure in the Appendix, similar to Figure 2.\n\n**Q6:** ***The paper could benefit from a little bit more depth, depending on the goal of the authors. If the data sampling method is the central contribution, then how does this approach compare to previous works? If it is the insight that dataset entropy, rather than variance, affects generation, what is the fundamental theoretical difference between these interpretations, and how do your results compare to variance-based sampling? Is this a new observation?***\n\n**A6:** We thank the reviewer for this insightful question, which allows us to clarify the central contribution of our work. The primary contribution is the novel insight that **model collapse can be analyzed through a transition from generalization to memorization** and that **data entropy serves as a direct and superior indicator** of this transition compared to variance. This perspective is new, and our proposed data sampling method is the practical application of this core insight.\n\nThis addresses the reviewer's questions in the following ways:\n\n1. **Entropy vs. Variance**: The fundamental difference is that variance measures simple deviation from the mean (which can be skewed by outliers), while entropy measures the 'informativeness' and diversity of the entire data distribution. As our results in Figure 4 already demonstrate, entropy has a stronger linear correlation with generalization, making it a more robust and direct metric for generalization. Based on the reviewer's feedback, we will expand this discussion and incorporate an explicit comparison against a variance-based sampling baseline in the revision. \n2. **Depth and Future Work**: Our current work provides the essential empirical and conceptual foundation for our claims. Building on this, we will add a discussion of promising future directions for in-depth theoretical studies. We conjecture that the loss of sample diversity is due to the architecture bias and sampling bias of our trained model, and we envision that a more formal theoretical analysis of collapse dynamics can be developed based on learning a mixture of Gaussians. We will add a discussion on these future theoretical directions to the manuscript.\n\n**Q7:** ***In a realistic setting, internet data contains images generated by many different models—how might your analysis change when considering this fact?***\n\n**A7:** Thank you for this insightful comment. We conducted simulation experiments with DDPM and EDM, extending the accumulate-subsample paradigm to a **multi-model** setting. In the first iteration, separate DDPM and EDM models are trained on the real CIFAR-10 dataset. We then generate synthetic images from both models and merge them with the original images to form a sample pool, simulating the Internet pool. In the next iteration, new DDPM and EDM models are trained on a subset sampled from this pool.\n\nThe table below reports the FID of these models across iterations in this multi-model scenario. We plan to extend the experiments to more iterations and provide a detailed analysis of the multi-model setting in the revised manuscript\n\n| Iteration        | 1 | 2 |\n|:------------------:|:----------:|:----------:|\n| DDPM | 28.0 | 36.6|\n| EDM | 6.1| 12.6|\n\n\n**Q8:** ***There is a deep question here when looking at Figure 2: why do later iterations of models always collapse to training examples from the original training dataset instead of collapsing to some other random set of synthetic faces?***\n\n**A8:** We would like to clarify that models in later iterations do not necessarily collapse to the original training data points. We analyze this behavior in Appendix D.1, where we visualize generated images across iterations. When the training dataset is small (e.g., $1024$), the model tends to **memorize** the dataset throughout the iterative loop, and later models indeed collapse to the original training images. In contrast, when the dataset is large (e.g., $32768$), early models generate novel images distinct from the original dataset, and later models collapse to **novel points** that are absent in the original data, rather than to the original training images.\n\n**Q9:** ***\"projecting high-dimensional images onto the subspace spanned by their top two eigenvectors.\" -> is this just referring to eigenvectors of the dataset's covariance matrix (i.e., PCA)?***\n\n**A9:** Yes, it is. We use eigen-decomposition to compute the top two eigenvectors of the covariance matrix. This is equivalent to directly performing SVD on the data matrix and getting the top two (right) singular vectors."}, {"review_id": "WeWOVlBXiL", "reviewer": "Reviewer_pXga", "summary": "This paper investigates a critical phenomenon in diffusion models: recursively training on self-generated synthetic data leads to a decline in generation quality and, in some cases, memorization of the original training data. Through empirical analysis, the authors demonstrate a strong correlation between the entropy of the training dataset and the model’s generalization capability within this self-consuming training loop. Motivated by this observation, they propose entropy-based data selection strategies that effectively mitigate performance degradation across recursive training iterations.", "questions": "## Questions and Comments\n\n### Appendix Section C.1\n\nThe results show that the vanilla paradigm using 2N images performs better than using N images, but still worse than the proposed data-selection method applied to N points. This suggests that simply increasing the number of training samples does not necessarily lead to better performance. One possible explanation is that the additional data in the 2N case may contain redundant or low-entropy samples, which could hinder generalization.\n\nIt would be helpful to include a more detailed analysis here. For example:\n- How do the generalization score and entropy vary across these different training regimes?\n- Is it the case that datasets produced by the data-selection method exhibit lower entropy compared to the full 2N image set?\n- Would these trends remain consistent across different datasets?\n\nSuch analysis could provide deeper insight into the role of data redundancy and entropy in recursive training.\n\n---\n\n### Appendix Section D.1\n\n- Do the generated images in this section come from the vanilla \"replace\" paradigm?\n\n- When the original training dataset is large, it is observed that after a few recursive training iterations, the generated images—while not exhibiting memorization—become increasingly fuzzy. Do the authors have an intuition for why this occurs? Is this phenomenon consistent across different datasets?\n\n- What visual patterns emerge from applying the proposed data-selection methods over multiple iterations? Do the generated samples still become fuzzy after a few rounds, or is the degradation mitigated?\n\n---\n\n### On the Data Selection Methods\n\nAs a disclaimer, I am not an expert in training diffusion models with self-generated data. However, my understanding is that the long-term goal of incorporating synthetic data in recursive training is to improve model performance beyond what is achievable by training solely on the original dataset.\n\nIn the current experiments, while the proposed data-selection methods help mitigate performance degradation over iterations, the model's performance in later stages still does not surpass that of the initial training iteration.\n\nTo strengthen the paper, it would be highly valuable to include an experiment—possibly unconstrained by fixed dataset size in later iterations but using a fixed neural network architecture—that demonstrates how data-selection strategies can lead to improved performance compared to training only on the original dataset. This would directly support the claim that such methods enhance long-term model generalization.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is generally well-written and effectively communicates its main takeaways.\n\n2. It presents a compelling empirical observation of a strong correlation between the entropy of the training data and the model's performance during recursive self-training. \n\n3. The inclusion of ablation studies provides meaningful insights into different aspects of the proposed approach.\n\n**Weaknesses:**\n\n1. Some empirical observations would benefit from more detailed analysis and discussion (see in the *Questions* section).\n\n2. The description of the classifier-free diffusion guidance experiment, as well as the explanation of how the proposed data selection methods enhance diversity, lacks depth and could be made more thorough and precise.", "was_revised": true, "final_justification": "I have read all the reviewers’ comments and the corresponding replies from the authors (including my own). In my view, all of the points I raised have been satisfactorily addressed. I believe the empirical observations and the proposed method presented in this paper will be of interest to the community. Accordingly, I recommend acceptance.", "rebuttal": "We appreciate the reviewer for the insightful questions. Below, we provide our response for your comments.\n\n## Regarding the comments on Appendix C.1:\n**Q1:** ***How do the generalization score and entropy vary across these different training regimes?***\n\n**A1:** In Figures 5 and 7, we have shown the generalization score and entropy of the vanilla \"replace\" paradigm and the selection-augmented method. The results show that by sampling $2N$ images and then selecting the $N$-size subset, we could improve the entropy of the training dataset, which leads to better generalization. As for training on the $2N$ dataset, we compare the entropy with the data selection method in the next question.\n\n**Q2:** ***Is it the case that datasets produced by the data-selection method exhibit lower entropy compared to the full 2N image set?***\n\n**A2:** No. The $N$ dataset produced by the selection method has **higher** entropy than the full $2N$ dataset. We compute the entropy of the generated $2N$ image dataset and its selected $N$ subset, obtaining values of $746.8$ and $796.4$, respectively. This demonstrates that the selection method yields a subset with higher entropy than the full $2N$ dataset.\n\nBy the definition of the KL estimator of entropy:\n\n$$\\hat{H}_ \\gamma(\\mathcal{D})= \\psi(|\\mathcal{D}|)- \\psi(\\gamma)+ \\log c_ d+ \\frac{d}{|\\mathcal{D}|}\\sum_ {x \\in \\mathcal{D}} \\log \\varepsilon_ \\gamma(x),$$\n\nthe last term represents the **average nearest-neighbor distance**, which is maximized by our selection methods. Although the dataset size decreases, the increase in average distance results in higher entropy.\n\n**Q3:** ***Would these trends remain consistent across different datasets?***\n\n**A3:** We observed these trends on CIFAR-10 and FFHQ in our work. But we think that whether selecting $N$ samples from $2N$ performs better than training on $2N$ also depends on the quality of the synthetic images. If the diffusion model learns pretty well and the generated distribution resembles the underlying image distribution, then more data could diminish the empirical estimation error, and training on the whole $2N$ images leads to better performance. On the other hand, if the quality of the generated samples varies, then our selection method can help us filter out more suitable data.\n\n## Regarding the comments on Appendix D.1\n**Q4:** ***Do the generated images in this section come from the vanilla \"replace\" paradigm?***\n\n**A4:** Yes. The images shown in Appendix D.1 are generated from the vanilla \"replace\" paradigm. We will make this clearer in the revision.\n\n**Q5:** ***When the original training dataset is large, it is observed that after a few recursive training iterations, the generated images—while not exhibiting memorization—become increasingly fuzzy. Do the authors have an intuition for why this occurs? Is this phenomenon consistent across different datasets?***\n\n**A5:** Thank you for highlighting this interesting phenomenon. We have also observed a similar behavior, which was previously reported in [1] (see Figures 14 and 15). Their results show that diffusion models can generate clear images when operating in either the memorization or generalization regime—corresponding to small and large dataset sizes, respectively. However, when the dataset size is around $4096$, lying between the two regimes, the model tends to generate fuzzy images. An intuition is that diffusion models can accurately learn the empirical distribution with small datasets and the underlying distribution with large datasets, but at the boundary between these regimes, the model reaches an unstable solution that neither memorizes nor generalizes well. A rigorous theoretical explanation for this remains an open problem.\n\nReturning to the fuzzy images in Appendix D.1: when the original dataset is large, the initial model generalizes well. However, after several recursive training iterations, the dataset entropy decreases, and the model enters the intermediate region between memorization and generalization, leading to fuzzy generations—consistent with the observations in [1].\n\n[1]: Huijie Zhang et al. \"The Emergence of Reproducibility and Consistency in Diffusion Models.\" Proceedings of the 41st International Conference on Machine Learning, PMLR 235:60558-60590, 2024.\n\n**Q6:** ***What visual patterns emerge from applying the proposed data-selection methods over multiple iterations? Do the generated samples still become fuzzy after a few rounds, or is the degradation mitigated?***\n\n**A6:** The selection method can effectively mitigate the performance degradation compared to the vanilla paradigms, as shown in the FID and generalization improvement. We will also add some generated images for the proposed selection method in the revision to better visualize the improvement. We do not see special visual patterns emerging from our methods.\n\n## Regarding the improvement of data-selection methods\n**Q7:** ***To strengthen the paper, it would be highly valuable to include an experiment—possibly unconstrained by fixed dataset size in later iterations but using a fixed neural network architecture—that demonstrates how data-selection strategies can lead to improved performance compared to training only on the original dataset. This would directly support the claim that such methods enhance long-term model generalization.***\n\n**A7:** Thank you for the suggestions. We fully agree that the ultimate goal of our method is to **surpass the baseline model** trained on the original dataset. To evaluate our approach in a more realistic scenario—incorporating **fresh data** and an **expanded training data size**—we conducted a new experiment comparing three models:\n1. **Model A (Baseline)**: The original model trained on $32{,}768$ real images, which then generates $10{,}000$ synthetic images.\n2. **Model B (Random Sampling)**: A new model trained on a $40{,}000$-image dataset randomly sampled from the pool of original real images, fresh data, and synthetic images.\n3. **Model C (Our Method)**: A new model trained on a 40,000-image dataset selected from the same pool using our entropy-based method.\n\nThe performance of these models is shown below:\n\n| Model        | A | B | C |\n|:------------------:|:----------:|:----------:|:------------:|\n| FID     |   28.0    | 30.8 |   27.5    |\n\nAs the results indicate, our entropy-based selection method (Model C) allows the next-generation model to surpass the baseline (Model A), a feat not achieved by random sampling (Model B). This directly shows that in a practical scenario with evolving datasets, our method does not just slow collapse but enables tangible improvement.\n\nThe current academic setup for studying model collapse—an iterative loop where no new real data is introduced and the model continuously consumes its own generated data—is valuable for **isolating the collapse phenomenon**, but it does not fully reflect practical usage. Our method is also applicable in **real-world scenarios**, where generative models are continuously updated with a **mixture of previously generated synthetic data and an incoming stream of fresh real data**, yielding improved models.\n\nWe believe this addresses the reviewer's concern about practical usefulness."}]

Author rebuttal:

Meta-review: This paper studies model collapse, wherein models successively trained on synthetic data become progressively worse. The authors identify that the entropy of the generated data decreases as model collapse progresses, and they leverage this insight for proposing mitigation strategies. The reviewers unanimously praised the paper for being topical, relevant, and convincing.

This paper is a clear accept, which I recommend for a spotlight presentation.
Decision: Accept (spotlight)