A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
Authors: Shi Fu
Yingjie Wang
Yuzhu Chen
Xinmei Tian
Dacheng Tao
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Synthetic Data
Transformer
Generalization Error
Learning Theory
Primary area: learning theory
TL;DR: The paper introduces recursive stability to tackle Self-Consuming Training Loops, offering generalization bounds for generative models like transformers.

Abstract: High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of *recursive stability* and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing.

Published as a conference paper at ICLR 2025
A THEORETICAL PERSPECTIVE: How TO PREVENT MODEL COLLAPSE IN SELF-CONSUMING TRAINING LOOPS
Shi Fu² Yingjic Wang'* Yuzhu Chen" Xinmei Tian
Dacheng Tao
1 GGnneaative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Univerrity of Science and Technology of China, Hefei, China fs 1s3i1@mail. uste. edu, en, xinmeiuustc, eyzkrau@mail ustc. edu. en, x xinmei @ust edu en, dacheng tao@gmail. com
edu. en,
ABSTRACT
High-quality data is essential for training large generative models, yet the vast reser- voir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming consuming Training Loops (STLs). However, the empirical results have been strikingly incon- sistent: some models degrade or even collapse, while others successfully avoid these failures, leaving significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model ar- chitecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sizer proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing.
INTRODUCTION
The quest of high-quality data is paramount in the training of generative artificial intelligence (AI), such large language models (LLMs). However, the vast reservoir of publicly available data on the internet has nearly been exhausted (Villalobos et al., 2022). pushing the research community to seek innovative yet plausible solutions. One promising approach is train the next generation of LLMs using synthetic data generated by earlier generations of the models themselves (Briesch er al., (023). Additionally, reliance on synthetic data has become almost unavoidable, as many existing datasets are already polluted with synthetic content (Schuhmann et al. (022), which proves difficult to detect self-consuming Training Loops (STLs), as illustrated in Figure 1 where generative models are recursively trained on mix of real and synthetic data generated by the models themselves. In theory, these STLs of data creation and refinement could propel models to new levels of capability, reducing reliance on external datasets.
reliably (Sadasivan et al. 2023). This has led to the development
However, despite their potential, the empirical results STLs have been highly inconsistent across studies (Shumailov et al. 2024 Alemohammad et al. 2024a Xing et al.) 2025 Dohmatob et al.) 2024b). Some studies (Shumailov et al. (2024) have encountered significant setbacks certain models have shown signs of stagnation, failing improve or adapt, while others have even regressed, leading to sharp declines in performance. Conversely, other works (Gerstgrasser et al., 2024 Gillman et al.) 2024 Alemohammad al.) 2024b Ferbach al.) 2024) have successfully avoided model collapse by incorporating sufficient real data, augmenting with synthetic data, or introducing guidance during the generation process. However, these observed phenomena lack thorough theoretical explanations.
When and how do STLs generalize effectively, thereby preventing model collapse from a theoretical perspective? Even among "refined" LLMs drawing from similar pools of model-generated data, the results vary significantly (Briesch et al. 2023 Fu et al. 2024a). These inconsistencies highlight the
Corresponding authors

---

Published as a conference paper at ICLR 2025
Generative Model Go
Generative Model Gi
Generative Model
Train
Train
Train
Sample
Real Dataset
Mixed
Mixed Dataset S2
Dataset
So
$
Figure 1: Self-consuming Training Loops: The initial model Go is trained on the real dataset Sp. For generation $ the model Gj is trained the mixed dataset Sy.
urgency of establishing theoretical guarantees for STLs by exploring the underlying mechanisms that determine when synthetic data generation either facilitates impedes model development. Initial theoretical explorations have started to address these gaps. For instance, [Shumailov et al. al. (2024a) demonstrated model collapse when models were trained exclusively on synthetic data, using simplified Gaussian models to illustrate this issuc. In a more detailed theoretical study, [Bertrand et (2024) derived upper bounds on parameter deviations for likelihood-bart based models in STLs, establishing convergence under strict statistical and optimization assumptions. Meanwhile, Fuu et al. (2024b) relaxed these assumptions and provided bounds on the divergence between synthetic and real data distributions for simplified diffusion model.
(2024) and Alemohammad
However, existing theoretical research lacks a unified framework and has yet to thoroughly investigate the generalization error of STLss Additionally, current studies often overlook the role of model architectures in preventing model collapse. Morcover, the behavior of transformers within STLs remains largely unexamined, leaving significant theoretical gaps in the literature. Notably, there is limited exploration of the theoretical trade-offs introduced by synthetic data augmentation. This paper aims address these gaps with the following contributions:
Theoretical Generalization Framework: This paper fills gap in prior research by being the first establish generalization error bounds. The key innovation, recursive stability, introduced address the core theoretical challenges, specifically the complex recursive structures and the non-i.i.d. nature of the data. Moreover, we demonstrate that the generalization error converges under the following conditions: (1) the generative model satisfies recursive stability, and (2) the proportion of real data maintained a non-negligible constant level, thus preventing model collapse.
2. Application to Transformers in In-Context Learning: This paper is the first to extend the theoretical framework transformer models in in-context learning. We prove that transformers in this setting satisfy recursive stability with a constant-leve proportion of real data, controlling output differences in STLs under small perturbations to the initial dataset. Consequently, we show that the generalization error bounded by O(n log" (n) + n 112 log(n) n 1/4).
3. Trade-off Analysis of Synthetic Data Augmentation: We investigate the trade-off in synthetic data augmentation. By employing decomposition techniques, we demonstrate that while synthetic data improves the generalization performance of each generation mixed datasets, it concurrently exacerbates distribution divergence across successive generations. Our theoretical findings further show that the optimal size of synthetic augmentation increases as the size of real dataset decreases.
RELATED WORK
This section reviews STLs research and algorithmic stability studies.
Self-consuming Training Loops. Recent research has increasingly focused generative models trained within STLs (Shumailov et al. 2024), with much of the analysis conducted from an empirical perspective (Martínez al. 2023). For example, Shumailov et al. (2024); Briesch al. (202))
2

---

Published as a conference paper at ICLR 2025
observe decline in diversity in language models when portion of the model's outputs is recursively used as inputs. Additionally, Wyllie et al. (2024) highlights that recursive training on synthetic data amplifies biases, resulting in significant fairness concerns. To mitigate model collapse, some studies suggest incorporating real data into the training process (Alemohammad et al. 2024a), expanding
the size of synthetic datasets (Dohmatob et al.) 2024c; Gerstgrasser al. 2024 Dohmatob et al. 2024a; Feng et al. 2024b), or providing guidance during the generation process GGilman et al. 2024
Alemohammad et 2024b Feng et al., 2024a).
While empirical research has extensively explored STLs of generative models, theoretical studies on
this process remain relatively sparse (Kanabar & Gastpar 2025: Seddik et al. [024] Marchi et al.
2024 Gerstgrasser et al. 1024,/2hg et al.) [024] Tao et al.
2024). Notably, (Shumailov et al. (2024)
and Alemohammad et al. (2024a) offer initial theoretical insights by analyzing a simplified Gaussian
model. In more comprehensive analysis. |Bertrand et al. (2024) derive upper bounds on parameter
deviations between those obtained within a STL and the optimal values, relying on assumptions about statistical and optimization error bounds. In contrast, Fuu et al. (2024b) propose bounds on the divergence between synthetic and real-world data distributions, without such assumptions. However, current research lacks a unified theoretical framework that accounts for the influence of different model architectures and does not provide generalization error bounds for STLs, thus failing to rigorously establish the conditions that guarantee the prevention of model collapse. Furthermore, the behavior of transformers within STLs remains unexplored, leaving substantial theoretical gaps.
Algorithmic stability. Algorithmic stability ensures generalization bounds independent of model capacity. A key measure, uniform stability, was introduced by [Bousquet & Elisseeff (2002) and has been instrumental in analyzing the generalization behavior of regularization methods. This measure was later extended to SGD (Hardt et al.] (2016), including non-convex and non-smooth settings (Charles Papailiopoulos, 2018] Bassily et al.) 2020; Lei 2023). Recent work shows that uniform stability can also provide near-optimal bounds with high probability (Feldman & Vondrak 2019
Bousquet et al. 20200 Klochkov & Zhivotovskiy 2021 Li & Liu 2022; Wang et al. 2024).
Building on these foundations, recent research has focused on stability in more complex, non-i.i.d.
settings. common approach models data from stationary and mixing sequences (Doukhan) 1994
Yu 1994), where weakening dependencies allow stability bounds through mixing coefficients (Mohri & Rostamizadehh 2010 He et al. 2016; Fu et al. 2023). However, estimating these coefficients remains challenging. Additionally, some studies (Zheng et al. 2023) address non-i.i.d. data by leveraging conditional independence properties. Nonetheless, current methodologies struggle with the complexities of STLs, as the non-i.i.d. nature of mixed datasets, where each generation's 's data is influenced by previous generations, presents unresolved challenges for stability frameworks.
Remark 1. Building on previous challenges, our work advances this area by developing a more comprehensive theoretical framework for analyzing generative models within STLs. Specifically, we present the first generalization error bound by addressing the additional complexity arising from the non-i.i.d. nature mixed datasets. To address this, we propose the key innovation of recursive stability, which quantifies error propagation across generations of synthetic data. Moreover, we are the first to extend this theoretical framework to transformers, explicitly utilizing error decomposition to illustrate the trade-off introduced by augmenting datasets with synthetic data.
PRELIMINARIES
In this section, we begin by formally describing the training process of generative models in STLs, then introduce algorithmic stability with a focus on uniform stability, and finally define recursive stability to address the challenges specific to STLs.
3.1 GENERATIVE MODELS WITHIN SELF-CONSUMING TRAINING LOOPS
Generative models have made significant strides in producing highly realistic data, such as images and text, which are frequently shared online and often indistinguishable from real content. Meanwhile, the supply of real data has nearly been exhausted. Consequently, deep generative models increasingly rely synthetic data, either unintentionally (Schuhmann et al. (022)) intentionally (Huang et al.. 2022). This reliance creates a recursive cycle where successive generations are trained on mixed datasets of real and synthetic data, a process known as STL,, shown in Figure T
3

---

Published as a conference paper at ICLR 2025
More concretely, we explore a stochastic process that evolves through sequential generations. In an STL, we start with an initial dataset Sg. consisting of real data points & E Z, sampled from the original real distribution Da. The initial generative model Go is trained on this real dataset So. producing the first generation synthetic dataset Si, whose distribution is denoted as Dj. Next, the real dataset So is combined with the synthetic dataset S1 in certain proportion form a new mixed dataset S1. with distribution D1. The next generation generative model G1 is then trained on this mixed dataset Sj. Moving forward, for each subsequent generation 1 S j S i, the mixed dataset Sj is composed of real data and synthetic data from previous generations. The generative model Gj is trained on S5,, producing the synthetic dataset S3+1. This STL proceeds iteratively until the maximum generation, denoted as i, is reached.
3.2 ALGORIITMICC STABILITY
Algorithmic stability measures the impact of modifying or removing small number of examples from the training set on the resulting model, a key concept in statistical learning theory (Bousquet & Elisseeff 2002p. Its primary advantage lies in providing generalization bounds independent of model capacity. Among various stability notions (Shalev-Shward et al.) 2010), we focus on uniform stability, the most widely studied form. Let S and S" be two datasets differing by one point. Then, we formally define uniform stability as follows:
Definition 1. (Uniform Stability (Bousquet & Elisseefi 2002)). Algorithm A uniformly 8n stable with respect to the loss function e if the following holds
VS, S' E z", Vz E Z, sup (((A(S)) x) (A(S') , x)1 < Bn.
Traditional notions of stability have predominantly been studied in the context of learning algorithms, such as SGD (Lei & Ying 2020). More recently, there has been significant progress in extending the concept of stability to generative models (Farnia & Ozdaglar 202T, |02T] Building on these advancements, we propose recursive stability to specifically Ily address generative models within STLs. This new stability measure is designed to quantify the differences in a generative model's outputs after multiple generations of recursive training when small perturbations are applied to the initial real dataset. The formal definition of recursive stability is presented below.
al. 2023 Li et al.) 2023).
Definition 2. (Recursive Stability) Let So represent the original real dataset, and Sy denote a dataset differing from So by a single example. A generative model G is said to be recursively 71:"-stable with respect to the distance measure d after the i-th generation of STLs, where the ratio of real to synthetic data is set to a, the following condition holds:
vS,, E Z", d (g (g (So). (S) S..0
where (b) denotes the output of the generative model at the i-th generation in the STLs. The distance measure d quantifies the deviation between the outputs generated from inputs Sn and Sy across STLs. Specifically, d can be defined using Total Variation (TV) distance, Kullback-Leible (KL) divergence, or various norms (e.g., &g norm), allowing flexibility in assessing the differences in generated outputs.
4 GENERAL THEORETICAL RESULTS
In this section, we present a general framework for analyzing generalization error. Moving beyond traditional analyses of parameter changes (Bertrand et 2024) and distributional discrepancies (Fu et al. 2024b), we focus on evaluating the utility of synthetic data after recursive training (Hittmeir et al.) 2019 Xu et al. 2023). Specifically, we examine the behavior of uniformly stable learning algorithm A trained on the mixed dataset S; in the i-th generation. Our goal is to study the generalization error of the hypothesis A(S.). Formally, we aim to bound [RD, (A(S.))-RR (A().)) where RD, (A())) Eg-DD (())) =)) represents the population risk of A(S.) under the real distribution Do, and Rgg (A(S.)) = 1/2 Ex.3,, ((A(S)). 2;) denotes the empirical risk on the mixed dataset. To derive this bound, we first decompose the generalization error as follows.
RD. (A(5.)) - Rg. (A(5.)) <
Rp, R (A())))
(A(55(())) Rg. (A(S.))
Cumulative distribution shift across generations
Generalization error on mixed distributions
4

---

Published as a conference paper at ICLR 2025
The first term captures the accumulation of error and distribution divergence over multiple gener- ations within the STLs. This heavily depends on the capacity of the generative model to preserve distributional fidelity across generations, requiring recursive techniques The second term reflects the generalization performance of the learning algorithm on the non-i.i.d. mixed dataset, where synthetic data points are influenced by the initial real dataset. Drawing on Zheng et al. (2023), we observe that while So satisfies the i.idd assumption, the synthetic datasets S; follow a conditional i.i.d. assumption given So. Leveraging this, along with moment bounds and concentration inequalities. we address the challenge of bounding the second term and managing dependencies within the STLs. We now present the following result.
manage eπror propagation.
Theorem 1 (General Generalization Bound), Assume that A is a Bn-uniformly stable learning algorithm and the loss function is bounded by M. Let n represent the sample size of the mixed dataset S, defined as S = o.Sg + (1 a3 for 1 S j S i, where O- < a S denotes the proportion of real data. Assume further that the generative model
recursively in -t-ste and the TV distance for each generation TV (D, D)+1) is of the same order, denoted by drv(n). Then, for any E (0, 1),
with probability at least 1 5, the following holds:
RD. (A(S.)) Rgg (A(())) X ShaM log(na))og(11/))) + 11nn Vlog
+ Br (log log (1/)) + / - a))ooo(11/))) drv (1 (1- (1-(1- (1- (1-a))) (I)
where 7h sup, (V (S)). D.tttaa)ss)) with So and Sy representing two real datasets of size n, differing by only a single data point.
Remark 2. Recursive Stability in STLs. In Theorem the recursive stability parameter is quantified using the TV distance to measure the divergence between the distributions of the n(1 a) synthetic data points generated by the model G. at the i-th generation. Notably, the concept of recursive stability, introduced in Definition 2 is adaptable to various metrics, making it applicable across different types of generative models. In Theorem 2 the recursive stability parameter for transformers is instead defined using the lo norm between tokens, allowing this concept to be generalized to broader range of model architectures.
Moreover, Theorem demonstrates that generative models with higher recursive stability exhibit better performance after undergoing the STL. Specifically, the results indicate that the convergence rate of recursive stability parameter is at least faster than O1/ n), which is relatively mild condition. Furthermore. Theorem 2 shows that, under mild assumptions, the recursive stability parameter for transformers in in-context learning settings achieves convergence rate of when measured by the & norm between tokens.
(1//n)
Remark 3. Effect of Real Data Proportion on Error Control. Previous experimental results (Shumailov et al., 2024 Alemohammad et al.) 2024a) have demonstrated that incorporating real data can mitigate model collapse and help control errors. This remark focuses on exploring the effect of the real data proportion a on the generalization error within the STLs. As shown in Theorem T the real data proportion a plays a significant role in the cumulative distribution shift across generations, specifically in the term 2M (1 - (1- - a)') a a- dn)(
-1
When a + O, we observe that [l=(I-0))) + i, leading to a linear accumulation of errors due to the Distribution Shift, making increasingly challenging to control the overall error. This observation aligns with the theoretical results reported in (Shumalov et al. (2024); Dohmatob et al. (2024a); Fu et al. (2024b). However,
important to note that the conditions on a for controlling this term are not strict. In fact, as long as a remains at a non-negligible constant level, the expression (1-(1-a))) a -1 remains bounded, effectively controlling the error. This aligns with theoretical intuition: when a is too small, the mixed dataset contains insufficient real data, resulting in a more severe distribution shift.
Moreover, the proportion of real data a also impacts the generalization error on mixed distributions, primarily through its effect on the recursive stability parameter 7 As a increases, the generative model becomes more recursively stable. We will further explore the influence of a the recursive stability parameter 7 for specific generative models, such as transformers, in Theorem particularly in Remark g
Remark 4. Convergence Rate of Uniform Stability Parameter. With respect to the uniform stability parameter Bn, we observe from the third term the right-hand side of inequality I that the
5

---

Published as a conference paper at ICLR 2025
convergence rate of Bn must be at least (L1/vn)) to adequately control the error. This is a relatively mild requirement.
For example, in the case of widely used algorithms such as SGD, it has been shown that the uniform stability parameter 3n converges at a rate of (log((n)/m) /n) under the assumptions of Lipschitz continuity and smoothness of the loss function (Zhang et al., 2022). Additionally, for regularization- based algorithms, such as kernel regularization schemes and the Minimum Relative Entropy (MRE) algorithm, has been demonstrated that B,, can achieve convergence rate of (1(mn) under certain conditions (Bousquet & Elisseelf 2002).
Remark 5. Convergence of the Distribution Shift Term (rv(n). Regarding the convergence the term 2M (1- (1 (1-al a))) a (n)v(n), as discussed in Remark when a remains a non-negligible constant, attention turns to the distribution shift term drv(n). This term critically depends the generative model's capacity and quantifies the divergence between the learned distribution and the input distribution in each generation.
Theoretical studies have provided various convergence rates for (ny across different generative models. For instance, in diffusion models, (n) has been shown to converge at a rate of O (1/ 1/4) (Fu et al. 2024b). Similarly, for GANs, the convergence rate is also O (1/n [Lingg 2021 More generally, by applying Pinsker's ' inequality to relate KL divergence and TV distance, the convergence rates for other models, such Bias potential models and Normalizing flows, have been explored in previous works (Yang] 2022). Additionally, we will further examine the behavior of transformer models in Theorem 3] demonstrating the flexibility of our theoretical framework across wide range of generative models.
Remark 6. Comparision with Previous Works. In the realm of theoretical research on the STL, where models are recursively trained on the synthetic data they generate, the foundational work was introduced by Shumailov et al. (2024) and Alemohammad et al. (2024a). They provided the initial theoretical definitions and analyzed the behavior of simplistic multivariate Gaussian toy model in such loops. However, their analyses were limited to basic theoretical insights and lacked in-depth exploration of more complex generative models.
Recent advancements in this field have primarily come from Bertrand et al. 2024 and Fu et al. (2024b). [ertrand et al. al. (2024) established an upper bound on the deviation of likelihoodd-ased model output parameters from the optimal ones, denoted as ||0; - 0* This was achieved by making direct assumptions on the upper bounds of both statistical and optimization errors in generative models, as outlined in their Assumption 3. In contrast. Fu et al. (2024b) derived bounds on the TV distance, addressing the distribution divergence between the synthetic data distributions produced by future models and the original real data distribution, with a specific focus on diffusion models. Our work makes significant theoretical advancements over both Bertrand et al. (002)) and Fuu et al. (2024b) in several key aspects:
Innovative Concept of Recursive Stability. A central technical contribution of our work the extension of the traditional notion of algorithmic stability. We define recursive stability, a crucial factor for controlling error propagation across generations. This novel concept tackles the theoretical challenges posed by non-i.idd data and recursive structures within STLs, while also incorporating the influence of model architectures into the generalization error. Moreover, recursive stability serves as new measure for assessing the stability of generative models within STLs. In Theorem 2, we further establish an upper bound on the recursive stability parameter for transformers under mild conditions, underscoring the broad applicability and robustness of our framework.
2. Establishing the First Generalization Error Bound for STLs. While Bertrand et al. (2024) primarily focused on parameter deviations in generative models and Fu et al. (2024b) concentrated on distribution divergence, our work emphasizes the utility of the generated data produced by STLs. Specifically, by utilizing recursive stability, we present the first generalization error bound that quantifies the gap between the population risk on the initial real data distribution Do and the empirical risk of the hypothesis A(S.), generated by applying learning algorithms to the synthetic data produced after multiple generations of STLs. This introduces a new layer of complexity compared to prior work, as it necessitates handling not only the distribution shifts within STLs but also the challenges arising from the no.-i.i.d. nature of the mixed datasets, where each generation's 's data is influenced by all preceding generations.

---

Published as a conference paper at ICLR 2025
3. A More General Framework Accounting for Model Structure. Our proposed theoretical framework is more comprehensive than previous studies. [Bertrand et al. (2024) restricted their analysis to simplified likelihood- based generative models, while et al. (2024b) focused specifically on diffusion models. Importantly, neither of their theoretical results accounted for the impact of different model architectures. In contrast, discussed in Remark [ our framework explicitly incorporates the effects of varying model structures, thereby extending its applicability to a broader range of generative models. Notably, are the first extend the theory of SLTs to transformer models, further broadening the scope of our approach across diverse generative model architectures.
Comprehensive Collapse Prevention Through Recursive Stability. addition to the existing theoretical work, which primarily analyzes conditions to avoid model collapse based on the proportion of real data (e.g., |Bertrand et al. (2024); Fuu et al. (2024b)), our work extends these analyses by considering the impact of model architecture. Specifically, Theorem demonstrates that under recursive stability condition and non-negligible constant level of real data, model collapse can be avoided across variety of model architectures. This analysis offers broader conditions for preventing collapse by incorporating recursive stability, deepening the understanding of how model architecture affects training robustness.
Remark 7. Proof Sketch of Theorem 1 We first decompose the generalization error of STLs into two distinet terms: (1) the cumulative distribution shift across generations, and (2) the generalization error on the mixed dataset.
Cumulative Distribution Shift: This term measures the shift between the real dataset Dn and the mixed distribution D; after the i-th generation. Using the TV distance to quantify the shift introduced by the generative model, we bound the difference as:
RD. (A(S.) R,, (A(5.)) s (L-a) RD,
Hs_,, +
By leveraging the recursive structure of the generative process, this cumulative distribution shift can be bounded across all generations as:
RD, (A(S:) RD, (A(S))) 2M (1 (1-1(111))) ddv(()))
Generalization Error on the Mixed Dataset: The second term quantifies the generalization error when training on the mixed dataset 5;, which consists of both real and synthetic data. Our goal is to establish moment bound on the generalization error, which can be decomposed follows:
RD, (A(5))
1
Σ ((A(S)).z,) + [I(1 - Ro, (A(5.) 2.ESu.a
1
Σ ((A(S)): 2:) 2.ES,,155
In this context, Su,a represents proportion of the data points in S., leading to total of data points. Similarly, St,1-a denotes a subset the synthetic dataset St, where S S S; and its size is (1 ax) a+)
[. For each term, we leverage the uniform stability 3n, of the learning algorithm
A and the recursive stability 7/ of the generative model to address the non-i.i.d. nature of the mixed dataset. The mixed dataset exhibits conditional independence (Zheng al. al.]|202 (023)). with synthetic data conditioned on the initial real dataset So, allowing the application of recursive techniques to derive the moment bound. Subsequently, Lemma X and Lemma are utilized derive the high-pro gigo-probbbylityy bound for the final result.
5 THEORETICAL ANALYSIS OF TRANSFORMERS IN IN-CONTEXT LEARNING
In this section, we first present the transformer in in-context learning (ICL) and its settings within SLTs in Section 5.J] In Section (5.2, we prove that satisfies recursive stability, followed by the derivation of the generalization error bound for transformers in ICL in Section [5.3] Finally, in Section we explore the scenario of synthetic data augmentation and investigate the associated trade-offs.
5.4
5.1 SETTINGS OF TRANSFORMER IN IN-CONTEXT LEARNING
In-Contex Learning Setting. ICL involves transformer model processing sequence of input- output examples to perform inference without parameter updates. Unlike traditional supervised learning, where model is trained on a fixed dataset and then makes predictions, ICI. allows the model
7

---

Published as a conference paper at ICLR 2025
to adapt on-the-fly to new queries based on the provided examples. We denote prompt, containing in-context examples followed by the (n + 1)-th query input, as S = (21, Ø2, 2ns 11 where (zz) (z,, y:) ;=1 Z X' ) represents i..d. in-context samples. and 1 X' is the query input whose label we want to predict. The transformer model, denoted as (F).), takes the prompt So as input and outputs the predicted label #n+1 for the query #n+11 :: Yn+11
(So))-
Recursive Data Generation in STLs with ICL. We extend the traditional ICL setting to an STL, where the transformer recursively generates new data using its own ICL predictions. Starting with an initial real dataset So. this serves as the initial real in-context examples for the transformer. The process begins by sampling the first generation queries { i..d. from the input distribution X. Each query æ1, is incorporated into the in-context examples from So as a new query 2p,++1+ and the transformer predicts the corresponding label D1.j- This produces a synthetic dataset S1, consisting 1 A mixed dataset S, then formed and
of inputs { and their predicted labels { used as the in-context examples for the next generation. This process continues, with each generation producing a new synthetie dataset Sj+1 based on the updated mixed dataset Sy.
5.2 RECURSIVE STABILITY OF IN-CONTEXT LEARNING WITH TRANSFORMERS
In this section, we demonstrate that transformers exhibit recursive stability within the ICL framework. Following the ICL setting from Li al. (203). we show that the model effectively controls error propagation from perturbations in the initial real dataset, ensuring stability across the STLs.
Theorem 2. Let So, Si be two initial real datasets that only differ at the inputs 2j (Iz yi) and z (z', v/)) where 1 S j S Assume the inputs and labels lie within the unit Euclidean ball in Rd. Represent the prompts So and So as matrices Zo. Zo E R(2n+1)xd Let (F(.) be an L-layer transformer. Given Zo the initial input, the k-th layer applies MLPs and reff-attention, producing the output:
Zu Parallel_ MAPT(ATTN (Zn-1)) where ATTN(Z) == softmax ( (ZW2 zT) ZV.
Assume TF is normalized as (IV 1, [||[ S Bw and MLPs obey MLP(z) ReLU 2) with [M I 1. Let TF output the last token of the final layer Zt. that corresponds to the query 2. 1-
Let n represent the sample size of the mixed dataset Sy. where S = aSo + (1 a for 1 S j S i. Then, we obtain:
((+1) Bw
TF(S.) - TF(S) <()
2n +I
where Bw (1 2Bw) eBBw and St denotes the mixed dataset at the i-th generation in the STL
when the initial real dataset is Sy. Additionally, if the measure d for the recursive stability parameter in Definition 2 is taken as the l norm, then the recursive stability v1 (1- (1-aa)
2n+1
Remark 8. Controlling Exponential Growth with Real Data Proportion. In this remark, we fur- ther investigate the influence of the proportion of real data As outlined in Theorem 2 the upper bound of the recursive stability parameter includes a term that grows exponentially with the number of generations i in the STL, specifically BB ((++1) However, we show that even a constant proportion of real data, a, is sufficient to control this growth.
the recursive stability of transformers.
Specifically, setting = 2(1 - By, ((++1))))) we establish that the recursive stability parameter in Theorem satisfies 7 S Tn+T. T Additionally, as the number of generations in the STL approaches
infinity, the proportion a asymptotically converges to - B Notably, the depth I. is typically small in practical settings. For example, studies on LLM performance in STLss such as Briesch et al. (2023), often employ models with L = 6. Furthermore, techniques like layer normalization effectively constrain the norm of weights Bw. ensuring numerical stability during training. Thus, with constant real data proportion a independent of the STL generation number i, the exponential (1-1)L can be effectively controlled, ensuring that 7/1 (11/n)
growth term BW
5.3 GENERALIZATION BOUND FOR TRANRPRRE IN IN-CONTEXT LEARNING
In this section, we investigate the behavior of transformers under the ICL framework in STLs. We select SGD as the learning algorithm A and consider binary task with y {0, 1). Applying
8

---

Published as a conference paper at ICLR 2025
general theoretical framework from Theorem I we derive the generalization error bound by addressing the terms 8n and drv(n) using recent results on SGD ZZhagg et al. 2022) and ICL (Zhang et al. 2023). The recursive stability parameter 7/2 is obtained from Theorem 2 We assume that the
loss function ((:2) is x-smooth and p-I.ipschiz, which are standard assumptions in related works
(Hardt et al. 2016 Lei & Ying. 2020), with formal definitions provided in Appendix A.T Examples
include logistic and Huber losses. We now present the generalization error bound:
Theorem 3. Consider an I-layer transformer under the setfing described in Theorem 2 Let
represent the sample size of the mixed dataset Sj, where S, aS, (1 - for S j S i.
Suppose that the loss function E(:; z) is k-smohth, p-Lipschitz and bounded by M 0 for every
Let A(S;) denote the output after running SGD for T Z n iterations with step size 7pp O [ on the mixed dataset St. Then, for any 5 E (0, 1), with probability at least 1 - 6, the the following holds:
1
RD. (A(..) Rs3 (A(S))) S n--nn 1/2
n-122 log(n) Mpaaa poaa a log
8
1/4 In (n)'p''2)))((((11 ((1))((1 - a) B w )'a log( )) + +n M (1-(1 a))) log( 8)
Remark 9. In this remark, we provide detailed explanation of the theoretical results of Theorem As discussed earlier in Remark 8 a is set to Bw' To enhance clarity and focus on the primary results, we omit constant terms and the log(1/5) factor. Consequently, the bound in Theorem can be expressed as follows:
(A(S.) - R3, (A(S.)) S n 11/2 log(n) + n log (n) + + 1/4
1/2 log(n) + n log2 (12) correspond to the generalization error on the In this bound, the terms -1/2
mixed dataset, while the term n 1/4 represents the cumulative distribution shift across generations, which is primarily governed by the learnability of the generative model.
It is evident from this result that the generative model's capacity plays crucial role in the performance within the STLs. The ability of the generative model to maintain distributional fidelity over multiple generations directly impacts the generalization error and determines how well the model can control the propagation of errors across generations.
5.4 SYNTHETIC DATA AUGMENTATION
The previous theorem addresses the scenario where the training dataset is unintentionally contam- inated by synthetic data, leading to STLss In contrast, many researchers intentionally incorporate synthetic data augment the real dataset, also creating STLs. Next, we explore this synthetic data augmentation scenario, where each generation's 's synthetic data is added to the mixed dataset, i.e.,
S S==Xx Ej Sy-
Theorem 4. Consider an L-layer transformer under the setting described in Theorem 2 Let n and An represent the sample size of the real dataset So and the synthetic dataset Sy, respectively, where 1 S < i. The mixed dataset 5; is denoted as Ej-o Sj. Suppose that the loss function ((:: k-smooth, p-Lipschitz and bounded by M > 0 for every z. Let A(S.) denote the output after rumning SGD for T Z n. iterations with step size πソ = O 2 on the mixed dataset S.. Then, for any 8 E (0, 1), with probability at least 8, the following holds:
1
RD. (A(3.)) - Rgg (A(S.)) Sn- log((1 + +i))) log
p² (1+ log(( (1 (1)m) 1)) log +n I+
1 8
Mi
+ an-11
iA
V
log 8
Remark 10. Analyzing the Trade-off in Synthetic Data Augmentation for STLs. In this remark, we examine the trade-off between generalization and distribution shifts from increased synthetic data, providing insights into optimal synthetic data size. At each generation, An synthetic data points are added to the mixed dataset. We analyze how the coefficient X, representing the scale of synthetic data augmentation, affects the generalization error in STLs. From the bound in Theorem that the Cumulative Distribution Shift Across Generations term is expressed as:
observe
1 log((1 (1 + i log(1/

---

Published as a conference paper at ICLR 2025
As the coefficient À increases, the cumulative distribution shift correspondingly grows, thereby amplifying the associated error. This behavior aligns with intuition, as an increase in À reduces the proportion of real data within the mixed dataset at each generation. Consequently, this reduction in real data leads to a greater divergence between the mixed distribution and the true underlying distribution, exacerhating the deviation and compounding the error across successive generations. In contrast, for the Generalization Error on Mixed Distributions term:
(1 iAy2 log((1 + iA) ( log
Mi I+
log
We observe that À increases, the corresponding error decreases. This outcome is consistent with theoretical intuition, as augmenting the dataset with synthetic data effectively enlarges the mixed dataset. larger dataset provides a more comprehensive representation of the mixed distribution, which in turn reduces the generalization error associated with this distribution. By incorporating more synthetic data, the mixed dataset better approximates the underlying mixed distribution, leading to improved generalization performance.
From the above discussion, we can conclude that the inclusion of synthetic data introduces trade-off: on one hand, it increases the error from the cumulative distribution shift, while on the other, it reduces the generalization error on the mixed distribution. This trade-off has been explored theoretically in Fu et al. (2024b), though they primarily provided theoretical intuition. In contrast, our work explicitly decomposes the error into two terms, offering a deeper understanding of this trade-off and its implications for model performance in STLs. As for the optimal augmentation coefficient X", it must satisfy the following condition:
x = inf n log((1 + M Mlg(1) A
P² (1+ log B (1+1) S n
Mi
1 log g
log +
1 ix
Unfortunately, obtaining a closed-form solution for X from this equation proves to be analytically intractable. However, we can derive the relationship between X*, the size of the real dataset from the above equation. Specifically, by omitting irrelevant constants and the log(1/5) term, we obtain that A should satisfy the following expression:
1B (1+1))
n3/4(1 + 127)2 n1/4(1 + iX*) log((1 + +**))) O().
We observe an important trend: the value of X" increases as the size of the real dataset n decreases. This aligns with theoretical intuition, as smaller real dataset struggles to adequately represent the underlying distribution, leading to higher generalization error. Consequently, more synthetic data required to control the generalization error of each generation on the mixed distribution. Conversely, when the real dataset is sufficiently large, the need for synthetic data augmentation diminishes.
6 CONCLUSION
As real-world data becomes increasingly scarce and existing datasets are progressively contaminated with synthetic content, STLs have emerged necessary strategy. STLs enable generative models to recursively train on a mix of real and synthetic data. However, empirical outcomes have varied significantly, revealing the need for theoretical foundation to guide their successful application.
In this work, we introduced recursive stability as key technical innovation and established the first generalization error bounds for STLs, which consider the impact of different model architectures. Our analysis demonstrated that preventing model collapse requires two critical conditions: maintaining non-negligible proportion of real data and ensuring that models satisfy recursive stability. Furthermore, were the first to extend this framework to transformers in in-context learning, showing that they also satisfy recursive stability and establish their generalization error bounds. Finally, we explored the trade-off introduced by synthetic data augmentation, balancing generalization improvement with potential distributional shifts. These contributions provide new insights into enhancing the stability and performance of generative models in STL.s.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This project is supported by the National Research Foundation, Singapore, under its NRF Professor- ship Award No. NRF-P2024-001.
REFERENCES
Sina Alemohammad, Josue Casco-Rodrigee Lorenzo Luzi. Ahmed Imtiaz Humayun, Hossein Babaei. Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go mad. In The Twelfth International Conference on Learning Representations, 2024a.
Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. Self-improving diffusion models with synthetic data. arXiv preprint arXiv:2408. 16333, 2024b.
Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33, 2020.
Quentin Bertrand, Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models their own data. In The Twelfth International Conference on Learming Representations, 2024.
Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory, pp. 610 626, 2020.
Martin Briesch, Dominik Sohania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint 1xXiv:2311.168/2 2023.
Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference on Machine Learning, pp. 744-753, 2018.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07711 2024a.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint arXi::2410.04800, 2024b.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference on Machine Learning, 2024c.
Doukhan. Mixing: Properties and examples. Lecture notes in statistics. New York: Springer, 1994.
Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradient- based minimax learners. In International Conference on Machine Learning, pp. 3174 3185. PMLR, 2021.
Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory, pp. 1270-1279, 2019.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires reinforcement. In ICML 2024 Workshop Theoretical Foundations of Foundation Models, 2024a.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, Julia Kempe, and FAIR Meta. Be- yond model collapse: Scaling up with syn-thesized data requires verification. arXiv preprint 2oXiv:24066075155 2024b.
11

---

Published as a conference paper at ICLR 2025
Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self-consuming generative models with curated data provably optimize human preferences. arXiv preprint arXiv:2407.09499, 2024.
Shi Fu, Yunwen Lei, Qiong Cao, Xinmei Tian, and Dacheng Tao. Sharper bounds for uniformly stable algorithms with stationary mixing process. In The Eleventh International Conference on Learning Representations, 2023.
Shi Fu, Yuzhu Chen, Yingjie Wang, and Dacheng Tao. On championing foundation models: From explainability to interpretability. arXiv preprint 1rXiv:2410.11444/ 2024a.
Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models. In Forty-first International Conference on Machine Learning. 2024b.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, In First Conference on Language Modeling, 2024.
Nate Gillman, Michael Freeman, Daksh Aggarwal, HSU Chia-Hong, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training. In Forty-first International Conference on Machine Learning, 2024.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225-1234, 2016.
Fangchao He, Ling Zuo, and Hong Chen. Stability analysis for ranking with stationary s-mixing samples. Neurocomputing, 171:1556-1562 2016.
Markus Hittmeir, Andreas Ekelhart, and Rudolf Mayer. On the utility of synthetic data: An empirical evaluation on machine learning tasks. In Proceedings of the 14th International Conference on Availability, Reliability and Security, pp. 1-6, 2019.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.16110. 2022.
Millen Kanabar and Michael Gastpar. Minimax discrete distribution estimation with self self-consumption. arXiv preprint arXhv:2501.19273. 2025.
Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with conver- gence rate O(1/n). Advances in Neural Information Processing Systems, 2021.
Yunwen Lei. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. The Thirty Sixth Annual Conference on Learning Theory, pp. 191-227. PMLR, 2023.
Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning, pp. 5809-5819, 2020.
Shaojie Li and Yong Liu. High probability generalization bounds with fast rates for minimax problems. In International Conference on Learning Representations, 2022.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoulos. and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594 PMLR, 2023.
Tengyuan Liang. How well generative adversarial networks learn distributions. Journal of Machine Learning Research, 22((28):1 20211
Matteo Marchi, Stefano Soatto, Pratik Chaudhari, and Paulo Tabuada. Heat death of generative models in closed-loop learning. arXiv preprint 2043:22404.023255, 2024.
12

---

Published as a conference paper at ICLR 2025
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Heráández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. In International Workshop on Epistemic Uncertainty in Artificial Intelligence, pp. 59-73. Springer, 2023.
Mchryar Mohri and Afshin Rostamizadeh. Stability bounds for stationary y--mixing and [ B-mixing processes. Journal of Machine Learning Research, 11(,), 2010.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi, Can ai-generated text be reliably detected? arXiv preprint arXiv:2303. 156, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pieme Youssef, and Merouane Debbah. How bad is training on synthetic data? statistical analysis of language model collapse. arXiv preprint arXiv:2404.00000 2024.
Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11:2635-2670, 2010.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631 2024.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXiv preprint 2404 14387, 2024.
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.066 0.3225, 2022.
Peng Wang, Li Shen, Zerui Tao, Shuaida He, and Dacheng Tao. Generalization analysis of stochastic weight averaging with general sampling. In Forty-first International Conference on Machine Learning, 2024. URL //tperreeee/eeeeeeeeeeeee net / http://rrmmmiiww///////ao
Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. Fairness feedback loops: training on synthetic data amplifies bias. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 2113-2147, 2024.
Xiaodan Xing, Fadong Shi, Jiahao Huang, Yinzhe Wu, Yang Nan, Sheng Zhang, Yingying Fang, Michael Roberts, Carola-Bibiane Schönlieb, Javier Del Ser, et al. On the caveats of ai autophagy. Nature Machine Intelligence, pp. 1-9, 2025.
Shirong Xu, Will Wei Sun, and Guang Cheng. Utility theory of synthetic data generation. arXiv preprint ar arXiv:2305.10013 2023.
Hongkang Yang. A mathematical framework for learning probability distributions. arXiv preprint arXiv:2212.111488 14481, 2022.
Bin Yu. Rates convergence for empirical processes of stationary mixing sequences. The Annals of Probability, pp. 94-116, 1994.
Yikai Zhang, Wenjia Zhang, Sammy Bald, Vamsi Pingali, Chao Chen, and Mayank Goswami. Stability of sgd: Tightness analysis and improved bounds. In Uncertainty in artificial intelligence. pp. 2364-2372 PMLR, 2022.
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305. 19420, 2023.
13

---

(N E), Published as a conference paper at ICLR 2025
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmenta- tion. Advances in Neural Information Processing Systems, 36:54046 -.544066-5000,, 2023.
Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou. How to synthesize text data without model collapse? arXiv preprint arXiv:2412. 146,,99 14689, 2024.
A APPENDIX
A.1 AUXILIARY DEFINITIONS
Below, we present some essential definitions.
Definition 3. (Lipschitz and Smoothness). Let constants K. p > 0. Consider the function W x 2 + R. We define the following properties:
Lipschitz Continuity: The loss is said to be p-Lipschitz continuous if ||((w1, z)
((w2.2)] 2)]l I
pl|w) w w2/l for any wi, W2, w
Smoothness: The loss / is said to be x-Smooth if | wlww,, z) V vwl(w2, z) k||w] W2|| for any w1, W2, 2.
A.2 EXPANSION TO GAUSSIAN MIXTURE MODELS
We adopt the setup from prior works Zheng et al. (2023) and consider binary classification task where Y 1, Given vector R with 14 2 = 1 and noise variance o² > 0, the data
distribution specified as follows: y ~ uniform(-1, conditional generative model using parameters Hyy and of where y E {-1,1) and [d]. For n data points, let ny represent the number of samples in class y. The parameters of the Gaussian mixture model are then learned as:
and y ~ N(yu, La We define the
Em=y
Rg Σ En. (Dik Pyy))
Byy
Fly
Then we can generate new samples from the distribution: y (-fform(-11,)) and æ Additionally, the learning algorithm functions as linear classifier, parameterized by 0 E Rd, with predictions given by: ý = sign(# x). The loss function is defined as:
",.....,))), o²). where Σ = diag( (
(x, 20 (x = y0).
Thus, the output is 0 Em Vi
In this setting, we demonstrate recursive stability for the Gaussian mixture model as follows:
Theorem 5. Let So, Sy denote two initial real datasets differing by single example. Let n represent the sample size of the mixed dataset Sz. where S, a.S + (1 a) a) for 1 j i. Choose m = (nnn) Consider the previously described sampling and learning steps, where real data samples drawn from the Gaussian Mixture Model distribution DD and the synthetic data for the i-th generation is generated from the learned Gaussian Mixture distribution of the i-th generation.
Then with probability at least 1 6, have:
Sn 1/2 11111(111110111 (1-a))) log(n//))))
where the measure for the recursive stability parameter is taken as the KL divergence.
As a approaches 0, indicating that no real data is incorporated during each generation of training, we observe
v S in 1/2 log nd 5
which suggests a linear accumulation of errors. This finding aligns closely with the theoretical insights presented in Shumailov et al, (0044) Alemohammad et al. 2024a), where Gaussian model
14

---

Published as a conference paper at ICLR 2025
trained without real data demonstrated linear divergence in variance. Thus, this underscores the validity of our theoretical results, confirming that the derived bound is meaningful and not vacuous.
Moreover, by leveraging the generalization error bound established in Theorem 1, derive the following:
Theorem 6. Consider the Gaussian Mixture Model in the setting outlined above. Let represent the sample size of the mixed dataset Sy. where S₁ a.S + S; for 1 S Suppose the loss function is defined as E(0, (x, y)) 20 2x= y0) (x y0). Let A(S.) denote the oulput of applying the linear classifier described above to the mixed dataset S. Then, for any E (0, 1), with probability least 6, the following holds:
RD. (A(3)) Rg. (A(5.)) 2 ( log(n/8)) log log(1/6)
11 - a (d log(n/0)) /dlognd/6)).
(d+
(4)
We observe that when is set to a constant (c.g., = 0.1), the generalization error can be effec- tively controlled, preventing model collapse. This result aligns with the experimental findings in Alemohammad et al. (2024a) for Gaussian models.
A.3 ADDITIONAL COMPARISON WITH RELATED WORK ON THEOREM
Dohmatob et al. (2024a) examined a linear regression setting, focusing solely on statistical approx- imation error without addressing the functional approximation error described in (Shumailov et al. (002)). They did not consider incorporating real data to prevent collapse and demonstrated linear dependency of degradation
the generation number in the case of fully synthetic data. Similarly, Alemohammad et al. (2024a) and Shumailov et al. (2024) provided theoretical insights using simple Gaussian models without incorporating real data, proving that the variance diverges linearly with the generation number. (eddik et al.] (2024) explored a linear softmax classifier and, while also neglecting functional approximation error, demonstrated that adding real data mitigate model collapse. Marchi et al. (2024) used asymptotic analysis to study parameter variance, assuming an infinite number of training generations and considering scenarios where the generative model is controlled via "temperature" parameter. They proved that parameter variance is bounded under these conditions.
In contrast, our work addresses much more complex and realistic scenario by introducing the novel concept recursive stability and providing the first generalization analysis for STLs. Our analysis accounts for statistical approximation error, functional approximation error, and optimization error during the training of generative models. Unlike the settings explored in prior theoretical works, such as linear regression (Dohmatob et al., 2024al Gerstgrasser et al.) 2024). Gaussian models (Alemohammad et al.) 2024at (Shumaloov et al. 2024). or asymptotic assumptions (Marchi et al. 2024), our framework accommodates more complex generative model architectures, such as transformers. Specifically, we reveal how both model architecture and the ratio of real to synthetic data influence the success of STLs. For example, in Theorem general generalization bound applies to transformer-base based generative models, providing a theoretical framework that aligns with practical and more sophisticated use cases.
demonstrate how our
Additionally, while Marchi et al. (2024) assumed an infinite number of training generations for their asymptotic analysis, we consider finite generations, which is more practical since most experimental setups limit generations to fewer than 10 (as noted in SShumailov et al. (2044)). Moreover, our results confirm that when Cumulative Distribution Shift (n) M (1 - aa))a="> grows linearly. This finding aligns with the theoretical results Dohmatob et al. (2024a); AAemohhmmaa et al. (2024)); (Shumailov et al. (2024); Fuu et al. (2024b). Furthermore, we show that introducing even constant proportion of real data significantly mitigates model collapse, aligning with experimental findings by Alemohammmad ct al. (2024a) and Bertrand et al. (204))
(i.e., no real data is used). the last term our bound, representing the
A.4 ADDITIONAL COMPARISON WITH RELATED WORK ON THEOREM 4
Gerstgrasser et al. (2024) also explored the use of accumulating data to prevent model collapse. They considered simple linear regression setting without accounting for the dynamic process of
15

---

Published as a conference paper at ICLR 2025
training generative models, focusing solely statistical approximation error. They demonstrated that under the assumption of fixed synthetic data quality matching the original real data, statistical approximation enror can be controlled.
By contrast, our work addresses a much more complex and realistic scenario, incorporating the dynamic behavior of transformer-based generative models, learning algorithms, and both statistical and functional approximation errors. Additionally, we allow for dynamic regulation of synthetic data size via À coefficient, enabling identify the optimal synthetic dataset size for avoiding model collapse in these more challenging settings.
A.5 AUXILIARY LEMMAS
In this section, we begin by introducing set of auxiliary theorems that will be utilized in the subsequent proofs.
Lemma (McDiarmid's Inequality). Consider independent random variables Zı, Zn Z and a mapping Φ 2" - R. for all E {1,...
,,,",,,, n.) and for all 21, 2n: 2/ E 2, the function ø
satisfies
(21 : 8141s ... 2n)) ① (z1, ,..,,,,-111
21 2/ Zit1, ... 2n) S
then,
-212
P (1 (7,,... ,,,) RDb (Z,,..., 7m) 2 11) S exp
nc²
Furthermore, for any p 2
| (Zi..... Zn) - EE EZ Zn)llp S 2 npc.
Lemma 8. ((Bousquet et al. 2020 Let z = (Zi..... Zn)) be a vector of independent random variables each taking values in Z, and let 91.... In be some functions 9: Z" - R. such that the following holds for any E [n]:
E [ox(2)) 2 S M,
2 = 0, E [9;(z)
gi has bounded difference 3 with respect to all variables except the i-th variable, that is. for all j * i, = (Z,,..., Zn) and x) (Z,....ZZ)...... Ln) E R", we have
[z( 9: (z)) 3.
Then, for any p 2 2,
Σ (z)
S 12v22m logn + 4M ypn.
Lemma 9. || I /pa + for any 2 1, then for any 5 E (0, 1), with probability at least 1 1-8, 8,
| srlse a lolo log ( b log )7)
In addition, we introduce the definition of the Total Variation (TV) distance follows:
Definition 4 (Total Variation Distance). Given two probability distributions p and q multidi- mensional space Rdd the Total Variation Distance between p and is:
1 / TV(p. 9) 2 ga
lp(z) {(2)) dz.
16

---

Published as a conference paper at ICLR 2025
A.6 PROOF OF THEOREM 1
In this Section, we prove Theorem 1 by first decomposing the generalization error into two compo- nents: the Cumulative Distribution Shift Across Generations and the Generalization Error on Mixed Distributions. We then proceed to bound the Cumulative Distribution Shift Across Generations by leveraging the properties of the generative model and recursive techniques. For the Generalization Error on Mixed Distributions, we follow the framework of Zheng et al. (2023). leveraging the fact that within the mixed dataset S;, the set S; satisfies the conditional i.i.d. assumption when So is fixed. Combined with moment bounds, this allows us to effectively bound the generalization error.
The main proof is follows:
Proof of Theorem 4 We begin by decomposing the generalization error as follows:
RD. (A(S.) - Rg. (A(5.)) v RD, (A(S)) - Rs, (A().))
RS, (A(S.) Rg. (A(()))
Cumulative distribution shift across generations
Generalization mixed distributions
Upper Bounding Cumulative Distribution Shift Term
For the term
RD, (A(Š.)) R5, (A((A) we first note that D. aDo + (1- a)D, Therefore.
we obtain:
RD. (A(S:)) R5g (A(5:))
(A(3)) aRD, (A(3)) - (1- (A(.
(1 a1) RD. (A():) RDD (A(5:))
(5)
Furthermore, we can further decompose it follows:
RD. (A(S.) RD, RR, (A(5.))
< (A(()) R. (A(3:)) + RB. RD. (A($:))
(6)
By substituting inequality 0 into inequality 5 we obtain:
Ro, (A(5.) R2, (A(5.))
(1 a) RD. (A().) Ro,,,, (A(5.)) + +1- R..,) RD, (A(3.)) (7)
Then, for the term RRD_, (A((:)) R.D. (A (A())))), we have:
RDD (A(5.)) = / ((A(3)),)) (
R Rb__,
(A(Š.)) RB_,(((5.))
IPD, (z) dz
< /. ((A(3)) (Po__,))) - -PD,(z))
< M / /|PPA... [PP....))) -Pr,
(z) -x),)))))))))
2MTV (D...
(8)
Incorporating inequality & into inequality
7
we arrive at:
|RD, (A(5,)) - R.s (A(()))
< (1 Ra), RR,,,, (A(S.)) + 1 a)MVV (D.....) Next, we apply recursive techniques to address the problem further. First, we obtain RR, (A(5.)) (A(_..(())))
(9)
< (1 - R),, (A(_((A())) + 2(1 - a)MTV (D.2.
(10)
17

---

Published as a conference paper at ICLR 2025
Plugging inequality into inequality nnto the inequality, obtain that:
RRD, (A((:)) - R5, (A(S)))
(5,_,(((SS)))) 2(1 - MTV (D12. +((1 - +2)1-a)MTV (D.....).
(1-
By recursion, obtain:
(RD((A(S)) RD, (A(5.))
a1 1 (A(5:)) RR5 (A(S.)) 2(1 a11a))- (Di, +...+ 2(1 -1)11)) (D.-1,,) 2 aaaaa''M'' MTV (Do. D1) + + - a1---)"-11a aaa-'M"'V (DI,
2(1 a)M (D..,..).
Let ny represent the sample size of the real dataset Su, and let n, denote the sample size of the mixed dataset S; in the i-th generation. Thus, TV (D.. Dy+1 can be written as function of n2.. Assuming that the sample size for each generation's 's dataset is identical, i.e., no = =1 = ... = n2 = n, and that the TV distance for each generation is of the same order, denoted by drv(n), we can derive the following result:
RD, (A(5.) RB, (A(5.)) S 2Mdrv(() [[1 a)* (1-
a) +... (1 a)] drv(n).
2M (1 a1-))) a -
(11)
Then we obtain:
RD, (A(S:)) Rgg (A(S.)) & (A(S)) Rs, (A(S:)) + IR (A(Si) Rg. (A(()))
< 2M - (1-(1-a))) ao dTv(n) + [Rs, (A(5.)) - Rgg (A(5.))
(12)
Upper Bounding Generalization Error on Mixed Distributions Term
Next, turn attention the term RR, (A(5.)) Rgg (A(S.)).. Our primary objective is to establish moment bound for this expression.
Rs (A(S.)) - R3.. (A(5:))
aRD, (A(5.)) + (1 - Ro,
(A(3.)
Σ ((A(5),,
Σ
((A(3)), zi)
2,E,0.a
2,ES;1
aRD, (A(S.) 1 Σ E(A(S)), zi)
+ RD.(s)
Σ 2:E8,11 a
E(A(St),
S
(13)
The newly sampled dataset, denoted as Sa... subset of the original dataset So, where So, C So and its size is a x |Su]. Specifically, So,a contains proportion a of the n data points in Sp. resulting in a total of
a data points. Similarly, S1,1-a - is a subset of the synthetic dataset S,, where Si,1-a S S;, and its size is (1 a) x [Sa). Specifically, Si,1 contains proportion 1 - of the data points 5,, resulting
(1 a:) data points.
We observe that for any function f(S), if there exists bound [|f (S₁) < C for some subset S, C S, then we have the following:
|||| = (EE (MSP S/ s (E S.
Fix So. then data in S; are independent. We use this property and Lemma g to bound the Term 2. We introduce functions (S,,1_a) which play the same role gj 's Lemma as
(S = ExI,yD,
U - z) - (AASS,. a 21.)
18

---

Published as a conference paper at ICLR 2025
where Zz,j is the j-th data in St,1aa+ and 5 - obtained by replacing 2G, by <'.g. Next, prove that fj satisfies the three conditions outlined in Lemma g
First, we demonstrate condition [f] M.
155l = [es, BBNDD ((ASS,, U U. 2) - C(A(SO, U. إاريح I ND, ((AASo,a 2) E(A(So.o U - إزيح
x M
We then continue by proving conditions |[
fj S i,1-a
Bei ~ Di BEND ((A(So, U (()1-))))) - e(A(So.a U. 2n5) s
DE (A (Sn
Son U U. .2) - EELL & (A ( (So, U US - [ 5
0 5/ 1
= O.
Finally, we prove that fg has bounded difference 23,, with respect all variables except the j-th variable. Let t * j, then we obtain:
|t, (St,1-a (a))
=IB=i ~D; EZND. (A(
- ((A(So.. US U.(11-a:
2nri =
EEi,,~D ~D, [BEND, ((A(So, U (S - ((A(So. U (S 13)11=>)))
a.')) 2z,)
Ber ND BEND, ((A(So,n U - (A(So, U US ) =)
Exz,,, ~D. [((ASSu,a U - 21,1) ((A(S,, U US z1,,)
<B, + 8n 23,..
Therefore, for any fixed So, by Lemma for any p 2 2, we have
n(1 Σ f3 (S (S))
< Spn(1 3) log(n(1 -1) M MM ppm(11a)
(14)
19

---

Published as a conference paper at ICLR 2025
We note that the difference between Term 2 and 2j=1 - fy is minimal. Consequently, for any fixed So, can bound Term 2 using inequality 14 follows.
(1 R))
Σ
((A(S)), 21)
2:E8
n(l-a)
(1 - RDDD (A(So.a U. - Σ
1(A(So,o U U((11a))
Fix So
/=1
a)
1
(EgDD. (A((o, US 12) - 6 (AA(So, U , (in))
Σ
ND BxNDD((A(SO,. U 2) ((A(S0, U - 2(,) 2)
+ a) 1133,l11
2
-=1
n(1-a)
1
Σ
+(1 - (11-a)|128nl) f3 (S;11-a) -a)
1
p(1 a) + a3
p(1 3) log(n(1 aa) .M
n
p(1 a)
- P1 3n log(n(1 a)) + M
n
Next, for Term 2, we derive the following result:
p(1 a(
(1 - Ro.
Σ
((A(5)) 2;)
(0 + M
2,E8,114
(15)
(A(2.(((3.)) - - ((A(3)), z0)1, We
Now, we use similar idea to bound Term decompose it as the following.
a aRD, (A(5:))
Σ ((A(5)), 2i)
2: 2.E8o.a
(aRD((A(5))
Σ ((A(S)), - E s. -a (aRp, (A(S))
Σ
E(A(3i), 2;))
ESo..
Term 3
E Eg P0000 (0-) (A((.))
1
Σ (A((S),
(16)
2,E80.a
Torm
We proceed by bounding each term. Specifically. Term 3 can be bounded using McDiarmid's inequality, as outlined in Lemma 7 and Term can be bounded by applying Lemma g
To bound Term 3, we begin by fixing So.a and utilizing the conditional independence property of S; once again. In order apply Lemma 8 must show that alt,, (A(S:)) - Σ Ex.eS... ((A(S,), z;) exhibits bounded difference with respect to S when So, is fixed. This expression can be
20

---

Published as a conference paper at ICLR 2025
formulated as follows.
aRD, U ((,1-)))-
1
Σ
((A(So,a U.i, 2i)
Th 2.E30.0
-aRDo(A(SSn) U. (1%11-a))++
1
E ((ASS,,a U U.
n
2,ESo,a
Sa RDo (A(S,, US RD, (A(S,, U U. -
Σ ((A(So,a (.)
E
( US? (0)1-0))22)
21ESo.a
I a, + a., = 2a.n.
Thus, by Mediarmid Inequality, we have
1 Σ ((A(Si), - E -paa--o (aRD. (A(5.)) T 2.E30,n
Σ ((A(Si),
2i))
2.ESp,o
S
2 1n(1-a)pas3,
(17)
We now introduce set of functions and apply Lemma 8 once more to bound Term 4. Specifically, we define hS), which serves a similar role to the gi 's in Lemma as follows:
h (So
= -Ea6 Do E s., a ( (St.a) BENDoL (S U.S.Ia), - (eAA( (5% US 20.3)
(18)
where 2o,j denote the j-th data point in So,a, and Si.a represent the dataset obtained by replacing
Z0,i with zú.j- Additionally, it is important to note that S D, ($).a) indicates that Si,1-a is the synthetic dataset generated after the self-consuming loop, following i-generations, and obtained by modifying single data point from the initial real dataset Sg. This complex scenario can be addressed using the recursive stability we have defined for the self-consuming loop in Definition
Moreover, similar to the process above, we observe that hyy S M and E hy Sh. 0. More
intricately, we will now prove that hy exhibits bounded difference. This will be demonstrated as follows.
1hy hi (S%.a)
Ez-Dz (A (A(5%bo (sh U US 10).--),,)) - -E (A(AA USL) .20.3) .0.s))
|E5 ~D -a (m) (sk.))
- -Ez,, ~D, B. ((8) (1))
EZNDD (A ((St,))))) US 2) (A ((Sh,a)) U US :00,)]
< /Bey, ~ Do E st pii (8%,.)
BENDEf (A (5/ (A(5
U U. ) 12) - -e (A(s U U.
:000. ,
EEzh ~D Eg Es Dp (s)
EENDDL (A ((Sb,a)' U U. .2) (A U. 220.
(19)
+ DD
~D S
E
(|)))))) (st.))
BENDA (A ((Sb,a)' U St ,)) - C(A U.S. _0,5]]
- -E2í, ~D, E S ~D (1)))) ((8) [EZDDe (A ((St,)))) US ,z) C (AA((()))))))) ((St,a)) (US 20.3)
(20)
21

---

Published as a conference paper at ICLR 2025
We bound equation 19 by applying the concept of uniform stability, resulting in an upper bound
of 23, Regarding equation 20 for ease of notation, let us represent & (AA ((S U S1,1-a) 2)
è (A ((5,,)) U U 20.j) as Q. Consequently, we obtain the following:
12 #o, ~D S1 Dmr (S%...) DO (A (15) US eA (A(( UU US 20.)]
Rzb ~o S S B (15h (15)..)) DE (A ((S U U 2) e (A((5))))) U. =0.)]
-EEy ~Do EENDO
St,1 a ( 0 E S1, u -- (1S) (( @
/ S-,1-0 (P (P 18%ba) P (Si. (Sh,a)) ") QulSi.1-a
D BINDO
S,,10a (P (Si,1-a) 18%..) - IP (S (5t,a) Q dS
< B BINDO
((P (S (b.o) P (PP(s (S (5%.a))) dS
< MB ~D ~DD ExwD
< sup
TV TV
D. (So))
=2M
(21)
Thus, h exhibits a bounded difference of 28,, + 2M with respect to all variables except the j-th variable. By applying Lemma we obtain the following:
no
Σ
h (Su,a)
< 12v2pma (28,, + 2Mx) log(na) 4M /pna
I pma (0n - M log(na) + M Vpna.
We observe that the difference between Term 4 and EEe h5 (So,a) is negligible. Thus, we can bound Term as follows:
Bg a DD
aRD (A(5.))
1 n
Σ (5.). 2.)
2,ESp.a
(A(aa)) - (A((3)):zz 23) )]
Se -n (11-))) Dpp(i-n)
So,
na
BEWDoL (A (Sg U USiL-a): ,²) -((A(S US =Ro.))) (a.aa)=o.))]]
B=, Do E S D (sb..)
2a 2a
1 h (So.a)
+ 12 + 20M2/l,,
p
Spe (8n + M log(na)) M Vpann + a, + aMi
Sp (3n + Myhh log(na) + M Vpaan..
22

---

Published as a conference paper at ICLR 2025
By substituting the above inequality and inequality 17 into the decomposition 16
obtain:
alvo (1(3)) - - E (A 21
E,,0
3 V n(1 - pa pa (3, + Myh log (na) M / ppamm
/P(v(2 (1 - a + MV an + pa (3,, + My.)) log(na).
(22)
Plug inequalities 22 and 15 into the inequality 57 then we obtain:
[R (A(S.)) Rgg (A(S)))
Sp(1 a, log(n( a:) M / p(1 ao)-I++ vD(V(1 a)naBn + MVan-1)
pa (Sn, M log(na)
=VD ( (1 3n + Mn 1/2 (vI a vα)
+P ((1 3n lo - a)) + a (3n + My log(na)).
(23)
By applying Lemma we can derive a bound on the generalization error with respect to the mixed Rs (A (5:)) Rg, (A (5.)) follows.
distribution.
Rs. (A (5:)) Rgg (A (5:))
$ (V(1- ajnas, + Mn (vI va)) /1og (
+ ((1 - log,))), - aa) + a (Bn + Myhh log(na)) log (1)
Finally, we conclude that:
RD. A (5.) Rg. AA (5:))
< lT. (A (A.5)) RT (A(5.)) + Rg. (A(5.) Rgg (A
5:
< 2M (1 (1 a)') a dry(nn (n)
((1 an)n log(n(1 a)) + a (Bn M1hh log(na)) log 1 3
(V(1- + +Mn 1/2 M (1 + vai) log ())
(24)
A.7 PROOF OF THEOREM 2
In this section, prove that transformers in in-context learning exhibit recursive stability. Specifi- cally, we utilize the framework and lemmas from Li et al. 2023), combined with recursive techniques, establish the proof.
Lemma 10 Li et al. (023))) ). Let 2,E RD be vectors obeying ||=||<< || =elle S Then, there exists a constant C (cC). such that
|| softmax(=)lle___ 5 20/n and ||oottmax()) softmax(z + E)lls, S ee'leelle,/
Proof of Theorem 2 Let Z [z....., 2n and E = [ 6,] be the input and perturbation matrices respectively. Given that the tokens z; lie in the unit ball, and assuming 2; + e; also
23

---

Published as a conference paper at ICLR 2025
lies in the unit ball, can proceed with the following. For matrix, let || ||2, denote the Ep- norm of the vector formed by the fn-norms of its rows. Therefore, we obtain |
7 22,00 S
and
| 2,cc = Z 2.00 I 1. Let the attention outputs be defined as A = softmax (ZW2) Z ZV
and A softmax (ZWZ) ZV. Define the perturbation
E Á A E En
Let us examine the attention output difference E A follows:
A, which be further decomposed as
E = softmax (ZWZ)
ZV softmax ZW Z ZV
softmax ZWW )
softmax ZWZ ZV + softmax (WWZ) Z EV
(25)
F1
P2
We first observe that preserves the norm, meaning that ZV satisfies | 2,00 S | S
and |
E|2, Moreover, for any pair of tokens, it holds that |z Wzj S Bw. Applying
2,1
Lemma 10, we can therefore derive the following:
E 12.1 |softmax (2W2) 2 EVI2, S 2B E ||12,11-
(26)
Subsequently, for E1, we establish the following expression
E 2.1
|softmax
(ZW2) 2
softmax (WWZ Z (EW2 1
2,1
soltmax (TW2T)
softmax (ZWZ
)le, |ZV ||,,00
softmax (ZWZT)
softmax ZWWz )
To advance the analysis, we introduce the d-scaled perturbation E' 6 Z, where is constrained within 0 < 8 < 1. Our approach involves first bounding the derivative as d + 0, and then integrating this bound along the path of E, effectively covering the interval from 5 = 0 to 1. Notably, 5 + 0, the quadratic terms proportional to 5² E diminish, simplifying the analysis at this limit.
|softmaxx (2'W2 2' 2TT) softmax (ZW2) Z )l
softmaxx Z'WZ softmax ZW Z )le,
) + ||softmax (ZWZ -softmax (ZWZ) ZT) Olle,
To bound the latter, we focus on each row separately. Consider a row from and its perturbed version Z + E', represented by the pair (z, + e²). It follows that for any cross product, Bw and |z Wzi Bw- Additionally, the bounds S W Z|1,, S Bwn ||='ll, and || WET I,, - Bw E I2.1 hold. Applying the perturbation
have the guarantees (z + +')² W z,
bounds provided by Lemma TO obtain the desired result
softmax
(= e') W
softmax (z Ww
< 2B |
softmax (z T W (Z + EY) softmax (z Wz
< Bwe 2Bw
E
E'112,1 /n.
Summing across all rows, obtain the following:
lim softmax ((Z + 8E) WZ softmax (ZWZ Ile, 18 BBwe
El
E
By integrating the derivative over the interval 5 to 8 1, we obtain the final expression,
|ssftmax (ŽWZ) Z softmax LZW Z ||e,
2.wee22w El
(27)
By substituting inequality 27 and inequality 26 into the decomposition we derive the following result:
Á
A 2.1 Bl1,,1 < (2Bw + B)e E
(28)
To continue, we aim to control the output for a specific index j where the input perturbation remains small, specifically ||e;|le, n
|| To address this, will apply the same argument, focusing
24

---

Published as a conference paper at ICLR 2025
on the j-th token. For the j-th token (omitting subseripts for clarity), let the inputs be denoted as 2, E,E = 2 - 2, and the corresponding outputs as a, a, É = a a. Similar to the previous decomposition, we can derive the following:
= V z
[softmax (ZW') T Z)
softmax (ZW 2)] V E' softmax (ZW T 2) (29)
By leveraging the fact that 2 W bound similar that in equation 26
Bw for all i, j, and applying Lemma 10 we can establish
Specifically, we can constrain the terms involved as follows:
e2Bw
E < E
softmax (ZWi) 2) 3)1le,
E 2,1
(30)
Similarly, for E,, we can derive the following:
| le, S [softmax (2WTz) 2) - softmax (ZW 2)]1|a
T
Z 20 ||softmax (ZW') TWT: =) softmax (ZW =)]le,
VA
softmax (ZWTa, 2) softmax (ZW 2) z)lle-
Now, considering the perturbation E' EE and letting 6 + O, we apply the triangle inequality to obtain the following result:
limo 6-00
-1
/sotmaxx ((Z + 6 W (z + +E)) softmax (ZWT x) =))
lim 8-1 6-0
softmax (lZ + 6E W 2) softmax [ZW 2)11e3
+ +
|sottmax (ZW (z + sE)) - softmax (ZWTs 22) lll,,
< <Bwe" 2Bw
|E||2,1/
+ 2B ||El|e
<2Bwe 2Bw
E 1/
(31)
In similar manner to the previous steps, can derive the following:
| (2B 2Bw |2.11
(32)
n
Next, we examine the effect of the MLP layer on the model's behavior. Let (M.) Rdxd represent the weights of the parallel MLPs that follow the self-attention mechanism. Given that [|M;] < 1, we denote the MLP outputs corresponding to the self-attention results and À as U and U. respectively. From this, we can derive the following relationship.
Let & denote the ReLU function, which is I-Lipschitz continuous activation function with (0)) First, observe that each row of U is given by u, á (M,,,), where M; E d represents the weights of the MLPs. Given the properties the ReLU function, we can derive the following bound:
|uillez < | (M.a:))
||Maaille, < ||aille, S
Next, we consider the difference between the perturbed and original outputs. We can express the difference as llu Tille, < l (M,a;) - ó (M,ā.)le which, due to the I-Lipschitz property of 6, is further bounded by M (a, ā. 1le,, lll,, S ||: a. azlle, Finally, we obtain:
Lu. u. Elle < aa -aill,
(33)
Thus, we conclude that the perturbations in the perturbations in A. Consequently, we establish the bound
of U controlled by the corresponding
| 0 2 - ||A Al12,1
Thus, from inequality 28 we derive the following result:
I Ū 12.1 < (2Bw 1) 2Bw E 2,.1
(34)
Furthermore, for any i E [n] such that ||E;lle, <
|| it holds that
1
l|u uille, _ n
(2Bw + 2B E 2,1
25

---

Published as a conference paper at ICLR 2025
where u, represents the i-th row of U. With this, have addressed the stability of the single-layer transformer. Moving forward, we will extend our analysis and focus on the stability of L-layer transformer. First,
derive the following:
|| [a, (s) 22,1 2 (1 + 2Bw)e 2B
[z -Z(k 12
where S h I L represents the number of layers in the transformer. Then, for L--ayer transformer, we have the following:
(1) ll.1 S ((1 + 2B Be y (() 2
(6))
What remains to perform induction on the difference between the last tokens zn Zn that, for all layers,
We claim
S - (11 ([1122B)e Z| (())) 11a.
This claim holds at i = 0 because the change in the last token is at most induction, the claim holds for all layers, and we conclude the proof by setting I., covering the entire depth of the I--layer transformer. Finally, we obtain:
Zío (0) 10.11,1, /n. By
| -1000)
1 (11 2B )e BB Z Z(u) 11..1
(35)
Next, we further analyze the self-consuming process. Let Sp = 120,1,-.. ..... Z0,j- =0.n] and S = [zo, "......, z0,nl represent two initial real datasets that differ only in their inputs, specifically
20,j = (zo.j: yo.)) and zú.j (zb..; Ho,), where I n. Since ||So - S,ll2,1 S 2, then, have the following:
TF (So) - TF (S))le, S 2n (11+2Bg )eaB 1|00 S,l12,1
(36)
((1 + 2B )c )e2Bw)&
$ 2n +1
Then, Sn and S used in-context examples, and i..d queries (01 are sampled from X. These queries, along with the in-context examples So and Sy are processed through the transformer model to predict their respective labels. As a result, the first generation of synthetic datasets, Si = [21.1,... Zi 21,m]] and Sf = [zi. zi,,,,,,, IT produced. Then we obtain:
|| 11-211112.1 2n +1 ((1 2Bw (e 2Br
2n
(37)
Given the mixed dataset Sy. where 3, a:S0 (1 83 for 1 S < i, we can proceed with further analysis based on the specified combination of the original dataset So and the synthetic dataset Sy.
| - 31 S allSo -1l12,1 + (1 [a]]]]]s S
2n
< 2a + (1 a)) 2n ((1 + )B
(38)
By reintroducing the mixed datasets 3, and 5 as in-context examples into the transformer model, and considering the query { as i.i.d. samples from the distribution X, we can derive the transformer's 's output according to Equation 36
TF (.
-TF (3) |
1
((1 + 2Bw 2Bw L $115; - -51112.
2n +
1
((1 + 2Bw 2B 1 ( 22 + (1 a)) +
2n
((1 ((1+2B (B )eBw)
< 2n +1
1
2n
((1 (1+ 2B )e 2B
a
2
((1 + 2Bw) eBw
(39)
(2n 1)²
2n
+
26

---

Published as a conference paper at ICLR 2025
From the above expression, further derive that
|| S2112,1 S - ())
(2n 1)2 ((1 + 2B 2e
2n
)Br 22
+
2n 2n +
((1 + )
Thus.
|| -2112.11
$ <al|So - Soll2,1 + a)lls - - -5il12,1
< 2a + (1 ar)
(2n + 1)2 ((1 + 2Bw 2Bw) (BB ) 26 + +a(1 a))
2n²
2n
((1 + 2Bw) e 2Bw )²
2n +
Similarly, for the 2-th generation, following analogous steps, we can derive that
TF $2 TF
$ 2n +
1
((1+ (L1+BBw) 2Bw
BB) 1. "133 - 5 2,1,,,1
< (1 1-aa a)²
(2n + 1)3 ((1 + 2Bw) B S2 + a(1 - a)
2n²
2n
((1 + 2Bw) e 22Br)
22
(2n 1)2
2
+ a
2n +1 (1++ + (1+2BB) c2Br)
(40)
Building on the above expression, we can further deduce that
15g S/l12,1 2,1
$ (1 - a²²
2n
((1 + 2Bw) BBw) 3L + - a)
2n
((1 + 2Bw) e
e2Bw) 2
(2n + 1)3
(2n + 1)2
+ 2n +1
2n
((1 + (L++BBw) ) B2Br)
(41)
The discrepancy between the mixed datasets is follows:
$ Sa Sol1,,1 + (1-a)) (1--a))15 - 5/112,,1
$ (1 a)3
2n
((1+ - ((1+)Bw) 2Bw )ar. 3r. + a(1 a)²
2n²
((1 + 2Bw) eBrr) 22
(2n + 1)3
(2n + 1)3
a(1 a) 2n ((1 + 2Bw) (Bw) 2aa
2n
(42)
Utilizing recursive techniques, obtain the following:
||5 3112,,1
$ (1 a)'s
2n
((1 + 2B2 2Bw )B) + a(1 a) 2n²-1
(2n + 1)i-T ((++BBw) eBBr (1-1))
(2n + 1)5
+ + a(l - a)²
(2n + ln2 ((1 + 2Bw) e 2Bw) 21 + aa(1 - a)
2n²
2n
((1 2Bw) eBw
2n +1
2a
< 2(1 - aa)
(2n + 1) ((1 + 2Bw) e 2Bw Sil
n
2a
(1 a)
(11 + (1+2BB)) 2BB cBBW) c2Br)
(1 a) n²
+1) ((1 (11+0 2Bw 2Bw
2n + 1
(2n + 1)2
(43)
27

---

Published as a conference paper at ICLR 2025
Ultimately, the discrepancy between the transformer outputs after generations of the self-consuming loop for So and Sy can be obtained as follows:
TF 5. -TF ( )1
((1 2Bw (ewre22 2er ||S: -5,12,1
2n +
$ (1 a)²
2n'
((1 + 2Bw) B2 (1+ + + aa)1-))
2n 2n'-1
(L++2 e ) ):r
(2n + 1)
(2n 1)²
+ + a(1 - a)²
2n
) ((1 2Bw) ) 36
+ +a(1 a)
2n
22 ((1 + 2Bw) c2Bw )ar
(2n + 1)3
(2n 1)²
1
((1 + BB ) 2B 7
2. 2n +
22(1 -a)
(2n + 1)4+1 11441 ((1 + 2Bw) e (1+1)L +2a
n²
((1 + 2B )e 1
2n 1
1 - (1 a) 2n +1 ((1 + 2B 2B L
(2n 1)J ((1 + 2Bw) e BB ) iz
n²
x
1 1- 1- a))
Subsequently, given that Bw = (1 (B+2 )e2 we define the measure d as the 2n-mo to quantify the output discrepancy of the generative transformer model after iterations of the self-consuming loop, starting from the initial real datasets Sn and S. In this context, the recursive stability parameter Th as described in Definition 2 can be bounded by the following expression, providing a formal measure of the model's stability across iterations:
B ((++) W
TF(S.) TF(5)) $1 (1-aa) a))
2n +
The proof is complete.
A.8 PROOF OF THEOREM 3
In this section, building on the general theoretical framework established in Theorem 1 we provide the proof of Theorem 3 by analyzing the terms 3, and (ry(n), leveraging recent advancements
in SGD (Zhang et al. 2022) and ICL (Zhang et al. 2023). The recursive stability parameter 7 is
derived from Theorem 2
Lemma 11. (Uniform stability of SGD in the non-convex case Zhang et al. 2022)). Assume is smsoohth and p-Lipschitz. Rumning T ž n iterations of SGD with step size
Choose the
stability of SGD satisfies
16p² log
Bn
TI
Lemma 12. (Zhang et al.) 2023) Let Fa represent the probability distribution induced by the transformer with parameter O. Additionally, the model TP pretrained by the algorithm:
0 argginn 9EB
log Po (2/211 ST))
72
where = 1.81,991.... .1.......... It, V.). Furthermore, we consider the realizable setting, where ground
truth probability distribution IP(- S) and Po- (-1 S) are consistent for some 0" O. Then, with probability at least - 8, the following inequality holds:
TV S), S),Pa(11)) S 1/22 log(1 + n) + n1/4 log(1/8).
(44)
where E denotes that omit constants that independent of n and 8.
28

---

Published as a conference paper at ICLR 2025
First, note that in the setting where the transformer generates data through in-context learning, the generalization error of the self-consuming loop is given by:
Proof of Theorem 3
Ro. (A(S.) - R2g (A(S:)) = (2- 2)
Σ ((A(S)), 2i) 2;ES:
(45)
Now, we are ready to prove Theorem 3 The main idea is to bound the uniform stability parameter 3n. the recursive stability parameter Tiu and the learnability of the generative model through the total variation distance (nv(n) as stated in Theorem [ First, as for the bound for the total variation distance drv (n) in Theorem [ For Equation 8 in the proof of Theorem I we can rewrite it in the setting of in-context learning as follows:
RD,,,(A(5 RD. (A((.))
E ((A)) x) (E =)|
TE (ENTRIBLL) 2) - E ())(() 2)
/ ((A(5)). (P (B(x1
(3__)) (=x15__))
Po ( (3...) dz
((A(()) 2) (P(x1 (1)1)1)) Pa (z 13,-1) dz
S M / PP (z (x1 (x13:-1) Po (x|,,_)) dz
2MTV (P(- (;-1), P. 33-1))
(46)
Where, the second equality holds because, the (i- 1)-th generation of the self- coff-mnnuuing loop, the mixed data distribution from the (i 1)-th generation is reintroduced as the ground truth distribution to train the transformer. a result, the transformer outputs the synthetic data distribution for the
i-th generation. Thus, TV (P(- 5y), 83)) corresponds drv (n) in Theorem
Finally, the
bound for the total variation distance drv (n) follows from Lemma 12
drv (n) S 1/2 log(1 n) 1/4 log(1 log(1/0).
(47)
Similarly, for the recursive stability parameter in the self-consuming loop, we rederive Equation 21 Junder the in-context learning setting:
from the proof of Theorem I
[Ez,, ~D, S. D (S6..) EENDD e (A ((S (U..-a)))) :2) - (A PUS. UUS,11a)) 20,5] .2,))
E
-Ez6, ND, s.,
(1)) [E (A 1a)) (A((())) ". USS =00))
||E2i, MD ESND Do E
[ S -p A (S) " P US ,z)
E -s ad () (A (SS US .2)
BeG ~D BEND [B
SA (A (St.a)) U U.S z0.)
- o) (1().)))) (A (S).aa U. =20.))
< 2n(1 - 3) TF ((tba) US - -TF (St ) S - 1
1 - ((1 (1--0)
a) Bu ji-1
22n(1 - a) 2n +1
((1 (1-
1 -
a B/v
= 2n - a333,1
(48)
29

---

Published as a conference paper at ICLR 2025
For the uniform stability parameter Bn of SGD algorithm, can derive the bound from Lemma 11 Substituting above results into Theorem we obtain the following conclusion:
RD. (A(5.)) R5,( (A((:))
((1 ((1-a)3, a)) + a (3n + a1-a) log(na)) log(z)
+ (V(1 -nnnnn + Mn 1 1/2 (2 + vai) / /1os(t + 2M (1 - a)) a dTv(())
+ log(na) log( () +a/11- (1-a)n 1
SBn (1 a) log(n(1 a)) log( 8
+ 1/2 PMMM(-aa vayy log 1 5) - 2ddv(n) (1 - (1-(1- -a)") aa)')aa a-
+ + 1-1 a(1 - =app² log log(
8
1
Sn 1/2 1/-1/3 log(n)))(
log
+ ((1 -a) lo (n) (n) log
+n-1/4 M (1- (1- a))) log(
1 8
(49)
A.9 PROOF OF THEOREM 4
In this section, we prove Theorem The proof follows similar approach to that of Theorem 3
however, it is more intricate due to the fact that the mixed dataset in Theorem 4 contains synthetic data from all previous generations. Each generation's 's synthetic dataset depends on the synthetic datasets of previous generations, leading to a more complex non-i.idd setting. Similar to Theorem 3
we begin by decomposing the generalization error into two components: the Cumulative Distribution Shift Across Generations and the Generulization Error on Mixed Distributions.
The main proof is follows:
Proof of Theorem 4 We begin by decomposing the generalization error as follows:
RD. (A(S.)) - Rg. (A(S.)) s
RD, (A(5.)) Rs, (A($.))
Rs. (A(5.) Rg. (A(5.))
Cumulative distribution shift across generations
Generalization mixed distribations
Upper Bounding Cumulative Distribution Shift Term
1 For the term RD (A(3.)) R2, (A(5:)) we first note that D; = -4IA
Do
A Dj I+IA
+ 2 D2 -
141x
I+X D. Therefore, we obtain:
RD. (A(S.)) R,, (A(3.))
RDo (A(5)) 1 i RD. (A(S.) 1 il RD. (A(3)))-... -.. 1 ++ Rv. (A(3.))
1
A
it
RDD (A((((()) (A()))
A
RD. (A($1))
X
RD. RD. (A(5)))
1+ it
I+ it
I+
X
RDa (A(S.) - RDD (A($:)) +...
(A(S.) - RD, RD, (A(3)))
1+ ix
1+
< ix
(A(3)) - Ro, (A(5.))
(50)
Furthermore, we can further decompose it as follows:
RD. (A(S)) RD, (A(3)) s RD. (A(S.) RD, (A(i))) (B,,((((5))) -RD, (A(S)))
(51)
30

---

Published as a conference paper at ICLR 2025
By substituting inequality 51 into inequality 50 we obtain:
(A(3)( - R5, (A(5.))
$ 1+ix ( (A(.() 1 + (B),,(((S)) - RD, (A(3:))).
(52)
Thus, from equation 46 in the proof of Theorem and lemma 12 obtain:
(((,,))()))) - RD, (A(5:))
< 2MTV (r (-15)-1) ,Ps (- (-15)-1))
x M
1/4 log æiーi log(1/6).
(53)
Incorporating inequality 53 into inequality 52 arrive at:
(A(S.)) R.g (A(S:))
$
A
Ro, (A(3.)) - RD,., (A(3.)) Σ
A
Mn; 1/4
log nj log(1/8). (54)
1 it j=1
1+ iA
Let ((1) = ΣØ= Mnj 1/4 ny
my log(1 1/). Then, we obtain: RR,, (A((,) - Rs, (A((:)))
I+iX RDD (A(5.) - Ro... (A(5:)) +...+ +
A
A
A RD. (A(S.)) - R5, (A(3.))
I+ 1 f(1)
I+ id
Similarly, we get:
RRD (A(S.)) RD_, (A(3.))
RDo (A(5.)) - RB_., (A(3:)) +...+
RD. (A(3.) - Rs, (A(5.))
1+ (1 - 1)A
1+ (i - 1}
A
1+ (i 1LX Sli S(i 1).
Then, we have
A
A A
(A($.) - Rs, (A(S))) & 1+ i f(() f(0) + 1+ (i 1)
f(i - 1)+ -1}+
A
I+ + 1+ 1)X )( RD. (A(3.)) - Rs.,, (A(3.)) +... + (A(3.) - R3, (A(3.)) )
(55)
Thus, by applying recursive techniques, we obtain the following result:
RD, (A(5.)) - Ro. (A(5)))
A
A
1
((i + 1+ i 1 (i 1)X
f(i 1)
00
(1 (1+ i2j2 1)5(i- 2)
i + (i 2)
A A
++11+1 + +0
(1()
(1 + iA)
X
f(i) +
f(i - +
X
f [ (1-2) 2) + +
(1-2)
i i+ =x)
A
i
1 (i - 1}
1 (i 2)
1 X
n4 1 log (n,-1) 1) + (1
$ M log
) log (1:
8 51 + it
1 (i - 1)}
X
I (1+
1+ (i - 2) n log(n, a) + ... + (1 + +
nn log(no))
(i - 1}λ
1+X
< n 1 log((1 + M log 1
(56)
31

---

Published as a conference paper at ICLR 2025
Upper Bounding Generalization Error on Mixed Distributions Term
Next, turn attention the term RR (A(5.)) R3, (A(S.))l. Our primary objective is to establish moment bound for this expression.
R5, (A(5.) - Rgg (A(5.))
1
X
A
A
RD (A(S.) + I+ iA
RD. (A(Si) + 1+ RD. (A(S.))
RD, (A(S.)
++ii i
1+ is 1
Σ ((A(S)). 2.)
Σ ((A(S)) 2n)
Σ
((A(5)).
(1+i
(1+m
+ idjn
2:E50
2;E81
2,E8,
Ro, (A(5:))
1
Σ ((A(5)), zi)
+
RD, (A(5.))
Σ ((A(S)),
1 1+i1
(1 + iAjm
1 id
(1+
z.ESo
z.ES,
Term
Term
X RD,(A(())
((AS)) 21)
(57)
iA
(1+
2.ES.
Term
Fixing So, S .I..., S the data in S; are independent. Following a similar approach to the proof of Theorem we utilize this property along with Lemma 8 to bound Term Consequently, from Equation 15 in the proof of Theorem [
we obtain:
1 ++ Ro. (A(5.)) - (1+ id)n
A
Σ E(A(S)), zi) E,
S PP iA ((+t2))n log(An)
+ 1 + iA
M
pd
n
(58)
Next, we consider Term Similar Proof of Theorem 3] we first introduce set of functions and apply Lemma 8 to bound Term O. Specifically, we define (S),, which serves a similar role to the gi 's in Lemma follows:
(So)
-Eb,,, LEWDO!
.2) (
.200)] (59)
where Zo.j denote the j-th data point in So, and S/ represent the dataset obtained by replacing 20.j
with zó... Moreover, following the procedure above, observe that hyy S M and EE h S
More intricately, we will now prove that h, exhibits a bounded difference. However, it is important to note that Si...., S₁ all depend on Sq, so when a single data point in So is changed, the corresponding datasets will also change. We denote these modified datasets as .i...., S and consequently, we
have the following:
(Sp) hy(SS]
= -IBzg ~D EZNDO (A (A(S8US) u.. ,2) - (A
=203]] U... US) 20.3)
-Eeby EENDD (A ((S u U...U U......)) -() (AA
S 288 (1S]) (1) + ||S| Sill, +... + ||S, [|s.-)(leea)
(60)
Thus, by applying the recursive stability established in Theorem 2 it is important to first note that in Theorem 2 the mixed dataset is defined as 5, aSo + (1 a).S whereas in this theorem, the mixed dataset is defined as S. Σラ Sy. Therefore, by following the proof steps outlined in Theorem 2 we can derive the following:
(hy(So) (S)) S (0(1)))) (11Bw)
32

---

Published as a conference paper at ICLR 2025
Thus, apply lemma
Σ hy (So) ==1
- 222
i!BW log(n) + 4M /pn
p
S p
log(n(1 + iA)) M Vpn
1 ++ it
We observe that the difference between Term 0 and can bound Term 0 follows:
1 (I+iA)m
Σラ= h1y(S) is negligible. Thus,
1
RDD (A(S:))
Σ ((A(S:): zi)
1+ iA
(1 + idjn
2,E80
S P (1 + 1
i.BW log(n(1
p/n.
(61)
1+ it
Using the same method, for Term where 1 i Si - 1, we can derive the following:
1 1+ RD, (A(S:))
Σ
((A(S)); 2i)
(1 id)m
#:ES1
(1 (1+iA)2n (1lB (og(n(1+1M)) iA)) + i
M Mp/n.
(62)
In summary, we can finally bound the Generalization Error on the Mixed Distributions term as follows:
Rg. (A(S.) - Rg. (A(5))
- P (1 + iA)nn log((1 i
0²
II)
Mi P
1 1 iA V n
Then, according Lemma 9 obtain, with probability at least - 6:
Rs. (A($.)) - Rg. (A(3.))
(1 iA log((1 + 1A) 1- 1)2
p²
log
Mi
log 8
+ iA
Then, combine the above inequality with inequality 56 we obtain:
RD, (A(3.)) Rgg (A(3.))
log((1 ii))) log j
1
0²
log((1 + AA)))
(+1)Z
log 1 it
Mi
log
(1 in)an
S n 1 1+iX
Mi 1 log +n
(1 i2j2 log((1 + iA)m)a!
2²
(1+1)4 log
log((1 + iA)n)Mlog
8
The proof is complete.
B EXPERIMENTS
In this section, we present some experimental results. Specifically, we trained transformer models to in-context learn linear functions within STLs.
33

---

Published as a conference paper at ICLR 2025
In these experiments, considered the class of linear functions:
F F= f(x) w x, w IR.
in d = 5 dimensions. We sampled ....... gg qquury and w independently from the isotropic Gaussian distribution (000, La). For each 2;, computed y/ w and constructed the prompt as:
P (I1, V1, 22, U2.
U2...... ................................ Lk yk Equery). )
We employed a 12-layer, 8-head GPT-2 model with hidden size of 256, trained I& linear regression task with 40 in-context examples. Two cases were considered:
Mixed Case: Fresh data and generated data were mixed a 0.5 ratio.
Full Synthetic Case: No fresh data was used.
The results of these experiments summarized below:
Loop
2
3
4
5
Full Synthetic
0.3817
1.4975
1.5396
2.0836
2.3912
2.8764
Mixed
0.3817
0.4208
0.4391
0.4503
0.4641
0.4702
As observed, the error accumulates progressively with more self-consuming loops, particularly in the full synthetic case, where the eтrоr grows rapidly. In contrast, maintaining a constant-sized proportion of real data effectively reduces the loss, which is consistent with our theoretical findings.
34


Reviews:
[{"review_id": "CcxId4shV5", "reviewer": "Reviewer_CLcc", "summary": "This paper theoretically studies Self-consuming Training Loops (STLs), where generative models generate their own training data. The key concept that the paper introduces is the recursive stability which the authors prove to be vital for generalization and for avoiding model collapse. After the general result on how the recursive stability is important, the authors move to provide an upper bound for the recursive ability for transformers. The theoretical results suggest a trade-off of synthetic data augmentation.", "questions": "see the questions in the above section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper introduces a concept called recursive stability that addresses the complexity and difficulty of self-consuming training loops. It can be used to derive an upper bound on the generalization error of learning algorithms trained on the self generated (mixture of self-generated and real) dataset after any iterations of the self-consuming loop.\n\n* The application to transformers is nice to have as it bridges to the most popular architecture in practice.", "weaknesses": "*  The reviewer is concerned about the validity of this recursive stability which is supposed to be the core contribution of this paper. Details follow. The recursive stability $\\gamma^i_n$ is a rather strong assumption, and it hides a lot of things underneath. For example, it is uniform not only in terms of the data point modification, but also uniform w.r.t. to any randomness in the STL process. It also depends on how the STL is performed, e.g., the ratio $\\alpha$ of the data mixture. The paper derives an upper-bound on the  recursive stability $\\gamma^i_n$ for transformer in Theorem 2 but the bound has an exponential factor $e^{B_WL}$ where $B_W$ is the bound on the norm of weights and $L$ is the depth. So it appears that $\\alpha$ is required to be very close to $1$, i.e., almost only using real data, for the guarantee on model collapse to stand.\n* Following up on the above concern, it is not really satisfying to see the paper jumps directly to the application to transformers in-context learning which is notoriously difficult to make the theory accurate. Why not apply the recursive stability and Theorem 1 to simple cases first, e.g., Gaussian models, where everything can be made accurate, to examine if the proposed recursive is indeed a valid property to look at? This means to examine the LHS and RHS of Theorem 1 accurately for simple cases and show that the bound is not vacuous.\n\n----------------after rebuttal--------------------\n\nThe authors have well addressed my major concerns, so the score is raised.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOMqCGz6Yv", "reviewer": "Reviewer_AyDT", "summary": "This paper theoretically examines the properties of self-training loops in training generative models. This is an area fo great interest and relatively limited theoretical analysis in current research. This paper aims to prove generalization bounds for self-consuming training loops. As they note, this can be difficult due to the distribution shift across training iterations, as well as the non-i.i.d nature of synthetic-real mixed training datasets. To address the challenge, this paper proposes a novel theoretical notion of recursive algorithmic stability, which allows for controlling the propagation of error across training generations. Leveraging this new theoretical concept, this paper proves a generalization bound. They show that this generalization bound resembles existing qualitative and empirical trends observed in the use of synthetic data. For example, the necessary role of including an appropriate proportion of real data. In addition to their general, model-independent results, this paper also studies the relevant problem of in-context learning in transformers. They provide a bound of the recursive stability of transformers on this task and demonstrate that a constant fraction of real data is sufficient to maintain stability. They additionally remark on the conflicting role played by self-consuming data: it increases the distribution shift in the training loop while decreasing the generalization error component of each step.", "questions": "1. Perhaps I missed something but what is the notation $\\sum\\limits_{z_{i} \\in S_{i}, \\alpha}$ referring to in line 358?\n2. To clarify, in the in-context learning setup, the \"synthetic data\" inputs are not generated by a model but sampled according to the ground-truth distribution correct? \n3. Follow-up from (2), I wonder if you could comment on the fact that in many \"real-world\" synthetic data settings both the \"input\" and \"output\" components are sampled from the model. For example, when generating instruction/chat tuning datasets, I believe that both instructions and responses are synthetically generated often. How do you believe that would change your analysis? Seemingly, it would make recursive stability worse due to the additional error in the input distribution.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, I am very supportive of this paper. Self-consuming data loops are currently an important topic in practical foundation model settings --both intentionally from the point of view of synthetic data and unintentionally from the point of view of leakage of model-generated content on the internet. While some existing works have theoretically demonstrated that self-generated data can lead to model collapse, as a practical matter self-generated data can often be leveraged in harmless or beneficial ways -- albeit without theoretical justification. This paper thus makes a significant contribution to placing the role of model-generated data on a more solid theoretical footing. They convincingly isolate some central technical challenges in understanding self-consuming training loops and propose an elegant and intuitive concept of recursive algorithmic stability to confront these challenges. This enables them to make a *key contribution* of presenting the **first** generalization bound on self-consuming training loops. The analysis of the transformer architecture is additionally interesting and instructive, establishing that their proposed recursive stability criterion holds naturally on realistic architectures and tasks. Throughout the paper, they demonstrate that their theory formalizes and quantifies many previous intuitions or folklore on properties of self-consuming training loops (such as the necessity of maintaining a proportion of real data -- or the benefits of synthetic data primarily arising in low real-data regimes). \n\nThe paper is extremely clearly written. The proof sketch was insightful, yet accessible. Moreover, the authors interleave their technical and theoretical content with real-world motivation and implications. I found the paper quite enjoyable and educational to read.", "weaknesses": "I don't have any serious concerns about this paper. One potential weakness is that the paper primarily focuses on the theoretical angle and  mostly relates their findings to empirical observations in prior work. It could also be interesting if the authors were to add some of their own numerical simulations justifying their findings. Although it is understandable to exclude this, it would be interesting to have some numerical backing in the paper itself. The construction of a simulated setting or dataset would also help in understanding the relationships between the setting studied in this work and real-world setups.\n\nThe authors primarily use their theory to explain or justify existing empirical or conceptual observations about self-consuming training loops. However, I feel it would be quite exciting if authors could propose and test (even in simulated settings) some novel predictions made by their theory. Perhaps, this might also be a question of merely highlighting the currently novel predictions made by theory a bit more explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmlM9NNIM5", "reviewer": "Reviewer_navH", "summary": "The authors attempt to present a theoretical framework to describe the generalization error of models when they are trained on synthetic data. They are motivated by the observation that multiple workflows presented in the literature on model collapse / self-consuming loops arrive at different conclusions. They introduce the notion of recursive stability, inspired by past notions of algorithmic stability like uniform stability, using which they quantify the propagation of error as model-fitting iterations progress. Finally, using these tools, they perform an analysis of in-context learning in transformers, obtaining generalization error estimates that describe the impact of real data in the model-fitting loop.", "questions": "1. In line 195, does $S_0'$ depend on $j$ or not? (the previous sentence suggests it does) If it does, then the provided formula doesn't make sense.\n\n2. With regards to equation 1, let's consider $\\alpha=0$ i.e. no real data is used. What is the behavior of the error gap? It seems to me that only the last term will matter and using the fact that $\\lim_{\\alpha\\to0} (1 - (1-\\alpha)^i)/\\alpha = i$, this predicts a linear rate of increase of the generalization gap. Is this correct?\n\n3. Following the point I make in 2 above, it seems to me that the discussion in lines 257-260 needs to change, because when $\\alpha$ is too small, yes $\\alpha^{-1}$ gets large but it's multiplied by $1-(1-\\alpha)^i$ which is small too. They balance each other out exactly giving $i$.\n\n4. I don't understand the final conclusion of Theorem 4. There are too many terms - which one matters the most? Gerstgrasser et. al. (2024), Marchi et. al. (2024) and Seddik et. al. (2024) have shown that the gap is like $\\sum_i 1/i^2$, which is kind of similar to the second term in the upper bound, but it has $n^{-1}$ in front which makes me think that this term is a higher order term. The presented bound makes me think that only the first term $O(n^{-1/4})$ matters, but then the rate $\\log(i)$ (essentially) seemingly contradicts past findings. I would urge the authors to actually connect this part with the more recent literature on accumulating data for avoiding model collapse.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Through their notion of recursive stability, the authors have considered a much more general class of algorithms / models than what the prior literature has described. They have been able to go beyond explicit model calculations or estimation strategies. This notion allows them to decompose the generalization error and explicitly attribute each term to a specific statistical reason. Interestingly, the authors are also able to identify, through these upper bounds on the generalization gap, the effect of proportion of real data in the STL.", "weaknesses": "The authors have tried to be too general - this has significantly affected the readability of the paper. There are no examples of models provided. What models are recursively stable? How do we know the TV bounds will be of the same order? The paper really needs some examples for clarity. As of now, it is too hard to parse. Understandably it is a math paper, but a lot of what is written needs to be appropriately motivated as the premise addressed by the authors is an important empirical problem (the problem of model collapse) which the authors are tackling theoretically. I would recommend the authors to rethink their presentation for greater impact.\n\nFurther, to me the bounds derived are a rather straightforward application of \"working with upper bounds\" when the correct assumptions are in place. Thus, I feel the technical novelty is limited, and I am not sure if we are learning something substantially new from these upper bounds, than what we already knew about STLs.\n\nFinally, the literature review is a little too crude. The authors should connect their results with those obtained by Gerstgrasser et. al. (2024), Dohmatob et. al. (2024), Alemohammad et. al. (2023), etc. because we know how specific workflows function already, and it is important to clarify that the present authors are not contradicting any existing result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)