When Models Don’t Collapse: On the Consistency of Iterative MLE
Authors: Daniel Barzilai
Ohad Shamir
Venue: NeurIPS 2025 poster ()
Keywords: Model Collapse
Generative Models
Maximum Likelihood
Generalization
Primary area: theory

Abstract: The widespread use of generative models has created a feedback loop in which each generation of models is trained on data partially produced by its predecessors. This process has raised concerns about model collapse: A critical degradation in performance caused by repeated training on synthetic data. However, different analyses in the literature have reached different conclusions as to the severity of model collapse. As such, it remains unclear how concerning this phenomenon is, and under which assumptions it can be avoided. To address this, we theoretically study model collapse for maximum likelihood estimation (MLE), in a natural setting where synthetic data is gradually added to the original training set. Under standard assumptions (similar to those long used for proving asymptotic consistency and normality of MLE), we establish non-asymptotic bounds showing that collapse can be avoided even as the fraction of real data vanishes. On the other hand, we prove that some assumptions (beyond MLE consistency) are indeed necessary: Without them, model collapse can occur arbitrarily quickly, even when the original data is still present in the training set. To the best of our knowledge, these are the first rigorous examples of iterative generative modeling with accumulating data that rapidly leads to model collapse.

When Models Don't Collapse: On the Consistency of Iterative MLE
Daniel Barzilai
Ohad Shamir
Weizmann Institute of Science daniel. barzilai @we zmann. ac i1
Weizmann Institute of Science ohad. shamir@we zmann. ac
Abstract
The widespread use of generative models has created feedback loop, in which each version of model is trained on data partially produced by its predecessors. This process has raised concerns about model collapse: A critical degradation in performance caused by repeated training on synthetic data. However, different analyses in the literature have reached different conclusions as to the severity model collapse. As such, it remains unclear how concerning this phenomenon is, and under which assumptions it can be avoided. To address this, we theoretically study model collapse for maximum likelihood estimation (MLE), in natural setting where synthetic data is gradually added to the original data set. Under standard assumptions (similar to those long used for proving asymptotic consistency and normality of MLE), we establish non -on-asymptotic bounds showing that collapse can be avoided that some assumptions (beyond MLE consistency) are indeed necessary: Without them, model collapse occur arbitrarily quickly, still present in the training set. To the best of our knowledge, these are the first rigorous examples of iterative generative modeling with accumulating data that rapidly leads to model collapse.
the fraction of real data vanishes. On the other hand, we prove
when the original data is
Introduction
Generative models such large language models (LLMs) and diffusion models are increasingly filling the internet with synthetic data. At the same time, web-scraped content remains a common source for training newer models. This creates a feedback loop in which each version of model is trained partly on outputs of all past models, and synthetic data gradually dominates future training sets. Because identifying and filtering artificially generated content scale is not always feasible, existing biases and artifacts in the models can become increasingly embedded in the training data and amplified from model to model. This phenomenon has recently been termed Model Collapse: A critical degradation in performance caused by repeated training on synthetic data [Shumailov et al. 2024 Bertrand al. 2024 Dohmatob et al., 2024a Gerstgrasser al. 2025
The effects of training on synthetic data appear to vary widely across settings. In some cases, even
small amount of synthetic data has been shown to significantly degrade performance Dohmatob
ct al.
2025] in others, model collapse can be avoided altogether [Gerstgrasser et al. 2025 Dohmatob
al. 2024a| synthetic data may even be beneficial [Jain et al. 2024 Dohmatob et al. 2024b).
To help clarify this picture, we theoretically study the behavior of maximum likelihood estimation (MLE) under iterative training with synthetic data. We focus on natural and practically motivated setting (also studied by Alemohammad et al. [2024), (ersttrasssr et al.) (2025). Dey and Donoho
2024 ), where we initially have samples from some ground-truth distribution, and at each iteration
T. the latest model generates n new samples that are then accumulated with all previous data and
used to train the next model.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recently, Dey and Donoho [2024] analyzed a similar setting specifically for distributions arising from exponential families, and showed that for any fixed T and in the limit as n + oo, the error of the iteration-T model is degraded by at most universal multiplicative constant compared to the initial estimator trained solely on real data. However, their guarantees are asymptotic in while T is fixed. Therefore, they do not quantify how the performance depends on the proportion of synthetic versus real data, since the fraction of training data that is real is a constant (given by 1/T) while n grows to infinity. Therefore, it is difficult to deduce from their results how concerning model collapse may be as the fraction of real data decreases.
In contrast, in this work, we prove in Theorem 4.1 non-asynnan asymptotic guarantees that remain valid even as the fraction of real data approaches zero. Under standard regularity and smoothness assumptions (of the kind long used to prove asymptotic consistency and normality of MLE), we show that as long as the number of samples per iteration at least polylogarihmic in the number of iterations, iterative MLE is consistent (meaning that it converges to the ground truth model as the sample size increases). These findings offer a sharper theoretical understanding of when model collapse can be avoided.
We complement this result with negative ones, that illustrate what can go wrong when these assump- tions are violated. In particular, we construct families of distributions for which MLE consistent when trained on real data, but nonetheless suffers from collapse when synthetic data is iteratively accumulated. Our negative results come in two flavors. In Theorem 5.1 for any fixed sample size n, we construct a family of distributions such that the first iteration gives an excellent approximation to the real distribution, but even the second does not with constant probability. Next, in Theorem [5.2] we show that there exists family of distributions such that for any model collapse will eventually occur, after number of iterations which grows arbitrarily slowly with n.
To the best of our knowledge, Theorem 5. and Theorem 5.2 are the first rigorous examples of iterative generative modeling with accumulating data that rapidly lead to model collapse. Recently, it has been suggested that model collapse does not occur when data accumulates across iterations Gerstgrasssr et al. 2025 Dey and Donoho 2024 Schaeffer et al. 2025 Our results show that such claims can only be true under structural assumptions beyond MLE consistency.
2 Related Work
Model collapse has recently drawn considerable attention, driven in part by the realization that many
datasets are already contaminated with synthetic samples (Alemohammad et al.) 2024). A growing
number of empirical studies have reported least some level of performance degradation in models
trained on such data (Shumailov et al. 2024 Alemohammad et al. 20224] Hataya et al. 2023 Bohacek
and Farid 2023 Briesch et al.) 2023 Guo et al. 2023].
Several types of synthetic data contamination settings have been considered. (Shumallov et al. [2024] considered a fully-synthetic setting, meaning that each model trains only on data produced by the previous model, without any real data. In such a setting, even simple Gaussian distributions can be shown to suffer from severe model collapse. In addition a fully-synthetic setting, AAleohhamaa setting, where data is mixed between real and synthetic data. They observed empirically that in such cases, model collapse may either occur slowly or be avoided altogether, depending on how much real data is added at each iteration. Since then, a few works have theoretically considered accumulating data settings [Gerstraassrr et al., 2025 Dey and Donoho 2024). These works suggest that data accumulation plays significant role and can mitigate model collapse. Our results show that this partially true: MLE in data accumulation setting can avoid model collapse if the models are sufficiently well-behaved, but there exist models that can suffer from severe model collapse even in such a setting.
ctt al. 2024] considered an accumulating-
Among theoretical works, several settings have been studied that differ somewhat from the setting of this paper. A notable line of works considers linear regression, taking advantage of the closed-form expression for the least squares estimate [Dohmatob et al.) 2024a Gerstgrasser et al. 2025 Dohmatob al. 2025). These works focus on discriminative models, where previous models are used to label new data, not synthetically generate new data as in this paper. Moreover, the setting of [Dohmatob] ct al. 2025 is non-iterative, and they analyze the test error when the training data contains some samples that are labeled from linear predictor drawn from "bad" synthetic distribution that differs from the real one. This a key difference from works such as Dey and Donoho (2024) that analyzed setting where data gradually accumulates.
2

---

There are quite few works that analyze specific family of generative models. For example, Gaussians and kernel density estimators have been analyzed in Shumailov al. [2024]. Kazdan et al.
[2024). He et al. 2025] Fu et al. [2024] analyzed model collapse for simplified one-hidden-layer
diffusion model. Dohmatob et al. 2024b) analyzed simplified token generators, including Hutter
L.Mss [Hutter 2021] and associative memories [Cabannes et al., [203]. Fu et al. (2025) analyzed
several architectures under a framework they called recursive stability, which bears similarities to algorithmic stability. In contrast to all of these works, our work applies to general families of distributions.
A few works characterize iterative generative modeling by analyzing MLE as we do here. Marchi al. [2024] assume that the differences between distributions subsequent generations form martingale difference sequence. However, this assumption is difficult to verify and somewhat unlikely in general. Seddik al. 2024), Bertrand al. (0244] analyze a setting where data is mixed from the ground truth model as well as the latest generative model. Our work focuses on more natural setting of data accumulating over time. There is also the work Dey and Donoho [2024]. which was
discussed in the introduction.
Lastly, we note that some works have proposed mechanisms to mitigate collapse through supervision or intervention. For instance. |eebbach et [ [2024), Feng et al. 20225. Amin et al. [2025] show that even minimal forms of ground- truth feedback can substantially reduce the risk of collapse. In contrast,
our work focuses on the unsupervised case, where synthetic data accumulates and no corrective signal available.
3 Setting and Notation
3.1 Notation
We use bold-faced font denote vectors, e.g. Rd, and denote by ||||| the Euclidean norm. We let [<] = (I,...,y)}, nn). Unless otherwise stated, ||-I| denotes the operator norm for matrices and 3rd-order tensors, where the latter is defined for 3rd-order tensor A as |||||| supv A(vi v2))3) ||vall aalllllv We use the standard big-O notation, with ((.). (-)) and (2(-) hiding absolute constants that do not depend on problem parameters. To specify constants that depend only certain quantities, put these quantities in parentheses. For example, C(K,, K2) would denote a constant that depends only K1, K2. For given vector and radius >> 0, let B, (v) {u [u v| < r) be the closed ball of radius r centered at v. For function f0), we write (0) for its Hessian, and 8 v (() for its 3rd-order derivative tensor, meaning [())(()) 00,80,00," f(0). For matrix A, we denote by (min(A), Amax(A) its minimal and maximal eigenvalues respectively.
may
3.2 Iterative Maximum Likelihood Estimation
In this section, we formalize the iterative MLE setting that will be studied throughout the paper. Let Ô be a of parameters and consider corresponding family of probability density functions (PDFs) over an input space X, given by Pe == (po (-) 0 }). Generative modeling aims to approximate unknown ground truth parameters 8 using some 9 E Ô Perhaps the most fundamental way to do this is through MLE (throughout the paper, we will also use this acronym to refer to the maximum likelihood estimator the meaning should be clear from context).
Definition 3.1. Let X be dataset with elements belonging to X, the MLE trained on X is given by
i argmax BE0
log (po(x)),
XEX
In the above definition, is not immediately clear why the MLE exists it unique. Existence is known to hold under mild assumptions, and throughout the proofs, we will explicitly show existence whenever necessary. Regarding uniqueness, the MLE may not be unique in general. However, under mild assumptions, it is known that the MLE converges to the real parameters 0* (e.g. [Wald] 1949) ).
and that given sufficiently many samples, the log-likelihood is strictly concave in neighborhood of 0*. As such, asymptotically, the MLE is expected to be unique. Nevertheless, formally treating this typically introduces unnecessary and undesired complications to the analysis. It therefore standard to simply assume that the MLE is unique whenever it exists (c.g. [Lehmann and Casella, 2006)). We
3

---

Algorithm 1 Iterative Maximum Likelihood Estimation
Require: Parameter space θ C R4; family of distributions {EE over input space Xt; number of samples per iteration n; target parameters E 6.
1: Set ((0) := 6"
2: for T 0, 1, 2,
................................ do
x) 4: Define cumulative dataset: X(SI) = -Ui= x(0
3: sample X = (x1
Porry ()) i.i.d.
5: Train model on ((S)):
@(T+1) = argmin (0), (()) = 1 Σ log (xo
7
6: end for
follow Bertrand al. [2024] in making similar, but slightly milder assumption that if there multiple parameter vectors maximizing the log-likelihood, the argmax may choose the one that is closest to a given reference point. This is, of course, made explicit in the proofs.
Throughout the paper, we will be mostly interested in what happens when MLEs are iteratively re-trained. We will be analyzing setting where synthetic data accumulates over time, as this is what one naturally expects to occur with web data (see the Related Works, Sec. 2 for a discussion on this). Let f" E e denote the parameters of the real underlying distribution, and set ((0) == d
x ( ~ Pacr) (-) i.i.d. and add these to the existing dataset, giving ((T)) U X )) Then, obtain ((T+1) as the MLE given the training data X(TT), We refer the reader Algorithm 1 for a complete description of iterative MLE. Note that for convenience, the algorithm is written as a minimization problem using the negative log likelihood (or cross-entropy loss).
For each iteration T = 0, 1, sample ( = (xl
(2) "..",,..,,"))
We are now ready to state our assumptions. They are minor variants of those long used to study MLE in classical statistical literature (since at least [1946), see also [Le Cam 1956 van der Vaart 2000 Lehmann and Casella 2006)). The first set of assumptions consists of standard regularity conditions (see e.g. [Lehmann 1999)).
Assumption (Regularity Conditions).
(A) There exists > 0 such that the closed ball (,* is contained in O.
(B) The probability density functions pe are distinct.
(C) The set points for which is positive does not depend on 0.
Assumption 1. B necessary quantify the distance between distributions pe. Pø using | l'| Note that one can always satisfy Assumption T. B by removing duplicates from Pe. or by considering the quotient topology in Redner (1881] Assumption [. C avoids pathologies and ensures that log (c() is well-defined throughout the iterative sampling process. In distributions modeled using neural networks, probabilities are usually given by applying a softmax, ensuring that they are always positive and thus satisfying Assumption I.
Classical analysis of MLE often require various smoothness assumptions on log (x() such as bounded third derivatives (see for example [Cramér 1946 Lehmann and Casella (006]). We will use the following (where r > 0 is the radius from Assumption Ap:
Assumption 2 (Smoothness). For any x E X and E O, log(pe(x)) is 3 times continuously differentiable in 0, the partial derivatives support differentiation under the integral sign and
Mcaning that we can exchange the order of differentiation and integration. This is mild assumption that is implicit in many papers.

---

(A) Bub-Gaussian gradients: There exists some K₁ > 0 such that for any 0 E (6*)
u2
Px ( log (xo())) 2 u) S 2xxp
Vu 2 0.
2K?
(B) Bounded Hessian: There exists some K₂ > such that for any x E and (0")
VV log (po(x))ll S K2.
(C) Bounded Third Derivatives: There exists some K3 such that for any E X and 0 E B, ). V log (po(x))] S K3.
Assumptions 2. A 2. B allow us bound the difference between sums of random variables and their expected values. Since our bounds are non-asymptotic, one cannot avoid some assumptions to bound these differences. Sub-Gaassianity is standard assumption in non-asymptotic works, and holds (for example) for bounded random vectors. Nevertheless, our assumptions need to hold only in a small neighborhood of 0*, making them relatively mild. It possible relax these assumptions further, but we do not pursue such generalizations, as it is not the focus of our paper.
Before stating next assumption, recall that the Fisher information matrix at some @ is defined as
I(0) = Ex Vo log (x Vo log (x ']-
The Fisher information matrix is well-known to play a central role in the analysis of MLE. Under other assumptions, it is straightforward to show that the Fisher information matrix is always positive semidefinite (see Appendix [or more information.. In fact, standard analyses of MLE (say, to establish asymptotic normality) require the matrix to be positive definite O [van der Vaart 2000 Lehmann and Casella 2006). Thus, to get our non-asymptotic bounds, it is reasonable to assume the following (where again, 0 is the value from Assumption I. A}:
Assumption 3. There exists some Ao 0 such that for any O E B,*)),, Amin (I(O) 2 λo-
We note that one can equivalently assume that I(0) is positive definite only at 0*, and pick r small enough such that by the smoothness assumption, this holds for the neighborhood. However, the formulation above is more convenient for our purposes.
We end by noting that the assumptions above are mostly satisfied (at least approximately) by neural networks. For example, the constant support assumption (Assumption [. is trivially satisfied in standard architectures, since the softmax function widely used to assign probabilities is always non-zero. The smoothness assumptions can be satisfied in various settings, especially when using techniques such as weight decay, which are standard in modern LLM training. Of course, the exact bounds would depend on the architecture and the setting. In general, we believe that weakening these assumptions is quite feasible and is an interesting direction for future work.
4 Consistency of Iterative MLE
In this section, we formally show that iterative MLE remains consistent under the conditions from the previous section. In particular, we provide non-asymptotic bound, which establishes that as long as the number of samples n is at least polylogarithmic in the number of iterations T,, then with high probability, all models remain close to the ground-truth parameters. This result highlights that model collapse is not inevitable, even when + oo and the fraction of real data vanishes.
Theorem 4.1. Under Assumptions 7 3 there exist constants e, C > O which depend only on K1, K2, K3, An and r, such that for any N, > and any n 2 c (log(T) + 1) log? 2 (7d7'), ) holds with probability at least 1 5 that
| - 8-|
s
c
log
71
For sufficiently large n, the bound in Eq. 1 is independent of T. and has only logarithmic dependence on the dimension d. The theorem is stated for a specific but union bound can easily provide similar result holding simultancously for all t [T', at the cost of log(T") factor.
5

---

Under the same assumptions as in Theorem 4.1 convergence of parameters also implies conver- gence in KL-Divergence and convergence in total variation (TV) distance. We refer the reader to Appendix A.1 I for background and details. In particular, for a suitable absolute constant C' > 0,
Theorem implies
log (#)
TV (po- Pocr,) 1 CV log (
DKL (po. Porr))
I C.
We now detail some ways in which Theorem 4.1 differs from past results on model (non)-collapse. In Bertrand et al. [2024), synthetic data does not accumulate across iterations, and for each iteration I E (T), most of the training data used to train ((4) is real. The maximal fraction of synthetic data was increased in the follow-up work |Ferbach et al. [2024] when assuming access the full distribution (i.e. = oc). Similarly, Dey and Donoho [2024] first fix '" and then analyze the limit of n + oo.
They do not provide finite sample guarantees that quantify the dependence between I and n. Seddik et al. (2024) bounded the expected value of the TV distance for distributions over finite vocabularies. When the amount of synthetic data is sufficiently large relative the vocabulary size, their bound scales as O (vk(n)) where k is the total amount of synthetic data. In the data accumulation setting,
k == - 1)n, in which case the bound becomes O (VT/m)
We note that while Theorem 4.1 considers a setting where the exact MLE is computable, we believe the theorem naturally be extended to also accommodate optimization error, where only an approximate MLE is available. Indeed, we empirically observe in Appendix G that for families of distributions for which an exact formula for the MLE is known, the results are robust to mild optimization error.
4.1 Proof Sketch of Theorem 4.1
We provide here the proof intuition for Theorem 4.1 and refer the reader to Appendix D for the rigorous proof.
As a preliminary stage, we first show using Proposition C.I that given enough samples, for any is small with high probability. The challenges of this step are that this is done in non-asymptotic way and takes into account data arising from all previous iterations.
t E [T], [| - |(
We note that Theorem 4.1 cannot be obtained naively as a direct consequence of the Proposition
C.I bound on || g(o)|| using the triangle inequality leads suboptimal dependence on T, since it doesn't it take into account cancellations from iteration to iteration. Instead, as we will show in the following paragraph, Proposition [.I]will will be used to ensure that for large n. ((++1) will be sufficiently close to ((t) to enable Taylor expanding the log likelihood around it. This idea draws inspiration from the asymptotic normality analysis of MLE Cramér 1946
Extending Proposition C.1
Lehmann and Casella 2006).
To that end, fix some E [T] and observe that for any such t, since ((++1) is the MLE on X(S),, stationary point of the log-likelihood function. As such, Taylor expanding, we show that there exists a matrix RG E [Rmx xm with || R&ll < le such that
1
Vly (D(+t1) D
Σ Vl, (BL)) (2e,(00)) (011+1) (01(41)) (1((0(+)))) -(00).
By definition, (00 is the MLE for X(5(-1) it is stationary point for the corresponding logelikelioood function and thus Σ Vl, (o())) 0. For notational simplicity, let Ht, Eo (2(,(8(0)) + RG, then the above simplifies to
0 == Vl, (0L)) + (0(())1)))01

---

In the full proof, we show that H2 is invertible. In such a case, we can rearrange the above equation to obtain
01+t1) - (0l1)..
Importantly, this allows us to express how the parameters evolve over many iterations by taking telescopic sum as follows.
|000)
-000||
0001
Σ
g(++1)
000)
Σ
H
Vle (Drc))
1=0
1=0
Vlc (oc))
-
Σ
Hi t I(0000)-1) Vlc (DCO)
0
The expected value of V&, (80) (conditioned on (()) can be shown to be zero, so that the first term forms martingale, which allows us to bound the norm essentially as if all samples were independent. Since each Vl, scaled by 1+1* the variance scales as (4+1)2- So the variance of the sum can be
upper bounded as E
# S S.
In summary, we show that with high probability
Ve, (0r)) 0o log [+1
1
Σ
The second term in Eq. ( has to be treated differently, as correlations between H;-1 and Ve, imply that each term is not necessarily mean-zero, and so the sum should be expected to have some dependence on T. This term somewhat complicates the proof, as bounding it requires knowing that ||0(t) - | is sufficiently small for all & < T. The proof thus works inductively, bounding this term from iteration to iteration. Roughly speaking, in the end, we show that with high probability
Σ
H
1 t+ -
)
1|ve (BO)) Vellog( 1) log
dT
1 Vn
n
t=o
where the last inequality follows from the assumption that n is sufficiently large.
5 Necessity of Structural Assumptions
Theorem 4.1 provides conditions under which the iterative MLE retains good performance, even if the proportion of synthetic data approaches 1. Clearly, this cannot always be true. In particular, there well-known examples of families of distributions which even standard MLE is inconsistent: Namely it will not converge to the ground-truth parameters as the sample size increases, even when trained purely real data (e.g. [Bahadur 1958 Ferguson 1982, Le Cam 1990]. In such situations, the whole question of model collapse is rather meaningless. Thus, a natural (informal) follow-up question is the following: In the setting where synthetic data is added to the real dataset in each iteration, is there family of distributions that is sufficiently well-behaved for MLE to be asymptotically consistent (when trained on real data), but still exhibits rapid model collapse? In other words, do there exist cases where the MLE can learn the real distribution, and yer model collapse still occurs when applying MLE iteratively?
In this section, we show that the answer is yes, and demonstrate different settings in which model collapse can occur when the conditions of Theorem 4.] are not satisfied. To the best of our knowledge, these are the first rigorous examples of iterative generative modeling with accumulating data that rapidly leads to model collapse.
We emphasize that, following the rest the paper, we focus here on a setting where synthetic data iteratively accumulates on top of the real data. A different model collapse setting studied in some previous works when at each iteration, MLE is performed purely on synthetic data generated by the latest model. In such setting, the real training data disappears already after a single iteration, and it
7

---

has been shown to lead to model collapse even for very rell-behaved distributions such as Gaussians [Shumailov et al. 2024]. has recently been suggested that if data added rather than replaced (as in our setting), the extent to which iterative MLE performance degrades is limited [Gerstgrasser al. 2025] Dey and Donoho 2024 Schaeffer et al., 2025]. We show here that this can be true only if further assumptions are made, beyond just MLE consistency (as we do in Theorem 4.1
To formalize our results, we will require the following consistency definition for MLE: Definition 5.1. We will say a family of distributions Pe is TV-consistent, for any 0* E 2 and n E N, the MLE 0 trained on i..dd samples from pg- exists, and
TV (po>, .P))
P, 0. 1-+00
Note that we use here convergence total variation, rather than convergence in parameters as in Theorem The reason is that to establish negative results, we have to make use of distributions that do not follow the assumptions of Theorem [.1] and in particular do not satisfy the smoothness assumptions there. Without smoothness, parametric convergence and convergence of distributions are no longer equivalent in general. Thus, using a probability metric such as total variation is more natural in our setting, as we are ultimately interested in approximating the ground-truth distribution.
5.1 Models Can Collapse Immediately
By definition, for a TV-consistent family of distributions, Pącn) is good approximation of the ground truth distribution pa+, assuming the number of samples is sufficiently large. Our first negative result shows that, perhaps surprisingly, one cannot hope to show the same even for Poe) without further assumptions. Specifically, for any n there is some family of distributions (that may depend on n), such that MIE on n samples from the ground- now augment the data with synthetic samples from the MLE solution, and re-run MLE, then the resulting distribution Pąes will exhibit model collapse with constant probability.
distribution will perform well, but if we
Theorem 5.1. There exists θ C RR2 and 6* E O, such that for any E N, there is a TV-consistent family of distributions {po Joea (that may depend on n) such that
1. with probability at least
TV (oo=) S log (n)
2. For some absolute constants c, C >> O, it holds with probability at least e that
TV pPca)) 2 C.
In the above theorem, as the number of samples grows, Poct) is very close to Pa- with high probability, but there is some constant probability that Paca) will be far from Para- This implies that statements similar to Theorem 4.1 are not possible for general TV-consistent families without further assumptions. Indeed, Theorem iimplies that the relative gap in total variation between iterations
find family of distributions such that
and t 2 can be arbitrarily large, since
TV (po-,, C n/ log().)
TV (po- Pgcn)
We now provide some intuition for the proof of Theorem 5.1 with the full rigorous proof appearing in Appendix E We consider a family of distributions, given by the following parameterized mixture of uniform distributions on I: 1 2
2 -.(( 1))
U(0, - (2o)) + a
+ -(((())))
+ a
U(n,
+ + f(a))),
4
where U(-) is the uniform distribution on interval, θ {(a, 11) | [0, 1e [ faa)] are the parameters, and f is a positive function that decays very quickly with so that the PDF of U U(H,, H + f()))) approaches a delta function (the exact form of f depends on n)). Let g:0) := 8" =
(a) = 0, (0))) = 0) such that Paro) = U(0, 1]). We show that the MLE ((1) = (a()) ()))) satisfies
11) max (m) 2 (0)
1
2n
8

---

As such, a converges very quickly to (0) as n increases, and we prove that this implies a rapid convergence of TV (paro), ( regardless of the value of (t)).
We now move on to analyzing the second iteration. Because (1) L then with some constant
2n'
probability (over the sampling of n new samples from Pœc1)), ), at least one of these samples x; will be inside the interval [2, 3]. When this happens, because f(a) is tiny for larger values of a (leading to high likelihood in the interval [M, μ f(a)), the MLE solution 6(2) = (a (x()) will be such that of(") [u(2), 10(2),(0)) [(a(2)) and (2) will be sufficiently large so that f (( is very small. In particular, (2) will be considerably larger than the ground truth (") leading to model collapse.
5.2 Arbitrarily Fast Model Collapse
Theorem 5. shows that without further assumptions, model collapse can occur already after a single iteration. However, the construction requires picking the distribution according to the sample size n E N, which is arguably unnatural. Below, we show that this requirement can be removed: Namely, there exists family of distributions where model collapse will occur for any sample size On the flip side, the model collapse no longer occurs after a single iteration, but rather after a number of iterations that grows with n (although the growth rate can be arbitrarily slow):
Theorem 5.2. Let Φ (0) + 20) be any strictly monotonically increasing function such that limm--oc ((n) = oc. Then there exists an absolute constant > O, a set O O* E and TV-consistent family of distributions Pe (which depends on o), such that for any 8 E (0, N,
holds with probability at least ö that
[ 8
( max (é(n),
TV (Pacrs. po-) 8
for some
T
log
Importantly, & can be chosen grow arbitrarily slowly. For example, taking d(n) = log log(n + 1). Theorem 5.2 iimples that one can exhibit model collapse in as few as Olog log(n1 + 1)) iterations.
The proof of Theorem 5.2 draws inspiration from the proof of Theorem 5.1 but the construction is more involved, as the distribution can no longer depend on the number of samples n. The family will consist of two types of distributions. The first, which we denote ha, has the form
Σ (1 -az)
II
where a [0, 11. and a has a finite number of non-zero indices. One way to think of these distributions is as sampling using an iterative process. where starting from j = 0, one flips a coin with bias ay, and either samples a point from (((j.jj + 2ay]) (with probability 1 1- a.). or with probability Oyj. increase j by one and repeat the process, until some point is sampled. We also include a family of distributions Jæ.J corresponding to
U100 1 U - 3+ 3(]))
1
where J E EENN (1), 3 [0, 1] and f is a function that decays very quickly as increases.
Now, consider the ground truth distribution to be ho (meaning a = 0 for all j), which is actually just U ([0, 1]). We show that at any iteration 6, the density /a that maximizes the likelihood out of functions of the form ha given by taking aj
(t+1) {[( max (50) n [i.j + 1]-
Thus, the general procedure is follows: For any J E N. once at > 0 for every - J 1. there (t) will reach interval [J, + 1]. ensuring d (t+11) O. We
non-zero chance that new sample x;;
choose the function f so that for any N E N. there is some J. such that if n & N and if there is some sample in [JN. JN + 1], then the MLE will be of the form Ia.J (as f(Jx) is sufficiently
small, leading to a high likelihood of the sample). We show that once this happens, the total variation distance will be large, and the proof will be complete.
The difficult part is showing that for any J E N, there some time E such that with high probability, there will be sample in [J.J + 1]. Morcover, this T can be chosen to be essentially

---

independent of n. Since we may let JN grow arbitrarily slowly in N,, and the number of iterations needed obtain a sample JN + 1] can be upper bounded independently of N, the number of iterations needed for model collapse can grow arbitrarily slowly with N (and thus with n).
5.3 Implications and Relation to Theorem 4.1
The results of this section inform how in the absence of the assumptions of Theorem model collapse can occur arbitrarily quickly. Even though the constructions of Theorems 5.1 and 5.2 are artificial, they highlight more general phenomena that are needed for model collapse to occur or to be avoided. In particular, in our view, the main difference from Theorem 4.1 is the smoothness assumption. Theorems [5.] and 5.2 crucially use highly non-smooth construction, in which slight perturbations of the parameters can induce huge differences in the resulting model, and we find it unlikely that a negative example would be possible without this behavior.
Discussion
We studied model collapse in a setting that has recently gained interest in the literature, where synthetic data accumulates over time. Focusing on MLE, we showed that collapse can be avoided under standard assumptions samples is polylogarithmic in the number of iterations. At the same time, when these assumptions not satisfied, construct scenarios where the MLE is consistent, yet collapse occurs arbitrarily quickly with synthetic data. These examples show that MLE consistency alone is not sufficient for preventing model collapse even in the accumulating-datt setting.
the proportion of real data vanishes, provided that the number of
While the assumptions in this work are rather classic, they may not the mildest possible while still allowing for positive results. Moving forward, it would be interesting to bridge the gap still present in this work between the assumptions in the negative and positive results and characterize assumptions that are both necessary and sufficient for avoiding model collapse. Our hope is that these results contribute to clearer theoretical understanding of model collapse, and lead to a more fine-grained perspective when does or does not occur.
Acknowledgments and Disclosure of Funding
This research is supported in part by European Research Council (ERC) grant 754705, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center and by research grants from the Estate of Harry Schutzman and the Anita James Rosen Foundation.
10

---

References
Sina Alemohammad, Josue Casco- Roddio-zdr Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaci, Daniel Leleune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go mad. In The Tweifth International Conference on Learning Representations, 2024.
Kareem Amin, Sara Bahakniya, Alex Bie, Weiwei Kong, Umar Syed, and Sergei Vassilvitskii. Escaping collapse: The strength of weak data for large language model training. arXiv preprint Siti::50220092244 2025.
RR Bahadur. Examples of inconsistency of maximum likelihood estimates. Sankhya: The Indian Journal of Statistics, pages 207 210. 1958.
Quentin Bertrand, Avishek Jocy Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. In ICLR, 2024.
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse. arXiv e-prints, pages arXiv-2311, 2023.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self self-consuming training loop. CoRR, 2023.
Vivien Cabannes, Elvis Dohmatob, and Alberto Bictti. Scaling laws for associative memories. arXiv preprint arXiv:2310.02984, 2023.
Harald Cramér. Mathematical Methods of Statistics. Princeton University Press, 1946.
Apratim Dey and David Donoho. Universality of x² /6 pathway avoiding model collapse. arXiv preprint arXiv:2410.22812, 2024.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: case regression. In The Thirty-eighth Annual Conference on Neural Information Processing Syxtems, 2024a.
Elvis Dohmatob, Yunzhen Feng, Yang. Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference on Machine Learning, 2024b.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. In The Thirteenth International Conference on Learning Representations, 2025.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification. In The Thirteenth International Conference on Learning Representations, 2025.
Damien Ferbach, Quentin Bertrand, Joey Bose, and Gauthier Gidel. Self-consuming generative models with curated data provably optimize human preferences. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL URR.https://// httpss://peereviice net. nww./frrummee..../ wtt.:eeeeee..ooom...doooooo...a
Thomas S Ferguson. An inconsistent maximum likelihood estimate. Journal of the American Statistical Association, 77(380):31 834, 1982.
Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models. In International Conference on Machine Learning, pages 14228-14255 PMLR, 2024.
Shi Fu, Yingjie Wang, Yuzhu Chen, Xinmei Tian, Dacheng Tao. theoretical perspective: How to prevent model collapse in self-consuming training loops. In The Thirteenth International Conference on Learning Representations, 2025.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. In First Conference Language Modeling, 2025.
Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. The curious decline of linguistic diversity: Training language models on synthetic text. arXiv preprint 1rXiv:2311.098.0 2023.
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference Computer Vision. pages
2023.
Hengzhi He. Shirong Xu, and Guang Cheng. Golden ratio weighting prevents model collapse. arXiv preprint arXiv:2502. 180499 2025.

---

Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04077 2021.
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024.
Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint
2019.
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David I. Donoho, and Sanmi Koyejo. Collapse thrive? perils and promises of synthetic data in self-generating world. arXiv preprint xcXiv:2410.16713 2024.
Lucien Le Cam. On the asymptotic theory of estimation and testing hypotheses, In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions the Theory of Statistics, volume 3. pages 129- 157. University of California Press, 1956.
Lucien Le Cam. Maximum likelihood: introduction. International Statistical Review/Revue Internationale de Statistique, pages 153-171, 1990.
Erich I. Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
Erich Leo Lehmann. Elements of large-sample theory. Springer, 1999.
Matteo Marchi, Stefano Soatto, Pratik Chaudhari, and Paulo Tabuada. Heat death of generative models in closed-loop learning. arXiv preprint arXiv:2404.02322 2024.
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics, 4:2111 22211-2245, 1994.
Richard Redner. Note on the consistency of the maximum likelihood estimate for nonidentifiable distributions. The Annals of Statistics, pages 225-228. 228, 1981.
Rylan Schaeffer, Joshua Kazdan, Alvan Caleb Arulandu, and Sanmi Koyejo. Position: Model collapse does not mean what you think. arXiv preprint arXiv:2503.03150, 2025.
Mohamed El Amine Seddik, Suci- Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training synthetic data? statistical analysis of language model collapse. arXiv preprint 2.00022404.00000, 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 63 (8022)))7575775 2024.
George Tauchen. Diagnostic testing and evaluation of maximum likelihood models. Journal of Econometrics. 30(1-2):415 443, 1985.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12:389 434, 2012.
AW van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Roman Vershynin. diim-siimesio probubility: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Abraham Wald. Note on the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):595-60,, 1949.
12

---

A Background on Likelihood Estimation
For any f E O the Fisher information matrix defined as
I(0)
Ex V log po (x) Vo log (o
We state here well-known results regarding the Fisher information matrix that will be used throughout the proofs (e.g. [Lehmann] [Sectinn 7.5)).
Theorem A.I. If Assumptions hold, then for any 0 (,()))),
Ex Vo log = 0.
Note that in particular, this implies that I(0) is the covariance matrix of the random vector Va log po (x) and is therefore p.s.d.
We will also need the following.
Theorem A.2. If Assumptions hold, then for any 0 (,(0)),,
I(0) Ex V log po
Note that the above theorem also implies |I(()]| < supx formally as the following corollary.
V2 loggoox))] s Kg. We state this
Corollary A.1. If Asssumptions 2 hold, then for any 0 E B,
[I(O)]| K2.
A.1 Parametric Convergence vs. KL vs. TV
Two common ways compare PDFs p, q over input space X' are the KL divergence:
p(x)
DKL (p)lg) = Exop log
q(x))
and the total variation distance
1 f (p(x)) ((x) dr.
TV (p.g)) := JEE
We note that while the TV proper metric, the KI. divergence not, as is not symmetric. Nevertheless, the two can be related by the well-known Pinsker's inequality:
TV (p. g) S DKI.
DKL (p 2)
is well known that under sufficient smoothness assumptions, convergence in parameters implies convergence in KI. and total variation. Indeed, for fixed x E X, consider second-order Taylor expansion of log (po(x)) around 8(0), which gives
1 log (Paco) (x)) + V log (pero (x)) T (0 - B(0)) + 2 B(0)) T log (pero (x)) (0 ((0)) R(x),
where the remainder R(x) can shown to satisfy [R()) < Ka ||0 g(0) under Assumption By Theorem the expected value of the gradient term is 0 and by Theorem A.2 the expected value of the hessian term is (I)).
6
13

---

As such, Taylor expanding at every point x together with Theorem A.1 and Theorem A.2 the KL divergence can be approximated as
DKL (Peto:
po)
Ex-ppro) [log (Paco) (x)) log (po(x))
2 00- (101) T I(0 ( B(0))
E [[(x)]
1
I(
|| 110 o(0)
||
K3
0 ||
2
6
K2
K3
0 g(0)
0
000
2
6
By Pinsker's inequality, this implies
TV (pocos, Pa)
S
DRL. (Poto: p1 5
K₂
|10 ((0)
1|
Ka
110
010) ||
4
12
v K2
K3
#
2 0
((0)
0
0
12
B
Concentration
We start with couple of known results that will be useful for approximating the gradient and hessian of the log-likelihodd. Theorem B.1 (Jin et al. there exist fixed 01...., of such that for all t E [T]. E[z, 2 Z1....,
2019 Corollary 7). Let 21..... Zr E Rdd be random vectors and assume
2r 1]
and
P ([ 2 21, 21-1) S 2xxp
Vu 2 0.
20
Then there exists an absolute constant C' > O such that for any 8 >>> 0, with probability at least 1 5,
T Z1
T log
2d
SC
Σ
of
Theorem B.2 (Tropp 2012
Theorem 7.1). Let {M.} be finite sequence of random symmetric d x matrices such that E. M, M,,..., M,-1]] = 0. Assume further that there exists a fixed sequence of
symmetric x d matrices {Ar] such that M/ 3 A? almost surely. Let o² = 12 A? for all 0.
P
Amax
ΣM,, Me
2
S dexp
S0Z
We bring Theorem B.3 to a slightly more convenient form for our uses.
Theorem B.3. Let [M, 17 be finite sequence of random symmetric d x d matrices such that such that [|Mell S K
E| MI.... Mr 1] = O. Assume further that there exists some
almost surely. Then for any 8 > 0, holds with probability least 1 - 8 that
T
2d
Σ
M
I
V8K
Tlog
Proof. Set :3 o² = TK² apply Theorem B.2 once to bound E, M, and again to bound E, M.. The corollary follows from the union bound.
OTT-1) constant > such that for any O, it holds with probability at least that
Lemma B.1. Under Assumptions I
ifo(1),.. (()))) (0()) then there exists an absolute
T
log 2d
Σ
ve((00)
< <CKi
14

---

[n] let ži,i := 2+1 V log Potrs (x) (x())) We order these Tn
Proof. For t {o....,T- 13,
random vectors z first by and then by i. Specifically, let p: [Tn] {0,.... T 1} [m] be this mapping of indices, such that
p(1)
pp()n) = 20.1..... Zu,n. 21.1.... 21 1,7
By Theorem A.1 for all [Tn]. ((),()))
pp(1). ................................ 2p(k Furthermore, by Assumption
2. A for any 7, and any a 2 0
P
1
V llg Potto (x(e)((())
2 2u) =P (lV (Paco (x(o (++ 1)u)
i+1
S exp (- (-(t+1)2uu 2K3
In particular, for all & E [Tn] letting on = Ki/(p(k) + (where ((k); is the t E ,0,...., T- 13 that corresponds to p(:) have
u²
P (l 2
ZB Zk-1) exp
Vu 200 O.
20
As such, by Theorem B.1 there exists an absolute constant C > 0 such that with probability least 1- 6,
Tn
2d 8
Σ p(k)
SC
Σ
o
log
k=1
Note that since Σ
22 I x² 6.
we have
Th
T
of =
π K2n.
1=1
We obtain with the same probability that for suitable altered constant C O,
T-1
Tn Σ p(k)
log (8
Σ t+1
Ve((0())
S SKK
n
n
can also obtain concentration for a single 0 E (0(%). We omit the proof as it is simplified version of Lemma B.T (speifiically, the assumptions and Theorem imm]yy that the conditions of Theorem B.T are satisfied, which gives the following result).
Lemma B.2. Let 0 (0(0)) and x Xn ~ i.i.d. Under Assumptions 1 there exists an absolute constant C' > such that for any 0 E B,()())), 5 > 0, it holds with probability at least 1 that
log 2d
V log (po (x,))
S SCK1
n2
m
We will also need the following result for the Hessian of the logllikelhood::
Lemma B.3. Under Assumptions /
if g(T-1) E (B(( then there exists an absolute constant C O such that for any O, it holds with probability at least 8 that
T-1
T Tlog 2d
Σ
(2 (0(1) < CCK2
n
1=0
15

---

Proof. Fort {O....,I- -...,,--1, [n] let Me, == V2 (Docs I(((()). We order these
Tn random matrices Mr first by l and then by i. Specifically, let p= [Tn] {0................ 10....,, 1} [n] be this mapping of indices, such that
M (())) ................................
, Ma(TT) == Mo.1....... Mo,n. MI.1...... Min...... Mr-1.
By Theorem A.2 for all [Tn]. ([ Ma(1)...... ................................ Mp(k- 0. Furthermore, by Assump-
tion 2. B and Corollary A.1 for any ho E [Tn].
M Mo00 - K2 + [ - 2KK2.
As such, by Theorem B.3 there exists an absolute constant C > 0 such that with probability at least
1 8,
Σ (0(00) (0(.)
1
Tn Σ
Tlog 2d
Ma
S <CK2
n
n
Once again, we can also obtain an analogous result for a single 0 (,(0))). The proof is also analogous to Lemma B.3
Lemma B.4. Let 0 (0(0)) and Xi...., Xy, ~ Pg i.d. Under Assumptions 1 there exists an absolute constant C such that for any 0 (,(0))), ö 0, it holds with probability at least
1 - that
log
log (po (x:)) -
< <CKs
C Preparatory Results
C.1 Non-Asymptotic Consistency
Lemma C.I. If Assumpttinn that is,
2 holds, then for every x E X,, V log is K3-Lipschitz on (0()):
v (o) v log (x) S K3 110 0'll, 00,0' E (,()
Proof. Fix E X and 0, (.(0)). Consider the line segment y [0,1] + (B(0)) given by (t) 1(0' 0). Note that the convexity of B, (0(0) implies that 7 (t) E (,((()) for all
t E [0, 1]. From the fundamental theorem of calculus,
V log (x) V log = ''' V logg log (x) = / Vi log (oe fdt,
dt
where
V log Px(e) (x)]0' 0l],, = Ei= 00,00,00 log (o(()) (e)) - 0]p-
Applying the operator norm and Assumption
V log vi lo (x)]
<
V log p. (x)10' 0]] di
< sup tE[0,1]
(() 1100 01l
< Kg ||6 01|
16

---

Lemma C.2. Under Assumptions ! for any E N. if ((*) E B, (, then there exists an absolute constant C such that for any with probability at least 1 -
(v
(3)
SC(K2 -
Ks)
t max
|0( 0000
1
Proof. By the triangle inequality,
v2 (0())) - - - Σ ( -3000))) -
Σ (e
-
By Lemma
C.1
(0) is K3 Lipschitz in 0. Using this and the triangle inequality, the first term is
bounded by
K3
g(0) g(0) - K3 [11000-000/
0001
- -
2Ks max (b g(0)
By Lemma B.3 there exists an absolute constant C >> such that with probability at least - 8 the second term is at most CK2y / (++1)log( 44)
concluding the proof.
Lemma C.3. Under Assumptions for any E N, if .... g(t) E (,()),),, then there exists an absolute constant C such that for any with probability least 1
Ka) (v
((++)) 1og
-e,(000) (1+ 1)
< <0(K2 (1+1)0000)
[060 |00) 1000-0000)
max 35t
Proof. By C.1 I(0) is K3 Lipschitz in 0,
(00)((()) (1 1)Z(00))
E
V2 (2,(0(0) - -I(OC))
+ (1 + +))((())
Σ I(06)))
==0
+ Σ
W
2 e (0(1)) I(B(O)
+ K3s
g(() g(0)
W
v2 22l (0(()) -
+ Kst max 35t
(00) g000
The proof now follows immediately from Lemma
C.2
We now prove the following proposition, which will serve a substantial role in the proof of Theo- rem 4.
Proposition C.1. Under Assumptions
there exist constants c(K,, K2, K3, Ao,
and C = C(K1, 2o) > O and a constant C2 = Cz(K2, K3) given by Lemma C.2 such that for any
t E,N, if maxx55i- (0() - |1o0|| I max
Ao ICs r/2
then for any > and n 2 clog (4d), with
probability at least 8
[g(() - g(c-1))|
c
log
4d
r
s max
t
2
17

---

Proof. Fix some > O that will be specified later, and let S := (B(t-11) be the sphere of radius with center at ((t-1), We will show that for sufficiently small a, with high probability it will hold simultaneously for all 0 on the sphere S, that 2j (0 o (0(t-))) As result, with high probability, there must be a local minimum of " (0( within the ball of radius a centered at g(t-1), This implie thht that ||0(++)) g(t) S a.
Assume for now that a small enough such that S. S (.00). We will later ensure this explicitly by picking a < r/2 (which is sufficient due to the assumption that ||0(0-)) 1) o(0) s r/22)
We first Taylor expand the normalized negative log-likelihood around 0it-1),
Σ
(0 (0-00--))++Q000
Q R(0),
(6)
where Q(0) the quadratic term, given by
1-1 Σ
Q(0)
g
g(t-1)),
and R(0) is the remainder term, which for some 0 between & and ((t-1) satisfies
|R(O)| 6
(00) (0 - -(1--))) g(c-1)), g(c-1))
-(((-1))
00,00,000 80
j=
t-1
tKg
s 6
a
(8)
1=
where the last inequality follows from 2. and by the convexity of (B(0)) which implies that θ E (0(%).
For the linear term, first note that if 2 then ((--1) is a stationary point of Lÿ (.), so
E Ve, (0lt-1)T O. So for any E N. EE (0(t)(())) ( Using this and
Lemma B.2 there exists a constant C = Ci (K)) > 0 such that with probability at least 8/2,
(0
V[l- g(c-1))
log (4 n
< Vel 000 s &lap
For the quadratic term, since the matrix V'l, (0 and the Fisher information matrices are symmet- ric, we have by Weyl's inequality, Assumption and Lemma C.2 that for C2- C2 (K2, K3) > 0 holds with probability at least - 5/2 that
1-]
1-1 Σ Iz 1=0
1-1
Σ v2 (,(ort-1)) 1=0
Σ ve (o(t-1)) -
Amin
2 Amii
tlog 4d
(0(5) - -8001
2tAo - C2
max
(10)
n
jst-1
2 Here we use that the argmax the definition of MLE not unique, chooses the parameters closest to ((t See the discussion following Def. (3.1 for more details.
18

---

Plugging Eq.
TO
back into the quadratic term given by Eq. 7 and using the assumption that
max
( 11 - A we have
log ( (1z)
Q
t 2
Ao C2 max -st-1
| |100 - C2
-21
a²
tn
t
-C3
log
id (
8
a²
(11)
2
where the last inequality follows by assumption.
log
Now take EAo (independent of 1) such that for any 2 e clog (fd)) , all of the following hold:
We can choose some constant e == cKi,, K2, K3, A0, >
1. < 2.
2. a < 20, VtC²
3. <
3da
4K3
The first condition was needed at the beginning of the proof. The second condition will allow us to bound Eq. (T)), since together with the choice of it ensures that
log
1d
aviC2do
Ao
C21
8C1
As result, Eq.
11
becomes
3
Ao
tdo a
A0
a
(12)
2
The third condition a ensures that the remainder term from Eq.
i8 negligible,
tK3
tho
IR(0)
Notice that the choice of a ensures that the bound for the linear term in Eq. D becomes
V0,(0 o2t-11) s Ciay
log
4d
tAu
Q
n
8
So overall, the Taylor expansion Eq. 6 satisfies
Q 2
Q 2
Ee(0 (0(t-1)))
Q
0.
==
So we have shown that for
8C1
log
and n 2 c log (#d) ), it holds with probability at least ((((t-)))) This implies the desired result as discussed
Oo
1-8 8 that for all 0 S., Σ/=
(0)
the beginning of the proof.
C.2 Lemmas for Theorem 4.1
Lemma C.4. Under Assumption 2 for any E N. if there exists some open ball B C such that < ++1
g(t) 0((++) E B, then there exists matrix R2 E Rdxd with || RRill that
2 K3 [((+1) - p(0)|| such
Σ Vl, (010) + (22;(010)) (g((+1) -B(0) ) + (.((((+1) g(0). 1=0
1
Vl, (ollt1)
=
0=
19

---

Proof. Fix some coordinate E [d] and consider the Taylor expansion of 00; Σj= & around 6(1),
which gives that for some 2; E Rd that lies the line segment between 0(0) and 0(++1),
①
e (
0
(0
ge
(((0)) g(t))
09: 1=0
00,
08
==0
a
(z(2)) g(tt) - -((t).. (13)
08,00 08,
where 2, E B (and in particular,
E Q). Let R₂ E Raxd be the matrix whose coordinates are given 88, (2(2) (2,2)(((t=1) (( Then Eq. 13) implies
by RReji. = Eij= Σ
0²
Σ Vl, (Dl+t))) 1=0
Σ
Veg (010) + (00)))))) (0(r+1) 0(0) -000)
-e(((it1)- 010)).
1=0
remains bound RR ||. By Assumption
2. C
we have
|
sup
v R
v (vi, v2, g((+1) )(1))
vI.2570
2 -=
t+1
K3 \v 1|vel
|ccc+11 00011
2
which shows ||&ll| - 1+1
K3 [((++) g(t)
2
Lemma C.5. Let A, B E Iddd be positive definite matrices, then 1|A B
||A- -B-11111 Amin (A) (B)
Proof.
1|A B
||---- = (B
||- -
Amin (A))min (B)
D Proof of Theorem 4.1
Theorem 4.1. Under Assumptions 7 3 there exist constants c, > 0 which depend only on
K1, K2, K3, An and r, such that for any N. > and any n 2 c(lo((T) + 1)² log" 2 (Tdl') holds with probability at least å that
| - 0r|
log s C
n
Proof. Let C = Ci(K1, K2. K3. Ag.
denote the maximum of the constants appearing in
the statements of Lemmas B.I B.3 be given by & := 6/T) for statement, whose values will be determined throughout the proof, and set
B.4
C.2
C.3
and Proposition C.I and let ōg. .o.,. > 0
and dT
8/2.
C and be constants as the theorem
24dT
TdT
2
N (log(T) + 1)² log 49
elog((T) + +1)² log
(14)
60
We will show inductively on t = O,..., T that for any n 2 N, it holds with probability at least Σラ= og that
1
1 160
2
log
2d
| -rro
o,
< min
C
VT E tt.....tt.
(15)
20

---

Note that in the case of t T, the probability of Eq. (15) holding becomes 1 - TT - = 85 2
1 - and the theorem follows.
For O the claim is trivial. Now, assume Eq. 15 holds for 1, and will prove it holds for t.
By Eq. (1)). for sufficiently large e and the assumption that n 2 N, the conditions of Proposition C.1
An are satisfied (if Eq. (15) is not < 4C2 as Proposition C.I requires, one can replace c by suitable larger constant that depends on the same parameters), so it implies that with probability least 1 - 6o/6 (using the union bound),
| / -0(0 < max
C1 +
log
24dt du
r 2
VT E (0,...,, 1}.
(16)
let A, denote the event that Eq. (15 and Eq. (16) indeed hold. By the union bound, P(Ai) Z
1 (1 - 1)60 - 0j 50/6 66
Consider some {0, to 1}.
((++1) is defined the MLE X(S)), which in particular means that it is stationary point of the log-likelihood function, so Σj= (l,(0(++1) = O. When
A occurs, the conditions of Lemma C.4 are satisfied, which gives us Taylor expansion for
Ej=o 1))
as
0= Vl, (o(T+))) j=0
Σ Ve, (B(T)) + (el((((z)) (B(T+1) - -B(1)) + - g(T)).
(17)
where Ry is matrix that satisfies by Eq. (16)
< (T + 1)K3
| -
CiK3 2
log
24dt 80
A
|R
(18)
2
2
(where again the last inequality assumes e is sufficiently large; if not, increase it).
By definition, for any т 0, ((7) is the MLE for (($+-1), so a stationary point satisfying
Σjóo Vl, (B(r)) O. For notational simplicity, let H, EE v²e,(0(1))) + R, then Eq. (17) simplifies to
0 Ver (B(ri) + H,(00++1) --((0r+1))g(())) g(r)).
(19)
To isolate 0(r+1) - ((F) we first show that H, is invertible. By Lemma 1 (6/(6)),
C.3 with probability at least
log 24dt
(2, (r 1 (B0))
max (06)) - 0001| S5
SC1
n
log
24dt
2d log
8.
SCI
TC'
7
log
24dt 60
< (C (C++))
(20)
21

---

where the second inequality follows from Eq. 155 and that ó; = 6u for r < T. Let A2 denote the even that Eq. (20) indeed satisfied for all T E .o...... 1}. which by the union bound satisfies P(Az) 2 1 6. When both A and A2 occur, using Weyl's inequality, Eq. (20) and Eq. (8) we have,
Amin (H,) 2Amin (IT + 1)Z (011))
Σ j=
V2 (2g((((7)) (r 1)I (000)
-1 II
-
log
24dt 60
Aa 2
2 (T [T+ A 4
2(++1)
(C1 (C)+
(21)
where the last inequality follows for sufficiently large e and the condition that 2 N. In particular, under these events, every H, is invertible so Eq. (19) implies
0(r+1) 0( = -- - (067))
Taking a telescopic sum, we obtain
[-1
[000) - 800)
g(r+1) (67)
Σ H Ver (0(7))
=
=
T I(0 -'Vr
VVe (ntT))
+
Σ
1-1
H
T + I(0001)-1) ) ve (0iT))
r=0
Ve, (p(:))
||
+
H 1
I(0(0))
- >
F|ve, (01-))
(22)
We will first employ an additional probabilistic bound for the gradient terms, By Lemma B. with probability at least 1 - 5./2 2
remains bound the terms in Eg 22
Σ
ver (01z7)
log
Ad d2
S SC
(23)
r=
Similarly, by Lemma B.2 and the union bound, it holds with probability at least 5 that
log
12dt 6
Ve, ( S C
|
Vr E to,...,t-1};
(24)
n
Let Ag denote the event that Eq. (23 and Eq. (24 are satisfied. Then letting A = A₁ n A2 n As
be the intersection of the desired events in this proof, we have P(A) 2 1 1 2 ビソ= 8, as desired.
Under the event A, from Eq. 18
20 21
and Lemma C.5 it holds for all r E {O....,t t 1) that
H
1
I(0(0))
H - (T + 1)Z(0(0)||
+
Amin ((T + ()(())) Amin (H2)
Σj=0 (2l2(0(T)) - (T + 1) (8(0)) + |
(T 1)dodmin (H,,)
4
C++C+ Ck3
log
2Ade Au
(25)
( 1)
2
22

---

Combining Eq. 24 Eq. 25 and the fact that Σi=1 7 S 1 + 1 1 dr I + log(t), we have for
suitable C" = C (K1,, K2. K3, Ao, r)
C' log 24dd 60
Σ
H
1
1(0000))11
Ve, (067)) 5
++11
n(T 1)
C" log
24dt
n
C' log
24dt du
(log(t) + 1)
1
VTi
S
n
where (+) follows whenever Ve 2 C' by the assumption that
24d7
2
N
(Iog(T)
1o
60
Using this and Eq.
23
Eq.
22
reduces to
|0() - 0100|
Ci Ao
log
2d
+
n
Taking suitable C' gives the desired bound. Lastly, for the induction we also need |10(0) - 000) s 5 This is indeed the case, taking sufficiently large
E Proof of Theorem 5.1
Construction Consider a fixed E N and let
a
(a)
Ya E R.
(26)
1
(12S)a2
Let L a [0 μ [2. f(a))}. Letting U denote the uniform distribution.
we define the family of distributions given by:
2 U((O,, 1J) -- U
2al) + a (U((2,)) + +((,)))UU((xx(())
2
Equivalently, letting TI denote the indicator function (where for any set A, (x) 1 if x E A and otherwise), the PDFs pa given by:
Po (x (x)
2 (x) + 2(1 2a. 21011-20(())
a
+ a (T222(00= + f(a) Tunat(os)()))))
1-a
+
11.11-20 - _ 111-2a.11(1) (x)
2 2(1 2a)
a (1 1 f(a))
[1 + T2..N
4
As such,
- -log Po()) = log
1
+ 2(1 2a)
a
10.1-20((x) - -log (2)
1
Ip (2)))
2
- log
1+ f(a))
1
-(nn+/(a)(())) - log
a 4p. (1)))) (27)
23

---

Lemma E.1. Under Construction 1 Pe is a TV-consistent family of distributions and g( exist.
Proof. Consider some dataset X S A' of size k N, there is finite number of values that po (a) can take, depending on the interval x lies in. This means,
.... (p.
0 E ++
< co.
As such, there must be some 0 that achieves this maximum. Consistency of the MLEs follows from Lemma F.10
Theorem 5.1. There exists θ S IR? and 0* E θ such that for any E N,, there is a TV-consistent
family of distributions (pol (that may depend on n) such that
1. with probability least
n
log (n)
TV (pox.Pow)
S
72
2. For some absolute constants c, C > O. it holds with probability at least that
TV (Uor,Paca) 2 C.
Proof. Consider the setting given by Construction with N = n and let ((0) = (a 0, (0 = 2). Existence of ((2) and ((2) as well as TV-consistency of Pe are given by Lemma E.
Because a (0) 0, Paço) is supported on [0, 1], meaning that E [0, 1] for every E [n]. As such,
(0) n
log
(po(x:)))
log
1
1 1
Σ
1011-2a1 (x: log
( (1)
2
Σ
11 20, (3F (0)
2
2(1 2(x)
71 1=1
2
n
i=1
1
1 1-a a
log
log
1 1 2a
|0.1 10,1-201 (x) (z:
2
1=1
where the last equality used log
1-a 201
log
(1 |
1=+)) )) log (+)
(1+ 1-
and that !0 E [0, 1].
Let maax = max xz (0) Note that whenever S I-mسu then every x; (0) is inside the interval
[0, 1 2n). Consequently, for all E
[0. Tuma ]. (0) log (1) log + 1-2a
1-a
Since
the function - log 1
1-a 2m
is monotonically decreasing a for all a < 2. (o()) is also
monotonically decreasing on [0, 1- As such, the MLE ((1) = (A p(() which minimizes (0) must satisfy
(1) Tmax
(28)
2
Consistency B(1; By Eq. 28 and Lemma F.8 for any > holds with probability least
1 8 that
a (1) 1 Fmax
log (1)
2n
24

---

Now using this and that Paco) = [z),) the total variation can be bounded as
1 TV (peros, Po(t)
/
1 / Poriy (a:)da:
Pocay (x)dr + 2 2
0
1
1-2a
2(1 20(1)/ ) dx
- (e(2)
1 f 1-20
1 1 2 dx
a1
1
0
(1
(1)
+ 7 aa
3
2
2
2(1
20(1i)
(1)
log
(1)
2n
Inconsistency of ((2) We will show that with constant probability, there will be some x;
(1) 32. Let denote the event that mmax
1. Since x!()) ~ U(O, 1]) i..d, we have
P(A) = 1
Conditioned on A, we have (2) I-Zmex 2 2n
so for each (1)
pgtt)
1
P
[2.3]
4
8n
Therefore, the probability that none of the x;
(1)
fall in [2 is at most
P Vi [n], 2!!" $ [2.3] A) 1 (1 (1- 8n
1 n
S 1e-1/8
Applying the law of total probability,
P Bi E [n]] such that 20!1) E (2,3) 2P(A) P (ii, x/ E [2,3] A) 2 1))
_ - e-18).
Thus, with constant probability, of the samples x (1) lies in [2.3]. The remainder of the proof is conditioned on this occurring. We will now show that the existence of E [2, implies that (2) will be far from a = O.
Now consider any [0, 1/10]. The function f (defined in Eq. 26 satisfics f(a) for any such As such, the term 1 %(a) is most for any [0, 1/10). Consequently, for any
[0, 1/10), the only term in Eq. 27 that is negative is the first one, meaning for any we have
Pog 2 - -log (1
1
2(1 2xx)
Since this bound is monotonically decreasing in a, have for any a E [0, 1/10),
4 (0)
log pa (2(2) 22g
1 1-a a
> -2
17
2(1 2a)
16
1=0
1=0
Now let ā = 1/8 and fix j such that there is least one sample in ja, p f(a)) (which we know
exists as there is some 2;(2) E (2. Plugging this 0 (ā, (i) into Eq. 27 using that the first term
25

---

is negative, and that there is at least one x;
(2)
3), we have:
1
1
1+ 1
1(3)) )
(0) S 2og
2 log
log
2
32
n
32
2 log
1
log
(1 + 32 (128) 2N -1)
32.
64
n
32
N
2 log
log (128) I -2log (2)
64
n
inf
L (0)
8: a51/10
(=0
As such, we have shown that (2) $ [0. 10.1/10). As result, the TV distance can be lower bounded as
2
1 1 1 a
TV (potos, Pg(M) >
Pa(a)
1
2
2(1 (2)
1
(2)
2
(2)
16
Proof of Theorem 5.2
Construction 2. Let
A A= [0, 1/4/Nu0 Bj. E eNs.Vj a₃ 0)=
namely, the set of all countable tuples in [0, 1/43 that have a finite number of non zero entries. For any a E A, let
86 has (2)
(1 a.))
II Okk 1 222,
T1st1-22,,
1=0
where we use the notational convention that II akk = 1.
To that this is valid PDF, first note that - ha (x) Ei az)) (ITZbbo)) consider any fixed M E N, then
Now
Σ 1=0
aaj))
II k=
akk
O
II a&
II k=0
Olk
k==
k=0
In particular, since Ork E [0, 1/4). this converges to as M - oc.
Let f: (2, - 1/2) be monotonically decreasing function that will be specified later in the proof. We also define for any 3 E [0, 1] and J E M.
1
g (x)
2J (.)) + 2f(J) 1--
The parameters 9 will consist of tuples (a,, 8, J. 8) where {0,1} is "selector" which tells us if we should choose the PDF ha or the PDF ga.J. Specifically, the parameter space is A x [0, (N) (1}) x {0, 1}. And the distributions Pe given by
(ha (ar) (g) =
8 0
(x(x)
26

---

Consider the ground truth distribution g to be such that
Pεto (x) = ho(x) = (0,1)
For each t, 8 an MLE given the data X(St), Existence will be guaranteed in Lemma F.
Regarding uniqueness, we do not use the fact that ((t) is the closest maximizer of the log likelihood to ((t-1), This is completely unimportant to the proof.
Lastly, for convenience, let
max (xlS0 nijj
+ - Br e nbi.
Mr.
0
else
where the maximum exists because the set is finite. words, Mr, denotes the maximal observed offset within the 'thh interval [j.j 1] up to time
Remark 1. Under Construction 2 for any X, if there is some non-negative integer j(z) such that x E [jxx),((x) + 2()())) then
((a)
on 1)
(() -1
log (ha (x)) = llog - 2az((z)
aj(a)
1+ =log log
Oj(a)
+
Σ
log (ax).
1 20(((s)
k=0
k=0
(29)
If no such (x) exists, then (x)) = 0 and log (ha())) undefined.
Lemma F.1. Under Construction Pe is TV-consistent family of distributions and (10) exist.
Proof. Consider some dataset X of size & and fix b E N such that X S [o, bj Following Remark 1 as see that for any æt, there is finite number of values that po(x)) can take, depending on the interval x lies in. This means,
well as the definition of ga,J, it is straightforward
(p. (z,,) E θ} < co.
As such, there must be some that achieves this maximum.
Note that any PDF in Pe has finite support. So w.l.o.g g we may assume that supp(Pacs) S [0, b] so that for any n, samples 2i,...,Iy from Pęc-) will all be [0, b).
In
and j 2 b, the parameters ay do not affect the log likelihood. Furthermore, since Ia.J = |
By Remark | for any 0
(I) the log likelihood is strictly decreasing As such, for the purpose of showing TVVconnistenccy, we may "discard" all values of J 2 b + 1 and all indices 2 J + 1 in a, treating θ as [o, 11 x [o, 1] x 2...., J.. +1+1 {0, 1}. This is closed and bounded subset of Euclidean space and is therefore compact. Furthermore, log Pato) (x) are uniformly bounded. So by Lemma F.10 the MLE consistent.
for VJ 2 6 ++ 1.
Lemma F.2. For any E NU (0) with s(t) 0. and any j [O) if M., > 0 then
((+1) (1 -M.)).
Proof. This is direct consequence of Remark Specifically, from Eq. (29) it follows that the log likelihood is strietly increasing in o (e+1), and is subject to the constraint that for all [.. +
holds that æ li.j + 2a, 2a,+1) ((+1)) In particular, this implies that any maximizer must satisfy
Mis = 1 fa
which is equivalent to what we needed to show.
The following lemma will be used throughout. It shows that for any interval [j.j j + 1], once there some 22(") E [i.j + 1], the values of My, and a (e+1) will remain the same in future iterations, as long the MLE takes the form ha.
27

---

Lemma F.3. Under Construction
2 for any j NN (0} if there exists some t, E N U (0} with = 0.
M > O, then Vt. > 1,, if (1s++)
1. Mrs My
(4/ (1++1) 2. 0) = aj 1 - (c,)
(1+1)
Proof. We prove the claim by induction t. The case of t t, is trivial.
Now, assume the claim holds for some time / Then a = aj (ta (1,+1) so following Remark | all
new samples X that are inside the interval [j.j + 1] must also be inside the interval
j.J + - 20, = j.j + 2a (1+1))
[j.j + Mc,,
where the last equality follows from Lemma F.2 Hence, no new sample in [i.j + 1] can exceed j + Me,,i which by the induction hypothesis was already the maximum. Thus
Mrj Mr Mej.h
and applying Lemma again gives
Mrj ((++1)
(L+1)
ag
= oj
2
This completes the induction.
Lemma F44. Under Construction 5 E (0, 1) let
let j.l E N and u = (1 (1-0))
(1)
IIk=0 0
10 > 0. For any
q ::= 2+ 2:=2+ e + log
let B denote the event that 3i E [n]] s.t x! E [i,j + 1] and for any E N, let Ag denote the event
that ii [n] x E [i.j 1|} 2 9. Then if s(t) = 0.
P (An ((2) 9l0) B S.
Proof. By construction, for any i E [n]. if s(t) = 0 (so that Paco) is of the form ha:0) it holds that
( E (b. + (())
o(t) = (1
ak = 0,
A==
+1] 1] and otherwise. Then conditioned on 0(1) o O(1), are i.i.d. Let b; be if x E [ Bernoulli random variables with parameter u, so applying Lemma F.7 completes the proof. O
Lemma F.S. Under Construction
2 let E ENu {0} and suppose that there exists some Lj such that (t, (tt++),
M3 Or..o.... M1,s O. Let (1 -1-ay a+f1
I ax
(1)+1) For any 8 (0, 1), letting
2 2 (3)
1++1==
1i 1 1
n 1 a (1)+1)
and
)
:= 2 2un + 2og
then it holds with probability least 8 that either (t))
for some E (t, + 1,..., t++1)
0 < Mr 5
4g
28

---

Proof. For any & E [0.....J]: 1.). by the assumptions that Mr, > 0, Lemma F.3
states that for all
I 12 4j. if g(1,+1,, s((+1) 0 then 0'k (++11 0% (1++1) [ (1 .k) > 0.
For any t, let by. be the bernoulli random variables that take the value of a; E [j + 1.j + 2 and 0 else. By Remark [ b are Ber(u) random variables. Let A, denote the event that Ab,, = 0.
For any t + 1, a are i.i.d. when conditioned on ((1), "ul......(.. and when s(t) 0, we get
P (A, 00) (0.0...,,0()))) = - u)" S exp (-nu),
Notice in particular that A, depends only on 8(1), ((+++1) and s(,+1), ,...,, st),, applying this argument inductively for each t E {t, + 1, ......,tj+1
,,+1] we get that
P P3te (tz + 1........411 s.t Maj+1 > 0) 2 1 cxp (-(t)-1 - 1)nu) 2 1 2
where the last inequality follows from the choice fJ+1 and the fact that aj+
(1,+1) 1/4
Mr,+1 > for some t E {ty + I...., ty+1} and s( ,..., g g(t.. 0 then Mi In summary, we have given the lower bound on Me needed for the lemma with probability at least 8/2.
By Lemma
F3
We move on the upper bound of the lemma. Suppose that there exists T which is the first timestep for which Mr 0 s(T)
1. If s(T) 1 we done, assume it is 0. Let B denote the event that 3i E [r]] s.t E [ [J++, 1,j + 2] and let A denote the event that
{i E [n]] x E [j +.j+ + 2}} I q. We want to bound Mr where we must condition on
the fact there is least one sample time that reached interval Recall that by Lemma E33
Ok = (1++1) for all < j. By Lemma R.4 using our choice of q we obtain
P (A (00) 000) B 2 4
Now suppose that this event indeed holds, so there are at most samples that land inside the interval [j + 1, 2] time Since nz(7)) are i.i.d. (when conditioned on ((1) ..... ,(r)) those that land in interval [ + 1,j + 2] are distributed within the interval as i.i.d. uniform random variables on
(0, 1 1.11-aa-p0 j+1
(+)
(which is included in [0, 1)). so letting 21 ..... 2g be i.idd uniform random variables
on [0, 1]. by Lemma F8 it holds that
8
Mr MrJ-151-
2
max 2,
S
2
4q
iElg]
4g
4'
So overall, the desired bounds hold with probability at least
Lemma F6. Under Construction for any J E N and for any E (0, 1), let
(Cloog
C' C'log (#)
L, =
8
where C. is some absolute constant. Then with probability at least 8, there exists some S t for which at least one of the following holds:
1. (t)
2. Mr,.J >
Proof. We begin by analyzing ((1), By Lemma
F2
Mo.0
0gg
2
29

---

By construction, Poto: is the uniform distribution on [0. 11, so Mo.o is the maximum of n i..d. standard uniform random variables. We thus use Lemma F.8 to bound Mo so holds with probability at least 1 5/2 that
log (3)
8 2n'
log (3)
Mo.0
1
0g
(1)
5
4n
2n
By Lemma
R3
this also means that for every /
0,
log Mr S 1
8
5
0g
(1+1)
log (3)
(30)
n
2n
4n
2n
If s11) we done. Assume not. We now move bounding aj for j > 0. Set to = 0, and for every E [J] we define
2 log (J
1=
4-11 1
(31)
IT: 0
(C) (1-1+1)
and
1-1
= 3 2c
log 5
A=
Note that the 9j defined here is slightly larger than the one defined in Lemma F.5 as (1-a111)))) 1-a0+11
(1++1)
By Lemma F.5 and I Lemma F33 for any j [J], if MG ..0..... Mr, 1J-1 U, with probability at least 1 - 6/ (2J). either there exists some S 1 with s(() = 1 or
Mty <
(32)
40
Note that by Lemma
F33 the same bound holds for any t 2 ty such that ((t)) ..... (1)) 0. It was
already shown for that Mr > 0, so for each j E [J], conditioning on fo...., f--1 holds
with probability at least jo/(2J) that either there is some / S Lj with (1)) = 1, or the bound on
MI given in Eq. (32 holds. Applying the union bound, this true for all j [J] with probability
at least 1 - 8 2.. From now suppose that for all S ty, (()
(otherwise we are done).
We now move to bounding the zj terms. Using the bounds on a (1)+1) from Eq. 30 we have
9j 33
e log (3)
O
(1)+
+ log
4
k=1
<3 e² log (2)
4
)
log
8
1 log
E [0, 1/4]. As such, by Lemma F2 and Lemma F33 it holds for all j E [J] that
for some suitable constant C² 0, where used that 0
(4y +1
8
(1+1) aj
(1 - M1,s) 2
>
8yj
8C" (1 + log (t))
So using this bound, Eq. 330.. and taking C = 88' 1). for any j E [J], the product can be bounded
5. (
j-1
(
5
5
n
1
c(11 log (+))
<
C(1 log (1)
k=0 0
30

---

Overall, Eq. (31 leads to
4J
t, 52J 2
(6++)) I Ok
==1
44//
C + log (
,
22 + log
8
6
The right-hand side is geometric series of the form Σj=1 r for > 2. Furthermore, for any geometric series satisfies Σj=1 r I 2nd. Using this, obtain
4J (T)
C' [ [ log (
1 22 + llg
Replacing C' by suitable larger constant C, this can be upper bounded as
C log (+)
t S
Theorem 5.2. Let db (0, - -0,
any strictly monotonically increasing function such
that lim,- 6(n) oc. Then there exists an absolute constant C > O, set Q 0* E O and TV-consistent family of distributions Pe (which depends on Φ), such that for any E (0, 1), n N,
holds with probability at least 1 å that
[ 8
log (7)
max (d(n), 1)
TV (pocrs, po-)
2
for some T T
Proof. TV-consistency of Pe and existence of ((t) for any / E N are given by Lemma F.1
For any J > and æ [0, 1]. {z) + S 1 but Patoy (x) So the TV distance between any (x) and Pato (x) is lower bounded as
1 f/20 (x) Pocos (x) dx 2 3 4 dr 3
1/11
2
As such, it suffices to show that there exists some time T such that with the desired probability, 93.J chosen the MLE.
As mentioned in Remark I there is some non-negative integer jxx) such that x E [i(x).((x)
1 20 (w)] then
j(x)
log (ha (x:) = loog 1 (j(())
+
Σ
log (ax)
(33)
1 205(a)
k=
If no such (xx exists, then log (ha (x:) is undefined. Note that the first term is increasing in ()(2))
and the second is negative (because I 1/4). As such,
ΣΣ log (hala(9) S nT log (1 1= t=1
4
< <n..
(34)
T
For the PDFs Ja,J, have
log (2J)
x [0, J| \ B,J 3
log (x))
log (2J) + log 2/(J)
x [J B,J - B f())
undefined
else
31

---

We now show that if T and J are such that T is sufficiently large and there is some sample in the interval [J 1, then a function of the form Ia.J will be the MLE. We will show that there exist some 98.f for which the log likelihood is bigger than for all PDFs of the form ha- The existence of the MIEE implies that there must some function of the form 9a.5 that the MILE
Now fix any J which will be specified later, and suppose momentarily that for some T E N, MT,J-1 (meaning that there exists some sample [J - JJ).
Let 83 == 1 M + [J) such that J - B3 - J 1 Mr.J- [J( and as such, by the
definition of Mr there must be some point in [J JJ.J B5 + f(J)]. Note that for any J E N,
since f assumed to satisfy S(J) S 2 holds that log 2/(J] 2 0, and thus
T
log (90,)(a(()) -I' -I' log())) + log 22(())
(35)
t=0 E1
In particular, to ensure the log likelihood of g3,.J is bigger than for any has it suffices for the right-hand side of Eq. (35) upper bound the right-hane side of Eq. (34). So we want:
log 2f(J))
2 nn (1 log(2 = n7 log (2eJ).
Taking the exponential of both sides and rearranging, the above is equivalent to
f(J) S 2(2
(36)
To that end, by Lemma F.6 for some absolute constant C' 0, letting
Va R, vbaa= = C log
( (
a-1
T (J),
holds with probability at least 1 - 6, that either (tt) = 1 for some t - T or MT.J-1 > 0. the first holds, we done, assume the latter.
Now, for any strictly monotonically increasing function é: 00) - (0, o0) with lima-too Φ(n) 00, let
S(J) (
(37)
Then to ensure Eq- (36) is satisfied, need (*))) 2 n. equivalently, J 2 max (b-))) 2) (where the 2 is because our domain includes only J > 2). In particular,
we take J == max (d(n)) 2). If = =, T= (2) Clog( ( and otherwise we can bound
T as
T (J} = & ([0-)((())))) = ((00)(((m)) +1) log (+)
- . (w (w-))))))= CClog (5)
Φ()).
8
In summary, we have shown that some timestep up T. it holds with probability least 1 that the TV distance is at least 3/8.
F1 Auxiliary Lemmas
Lemma F.7. For all E [n] let by Ber(u) be i.i.d. Bernoulli random variables with parameter Then for any E (0, 1).
n
Σ
2 2 2un + 2 2og
Σ
by 2
58.
32

---

Proof. Since b, are ii.d., using the inequality 1 - - exp( (x) for any 2,
P
by
2
= =1- u)" 2 exp( un))
By Chernoff's 's inequality (c.f. Vershynin 2018 and the chain rule of probability for any >>
un,
b 2 9
Σ by
21
==1 by 2 q)
S exp(-uu)
ewn 1
eun
P
+1 b 221 1 - exp( un)
un
(38)
where the last inequality uses that e depending on un. First, if un & 20 < 1, for any 2 2, Eq. 38 becomes
1-e- 1/2 for any 0. Now split into two cases
n
6² <
Σ
b 2
b
1 un
= eun
(un)
4 un S 8.
On the other hand, un > - taking 2 2 + un + 2 log (#) (which particular ensures that
edt
S 1/e and 2 2 + 2og (1)) ) Eq. (38) becomes
1
())
(
e² <
P
bj
2 4
ba 2
S
exp 2 - 2
48 2
1=1
1=1
In either case, q 2 2 e²un + 2 2og (3) suffices to ensure that desired bound.
Lemma F8. For E N let xı, For any > holds with probability least 5 that
U(0, 1)) be i.i.d. uniform O0, 1] random variables.
max TG
S 1
(39)
<E(M
2. For any > it holds with probability least
5 that
log (1) n2
max 2
(40)
RE(D]
As a result, for any > 0, it holds with probability at least - 6 that
log (2)
S max 24 S 1
(41)
(E(m)
2n
Proof. Since x; are i...d,, the CDF of maxie[n] 2, given by
IP
max
Px: $1 uw) (1)==11-))) u)
iEIn]
To prove Eq.
39 by Bernoulli's inequality (1 u)" 2 wn, it suffices take 8
To prove Eq.
40
we use the well known inequality u S exp(- u) to obtain
IP
max .; w
u)" > 1 exp( um))
Taking u log completes the proof.
Eq. 41 follows from Eq. 40 and Eq. (39) with 8/2 and the union bound.
33

---

The following lemma gives a version of the uniform law of large numbers that is suited for TV- consistency. We note that the conditions can be made even milder (c.f. (Tauchen 1985]), and are relatively similar to those of [Wald]]
1949
Redner 1981].
Lemma F.9 NNwey and McFadden 1994] Lemma 2.4). Lef ④ S Rdd be compact, ( θ let {zr)i=1 ~ Pacon be i..dd and let f(x, 8) be function which for any E measurable, continuous for almost all zs, and satisfies [f , 0)1 I ((x) for some function @ with Exwp,,, [o(x)] < oo.
Then (x, 0)] is continuous in and
[1 0) p (1, 0)] P, 0.
sup AEB
-=
Lemma E.1. Let θ C RR be compact, i θ and assume that for any 0 E O (og(po(x)) is measurable, continuous for almost all x, and satisfies (og(po(x)) S Φ(x) for some function 6(x) with Ex < cc. Then for any n, there exists an MLE ((") with respect to n i.i.d. samples from 0, it holds that
TV (pg. Pāco))
P, 0.
12-000
Proof. By Lemma F.9 for any Ó, > 0, there is some no such that for all ng.
sup (le(0) - Ex log (x+())) S { 2 2
(m) minimizes E, implying (((()) S ((O) and thus with probability at least
2 ((0(m) - ((0) 2 Ex-ps1 log (Poc.) (x)) + log (po(x))] - 2e² DKLL (po (p( 2ea.
Rearranging and using Pinsker's inequality,
TV (pp.) S V/ DKL (pa Pācn)) S
1 112 26 /02
G
Experiments
0.3
Samples
Samples n
0.25
100
Eror
20
20
0.10
0.05
60
80
100
100
Time Step
Time Step
(a) Exact MLE
(b) Optimized MLE
Figure I: MLE for one-dimensional Gaussian distribution.
There are by now many experiments the literature that support our finding from Theorem 4.] [Alemohammad et al. 2024 Gerstgrasser et al. 2025 Dey and Donoho 2024]. In particular, in those papers, the error does not increase much from iteration to iteration when synthetic data added gradually.
Rather than repeating experiments, we analyze the difference between exact MLE solutions and those that are obtained via optimization. To this end, we pick several families whose MLE has a known closed form. These include a Gaussian (where the parameters are the mean and std) in Fig. Exponential distribution in Fig. and family of Beta distributions with PDFs given by
34

---

025
Samples
0.226
Samples i
500
0.200
500
0.00
0.176
100
Eror
9.100
20
0.126
0.10
0.100
0.076
0.050
Time Step
Time Step
(a) Exact MLE
(b) Optimized MLE
Figure 2: MLE for one-dimensional Exponential distribution.
0.20
Samples
Samples
500
0.55
500
100
0.20
100
Emor
20
20
0.12
0.10
0.08
0.88
20
40
60
80
100
20
40
60
100
Time Step
Time Step
(a) Exact MLE
(b) Optimized MLE
Figure 3: MLE with respect to a Beta distribution family with PDFs given by p(x; 0) 0x for 0 and x E 10.
p(x;0) 0x for and I E (0, in Fig. The real parameters are (p 0, for the Gaussian and 0g = 1 for the other distributions. When optimizing numerically for the MLE, we use siipy.optimize.miinn minimize on the negative log likelihood to find the parameters. We opt for this built-in function to remove any uncertainty regarding the quality of the optimization code itself. be one of 20, 50, or 100. We run the iterative MLE algorithm
We take the number of samples as specified in the paper for up to T = 100. All values are averaged over 50 runs. The error is measured by the norm relative to the real parameters, meaning ||o(e) - 0ol| In all cases, the error at all timesteps similar the error time 1, our theory would suggest. Furthermore, we observe the model (non)-collapse behavior between the exact MLE and the optimized one to be similar.
We also consider various Ag going from 0.1 1 for the Beta distribution, where smaller &g corresponds to a neighborhood the parameters that are less smooth. In cases, we plot the ratio between the error time 7 to the error time 1. For 1, the error increases by factor of only 1.25 across 100 iterations, but for the "less smooth" 0p 0.1, the error increases by factor 3.27. These confirm that negative results hint
general phenomenon and support results.
8,
1.0
Relative Error
0.5
2.5
0.2
2.0
0.1
1.5
1.0
20
40
60
80
100
Time Step
Figure 4: MLE with respect to a Beta distribution family with PDFs given by p(a; 0) -- for 0 and x E (0, 1), for various choices real parameter 0g
35

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The abstract and introduction summarize the theorems and reference them. All claims are supported by the theorems.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the "Discussion" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
36

---

Justification:
The setting and notation are discussed in detail in the "Setting and Notation" section. All theorems are rigorously proved in the appendix. We also include proof sketches in the main paper to accompany the proofs in the appendix. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu http:///ggiddeeeeeouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
38

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with NeurlPS Code of Ethics http:///wwrgg...ooooaaa///ccaaaa
Answer: [Yes]
Justification: The research conforms in every respect the code of ethics, and do not foresee any negative implications of our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work is completely theoretical; We do address the potential societal impacts of model collapse (e.g. in the introduction), but we do expect any such impacts to be directly influenced by the current work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
39

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
40

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
41

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy (https: hateap///eerrpppc.cc///offeerers for what should or should not be described.
42


Reviews:
[{"review_id": "t9BG7zmdei", "reviewer": "Reviewer_8uXS", "summary": "This paper studies iterative MLE, a stylised setup in which parameters are updated first with MLE on the original dataset and subsequently with MLE on the original data plus synthetically generated data from the previous iteration. The main contributions are 1) sufficient conditions under which iterative MLE estimator converges to the true parameter in l2 norm, and 2) necessary conditions, failing which the estimates with synthetic data are bounded away from the true parameter, in terms of the TV distance between the probability distributions they parameterise. In particular, the negative examples are constructed such that MLE is consistent but iterative MLE is not.", "questions": "Questions:\n- Definition 3.1: \"Given a data set $X \\subset \\mathcal{X}$\" -- This notation would suggest that $X$ is a set of distinct elements. Perhaps the authors mean \"Let $X$ be an $\\mathcal{X}$-valued dataset\"?\n- Proof sketch of Thm 4.1: The role of Proposition C.1 is not clear in the current presentation of the proof sketch. My understanding of Proposition C.1 is that it plays role in ensuring $H_t$ is more or less stable / can be viewed as something with negligilble influence from $\\theta^t$  --- can the authors confirm if this is true?\n- Proof sketch of Thm 4.1: $|| \\sum_{t=0}^{T-1} \\frac{1}{t+1} \\nabla l_t (\\theta^{(t)}) \\| <= O( ... )$    I think the authors want to use $ = O( ...)$ and the authors probably missed \"with high probability\" in this statement?\n- Proof Sketch in 5.1: While I think the specific proof sketches can be shortened / partially moved to the appendix, having the explicit formula  of the distribution here is nice. However, I think the authors may want to consider having pictorial illustration of the distribution which makes it easier to see how the second iteration becomes problematic.\n- Proof Sketch in 5.2: My understanding is that most of the intuitions are similar to that of 5.1 and I suggest the authors considering substantially shortening this / moving some details to the appendix, in favour of a discussion of the broader consequences of these negative examples on more practically relevant setups.", "limitations": "Partially. The authors acknowlege briefly in the conclusion that the assumptions nin this work may not be the mildest possible and that more works need to be done to bridge the negative and positive results. However, I do think that even within the context of this work, to make this paper useful and interesting to the NeurIPS audience, it is important to bring out more interpretations and understanding of the currently presented negative results, which are limited to theroetical toy constructions. I am however very open to raise my scores if the authors can provide discussions or examples of this nature during the rebuttal.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: The paper is well-motivated. I find the necessary conditions (or rather, negative examples in Section 5) very interesting, and a useful complement to the understanding on model collapse. The observation that model collapse can happen even within a setup under MLE consistency is new to the best of my knowledge.\n\n\nWeaknesses: \n- There are some small expositions that are unclear to me but do not affect the overall flow of the paper. I have included them in the Questions section. In particular, I find that the proof sketches are not very well-presented (see Questions) and I wonder whether the authors could consider moving them to the appendix, both to gain more space in sketching the proofs in the appendix, and to have some space for elaborating on the consequences of the results.\n\n- The sufficient conditions are expected, though interesting nevertheless. The negative examples are rather specific constructions. While the negative examples are very interesting, it would be more useful if there are some general messages that can be extracted from them -- e.g. do they inform us about verifiable conditions that may be necessary  (not necessarily in a rigorous way) to prevent model collapse? Or do we expect the irregularity of these specific constructions to manifest in more concrete, real-life ML examples? These will strengthen the impact of the paper substantially, and would have much higher value than the elaboration of the specific proofs that go behind the negative examples.", "was_revised": true, "final_justification": "I was previously concerned of the significance / contribution of the paper, because I felt that the positive results were on a limited / standard theoretical model and rather expected, whereas the construction for the negative result seems very restrictive and uninformative. After the authors' clarification, I see that the negative results can be used as an example to understand how smoothness has different impacts on MLE consistency in a traditional setting vs consistency in an iterative synthetic data training setting, and I think this insight is valuable to the community. I also appreciate that the authors will add more explanations on the intuitions behind the negative examples, which will improve the paper's accessibility. Therefore I decide to raise my score to Accept.", "rebuttal": "We thank the reviewer for their constructive comments and their time. Below, we address all of the comments raised by the reviewer. If there are any concerns remaining after this response that are preventing the reviewer from raising their score, please let us know so that we can address them.\n\n- Regarding proof sketches: Thank you for pointing this out. We are happy to clarify them and add more discussion, elaborating on the consequences of the results. \n  - Role of Proposition C.1: To prove Theorem 4.1, it is important to show that when summing the differences in parameters at each time $t$, meaning $\\sum_{t=1}^T \\theta^{(t)} - \\theta^{(t-1)}$, we need the vectors in the sum to partially cancel out so as not to compound too quickly. We follow a two-step argument,  partially inspired by classical results in the statistics literature, where we first show that each $\\theta^{(t)}$ should be close enough to $\\theta^{(t-1)}$ to enable a Taylor expansion, and then analyze the expansions. Proposition C.1 provides the first part, ensuring that given enough samples, the parameters of the generation $t$ model $\\theta^{(t)}$ should well approximate those of the previous generation. Your intuition was correct, in that by doing this, $H_t$ can be well approximated by second-order terms, whose expected value can be expressed using the Fisher information matrix. This approximation allows us to, in turn, obtain good convergence rates for the sum. \n  - Probability in the Proof Sketch of Theorem 4.1: Indeed, the equation after line 240 is with high probability (where the explicit probability is written in the full proof). We will clarify this.\n  - Proof Sketch 5.1 and 5.2: Thanks for the suggestions! We agree that it makes sense to shorten the proof sketch of Theorem 5.2 in favor of a discussion of broader consequences. We will indeed make sure this discussion is present in the camera-ready version. We will also add pictorial illustrations to the paper to help clarify the constructions.\n\n- Interpretations and understanding the negative results: We are happy to further elaborate on this in the paper, and thank you for the suggestion. Indeed, Section 5 (the negative results) informs us about what conditions are needed to prevent model collapse. Even though the constructions are specific, they highlight more general phenomena that are needed for model collapse to occur or to be avoided. In particular, in our view, the main difference from Theorem 4.1 is the smoothness assumption. Theorem 5 crucially uses a highly non-smooth construction, in which slight perturbations of the parameters can induce huge differences in the resulting model, and we find it unlikely that a negative example would be possible without this behavior. Regarding real-life examples, even though neural networks can express very poorly behaved distributions, in many cases, they can indeed be shown to follow the smoothness assumptions, especially when using techniques like weight decay.\n\n- Definition 3.1: Thank you, this will be clarified.\n\n\n\nWe hope our response addresses the reviewer’s comments, and will be happy to further clarify any point if needed."}, {"review_id": "51y7jzXy9h", "reviewer": "Reviewer_aPhR", "summary": "Consider the following method for training a generative model $p_\\theta$:\n\na) Start with $n$ samples from the target distribution\n\nb) Repeat:\n\n-- Find the MLE model for all the data so far\n\n-- Generate $n$ more samples from this learned model\n\nQuestion: Does this procedure diverge from the target model?\n\nThe authors have two sets of results:\n\n1. Positive results.\n\nSuppose the family of generative models $p_\\theta$ is very smooth, in the sense that for any $\\theta$ near the target model, (i) the log density $\\log p_\\theta(x)$ has bounded first, second, and third derivatives, and (ii) the random variable $\\log p_\\theta(X)$ [for $X$ from the target distribution] is sub-Gaussian. Suppose also that the Fisher information is lower-bounded (in all directions) near the target model. Then:\n\n-- All models ever generated by the procedure above are O(1/sqrt{n}) close to the target model in total variation distance.\n\n2. Negative results.\n\nThe authors present a family of models that are hardly pathological (they are mixtures of uniform distributions over intervals of the real line) but do not satisfy the smoothness assumptions above. They show that although the first model generated by the learning process above is O(1/n)-close to the target distribution in total variation distance, *all* subsequent models are Omega(1)-far.\n\nIn this negative example, the target model is chosen based on the incremental sample size n. The authors also present another construction in which the target does not depend on n, and in which the models produced by the learning algorithm are Omega(1)-far from the target after f(n) iterations, where f(n) can be chosen to be as small as log log n.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a timely paper, given that the data used to train generative models increasingly includes data created by prior such models. There has been a line of previous work on the same topic, and indeed the learning model used here was introduced in earlier work by Gerstgrasser et al (\"Is model collapse inevitable?\"). The results here complement earlier work primarily by looking at a somewhat broader class of models as well as obtaining finite-sample rather than asymptotic bounds. In particular, the negative results in this setting are a striking contrast with earlier positive results for simpler families of models (linear models and exponential families).\n\nOverall, this is a worthwhile contribution to the literature on generative modeling.", "was_revised": true, "final_justification": "My assessment is unchanged after reading the other reviews and author feedback.", "rebuttal": "We thank the reviewer for their very positive feedback and their time!"}, {"review_id": "J5sSY5Qwwv", "reviewer": "Reviewer_a93t", "summary": "This paper proposes an abstraction for training generative models using synthetic data, formalized in Algorithm 1. Within this framework, the authors conduct a non-asymptotic analysis of distribution estimation under various assumptions. In particular, Section 4 demonstrates that iterative MLE remains consistent under Assumptions 1–3 (i.e., regularity conditions and smoothness), while Section 5 shows that without these assumptions, there exist distribution families for which iterative MLE yields a solution that deviates from the true distribution by a constant gap, indicating inconsistency.", "questions": "1. Given the first and second weaknesses mentioned above, could the authors either:\n\n  - propose variants of the iterative MLE algorithm that better reflect practical setups, or\n\n  - justify the current modeling assumptions with concrete real-world examples (e.g., specific use cases where the assumptions approximately hold)?\n\n2. Given the third weakness mentioned above,  I strongly recommend including simulations and, if possible, real-world experiments to illustrate the behavior of iterative MLE.\n\n3. Regarding the analysis of the theoretical results:\n\n  - Are the key distinctions between Theorems 4.1 and 5.1–5.2 primarily the presence or absence of Assumptions 1–3?\n\n  - If so, can the authors clarify why these assumptions are critical to ensuring consistency of iterative MLE (for instance, the smoothness might be technically necessary for controlling cumulative estimation errors across iterations)?", "limitations": "The Discussion section currently only addresses the limitation of weakening the assumptions. Given the practical concerns raised above, if the analysis of more realistic training setups is indeed technically intractable or out of scope, I recommend the authors explicitly acknowledge this as a limitation.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The theoretical content and proofs presented in the paper are based on analyses of MLE consistency and normality, which appear sound (though I have not verified the proof details).\n2. The logical progression from Section 4 to Section 5 is well-motivated, allowing me to follow the authors’ idea from \"classical assumptions yielding positive results\" to \"relaxed assumptions yielding negative results.\" \n3. The results in Section 5 are particularly interesting, as they highlight concrete scenarios in which iterative MLE fails under more general assumptions (compared to the strong smoothness assumptions in Section 4, which are hard to verify in complex neural networks). \n\n**Weaknesses**\n\nThe paper adopts a highly theoretical focus but does not sufficiently connect its analysis to practical generative modeling scenarios (while a purely technical contribution with novel theoretical insights would justify a strong theoretical significance, this paper does not appear to aim for such breakthroughs). Instead, its motivation is to inspire real-world insights. Thus, the practical relevance of the setup and assumptions becomes essential. My main concerns are:\n\n1. Iterative MLE algorithm: The proposed algorithm differs from how synthetic data is used in practice. In real-world settings, synthetic data rarely dominates the training set as much as assumed here (e.g., in foundation models, data is often filtered). Moreover, synthetic data is often deliberately designed rather than directly sampled from model distributions (e.g., instruction-following data, CoT data). These discrepancies reduce the practical relevance of the proposed framework.\n\n2. MLE: Modern large-scale generative models are typically not trained to full MLE convergence (e.g., current LLMs typically stop training while the loss curve is still declining). As such, the assumption of MLE may not reflect actual training dynamics.\n\n3. Experiments: The absence of experiments is a notable limitation. Simulations that validate the theoretical findings under the assumptions in Sections 4 and 5 are necessary, and real-world experiments would significantly enhance the paper's significance.\nThese concerns are the primary reasons for my current overall score. I would be willing to raise my score if they are adequately addressed.", "was_revised": true, "final_justification": "The authors have provided detailed clarifications and supporting materials, which have well addressed my concerns. I adjust my score to 4.", "rebuttal": "We thank the reviewer for their constructive comments and their time. Below, we address all of the comments raised by the reviewer. If there are any concerns remaining after this response that are preventing the reviewer from raising their score to acceptance, please let us know so that we can address them.\n\n- Theoretical paper: It is important to note that this is, in fact, a theoretical paper and should be judged as such. Of course, the theory aims to better understand a phenomenon that has gained much practical interest in recent years, but it is still a theoretical paper.\n\n- Practical Setting: Thank you for this comment; it is indeed important that the paper accurately conveys the practicality of this setting, as it has indeed become highly relevant recently. We will clarify this in the paper.\nThe scenario we are analyzing is synthetic data appearing inadvertently in datasets, due to the rapidly widening use of generative models (this is different than synthetic data that is intentionally designed and added to the training data). The iterative setting that we study (and variations of it) has already gained huge interest with many papers on model collapse over the past few years (e.g. [1-6]. A comprehensive list of papers on the topic is given in [5]), which have even led to news articles in the New York Times and Wall Street Journal. \nAs a simple example that demonstrates the importance of such a setting, consider an LLM with some cultural/linguistic biases. People using the LLM would increase the amount of web data with these biases, and it is not easy to filter unless huge chunks of LLM outputs are copy-pasted as-is. Even in such cases, filtering all LLM outputs on large datasets is unrealistic. It is therefore inevitable that these biases will be more present in future training sets and could thus potentially be amplified in future iterations.\n\n- Justification of the current modeling assumptions with concrete examples: \nThe modeling assumptions are very general and can capture standard neural network training. We will add to the paper a discussion showing how our theory captures practical settings. In particular, smoothness assumptions can be satisfied as long as the weights of the neural network don't diverge to infinity (indeed, in practice, they typically don't). Positive definiteness of the Fisher information matrix can also be shown to be satisfied in various settings, and there are entire papers devoted to analyzing the eigenvalues of this matrix (e.g. [7]). For the regularity conditions, a constant support (Assumption 1.C) is trivially satisfied in standard architectures, since the softmax function widely used to assign probabilities is always non-zero. The fact that the models are unique (Assumption 1.B) is made for convenience, and is not essential as discussed in the paper.\n\n- Simulations: We will be glad to add simulations to the paper in order to complement our theoretical findings. Note that we cannot directly display them in the rebuttal as per the new NeurIPS policy, but we will add these to the camera-ready version. We also note that there are already some experiments in the literature in our setting (e.g. [2]) which align with our theoretical results, and we will reference these as well.\n\n- Approximate convergence: The extension of the results to approximate stationary points is of course, possible. However, it is common to omit such extensions in generative modeling papers as they unnecessarily over-complicate the analysis, hiding the main points of the theorems. Analysis of the exact MLE is something that has been common for many decades and continues to interest many people in the community. In particular, this is standard in previous papers on model collapse  (e.g. [3], [6]). \n\n- Difference between theorems 4.1 and 5.1-5.2 - Yes, the reason for the difference is the assumptions. Theorem 4.1 follows assumptions that are standard in the MLE literature, and Theorems 5.1-5.2 complement this result by showing what can go wrong when these assumptions are violated.\nThe main assumption that is critical is indeed smoothness. There will typically be some difference between the real model and the learned one. If arbitrarily similar parameters can result in models that behave very differently, we believe it is unlikely that a theorem like 4.1 would be possible in general. To prevent uninteresting counter-examples to Theorem 4.1, we require a minimal assumption of consistency in Theorems 5.1-5.2. Finding any cases in which iterative MLE fails in the data-accumulating setting was an open question that interested many people in the model Collapse community, and that we address in our paper.\n\n\n\nWe hope our response addresses the reviewer’s comments, and will be happy to further clarify any point if needed.\n\n\n[1] AI models collapse when trained on recursively generated data, Shumailov et al. 2024\n\n[2] Self-Consuming Generative Models Go MAD, Alemohammad et al. 2023\n\n[3] On the stability of iterative retraining of generative models on their own data, Bertrand et al. 2024\n\n[4] Strong Model Collapse, Dohmatob et al. 2025\n\n[5] Position: Model collapse does not mean what you think, Schaeffer et al., 2025\n\n[6] Universality of the π2/6 Pathway in Avoiding Model Collapse, Dey and Donoho, 2024\n\n[7] The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network, Pennington and Worah 2019"}, {"review_id": "GnT6TM5DVf", "reviewer": "Reviewer_15P5", "summary": "- Theoretically studies model collapse for maximum likelihood estimation (MLE), focuses on setting where synthetic data is added to the original dataset\n- Under standard assumption, establish that model collapse can be avoided even as the fraction of real data vanishes", "questions": "- Introduction lines 42: Regarding Dey and Donoho: “Because in their setting, there is always a constant fraction of training data that is real.” Looking at Workflow 1 of Dey and Donoho, I do not believe this is true but I am possibly misunderstanding. Can the authors please clarify?", "limitations": "Yes. I think the authors could more explicitly state that the contributions are entirely theoretical and more work is necessary to connect to empirical phenomena.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strength: The paper is extremely well written: easy to follow, easy to understand\n- Strength: Theorem 4.1 is clean, elegant, meaningful\n- Weakness: Section 5 is very well motivated but the results are a bit odd. The distributions that appear feel carefully constructed and “unnatural”. It’s difficult to know what to make of the significance of Theorem 5.1 and Theorem 5.2 because I don’t know how general these distributions are (or what the salient properties are). In comparison, Theorem 4.1 feels much more general. One would like to know: are large scale deep generative models likely to produce model collapse? Does web-scale data / synthetic data have such distributional properties that would cause model collapse? I don't know after reading Section 5.\n- Weakness: The paper feels overall short. Section 3 is a lot of notation, preliminaries, algorithm 1, assumption 1, assumption 2, assumption 3. Most of Section 4 is a sketch of the proof (I like hearing the sketch, but I could imagine some may prefer it to be deferred to the appendix). Section 5 similarly spends a lot of time on sketching proofs.\n- (Semi)Weakness: Many papers concerning model collapse reach (softly) contradictory conclusions. For example, Strong Model Collapse argues \"Our results show that even the smallest fraction of synthetic data (e.g., as little as 1 per 1000) can still lead to model collapse: larger and larger training sets do not enhance performance.\" I think this paper could be improved by connecting more to prior work and explaining whether this paper's contributions and other papers' contributions are consistent and/or inconsistent, and explaining why different conclusions are reached.", "was_revised": true, "final_justification": "I thought the paper was good overall, and the authors have agreed to make a few minor adjustments (primarily towards the discussion and context) that I think will improve the paper.\n\nI still feel like the paper is on the shorter side. While its contents are high quality, they feel stretched out and the manuscript overall doesn't feel like a full conference paper.", "rebuttal": "We thank the reviewer for their constructive comments and their time. Below, we address all of the comments raised by the reviewer. If there are any concerns remaining after this response, please let us know so that we can address them.\n\n- \"are large scale deep generative models likely to produce model collapse?\": Generally speaking, neural networks do fit the setting of Theorem 4.1 (the positive result). For the regularity conditions, a constant support (Assumption 1.C) is trivially satisfied with standard architectures, since the softmax function, which is used to assign probabilities, is always non-zero. The fact that the models are unique (Assumption 1.B) is made for convenience, and is not essential, as discussed in the paper. Positive definiteness of the Fisher information matrix can also be shown to be satisfied in various settings, and there are entire papers devoted to analyzing the eigenvalues of this matrix (e.g. [1]). Smoothness assumptions can be satisfied in various settings, especially when using techniques such as weight decay. Of course, the exact bounds would depend on the architecture and the setting.\n\n- Regarding section 5 (the negative results): Even though the constructions are specific, they highlight more general phenomena that are needed for model collapse to occur or to be avoided. In particular, the main difference from Theorem 4.1 is the smoothness assumption: Theorem 5 crucially uses this non-smoothness, and we find it unlikely that a negative example would be possible under the smoothness assumption of Theorem 4.1.\n\n- Connecting to prior work: We agree that many model collapse papers appear to reach differing conclusions, largely caused by the differences in the settings. For example, the excellent “Strong Model Collapse” paper analyzes a particular version of linear regression, which is different from a generative setting as in this paper. Moreover, their setting is non-iterative, and they analyze the test error when the training data contains some samples that are labeled from a linear predictor drawn from a \"bad” synthetic distribution that differs from the real one. \nBeyond this paper, another notable difference between past works that can influence model collapse is whether or not synthetic data is *added* to the original dataset, as opposed to *replacing* it. We are happy to expand the discussion in the paper comparing these differences in finer detail.\n\n- Regarding Dey and Donoho: In their paper, the theorems hold in a setting where for each FIXED iteration $t$ (they denote this by $G$ in their paper), the number of samples per iteration $n$ tends to infinity. So the proportion of real data is $1/t$, which is independent of $n$. In contrast, our Theorem 4.1 in particular holds whenever $t$ is sub-exponential in $n$, and the fraction of real data given by $1/t$ may vanish.\n\nWe hope our response addresses the reviewer’s comments, and will be happy to further clarify any point if needed.\n\n[1] The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network, Pennington and Worah 2019"}]

Author rebuttal:

Meta-review: This paper offers a solid theoretical analysis of iterative MLE in the context of model collapse. It establishes sufficient conditions under which iterative MLE remains consistent, while also presenting carefully constructed counterexamples that demonstrate how inconsistency can arise in non-smooth settings. These negative results are not only technically interesting but also provide valuable insights into why smoothness plays a crucial role in preventing collapse, thereby enriching our understanding of generative model training on synthetic data. The work is clearly written and well motivated, and reviewers uniformly provided positive scores. I recommend acceptance as a poster.
Decision: Accept (poster)