Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression
Authors: Anvit Garg
Sohom Bhattacharya
Pragya Sur
Venue: ICLR 2026 Poster (2026)
Keywords: Model Collapse
High Dimensional Regression
Overparametrization
Primary area: learning theory
TL;DR: This paper establishes optimal ratios for mixing real data with synthetic data for high dimensional interpolation learning and ridge regression that optimizes prediction performance while preventing model collapse.

Abstract: Model collapse occurs when generative models degrade after repeatedly training on their own synthetic outputs. We study this effect in overparameterized linear regression in a setting where each iteration mixes fresh real labels with synthetic labels drawn from the model fitted in the previous iteration. We derive precise generalization error formulae for minimum-$\ell_2$-norm interpolation and ridge regression under this iterative scheme. Our analysis reveals intriguing properties of the optimal mixing weight that minimizes long-term prediction error and provably prevents model collapse. For instance, in the case of min-$\ell_2$-norm interpolation, we establish that the optimal real-data proportion converges to the reciprocal of the golden ratio for fairly general classes of covariate distributions. Previously, this property was known only for ordinary least squares, and additionally in low dimensions. For ridge regression, we further analyze two popular model classes -- the random-effects model and the spiked covariance model -- demonstrating how spectral geometry governs optimal weighting. In both cases, as well as for isotropic features, we uncover that the optimal mixing ratio should be at least one-half, reflecting the necessity of favoring real-data over synthetic. We study three additional settings: (i) where real data is fixed and fresh labels are not obtained at each iteration, (ii) where covariates vary across iterations but fresh real labels are available each time, and (iii) where covariates vary with time but only a fraction of them receive fresh real labels at each iteration. Across these diverse settings, we characterize when model collapse is inevitable and when synthetic data improves learning.
We validate our theoretical results with extensive simulations.

Published as a conference paper at ICLR 2026
PREVENTING MODEL COLLAPSE UNDER OVER- PARAMETRIZATION: OPTIMAL MIXING RATIOS FOR INTERPOLATION LEARNING AND RIDGE REGRESSION
Anvit Garg', Sohom Bhattacharya Pragya Sur!
Harvard University, Cambridge, MA 02138 University of Florida, Gainesville, FL 32611
ABSTRACT
Model collapse occurs when generative models degrade after repeatedly training their own synthetie outputs. We study this effeect in overparameterized linear regression in a setting where each iteration mixes fresh real labels with synthetic labels drawn from the model fitted in the previous iteration. We derive precise generalization eтror formulae for minimum-ez-none gression under this iterative scheme. Our analysis reveals intriguing properties of the optimal mixing weight that minimizes long-term prediction error and provably prevents model collapse. For instance, in the case of min-e2-norm interpolation. we establish that the optimal real-data proportion converges to the reciprocal of the golden ratio for fairly general classes of covariate distributions. Previously, this property was known only for ordinary least squares, and additionally in low dimensions. For ridge regression, we further analyze two popular model classes the random-effects model and the spiked covariance model demonstrating how spectral geometry governs optimal weighting. In both cases, as well as for isotropic features, uncover that the optimal mixing ratio should be at least one- half, reflecting the necessity of favoring real-data over synthetic. We study three additional settings: (i) where real data is fixed and fresh labels are not obtained each iteration, (ii) where covariates vary across iterations but fresh real labels are available each time, and (iii) where covariates vary with time but only a fraction of them receive fresh real labels at each iteration. Across these diverse settings, we characterize when model collapse inevitable and when synthetic data improves learning. We validate our theoretical results with extensive simulations.
interpolation and ridge re-
INTRODUCTION
Modern AI models are increasingly trained on their own synthetic outputs. However, this practice can lead to model collapse, where prediction performance degrades progressively with iterative training on Al generated synthetic data (Shumallov et al. (2024 2023). The phenomenon has been empirically observed across a wide array of settings Alemohammad et al. (2024) Bohaack & Farid (2023); [Briesch et al. (2023); Hataya et al. ):023): Martínez et al. (2023a b). For instance, Ale- mohammad et al. (2024); Shumailov al. 22024 2023) Martinez al. (2023a b) studied settings where a model is iteratively trained on synthetic data generated from the previously trained model, starting from original real data. They also studied settings where small fraction of the original real data is used in conjunction with the generated synthetic data. They demonstrated that model collapse occurs in both cases. These concerning observations motivated a surge recent work that studies model collapse rigorously and attempts to develop method that mitigate it where possible Bertrand et al. (2024); Shumailov et al. (2024): Dohmatob et al. (2024a by 2025); Feng et al. (2025); Dey & Donoho (2024); Gerstgrasser et al. (2024); Kazdan et al. 2024). However, previous work remains limited to dindeninniional settings or Gaussian features, creating a significant gap in our understanding, This paper breaks this barrier by addressing two fundamental questions: Can mixing fresh real data with synthetic outputs mitigate model collapse in overparametrized problems? What
"Corresponding author; pragyalfas, www.aaawwaa......

---

Published as a conference paper at ICLR 2026
is the optimal mixing ratio that minimizes prediction error in the long run? We provide rigorous swers for overparametrized linear regression, demonstrating how model collapse can be prevented under overparametrization.
Prior work has rigorously studied model collapse across several problem settings, though with im- portant limitations. For low-dimensional Gaussian distribution estimation, (Shumallov et al. (2024) shows how repeated use of synthetic data causes the estimated covariance matrix to collapse to zero almost surely, while the sample mean diverges. Crucially, they consider settings where the original real data is fixed and fresh real data is unavailable at each iteration. Similar results hold for linear regression for Gaussian features Dohmatob et al. (2024a). Recent work Dohmatob et al. (2024b 2025): Feng et al.] (2025) attributes collapse to change in scaling laws, with applications to text generation and Gaussian mixture problems. To mitigate model collapse, [Bertrand et al. (2024);
Gerstgrasser et al. (2024); Dey & Donoho (2024); Kazdan et al. (2024); He et al. (2025) develop mixing framework, where models are trained on a mixture of real and synthetic data each iteration. This approach prevents collapse by ensuring that the test error remains bounded even as the number of iterations increase. Crucially, these papers focus exclusively on low-dimensional problems.
interpolators (Section 3.1 and ridge regression (Section 3.2 and Section 3.3). Modern machine learning (ML) algorithms frequently exhibit implicit regularization Zhang & Yu 2005);[Soudee al. 2018); Gunasekar et al. (2018a)
We study two broad classes of prrimators-minimum---
with appropriate initialization and step sizes, algorithms converge to predictors that achieve remark- able generalization in overparameterized regimes. Implicit regularization has become cornerttone for understanding why overparameterized models generalize well Bartlett et al. (2020). Within this framework, min-norm interpolators have emerged as fundamental class of predictors that com- monly arise as implicitly regularized limits of gradient-
algorithms Bartlett et al. (2020); Deng 2022 : Gunasekar et al. 2018al by; Liang & Sur (2022); Montanari al. 2019p; Muthukumar ct al. et al. (220);; [Soudry et al. (2018); Zhang & Yu (2005); Wang et al. (2022); ;; Zhou et al. (2022).
At the same time, ridge regression represents a fundamental learning paradigm that has historically provided valuable insights into complex algorithms,, often illuminating phenomena observed in deep networks Hastic et al. (2022); : Patil et al. (2024). We study these popular classes of predictors for understanding model collapse under overparametrization.
Our work here is motivated by findings in Alemohammad et al. (2024), which suggest that among interesting possible combinations of real and synthetie data mixing, fresh data augmentation is the only scenario where model collapse can be prevented. We work under this fresh data augmentation framework, which describe below. This framework also appeared in He al. (2025), , though they focused on din-dimensional problems. Given covariates X, at iteration f, we generate a new hatch of real responses yt alongside synthetic responses y, produced using the estimator from iteration (t- 1). At iteration f, the estimator is formed using weighted mixture of these real and synthetic responses. Concretely, we define
B. (X
X
X (wy, + (1- (1-w)); w)
B ( X + mid) 'x (wyr + (1 w) Fr.2).
(1.I)
Above, denotes the pseudoinverse. It is well-known that B. -im B.x In the aforemen- tioned setting, our main contributions are as follows:
(i) Quantifying the generalization error. In overparametrized regime (stated precisely in Sec- tion we characterize the generalization error as -> Do for both the min-(2-norm interpolator B, (Thoorem 3.1 and the ridge estimator Pr, (Theorem 3.4p. Our results capture the precise depen- dence of the limiting risk on key problem parameters: the signal strength, feature covariance matrix, regularization level X, and mixing proportion w. Our work substantially advances the growing lit- erature on interpolation learning and high-dimensional ridge regression Montanari et al. (2019); Deng al. (2022); Liang & Sur (2022); Wang et al. (2022); Zhou et al. (2022); Bach (2024); Patil ct al. (2024): Mallinar et al. (2024); y Song et al. (2024) (see also data selection strategies Rezaci et al. (2055) which has previously not examined the impact of synthetic data on these learning problems.
(ii) Characterizing the optimal mixing ratio. We characterize the optimal weight on real labels that minimizes the long-term prediction error across different settings. For min-l2-norm interpola- tion. we establish that the asymptotic risk is uniquely minimized at "= = 1/4p (the reciprocal of the golden ratio) for any feature covariance matrix with bounded eigenvalues (see equation (2.3) This
2

---

Published as a conference paper at ICLR 2026
phenomenon previously proved only for maximum likelihood estimators in generalized linear models and nonparametric estimation in He et al. (2025). But the focus here was on low- diw-dmmnsiioal settings. For ridge regression, prove that the risk is log-convex and admits unique minimum at w* under several important scenarios: when the covariance is isotropic (Theorem (33)) or when the covariance follows spiked model (Section B.4.2). or when the signal follows random-effects model (Theorem (.2). Across all settings, we show w* 2 1/2, highlighting the necessity of weight- ing real-data more heavily than synthetic data. Such rigorous analysis of the mixing ratio provides concrete guidance for mitigating model collapse in overparametrized problems, complementing re- cent theoretical studies that were limited low-dimensional regression Gerstgrasser al. (2024);
Dey & Donoho (024).
Paper Structure The rest of the paper is structured as follows. Section formalizes the prob- lem setup and aata-augmentation framework. In Section B we state our theoretical results: Sec- tion 3.1 considers the min-f2-norm interpolator and Sections 3.2 and 3.3 consider ridge regression for isotropic and anisotropic covariates, respectively. Sections of our main results for random effects and spiked covariance models, respectively. Section 4 and presents simulations to complement our theoretical findings. We conclude with a discussion and outlining of future directions in Section 6 Proofs are deferred to the supplementary material.
and 3.4.2 3.4.2 applications
PROBLEM SETUP
We consider the fresh data augmentation framework He et al. (2025), but in the context of over- parametrized linear regression. Suppose we observe a dataset (y, X) from linear model, i.e.,
y = -x with y, E R" RP X R
(2.1)
We compute initial ridge estimator o using (y, X). For t we iteratively generate synthetic response vectors Yt, using f then augment these with fresh real responses yr. At each step, the next ridge estimator is computed using a mixture of the real and synthetic responses, yr and Yr, respectively, with mixing proportion (0, 1). The procedure is outlined in Algorithm ]
Algorithm Iterative ridge with real/synthetic data augmentation
1: Input: Dataset (y, X); regularization parameter À > 0; mixing proportion w E (0, 1).
2: Initialize: Po,, + [ X + - x
3: for 1 do
4: Generate real responses: yr <- XB + E..
5: Generate synthetic responses: Ne,A XBB-L.X E.
6: Update estimator:
B + (X ( (wy, (1 w))x.).
7: end for
To capture an overparametrized regime, we assume that p n with both diverging at a com- parable rate, i.e. p/n 7 > 1. This means we work with sequence of problem instances (y(n), X(n), (()),)()))221 with X(n) e g=xp(1), v(n), E RR" B(n) Re((), satisfying equation (2.1) and further assume that
lim [[(m)]]== b. e (0) 20
(2.2)
Below we suppress the dependence on for conciseness. This regime has seen incredible success in modern ML theory explaining phenomena observed for deep neural networks and other prac- tical algorithms Hastie et al. (2022); Adlam & Pennington| (2020); Montanari et al. (2019); Mei & Montanari (2022): Liang & Sur (2022); Cui et al. (2023): ); Paquette et al. (2024): Emrullah Ildiz et al. (2025); ; Lu al. (2025). ). The regime has also
enormous utility in high- -ighedimensional statisties, particularly for the development of new theory and methods in challenging contemporary inference
problems [Bean et al. (2013): El Karoui (2018): Donoho al. (2009): Bayati & Montanari (201))
3

---

Published as a conference paper at ICLR 2026
Wang et al. (2017); y Sur & Candès (2019); Sur et al. (2019) Fan 2002
Li & Sur (2023) Celentano
et al. (2023 : Jiang et al. (2025).
In the sequel, operate under the following assumptions the covariates and errors that commonly seen in random matrix theory Bai & Silverstein (2010).
Assumption 2.1. (i) The covariates satisfy X Zy where Z E Ruxp are random matrices whose entries Zip are independent random variables with zero mean and unit variance. We further assume that there exists constant T > 0 by which the v-th moment of each entry is bounded
for some u > 4, that is, [| S T We will also assume that E has bounded eigenvalues $1...... ppi
8p S81ST
(2.3)
(ii) The noises Et, Ez (defined precisely in Algorithm] 7 are assumed to have i.idd. entries with mean variance o², and bounded moments up to any order. That for any & > O, there exists a constant Co such that E.
| < Co. E
1 Cs.
Together with equation (2.2) Assumption (. implies that the signal-to to-soise ratio (SNR) SNR ==
b./o² remains finite as n, p + ensuring that we work under a non-trivial and interesting regime.
2.1 RISK
The out-of-sample prediction risk of estimator B (hereafter simply referred to as risk) new data point (yo, Xo) is defined as
R( : BI(xx
(2.4)
where for vector and matrix E, define liv v Ev. Note that this risk has o² dif- ference from the mean-squared prediction error for the new data point, which does not affect the relative performance and is therefore omitted. As defined, the risk involves expectation over both the randomness in the new test point (yo. Xo) and that in the noise variables. We define the risk con- ditional the feature matrix X, and risk characterization results are high probability statements over the randomness in the covariates. Despite this dependence on covariates, we use the notation R(B; B) since the context is clear. The risk admits bias-variance decomposition:
R(; B) = || X) - B113 + Tr Tr([ov(8
(2.5)
B(
v(8.9)
We next state our main results, which involve precise characterization of the risk of the estimator Bi, from Algorithm and B, defined by equation 3.3)
3
MAIN RESULTS
We begin by introducing two measures that feature crucially in the risk the estimators Br,, and B,- -E Syus Define the probability measures:
Let 0 up denote the eigenvectors of E: Σ
(x) =
(x) = B
Ukk 1 Su
(3.1)
1=1
Throughout we assume H. and Gp converge weakly to probability measures H and G respectively. For any z C/RR,, define m(2) be the solution
m(2) m +2= / 1+ dH.
(3.2)
Remark 3.1. Existence and uniqueness of m(2) is well- (c.f., Knowles & Yin 2017 Lemma 2.2)). Further, m(z) is the companion Stieltjes transform of the free convolution of H and M M,,,, where M is the Marchenko-Passe distribution with parameter Marčenko & Pastur 1967).
4

---

Published as a conference paper at ICLR 2026
3.1 MIN-(2-NORM INTERPOLATOR
In this section, we will analyze the behavior of R(B B), where we use
B, = [ X)(XX (wy, + (11-w)y)) Y: = -
(3.3)
in place of B Vr, in Algorithm This estimator is convex combination of the min-(2-norm interpolator computed on (X,y)) and (x) The generalization error of B, is characterized below.
Theorem 3.1 (Interpolator Risk), the setting of Section the risk of B.. defined by equation (.3 satisifes the following. For any w E (0,1), we have almost surely over the randomness in the covariates,
lim
lim R(B,;B) = = b.B
(3.4)
n-200 t1000
with c(w) = (w² + (1 - w)²) (2 w) and
m' ml(0)
m0)² ( m1
m' (0)
x
1,
(3.5)
m(0)
Moreover, the limiting risk is minimized at wn x- where 4 (1 + v5)/2 is the golden ratio.
Note Theorem 3.1 applies for any Σ obeying our assumptions. For the special case of isotropic features, i.e., Σ I, the limiting risk simplifies to
lim lim R(B;B) d'o(w) 1
by 61(1
) )
n-000 1-000
Proof of Theorem 3.] is available Appendix A
Effeet of number of iteration / and mixing parameter w: In equation (3.4), the first term corre- sponds to the variance while the second to the bias (recall definitions from equation (2.5). Note that the quantities v² and b coincides asymptotic variance and bias of min- norm interpolators
in overparametrized regression (Hastie et al. 2022 Theorem 2). In fact, the bias term B) is independent of both the iteration ł and mixing proportion w. The impact of mixing on the general- ization error arises through only the variance, captured by the function c(w), which minimized at w" sp- s 0.618. This golden-ratio weighting phenomenon was previously observed for host of models in now-dimensional settings He et al. (2025).
If w = O, then the generalization error R( + 3o as + oo, even for finite sample size n. This implies that training solely on synthetic data results in model collapse, as seen also by Shu- mailov et al. (2024): Dohmatob et al. (2024a) for low-dimensional learning problems. Moreover, if the mixing proportion
1/3, then by equation (A.2). we have the variance V B) decreases monotonically with t for any fixed n. Since w" = > 1/3, we observe that the generalization error also decreases monotonically for optimal mixing, thereby preventing model collapse.
Dynamic mixing: One might wonder whether the limiting generalization error can be further re- duced by selecting the mixing proportion w; adaptively at each generation to minimize R(B;; B) for any finite sample size n. We show Section C that the optimal choice ws in this sequential setup satisfies the recursion
+ W
w
Wp
1.
+ w
It follows immediately that wi is decreasing, so if one is free to adjust the mixing proportion at every generation, the optimal strategy places progressively more weight on the synthetic data. Moreover, using w w² (as defined in Theorem 3.1 in the long run, the asymptotic risk is the same whether one uses a fixed w* across all generations or adapts wi dynamically, as proven in Appendix C Our findings can also be seen empirically in Figure 3 (b), where we plot the risk vs number of iterations for different values of the weight w. The difference between w = pp and dynamic w is minimal and only visible t 1. Further, this difference is minimal compared to the difference between and to - The empirical risks were calculated at 200, p 400 with SNR = 1.
Next, study ridge regression. Here, both the variance and bias terms will depend on X, t,
5

---

Published as a conference paper at ICLR 2026
3.2 RIDGE REGRESSION: ISOTROPIC COVARIANCE ( = al)
In case of isotropic features, i.c., Σ al. the bias and variance of B.,, simplifies since sq a. Hence, by equation (3.1). G = H = da. where òg denote the Dirac probability measure at x. This
implies m(>), from equation (322). simplifies be the unique solution to
m(=) + 1 am(2)
a
(3.6)
We use the notations m. := ml- an(-\/w), m2 m= m(-1/(2 w))
Theorem 3.2 (Isotropic risk). Assume Σ = al and the setting of Section 2
For < w <
\ 0,
we have almost surely over the randomness in the covariates
lim lim R(B = 0² c(w)) VA + by Bx: n 1000
(3.7)
where c(w) = (w² + (1 - )//))//22222))2)) w) and
a/(1 + )21)²
w w) 7
B^ 1 1 10-7amm//0 + am1))²'
V^ 2(1 - w) amm1 am
(3.8)
+ amm
Moreover, the limiting risk is a log-convex function of w and has a unique minimizer.
Theorem 3.2 characterizes the precise asymptotic risk as function of the regularization parameter and the mixing proportion w. Furthermore, the proof of the result shows that the map w + c(w) V> log-convex, and the map w F Bx is decreasing and log-convex. Hence, we obtain that the limiting risk is log-convex in w, thereby admitting unique minimizer. Further, it can be shown that both VA and Bx are continuous functions of À and lim VA = V and lim BA = B, with V, B defined as in equation (.5). The proof of Theorem 3.2 shows that the bias at the i-th iterate
x B(B B) depends on both and w. unlike the bias of the min-la-norm interpolator. The following function of X.
result characterizes the behavior of the optimal mixing parameter
Theorem 3.3 (Isotropic Optimal Mixing). Under the assumptions of Theorem 3.2 let w (A) be the unique global minimizer of the limiting risk defined in Theorem 3.2 Then w (A) is increasing
function of SNR, a continuous function of X and satisfies (i) w (A) 00.,, 1}. (ii) us (A) + x 0.618 as X $ 0 and (iii) w" (X) + 11a as X 1 cc.
The optimal mixing parameter minimizing the asymptotic risk is always at least one-half, empha- sizing the importance of favoring real-data synthetic data. In Figure (b). show empirically that w² can be arbitrarily close to 0.5 for low SNR. Proofs of Theorem 3.2 and Theorem 3.3 are available in Appendix A and Appendix respecivelly
3.3 RIDGE REGRESSION; CORRELATED FEATURES
We now state our most general result for anisotropic covariates, which calculates the limiting gener- alization error of ridge regression for arbitrary measures H1p Gp (recall equation 3.1
Theorem 3.4 (Ridge risk under correlated covariates). In the setting of Section (0, 1), and > O. Define m m(-X/w),, m2 m(-2/(2 w)), where ml.) the unique solution to equation (32). Then almost surely over the randomness in the covariates, equation (.7) holds with
2
suppose w E
B₂ ( Uitmpppdo
(1-0
mz
m dG
dH (1 + M12)²
(3.9)
(2 - w)
x dHl
x
V> 2(1 w)) X
m2 dil
(3.10)
mi
The proof is deferred to Appendix A Theorem 3.4 is our most general result. shows for any
W E (0, 1) and any À > 0, : does not diverge even when increases. The result highlights the necessity of mixing real-data with synthetic outputs to mitigate model collapse. In its most general form the risk in equation (3.7) is involved to analyze. In the sequel, we study two popular models where the risk simplifies and the optimal mixing ratio be studied analytically.

---

Published as a conference paper at ICLR 2026
3.4 EXAMPLES
In this section, we study two popular classes of examples: (i) the random effects model (Section 3.4.1 I] and (ii) the spiked covariance model (Section 3.4.2 where structural assumptions the signal or the population covariance matrix render the limiting risk more tractable.
3.4.1 RANDOM EFFECTS MODEL
Modern applications, ranging from text classification to economics and genomics, are characterized by dense but weak signals spread across many coordinates (Joachims) 1998 Boyle al.) 2017 Yang et al. 2010; Shen & Xiu 2025). This setting is well captured by a random-effects model, where each feature contributes a small, independent effect, and provides simple yet sophisticated
framework for rigorously analyzing interesting nigh-dimensional predictors (Dobriban & Wager mandom-effects framework, we assume that each coordinate B, of the signal drawn i.i.d. with E,; and (ar(B:) = b,/p 0.
2018). Adopting
Proposition 3.1 (Ridge risk under Landom-Effect, Models). Suppose 3r ldd. (0.b./p). Assume the framework of Section and fix O w < 1, X > O. Let m(.) be the solution to equation (32)) Then almost surely over the randonness in the covariates,
and define f(2) m(-
equation 3.7 holds with
1 (
A
B^ = -F1N/))
2-
VA
(3.11)
2-30
Moreover, Bx is decreasing and log- log-convex and c(w)V, is log- log-convex.
With this characterization for the risk, we show the optimal mixing ratio satisfies the following.
Proposition 3.2. Under the setup of Theorem 3.1 (i) The generalization error has a unique min- imizer w*, (ii) w" E [0.5, 1), with w² + 1 as X t and wt x-1 with X O and (iii) w" increases with SNR.
We provide some context for the random effects model. Our main Theorem 3.4 makes it clear that in presence of general covariance matrices, the variance in the limiting risk (first term in RHS of equation (377) is determined by the spectrum of E, as captured through the limiting spectral distribution H. If the signal were deterministie, without additional assumptions, the bias (second term in the RHS of equation (.7h) would depend on how B aligns with the cigenbasis of Σ. This is captured through the measure G. But in random-effects setting, B lies in generic position relative to Σ, which simplifies both the limiting bias and variance making equation (3.7) tractable to analyze
function of w. Proofs of Propositions 3.1 and 3.2 are available in Appendix B
recent work Dohmatob et al. (2025) studies model collapse in Gaussian random-effects models but they consider setting where the real and synthetic data are generated from different distributions, subsequently pooling these datasets and studying when model collapse occurs. Crucially, DDohmatob
al. (2025) do utilize synthetic data generated from fitted model, unlike our setting. This leads to fundamental differences in our framework compared to theirs. Additionally, we consider broad class of undom-effects models, without requiring Gaussianity on the signals, and additionally provide guarantees for the optimal mixing ratio.
While Propositions 3.1 and 3.2 are stated under the random effects assumption, the conclusions continue to hold for a broader class of parameters (B, E). If the probability measures G and H, from equation (3.1) converge weakly to the same distribution, i.e., G = H, we have the same conclusions. In fact, prove in Lemma D.I that the random-effects assumption can be seen as a special case of G H. Other natural examples where G = H include the isotropic covariance case, Σ = al, where B is drawn uniformly random from p-dimensional sphere, or Σ is drawn from an orthogonally invariant ensemble.
Remark 3.2.
3.4.2 SPIKED COVARIANCE MODEL
For our second application, we consider a popular class covariance matrices the spiked co- variance model Birnbaum et al. 2013); Johnstone (200T); Johnstone & Onatski (2020). In the past

---

function of mixing weight for two different classes of covariance matrices: i. Σ I (Panel (a)) and ii Σg matrix of an AR(1) process. We vary 7 = 1,5, 2. The choice of the remaining parameters are as follows: sample size = 200, number of features p -yn, and number of iterations t = 5. To lid. Published as a conference paper at ICLR 2026
two decades, this covariance class has exciting applications in population genetics Patterson let al. (2006); Price et al. (2006), finance Knight et al. (2005); Ledoit & Wolf (2022) and signal processing Johnstone & Lu (2009); Wang et al. (2224). among others.
sov for some v E IRP, with ||l||2 = 1, and s O. The results below extend naturally to spiked models with multiple but finitely many spikes, but for simplicity, we study the problem for the case of a single spike. The limiting risk takes the following form.
Formally, we assume Σ =
Proposition 3.3. In the setting of Section 2 assume that Σ = I I+ svv" and the signal takes the form B Or + vI - 02x while satisfying equation (.2). with v u = 0 and || 1. If (() + O.. then we have almost surely over the randomness of covariates that equation 3.7
holds with V> same as in Theorem 3.2 and
B₂ BA= ( ( m1(1 + 8))2 + (1- (1 mz)a
1 1+s
m2)² (1-
mi
Further, the limiting risk is uniquely minimized at a w² satisfying the conclusions of Thhorem 3.3
Note that, 0, * O, then G + H. Hence, this spiked matrix case differs fundamentally from the settings discussed in Remark 3.2 and the proof of Proposition [33 does not follow directly from the proof of Proposition 3.2 Proof Proposition 3.3 is available in Appendix B
SIMULATIONS
We conduct numerical experiments complement our theoretical findings. For all plots, we gen- erate X = // where Zij (N, 1) and vary Σ across different plots. We display both the theoretical risk predicted by our formulae (solid lines) and corresponding empirical estimates (x markers). For the first two plots, the empirical risks calculated by averaging over 100 runs.
Risk of min- min-l2-norm norm Interpolator In Figure [ we plot the asymptotic generalization error of the min-f2-norm interpolator
|i-j (Panel (b)), corresponding the correlation
generate B, we first simulate 8 with 3g ~ Bern(0.1). Set B = B/11 which yields b = 1.
We observe that the empirical risk matches with its theoretical counterpart even for moderate sample size. Further, the generalization eтror is always minimized at w- = = 0.618 (dashed vertical line), consistent with Theorem 3.1 Panel (c) shows the risk of the min-é2-nome interpolator function of iteration 1. We have computed the risk at optimal mixing weight w y and Σ = I. We observe that both theoretical and empirical risks stabilize after only few iterations.
Optimal weights in ridge regression In Figure 2 we plot the optimal weight w" as function of
X. We set n = 200, p = -n,, and t = 5, and vary 7 = 1.2, 2, 4. Figure 2 (a) considers isotropic covariance Σ = I with high noise variance o² = 64. The plot demonstrates that for low SNR, w² can be arbitrarily close to 0.5. Further, w (A) is neither monotone, nor convex as a function of with w² (A) 6p as À 0+.
......,,. Figure (b) corresponds to the spiked covariance model Σ I SejeT, where e Ep denote
canonical basis vectors. We set 31 0.5 and B2.p v1 - 0.5 x /||]3]| where 3. iid Bern (0.25).
This implies that ||
and 6. 0.5, where 0, defined as in Proposition 3.3 The plot shows
that for large regularization parameter X, we have w = 1, consistent with Proposition
3.3 Figure
2 (c) plots the risk of Be, with の² 1, -1 We take Σ (1 p/vp)/ (p/vr) with
p 1/2, and B4 lid pl.. The empirical risk is computed with 0² 1,t= 10, n 400.
Note that Σ does not satisfy bounded eigenvalue condition Assumption 2.1 as its largest eigenvalue
2 CD for some C > Still, the generalization error is accurately predicted by Proposition 3.1 as
long as 3 lies in generic position relative to Σ. This demonstrates the robustness of our theoretical findings. Figure 3 (a) plots theoretical values of w" as a function of y for different values of À (we
fixed SNR 1/4). As expected, when À is close to O, w² x -1 However, À increases, the dependence on 7 becomes more visible. This dependence of the optimal weight on y is important to note, since it illustrates the importance of studying our high-dimensional framework.
8

---

Published as a conference paper at ICLR 2026
(a) Isotropic
AR(1)
(c) Isotropic
4
4
3
3
1.25
2
2
1
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
13 17 Iteration (t)
Weight on the real data (w)
Weight on the real data (w)
y= 1.5
y=
Y 3.0
Empirical estimates
Figure| 1: Generalization error of min min-l2-norm interpolator as function of weight w (Panel (a) and (b)) and iterations (Panel (c)) for different values of T.. Panel (a) considers isotropic covariance = I and panel (b) considers anisotropic with Σ,j = 2 2-11--11 -- which corresponds covariance matrix of AR(1) model. In panes (a) and (b), the risk is minimized at w² = 1/p, as proved by Theorem 3.1
Panel (c) shows that both empirical and theoretical risks stabilize in few iterations.
(a) Isotropic
(b) Spiked
(c) Equicorrelated
1.0
2.0
0.8
Risk
1.5
1.0
0.1
10
0.1
1
10
0.1
10
À
À
y= 1.2
y=2 Y=
Empirical estimates
Figure 2: In (a) and (b), we plot the optimal mixing weight w² as a function of for different values of y and classes of covariance matrices: Panel (a) considers Σ I with high noise o² = 64, demonstrating w" can be close to 0.5 for low SNR. Panel (b) plots w* for the spiked covariance matrix showing w* for large Panel (c) plots the generalization error of À for Σ equicorrelated matrix. Here, empirical risks align with theoretical predictions given by Proposition 3.I. though Σ violates Assumption 2.1 illustrating the robustness of our results.
function
5 BEYOND FRESH DATA AUGMENTATION
In this section, we provide rigorous evidence when and how model collapse can be prevented beyond the fresh data augmentation framework equation [..pp. In the interest of space, we focus on the min-é2 norm interpolator, but our results can be extended for ridge regression.
No fresh real data First, we consider the setup where fresh real data is unavailable beyond the original data (y, X). Therefore, one augments the initial labels y with synthetic labels as follows
D == (X (wy + (1 w)..) where y, xB + E
(5.1)
We assume B0¹ is the min-ly-norm interpolator calculated using the original (X,y).
(r)
Theorem 5.1. The risk of defined by equation (5.7), satisfies the following, For any (0, 1), we have almost surely over the randomness the covariates,
lim lim R(B (1) :B) -8)=0= w(2 w)
V b.B
(5.2)
with V, B defined as in equation 3.5 The limiting risk is minimized at w² =
In Theorem 5.I V. do not depend on the weight, thus the asymptotic risk monotonically de- creasing in w, yielding w² 1. Thus, the asymptotic risk is minimized when the weight the

---

Published as a conference paper at ICLR 2026
0.9
y- 1.5
1-10
y- 2.0
0.8
2=20
1.4
3.0
4=4.0
Risk
Risk
W - Dyn.
s 0.7
- 4
1.2
0.6
10
10
10
10
20
0.25
0.50 0.75
1.00
Number of Retraining Steps (t)
Weight on the real data (w)
Figure 3: In (a), we plot the optimal w* for ridge regression as a function of 7. In (6), we plot the empirical risk for the min-norm interpolator to demonstrate the dynamic mixing scenario described in Section 3.1 Note that, the risk under dynamic mixing and g heavily overlap for onward. In (c), we plot the empirical (the points) & theoretical (the solid lines) risks for the no fresh data augmentation case described by equation (5.1) and Theorem 55.1 Note the risk is optimized at w* 1. Throughout E
real data is 1, suggesting adding synthetic data following this mechanism fails improve perfor- mance. This can be seen empirically in Figure (c), where we plot the empirical risk as well as the theoretical risk. Here 500, SNR 1 and we consider 50 iterations.
Time-varying covariates Next, we consider the setup where different covariates are used to train different iterations of the model. Formally, for t 2 as synthetic labels: yt +- X, + ēt. The estimator for 1-th iteration is defined
have real labels: yr <- X, E, well
B = X X.) X (wy: + (1 w)
(5.3)
(1) We provide the precise generalization error of Be in Theorem E. (for simplicity we assume Σ = I, see also Remark E.I regarding limit orders) and characterize the unique optimal weight which the risk is minimized. Unlike Theorem 3.1 both the bias and variance depend on w.
Time-varying covariates with fresh labels in fraction Finally, we consider a setting where at each iteration, fresh labels are available only for a fraction of covariates. Since all covariates do not both see real and synthetic labels, instead of our mixing framework from equation (1.1). we consider pooling the datasets with real and synthetic labels. Formally, for 2 1, we observe real responses:
Y+ X, e, and synthetic responses: yt
X. + ē.. Define
(a)
B
X
In Appendix E.3 (specifically Theorem E+ we show that there exist ranges of values y where preventing model collapse is not possible in this setting.
6 DISCUSSION
We provide a rigorous analysis of model collapse under overparametrization for linear models. As overarching theme, demonstrate how mixing real-data with synthetic outputs mitigates model collapse, and identify optimal mixing ratios that minimize prediction error in this context. As promising direction, understanding how model collapse affects interpolators in other ép ge- ometries would be crucial. Such interpolators arise as implicit regularized limits of popular algo- rithms Gunasekar al. (2018b) and typically require techniques beyond random matrix theory- critical tool employed in our analysis. Additionally, approaches that extend linear arguments to non-linear high-dimensional problems (c.f., Sur Candès (2019); Hu Lu (2022) should enable our qualitative conclusions to generalize to structured non-linear models. This includes generalized linear models, single-index models, and even non- non-parametric models through parametric-tu to-non- parametric equivalence techniques introduced in Lahiry & Sur (2023). Finally, extending our results to more complex architectures remains an important challenge. One promising approach involves studying multi-index models and their sequential variants, which capture classes of neural networks and transformer architectures Troiani et al.) (2024) 3025); Cuil (2025). Investigating model collapse and mitigation strategies in these contexts presents exciting avenue for future research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ben Adlam and Jeffrey Pennington. Understanding double descent requires fine-grained bias- variance decomposition. Advances neural information processing systems, 33:1 33:11022-11032, 2020.
Sina Alemohammad, Josue Casco-Rodriguee Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaci, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go mad. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 53581-53608, 2024. URL httpp:::ppocceeed wtt.s/s............eeeeeeeeeeeee cc/paper http://teeeeeeeeeeeepeeeeeeeeeee httppppee//22//1eee/ (etee//666eee551100cccccccllaaa
iclr.
Paper Confereccne
Francis Bach. High-dimensional analysis of double descent for linear regression with random pro- jections. SIAM Journal on Mathematics of Data Science, 6(1):26-500 2024.
Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117749))0003333007) 2020.
Mohsen Bayati and Andrea Montanari. The lasso risk for gaussian matrices. IEEE Transactions Information Theory. 58(4):1997-2011 2011.
Derek Bean, Peter Bickel, Noureddine El Karoui, and Bin Yu. Optimal m-estimation in high- dimensional regression. Proceedings of the National Academy of Sciences, 110(3)):14563- 14568, 2013.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining generative models on their own data. In ICLR, 2024.
Aharon Birnbaum, Iain M Johnstone, Boaz Nadler, and Debashis Paul. Minimax bounds for sparse pea with noisy high-dimensional data. Annals of statistics, 41(3):1055, 2013.
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse. arXiv e- prints, pp. arXiv-2311, 2023.
Evan A Boyle, Yang Li, and Jonathan K Pritchard. An expanded view of complex traits: from polygenic omnigenic. Cell, 169(7):1177-1188 2017.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the colf-consuming training loop. CoRR, 2023.
Michael Celentano, Andrea Montanari, and Yuting Wei. The lasso with general gaussian designs with applications to hypothesis testing. The Annals of Statistics, 51(5):2194-2) 222000 2023.
Hugo Cui. High-dimensional learning of narrow neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2025(2).023422, 2025.
Hugo Cui, Florent Krzakala, and Lenka Zdeborová. Bayes-optimal learning of deep random net- works of extensive-width. In International Conference on Machine Learning, pp. 6468-6521.
PMLR, 2023.
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2): 435-495, 2022.
Apratim Dey and David Donoho. Universality of the 2²/6 pathway in avoiding model collapse. arXiv preprint 0rX1v22410.22812, 2024.
Edgar Dobriban and Stefan Wager. High-dimensions asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46( (1:2277772 2018.
11

---

Published as a conference paper at ICLR 2026
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The of regres- sion. Advances in Neural Information Processing Systems, 01146999-470115 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: model collapse as a change of scaling laws. Proceedings of the 41st International Conference Machine Learning. ICML''2. JMLR.org, 2024b.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp- 15656-15691, 2025. icl wct.ppaceppppcc.eppppeeee....ee. fills//paaer/2222/1111 2BAAGC22DEE610221552233300--0002 Paper -apee-rooeotteeeeeeppppo
URL
David I. Donoho, Arian Maleki, and Andrea Montanari. Message-passing algorithms for com- pressed sensing, Proceedings of the National Academy of Sciences, 106(45):19114-18919 2009.
Noureddine El Karoui. On the impact of predictor geometry the performance high- dimensional ridge-regularized generalized robust regression estimators. Probability Theory and Related Fields, 170(1):95-175, 2018.
M Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. ligh--iimensooal analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In 13th International Conference on Learning Representations, 2025.
Zhou Fan. Approximate message passing algorithms for rotationally invariant matrices. The Annals of Statistics, 50(1):197-224 2022.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification. In The Thirteenth International Conference on Learning Representations, 2025.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse evitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp. 1832- 1841. PMLR. 2018a.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent linear convolutional networks. Advances in neural information processing systems, 31, 2018b.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 20555, 2023.
Hengzhi He, Shirong Xu, and Guang Cheng. Golden ratio weighting prevents model collapse. arXiv preprint arXiv:2502. 18049, 2025.
Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932-196) 2022.
Kuanhao Jiang, Rajarshi Mukherjee, Subhabrata Sen, and Pragya Sur. A new central limit theorem for the augmented ipw estimator: Variance inflation, cross-fit fit covariance and beyond. The Annals of Statistics, 53(2):647-675 2025.
Thorsten Joachims. Text categorization with support vector machines: Learning with many relevant features. In European conference on machine learning, pp. 137-142, Springer, 1998.
12

---

Published as a conference paper at ICLR 2026
lain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis.
The Annals of statistics, 29(2):295 327, 2001.
lain M Johnstone and Arthur Yu Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486):682 69006366266669,- 2009.
lain M Johnstone and Alexei Onatski. Testing in high-dimensional spiked models. The Annals of Statistics, 48(3), 2020.
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David Donoho, and Sanmi Koyejo. Collapse or thrive: Perils and promises of synthetic data in a self- generating world. In Forty-second International Conference on Machine Learning. 2024.
John Knight, Stephen Satchell, and Chris Adcock. Linear factor models in finance. Elsevier, 2005.
Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169(1):57-3352 2017.
Samriddha Lahiry and Pragya Sur. Universality in block dependent linear models with applications to nonparametric regression. arXiv preprint arXiv:2401.00344, 2023.
Olivier Ledoit and Michael Wolf. The power of (non-) linear shrinking: A review and guide to covariance matrix estimation. Journal of Financial Econometrics, 20(1):187-218, 2022.
Yufan Li and Pragya Sur. Spectrum-aware debiasing: modern inference framework with applica- tions to principal components regression. arXiv preprint (xXiv:2309.078111 2023.
Tengyuan Liang and Pragya Sur. A precise high-dimensional asymptotic theory for boosting and minimum-//-norr interpolated classifiers. The Annals of Statistics, 50(3), 2022.
Yue M Lu, Mary Letey, Jacob A Zavatone- Anindita Maiti, and Cengiz Pehlevan. Asymp- totic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, 122(28)::202259912, 2025.
Neil Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. arXiv preprint 2rXiv:2404.00522 2024.
Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 144):557, 1967.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Mare Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint arXiv:2303.0 2023a.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the inter- net. In International Workshop on Epistemic Uncertainty in Artificial Intelligence, pp. 59-73. Springer, 2023b.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max- margin linear classifiers: Benign overfitting and high dimensional asymptotics the over- parametrized regime. The Annals of Statisties (to appear), 2019.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute- optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459- 16537, 2024.
13

---

Published as a conference paper at ICLR 2026
Pratik Patil, Jin-Hong Du, and Ryan Tibshirani. Optimal ridge regularization for cuttof-diisributtion prediction, In Proceedings of the 41st International Conference on Machine Learning, pp. 39908- 39954, 2024.
Nick Patterson, Alkes Price, and David Reich. Population structure and eigenanalysis. PLoS genetics, 199 2006.
Alkes L Price, Nick
Patterson, Robert M Plenge, Michael E Weinblatt, Nancy A Shadick, and David Reich. Principal components analysis corrects for stratification in genome- association studies. Nature genetics, 38(8):904-909, 2006.
Parham Rezaei, Filip Kovačević, Francesco Locatello, and Marco Mondelli. High-dimensional anal- ysis of synthetic data selection. In EurlPS 2025 Workshop on Principles of Generative Modeling (PriGM), 2025.
René I. Schilling, Renming Song, and Zoran Vondraček. Bernstein functions: theory and applica- fions. Walter de Gruyter, 2009.
Zhouyu Shen and Dacheng Xiu. Can machines learn weak signals? Technical report, National Bureau of Economic Research, 2025.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023.
Ilia Shumailov,, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(80222 2024.
Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpolators in transfer learning. arXiv preprint arXiv:2406. 13944, 2024.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data, Journal of Machine Learning Research, 19(70): 2018.
2018.
Pragya Sur and Emmanuel Candès. A modern maximum-likelihood theory for high high-dimensional logistic regression. Proceedings of the National Academy of Sciences, 2019.
Pragya Sur, Yuxin Chen, and Emmanuel J Candès. The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled chi-square. Probability theory and related fields, 175(1):487-558, 2019.
Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, and Flo- rent Krzakala. Fundamental computational limits of weak learnability in high-dimensional multi- index models. arXiv preprint arXiv:2405. 15480, 2024.
Emanuele Troiani, Hugo Cui, Yatin Dandi, Florent Krzakala, and Lenka Zdeborová. Fundamental limits of learning sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholds. arXiv preprint 2rXiv:2502.0090 2025.
Guillaume Wang, Konstantin Donhauser, and Fanny Yang. Tight bounds for minimum 41-norm interpolation of noisy data. In International Conference Artificial Intelligence and Statisties, pp. 10572-10602. PMLR, 2022.
Shuaiwen Wang. Haolei Weng, and Arian Maleki. Which bridge estimator is optimal for variable selection? arXiv preprint arXiv:1705.00 170.00861, 2017.
Zhichao Wang, Denny Wu, and Zhou Fan. Nonlinear spiked covariance matrices and signal propa- gation in deep neural networks. In The Thirty Seventh Annual Conference on Learning Theory, pp. 4891-4957, PMLR, 2024.
14

---

Published as a conference paper at ICLR 2026
Jian Yang, Beben Benyamin, Brian P McEvoy, Scott Gordon, Anjali K Henders, Dale R Nyholt, Pamela A Madden, Andrew Heath, Nicholas G Martin, Grant W Montgomery, et al. Common smps explain large proportion of the heritability for human height. Nature geneties, 42(7):565- 569, 2010.
Tong Zhang and Bin Yu. Boosting with early stopping: Convergence and consistency. The Annals of Statistics, 33(4):1538, 2005.
Lijia Zhou, Frederic Koehler, Pragya Sur, Danica Sutherland, and Nati Srebro. non-asymptotic moreau envelope theory for high-dimensional generalized linear models. Advances in Neural Information Processing Systems, 25:21286-21299, 2022.
15

---

Published as a conference paper at ICLR 2026
Supplementary Material
A PROOF OF THEOREM 3.1 THEOREM 3.2, AND THEOREM 3.4
A.1 NOTATIONS
Our objective is to derive the asymptotic generalization error of Bi,, given by Algorithm and B, We define the scaled sample covariance matrix and its resolvent as
defined by equation (3.3)
M x
A = (M + n
Note that A and M are simultaneously diagonalizable and thus commute with each other. If X = 0, we use the notation A (XTx)* The mixing proportion of the real-data is denoted by w and we define W 1 For two sequence of random variables Un Un we use the notation Uh if
P as n + Cc.
Un/ On
A.2 VARIANCE
We use the notations of Subsection A.1 throughout the proof. First, we consider the case À > Lemma A.1 (Ridge BrA Covariance). For > O, the covariance matrix Cov(Br.X (X] is given by
o²
(w² + w
2( 2
A +
21+1 A 21--2
(A.1)
Proof of Lemma A.
We will prove equation A. by induction. For the base case (t = 0), we have
Bo, A,X y. Hence.
CoviBo.A X
A Coviy/X|XAx
A, (O'1)XA
o² A
This proves the base Now let's assume equation (t + 1), we have
A.I holds for some t. Then, for iteration
Cov1BI+, = A,X +
= A, XWT (W TH XCoo BLN + IXXA
XX/MA) + XIIMAA
M
)
12 - شن M A A MA
= -AM 0² (w (W2+
+ 0² 0 + + MA
(w2 +
M 2k+1 A 2 21+
2 2 A
21+4
k=
+ 2 MA
2
2
2
2 A 2k 2
21+3 2 A 2144
This completes the proof of equation A.I
Corollary A.1. Using equation 2.5
(B i B) is given by
V(Brx;:)) 2²
(w²
T:
+ Th Tr
(A.2)
where B 128-11265
16

---

Published as a conference paper at ICLR 2026
Next, note that max (MAx) 1 and therefore CoviB,,, X converges in matrix norm the fol- lowing limit:
00
lim Cou(B,,|X] 1-000
= (n² + (1 11-1) -ww)) Σ w) 2 XX 2k+ A 2
= ou + (1 w) ) ) XA (1 - (x)
= o 10 ) X (nd w x) (nd (2 1) X X)
o' + (1- (1-
(wx X + + ((2 w) X T X +
2(1 - ww)
0² (w² + 2) lim V(B.xiB)
1
TY E
1r|Ax/(2-))) - و
1-000
2(1 w)
w
2-w
where the second to last equality follows using the following equality
x (nA+ wz) + (2 - w)
2(1-w) w)
nd + we
nd + (2 w)
combined with diagonalization of X X. Thus, compute the limiting variance, it is enough to calculate
lim Tr(A- lim n-
Tr [[n
X X 2I -
for 2 E C/Rt. Using averaged local law Knowles Yin 2017). we obtain
x X xIp)
?
mz + dH(r),
x
n
where m(2) is as defined in equation obtain that
3.2
Combining the above display with Corollary A.1 we
lim lim V(B = -2 (w + (1 11--w9 w)²) 2(1 - w)
V
I dH
dH
(A.3)
+ mi
1 m2
where my and m2 are as defined in equation 3.10 This completes the proof of the expression of variance for 0.
Next, we turn our attention to the case À = 0. Here, we have the following expression of covariance.
Lemma A.2. For 2 1, the covariance matrix CoviB, [ is given by
Cov|B,|X| = xx (w² (w²+ (1 w) Σ (1
w) 2(k-1) (1 - w=)
(A.4)
k=1
where B, is given by equation 3.3
The proof follows the same steps proof of Lemma
A.I
once
note
that
(X ( x)t)x2 = (X x) (xx)) for any 2
As consequence, have
lim V(B;8)= = - u (1 w) t0000
(A.5)
w w)
The limit the above quantity as n + oo is given by (Hssie et to the variance given by equation (3.5
2022 Theorem 2) and equal
A.3 BIAS
We the notation of Subsection A.I
throughout this subsection.
17

---

Published as a conference paper at ICLR 2026
Lemma A.3 (Ridge Br, Expectation). For À > O, we have
B = (WAAM)'A.MB + A
(A.6)
Proof. We prove equation A.6 by induction. Let's start with the base case (t = 0).
b = (x _x X + nA) X E
AB
This proves the base case. Now assume equation A.6 holds for t Let's prove it for k 1.
EB K+1AA X AxM(uB + (1 w B X
= =A + AA XJ)
= A.
( + Σ MA AB
(MMA w
(AA
This completes the proof of Lemma A.3
Corollary A.2. Bias for the Ridge Estimator given by
nd
B(B
||A 2/10 (I
( 1
(A.7)
w
Y1
nd
lim B(B
//
Tr[B3T Ax// EAs/l.
(A.8)
100
Proof. From equation
A.6
we have
B - B-EB = - (-AAAM) MB
w(I - ( (M))
(A.9)
Diagonalizing A,M and simplifying using the following identity
x.
(1
x1 11++
w [+1
I
1 WT
1 W.
tells us that
B = (I- - (MA, H+)
Next, we diagonalize M use the following identity on product of first two matrices.
(1-- x (-=tnn)) (1 (1- + nd
x
Wit
nd
nd
+ Tl
WE +
nd
conclude (I - AI TH A nd A
A/w- This completes the proof of equation A.7
w
equation (A.8) follows since Tuax (MA.
1.
Lemma A.4. Suppose Assumption 2.1 holds. For any deterministic sequence symmetric matrices C' e RPXp with bounded operator norm and z E C IR.t, we have
2² (T)(CQQ())EQ.(()))) Tr(C + -m==))) " 1 ddfo
(A.10)
where df²(H) 2 (IT + Σ) and Q=( (n- In-IM -I)-
18

---

Published as a conference paper at ICLR 2026
Proof. By plugging in A C' and B = Σ in Baahh 2024 Eq (3.9) and noticing that (I + m(2)) and Σ are simultaneously diagonalizable (and therefore commute), we get
(=T((Q)())EQN(())) (+[C(++min m(=) Σ
Tr (m(z) -1 1)-
+r[CC(+++ m(2) "Σح
n dfe(m((2)-)
1 df2(m(())1)
Tr(C(I -())=)
n - dfe(m(()))
Tr|C(I+ ()))))
1 n dff22(((()))
This completes the proof of equation A.10).
We now show that Equation 3.9 holds. Start by noticing that nA, Qn((2)). Then, equa-
tion A.7) combined with Lemma A.4 tells us that
nd w
lim B(B
TriB3T Ax/W EAx/w
Tr|BBT (T + m1 m1
Σ) Σ
(m2
where my =m(-/w).. All that is left to get equation
is realizing that 3.9h
lim Tr (I + mi E) " =b, / (1+max) dG,
n noc
and lim n (f2 / 100
mfx²
(1 + m)xx2 dH.
(A.11)
This completes the proof of limiting value of bias.
PROOF OF THEOREM 3.4
The asymptotic variance of Be, is given by equation (A.3 for general E. The asymptotic bias is given by equation A.ITI. This completes the proof of Theorem 3.4
PROOF THEOREM 3.2
Plugging H G 6, in equation (3.10) and equation (39) yields the asymptotic variance and bias
for B., when Σ = I. The proof of log-convexity risk follows from the proof of Theorem 3.3
PROOF THEOREM 3.1
The asymptotic variance of B, obtained by equation A.5 Dp.. To obtain the bias, recall that B (X x) (wy, + (1 - ).). Since B, = [ x) X we have B(B,IX) = (x x) x xB.. Now, we want to prove by induction that
E E(B = [X X XB
(A.12)
for all 6, To this end note that,
E(B = XX X X
(E(y:|X) + (1 (1-w))B(y(
(A.13)
Since y, XB+E, we have E(y, X X,B.
For the synthetic data, 5, -X,, Therefore,
E E X)= XIE
XB8
by induction hypothesis. Using equation (A.13) we have
E(, = (x X wxB + (1-1)x((X-) X3) =x -XX T
19

---

Published as a conference paper at ICLR 2026
This proves equation (A.12 by induction. This in turn implies using equation 2.5 that for any
-
where Px = - - XX (X X). Using (Hastic et al. 2022 Theorem 2), we obtain the analytic value of the bias.
Finally, note that the asymptotic risk depends on w only via c(w). Since c(w) is minimized at 1/p, this completes the proof of Theorem 3.1
B PROOFS OF REMAINING RESULTS
In this Section, we will show that the expression of generalization error R(Br,>) can be simplified
further if the probability measures Gp and Hp defined by equation 3.1 weakly converges to the same probability distribution, i.e., H. this special case, the generalization error has unique minima w.rto
Recall the definitions of VA and B3, given by equation 3.10) and equation (3.9 respectively. We will first rewriting Va and Bx in a different form. To this end, differentiate both sides of equation 3.2 w.r.to & to obtain
m'(2 m2
/ m') dH. (1 (mmmmm)))))))
+1 +1=
(B.1)
The above equality will be helpful in writing the integrals concisely. Also define f(2) ml -2) 2. We know that m(=) is a Sticltjes transform We will need the following technical Lemma whose proof we defer.
non-negative random variable by Remark 3.1
Lemma B.1. There exists some measure μs R+ with [S(1) < co such that
f(z) = = f
+ uldt).
2
(B.2)
Invoking Lemma B.
equation
(310) tells us that
2(1 w)
/
dH
dH
w 22 w)
+
miæ
+
m2
1
X
1
A
(By equation
3.2
(B.3)
m71
112
2
A
X
f
(B.4)
w
2
À + wt u(tt) -
x+ (2 w)t uldt)
2(1 w)t
(A () (A (2 w)() u(dt).
Thus, we obtain that for some measure jr + we have
w(2 w)
VA (A+ wt) (2 - wit) ((dt)
(B.5)
Similarly, we can also simplify the bias. If H. G. we have from equation (39
Bx = (
(1-7/ aa dH
mfI²
(1 (1+ m
(1 m
dH
We first multiply both sides of equation
B.Il by -mद/ (-\/)/ to obtain
=
m
dH = m' m'(-\/w)
(1+ m)
20

---

Published as a conference paper at ICLR 2026
Next, we decompose the first term of the bias as
I
m1x²
+ 12
(1 m))))²
7
(1+mpp dH =y /
x
m m1
dH
HH
+ mj
(1 + m2æ)²
1
m
m1
m1
m m'(X/w)
m1
m' mn/ww)
Combing the two equalities above, get
Bx = y-
m
A
ml m'(- (~\/w=)
(m'(-A/au
w
mi
m'
(-2)
y-
m( (
ml
=
=y- f(\/w) ((A/w)
(B.6)
Let N/w, we use equation
B.2 to get
yB2 = a /
2
(z +2+ + (2+t)z/0 ((t)
zt
2 2+t
=a + /
22
(a + t)2 u(dt)
X²
=0+ +
(A + wty l((dt
(B.7)
Since the sum of log-conves, functions are log-convex, this implies that Bx is decreasing and log- convex. Next we show that V(w)Va is log-convex. By equation B.5). we have
w² + (1 w)²
c(w) = ( (A wl)() (2 w)) u(dt)
The log-convyxity of the above expression simply follows from the fact that w² + (1 w)², (A+ + wt)- and (A + (2 w)t)-1 are all log-convex in E [0, 1] for all t, x and the fact that sums and products of log-convex functions are log convex.
As long μ + dg. further have that v(w)V> is strictly log log-convex, fact erucial for uniqueness of the minimizer of the risk. It can be readily verified from the definition of f(z) that = 60 = m(=) (a-z)-1 that is m must be Stieltjes transform of degenerate random variable. However, recall that by Remark 3.1 m is the Steiltjes transform of a free convolution between MP,, and H. Since MP, is non-degenerate, we must have that m(2) + (a (a==))- 2)-1 and hence the variance is strictly convex.
As long as ((R+) > 0, we further have that V( is strictly log-convex, a fact crucial for uniqueness of the minimizer of the risk. It be readily verified from the definition of f(=) that = 0 = m(z) (a - z)-1, that is m must be a Stieltjes transform of a degenerate random variable. However, recall that by Remark [3.]] is the Steiltjes transform of free convolution between MP,, and H. Since M M,,, is non-degenerate, we must have that m(x) # (a 2) and hence the variance is strictly convex.
Next, we show that w" is a continuous function of X. Define the quantity
R(w, X) lim lim R(B
Note that, R(w, A) is continuous in both w and A. Define any convergent sequence X + X. Let wi be the unique minimizer of R(w, Xx) (unique minimizer because R. is strictly log-convex is
21

---

Published as a conference paper at ICLR 2026
all three results of this section). Since wa is sequence in the compact set [0, 1}, there exists convergent subsequence wH, converging to some limit w" For any Aw E [0, 1],
R(wng Ana) S R(w, Ama)
R(ü, X) S R(w,)).
That is, w" is the minimizer of R(: X). Since the convergent subsequence mi we picked was arbi- trary, we have shown that every convergent subsequence of wi converges wi and thus must have that w + w. This proves that w* is continuous function of X.
Next, we propose to show that under G H. the optimal mixing proportion w" is in [1/2, 1]. with w" w²(A) (A) + o as A & 0 and w*(A) + as A + oc. Recall that both c(w)V> and Bx are continuous and log-convex, and B also decreasing. This immediately tells us that the limiting generalization error is log-convex and hence it has unique minimizer.
lim V( A-0
w²2 + (1 - -2) w22 w) 7 f Fu(ar) and
lim B₂ X-0
a
(B.8)
7
The minimizer of the of the risk is clearly only dependent on c(w), which minimized at ø Next, as À oo, have c(w)V> O. However, since 5x is decreasing function of and lim inf Bx > 0, we must have that wn + 1. Finally, we need to show that w* 2 1/2. To this end, A-000
write the variance as
ty
<wwV2> (w² (1-++1-w)) (A + wt) + (2 w)t) w)t u(dt)
The expression outside of the integral is minimized at 1/2. while the factor inside the integral a decreasing function of w for w E [0, 1]. This can be seen by calculating the derivative of the integrand
20 w) (A+ wt)²(A + w)t)²
(B.9)
dw (A wt) (2 w)t)
Since B3 is also a decreasing function of w, we must have that risk at w < 1/2 must be larger than the risk at 1/2. This completes the proof that w 2 1/2.
PROOF OF PROPOSITION 3.1
We obtain the asymptotic bias and variance for the random effects model from equation B.6 and equation (B.4) respectively. The log-convexity of variance follows from equation (B.5)
B
log-convex and decreasing using equation (B.7). This completes the proof of theorem 3.1
PROOF OF PROPOSITION 3.2
By Corollary D.1 we obtain that G H. Further, under the assumption 33 Lid (0, b.pp. we obtain || b, almost surely. Then, by the argument above, we have w* E [1/2, 1]. with w² (A) Φ- as À + 0 and w² (A) + 1 as + xc. Finally, to see that w² increases with SNR = b./o² recall that the risk is R(w, b.) o + b. By. Note that dividing the risk by x² does not change does not change w*. Thus, it is enough to show that wn increases with b. By the implicit function theorem.
Oub. R
Bx D.n.
0, w
Oun
Since Bias is a decreasing function of w and risk is a strictly convex function of w, must have that Ob, w 2 0, thus proving w² is decreasing function of bf 3.2
PROOF OF THEOREM 3.3
If Σ aT, have G H. and we obtain the desired conclusion by the argument above.
22

---

Published as a conference paper at ICLR 2026
PROOF OF PROPOSITION 3.3
For spike covariance matrix Σ, we obtain H, = -p-1 + (1 (1-p-1)61 where o, is the Dirac measure at the point J. Therefore, Hp = H ô1. To compute Gp. we write the signal B as B tr /1 pv' where v.v' 0 and || ll 1. This implies || 1 and hence
G 0² 01+. + (1 0r. If 0 d(n) + as + oc, we obtain G3 H ôj and the conclusion of Theorem 3.2 holds. Hence we will restrict ourselves to the case ((n) - 0. * O. Here we have G 0 81 + 02) 61. Since H = 61. the limiting expression of variance is still o'c(w) where c(w) and Va are the same as Theorem 3.2
Turning the characterization of asymptotic bias, again use the function m(z) satisfying m(2) + 1+m(2)) Defining mi m(-X/w), we have
1+8
(1 + m1)² ) (1-9
m
Bx
6%
+ (1 6%)
(1 + m1 (1 + s))2
(1 m1)²
We know from Theorem 3.2 that Φ1 (m1) (1+ma)2-ym+ is decreasing and log-convex function w. From the above display, we obtain that
(1 mm)²
Bx 02
m1 (1+ Φ1 (m)) + (1 (1-65)0 ())
(2(m))
Since (2(m)) is also decreasing and log-convex function of w, this implies that limm-+oo lim BB B) is decreasing and log-convex in w. Therefore, asymptotic general- ization error has unique minima w². To see the properties of w* note that by equation (B.9). we have w 2 1/2. Using equation (B.8), we obtain that w + 1/4 as - 0+. Finally similar to the and lim Bx 0. Since Bx is decreasing, 2000
proof of Theorem 3.2 have V>= 0 as À
again have lim, oo (A) This completes the proof of Proposition 3.3
We conclude the section with with the proof of Lemma B.I
PROOF OF LEMMA B.I
We need the following definition.
Definition B.1 (Stieltjes Function (SF)). A function f R - RR is called a Stieltjes function if can be written
f(x)
a + b+
JRt +t u(dt),
1 where a, b 2 and p is a positive measure on R+ with Jg- 1t1 p(dt) < oc.
Recall, the definition m²=) E(A - -1 for some non-negative random variable A. Using the definition above, the map m (n(=)) is Stieltjes function. Further v²2) 1/2 is Stieltjes function by Definition B.T Using (Schilling et al. 2009) Thcorem 6.2(1i) and Corollary 7.9), we
obtain that 1/(2m( =)) is Stieltjes function. Note that (2m(-2)) 2- =-1f(2) Hence, by Definition B.1 we have
f(z) +== bz + /
2 u(dt) ((dt)
2
with [f(1)] < co. It remains to show that b = 1. This follows from
f(>) lim +(2(A++))+111
20000
20000
where the last equality follows from dominated convergence theorem.
23

---

Published as a conference paper at ICLR 2026
C DYNAMIC MIXING
In this Section we prove the following claim for min-l2-norm interpolator: Suppose we select the mixing proportion wa adaptively each generation to minimize R) for any finite sample size n. Then wy satisfies
1 wi-1 w = + w7-1
wh = 1.
(C.1)
Further, under this setup
lim lim R(B;: =0²c(=") B. 12+00 10000
(C.2)
(3.5 and w" 1/p. To the claim, note that the bias of B is independent of mixing proportion, hence converges to B. Regarding the variance, we will prove the following by induction
where V, B as in equation
" = min ((B:B), Cov|Br|X] = (x
(C.3)
Suppose 1. For any w1 1, we have
Cov|B,|X] (X x X (wfo² 2(1 wi )0 X) x XX(X
= [w + 2( - w/ (x)
Hence the variance is minimized wi 2/3 (1 + wgg (2+)() Further, 2 wi proving equation (C.3) for = 1. Now, for any and any mixing proportion wt. we have by
induction hypothesis
CoviB,1X] [X
x X (wi
3 ( x)
=ui0 2 X x
w 2 [X X) 'X X
XCov(B,_ XXX + to x(Xxx)
(w] + (1 w -i- 1)) 0) (X x) 1+ which is minimized at = = 2+ Further we have wt² + - w7 ( w w prov-
ing equation (C.3) by induction principle. Since by equation (C.I) we have w + w², we immedi-
ately have equation
D RANDOM EFFECTS
Lemma D.1 (Equal weak limits). Let Eh laa with eigenvalues $1,..., 8p and or- thonormal eigenvectors u1,..., Up Define
1 1
P
Hx(x)
Gp(z) 1|8113
Σ Ukk B)2 1m
D k=
Assume B = .B 3p) has i.i.d. entries with E3; = 0. Esp = 72 E (0,00) and ES < co, independent of E.
Then, conditional on E, for every bounded Lipschitz, test fanction ab.
f N/GdG. -
-f
ff f dit, P. 0
as p co. Consequently if f. = H weakly, then also G H weakly.
Proof. Fix any bounded Lipschitz continuous function sb R + R such that || S M. Set ag := (8E) (which is deterministic given E). Let & = (vx, B). Since Bi's are i..dd. and independent of
24

---

Published as a conference paper at ICLR 2026
Σ, orthonormality gives that, conditional E. the variables & identically distributed and mutually uncorrelated with E5 = 0, E
= T². E < co. Then
/ /vdG v dā Σk=1 4x c2 Σラ= E)
/vdA, Σ
as
Ap
k=1
Define centered averages
1 Bp ==
a (E - --²,
cp== Σ
1
(5) - -.).
P k=1
P 1=1
Then
Σx ax Ap + Bp)
Bp - Ap
Ap +
+
+ Cp)
72 + Cp
Since bounded, ags $ M. Using B(E = T2 and E(st) oc.
1
M² d (ar((2) _ Var(s?) = o - p
Var(C,) = - Varr(E)) -a)(5))=00( =0( p
Var(Bp)
p
Hence Bp = 01(p-1/2) and Cp = 10plp11/'
conditional on E. Also AAp S M and 7² + Cp +
72 > 0. Therefore
I vb dG, - ff
-f
Bp ApCp 72 + Cp
Op
E 0,
again conditional on Σ. By characterizations of weak convergence, we have that Gp - A + 0 in probability, and thus any weak limit of Hp is also weak limit of Gp.
Corollary D.1. Let Hp, H, Gp. G and E be defined in Lemma D. "pp * some arbitrary sequence of real numbers and (Wi, w has i..d. entries with Ewi 0, Eu? x2 oo) and East sc, independent of E.
Assume B Thd where
Then, conditional on E, for every bounded Lipschitz test function vb.
J
|/ [b - f d E, 0
as p - co. Consequently, if H. + H weakly, then also G, H weakly.
Proof. Follows from the fact that
Gx(z)= = B
(u,, B)² 2
uu w) 10s5 =: Gx(z),
|wl
and applying Lemma D.1 on Gy..
E DETAILS OF SECTION 5
E. ABSENCE OF REAL LABELS
In this subsection, we analyze the following algorithm, where we generate real data once, and then compute the following estimator based on synthetic data generated in each iteration.
Next, we prove Theorem 5.1 which analyzes generalization error for B.
Proof of Theorem 5.7 First, we show that, similar to the main manuscript, bias ((a B) does not depend on w. To this end, note for / >
E[ x) (X x (wXB + ww X x)) (xx xB.
25

---

Published as a conference paper at ICLR 2026
Algorithm 2 Iterative interpolation with no new real data
1: Input: Dataset (y, X); mixing proportion w (0, 1).
(r) 2: Initialize: Bo + (XT [xxx) y.
3: for 2 1 do
4: Generate synthetic responses: Yc < 1-X,,1 + E
5: Update estimator:
3 + (X (wyo + (1- w)).).
6: end for
using the definition Bu in Algorithm By induction, E(B, [X) = (xxx) x (wXB + (1
(r)
w x)
(r)
(X X) XB8 yielding our conclusion that the bias converges similar the
proof of Theorem 3.1 To compute the variance, we will inductively show that for / 2 1,
B. (x xxXxxx
yo E w)(ti-1E))
(E.1)
j=1
If 1. we have
8/ = (X xX (wyo + - w) = (X 'X (wyo + (1 - w XBO + w)ē1)
(r)
= =(X -XTX) X) X (wyo + (1 w) X(X X x yo + (1 (1-w)ē1)
-XTx) x (yo + (1-w)ē) i)
This proves equation
E. for 1. Suppose the claim is true for general t. Then
p(n)
P+1 = (X x)
wyo + (1 - -++++1))
=X"xxtxx (wyo + -11
w) XB, +(1 w)
1 yo
xX-X
wyo + (1 w) X(X X)
w) (1--w)-1--13 + +(1 (I-WEE
j=1
Yo Σ (1 w) 42-
proving claim. This further yields that
lim Tr(Cov(B,") (r) (())= o²
Σ
(1 w)²
Tr (x
i=0
o²
Tr [X X
w(2 w)
(r) Hence, the risk is minimized at w² 1 and the quantity lim lim V VB. (B.",, .B) can be obtained similar to main manuscript.
E.2 TIME-VARYING COVARIATES
In this section, we want to consider the setup where X, is generated independently for each I, with
Σ = I. Formally, we consider the estimator B (1)
and we want to characterize, for each f,
B, = lim B(B (v) ;),
V = lim v(B.") (v) ;
000
200
26

---

Published as a conference paper at ICLR 2026
Algorithm 3 Vireinarying covariates
1: Input: Dataset (y, X); mixing proportion w (0,
(r) [ x)txTy.
2: Initialize: Bo
3: for 1. 1 do
4: Generate real responses: Yc x,B+ E..
(1)
5: Generate synthetic responses: y: t X,B,-1 + Ec-
6: Update estimator:
B (X, X,) X (wye (1 w)..).
7: end for
Remark E.1. Note that in both Theorem
E.1 and subsequent Theorem E.2 we analyze the asymp-
totic risk in the order lim lim RBB B), rather than lim lim R(B :) used in the main 1-100 n noc
n--00 t-
manuscript. This ordering is necessary because the covariates are generated independently each iteration Theorem E. (Xx) and Theorem E.2 (X,,X, X2). Consequently, for each fixed iteration t,
the risk must first be averaged over the randonness of the newly generated covariates as n - c0.
After the proof of our result, we provide two simulations to complement our findings. Figure 4a plots optimal mixing parameter w" function of 7. Our plot shows, unlike Theorem 3.1 w depends on Further, Figure 4b shows empirical risk matches with our theoretical prediction in moderate sample size.
Theorem E. Assume (0)
is oblained via Algorithm 3 and E I. Then we have
b. (1-3) 1
(w² + (1 3))
lim Be
lim V2
(1 w))²
(7 1) - (1 - -w)
Hence, risk is optimized at w 1 (v(c+27) 27) + 1 2c - (c+ 27 - 11) where
1)2--1
(v)
Proof. We begin by characterizing the bias B(B." ;8) conditioned on X (Xj)
BB, :B) = |B ED" (1)
(e)
X x
-118 - [ x,) ) X (uB + (1 11-w)BBB (1)
(1
1B (v)
-I( - (X, Xx. X.) + -1-
X (B
X n
= (I- X B (1 w)- E x)((-1))) TXX tx Bβ (
Therefore, conditioned on X {i(-t we obtain the high probability limits of the two summands above separately, For the first summand, note that (Hastie et al. 2022 (114)) that
BT (T - (X, X, x x - b, "
with high probability where b. = lim || 2 Similarly, the second summand converges with high probability, to
1 (1 w) w) ET X
Hence, we obtain that B, lim BB,
(1) ;) satisfies the recursion
40000
B, = b, (1 (1- 1 + 1 - -w)² B--1-
27

---

Published as a conference paper at ICLR 2026
(1- 1- (1- as -
Since ) w)² < 1, the sequence converges
Next, we characterize the variance V(B.") ;) conditioned on X (Xj)ssse- To this end, note that,
VBB (2) :B) = Tr(Cov(B, (2) [1:)]
w² or X + (1 II> X,) { X,Cov X 1)) + to
(w² + (1-w)) o 2x xI)
+ - u.) (X{( X,)
(x, (x, X,) (
V(B."' B), we have, by argument similar above, that n
Recalling that V lim
V = 7 (w² + (1 3)²) + (1 - w)² V -
(w2+11 -1w })
Therefore, we obtain V4 converges to
as t + 00. 00.
(x-1) 1- (1 w)
0.72
SNR
SNR=0.3
4.0
y=15
y-3.0
0.70
SNR =20
3.8
0.68
0.66
3.6
0.64
3.4
0.60
3.0
10
10'
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Weight the real data (w)
(a)
(b)
Figure 4: Panel (a): We plot theoretical values of optimal mixing parameter w" as function of as obtained in Theorem E. We vary SNR = (0.5, 2). Our plots show that
non-monotone as function of 7. Panel (b): We empirically verify the generalization error of Theorem E. For 7 = (1.5, 3) we plot risk as function of w. The dotted lines correspond to empirical risk and the solid lines represent theoretical risk. The vertical dashed lines correspond optimal mixing value w". The empirical risk computed with SNR and n 1500, itcration t 20.
E.3 CONCATENATION OF REAL AND SYNTHETIC DATA
Algorithm Concatenation online setup
1: Input: Dataset (y, X); mixing proportion w E (0, 1).
<(a)
2: Initialize: Bn + [X] X)tXTy y.
3: for 1 do
4: Generate real responses: Yt t- X, E (a) 5: Generate synthetic responses: ÿ. X,B +
6: Update estimator:
3(a) < (X. X, + XXX +XT7).
7: end for
28

---

Published as a conference paper at ICLR 2026
Here, consider the setup where X,, X, both are Gaussian data matrix with Σ I. We want to analyze whether model collapse can be prevented by constructing an estimator by pooling the real and synthetic labels delineated by Algorithm 4 We have the following result which shows for certain regimes of overparametrivation, model collapse might not be preventable. Figure indeed shows that the asymptotic generalization error diverges.
Theorem E.2. Assume B.a) is obtained via Algorithm 4 > and E = 1.
> then 7(5-2)
(2-1)
lim lim R(B, (a) :
(a) Proof. We begin by characterizing the bias B. lim BiB B), and R. lim RiB,"), B)
where we compute the bias and variance conditioned on X = j x))ssse- Note that, to define the estimation B, in Algorithm 4 need p 2n, hence will consider y > 2. For any (a) finite n, BBB B) satisfies the decomposition (Song al. 2024 (2.13)):
(a) BiG :B) = 3TSS 1)
B__1( X, X
X, X
B L-1
(E.2)
2n
(B:8)
B2(,,10)
where B :- B 1 Σ
[X]
X1 X X,). Using Song al. 2024
Theorem 3.1),
2n
obtain with high probability,
B, 2 (7 - 1)
Bt-1,
(F.3)
7(7 - 2)
Therefore, if (7-1) Theorem.
2) 1, we obtain that Rg 00 as + 00. This completes the proof of the
160
y=
140
y=
y-
120
(Risk)
100
60
20
40
60
80
100
Number of Retraining Steps (t)
Figure 5: We plot generalization enror of 3((0) to complement theoretical findings of Theorem E.2
The dotted lines correspond to log(Rssk) with SNR 1, and n 100 and different values of 7. The z-axis corresponds to number iterations i, which means risk increases exponentially with iterations and resulting in model collapse.
29


Reviews:
[{"review_id": "xfG3m2vMYQ", "reviewer": "Reviewer_8T4F", "summary": "This paper studies model collapse in the context of overparameterized linear regression. The authors analyze the minimum-$\\ell_2$₂-norm interpolator and ridge regression, in a setting where each iteration mixes real and synthetic labels with a fixed proportion.\nThe paper provides precise asymptotic risk characterizations using random matrix theory and discuss the dependence on the mixing proportion and the interplay with regularization and SNR. They validate theory with simulations.", "questions": "1. Could the authors better justify the choice of this framework from a practical point of view? In particular, are there real-world scenarios that resemble the training procedure illustrated here? For instance, how should one interpret the assumption that at iteration $t$, the statistician looses access to the true labels $y_{t-1}, ..., y_1, y$, yet maintains constant access to new labels for the same covariates?\n2. Do you have intuition on how would the results change in a setting where, at iteration $t$, the dataset is obtained by union of the fresh and synthetic labels rather than their mixing? Would this alternative formulation still prevent model collapse?\n3. As a suggestion, the paper would benefit from a more extended discussion of prior literature. While several recent works on model collapse are cited, a short summary of their different frameworks and main findings would help clarify how the present contribution fits within the literature and emphasize its novelty.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper provides a novel and technically solid contribution to the study of model collapse. It builds on a rigorous mathematical foundation, leveraging random matrix theory to generalize prior results that were limited to low-dimensional or Gaussian settings. The presentation is clear and well-structured, offering useful intuition for interpreting the theoretical findings, which could inform strategies for mitigating model collapse, when fresh labels are available. The numerical experiments are extensively detailed and effectively complement the theoretical analysis.", "weaknesses": "The proposed framework may appear somewhat artificial, and its motivation is not sufficiently discussed in the introduction. This limits the perceived practical scope of the work and raises questions about its connection to real-world training dynamics. See also the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qxnOrJN8NE", "reviewer": "Reviewer_EFFX", "summary": "This paper studies model collapse – i.e. when generative models experience performance degradation when repeatedly trained on its own synthetic data – in overparameterized linear regression for iterative training with a mixture of fresh real labels and synthetic labels. For min-l2-norm and ridge regression, the authors derive asymptotically precise generalization errors and uncover meaningful insights. Core results include: (i) for min-norm interpolation, the optimal mixing ratio equals 1/ golden ratio for general covariance ${\\Sigma}$ under certain assumptions; (ii) for ridge, the formula is log-convex in the mixing ratio w, and the optimal ratio is an increasing function of SNR. The paper also treats two case studies on random effect & spiked covariance models and validates theory with simulations.", "questions": "Some questions are raised in the Weakness section. Additionally, \n\n1. In the interpolation risk (Thm 3.1), the mixing ratio only affects the variance, but it affects both for the ridge model (Thm 3.2). This is an intriguing phenomenon (something similar also happened in double descent literature as I recall). Do you think there is any possible interpretation for this? \n\n2. How can the results possibly relate to the scenarios using real data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The precise risk formulae are clean and leverage tools from random matrix theory (beyond isotropic features for the ridge model). The optimal mixing ratio is a meaningful extension from prior works on OLS and low-dimensional results. \n* Though technical at times, the paper is written clearly and easy to follow. It shows precise formulae for both min-l2-norm and ridge, including random effect and spiked covariance as examples for the latter. \n* The empirical results validate the theory well and show robustness even when certain assumptions are violated.", "weaknesses": "I do not identify key weaknesses, and overall the paper is technically sound. There are some small concerns I have: \n\n* **Modeling Gap to Practical Model Collapse**: In iterative training, the synthetic labels are generated from the previous linear weight on the same covariates, but in practice, the model collapse setting touches both features and labels. The bridge from this linear supervised setting to practice seems under-argued. While I appreciate the need for tractable theory, I would like to see some discussions regarding this. \n\n* **Dynamic Mixing**: The idea of dynamic mixing in Section 3.1 is an interesting extension. Though the end result is somewhat similar, I believe it is worth having some synthetic simulations on this too. Additionally, would it be possible to verify these empirical observations on any small-scale experiments with real data? \n\n* **Notations**: at times, vectors and matrices are not bold (e.g. to name a few, Lines 404, 412, 419). It is good to keep the notations consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2Lsr6N0Z8", "reviewer": "Reviewer_Fx5M", "summary": "The paper presents a theoretical analysis of model collapse in overparameterized linear models in the ridge and ridgeless regression regime. The paper considers the fresh data augmentation framework with fixed covariates and fresh labels at each iteration. Interestingly, the authors identify optimal mixing ratios that minimize prediction error in this context which demonstrate how mixing real-data with synthetic outputs mitigates model collapse, notably with the appearance of the golden ratio. Specifically, the authors consider a fixed design matrix $X$ with i.i.d. rows in the proportional regime, and repeatedly refit on responses $ w y_t+(1-w) \\tilde{y}_t$. The paper considers the limit of $n/p \\to \\gamma > 1$ when $t \\to \\infty$ and derives the exact asymptotic risk as a function of $w$.", "questions": "* My main questions are in the weaknesses section. The following are additional questions that I think the paper could incorporate to discuss. Overall I think the paper is technically sound, but the contributions in high dimension is somewhat tangential to the main theme of model collapse.\n* What do the authors think of model collapse if fresh labels are only provided in a selected portions of covariates? It is more natural to make use of unlabeled covariates by synthetic labeling. How would this perform in an online setup?\n* I appreciate the numerical simulations that unpack the mathematical formulae---however, how would those connect to empirical findings in model collapse? I would like to see more discussions.\n* The general $\\Sigma$ with bounded spectra is essentially a corollary of the isotropic case. Would the authors comment on model collapse in the scaling law setups? Namely the spectra decay to 0 with infinite number of features such as in Cheng & Montanari (2024). \n\nCheng, Chen, and Andrea Montanari. \"Dimension free ridge regression.\" The Annals of Statistics 52.6 (2024): 2879-2912.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-organized and clearly written. The theory is sound as far as I checked. \n* The paper provides high dimensional generalization of the conclusion in He et al. (2025), showing the same optimal mixing ratio of $\\varphi^{-1}$ persists in high dimension.\n* The technical result of the paper is fairly generic, covering general $\\Sigma$ structure and ridge regression with penalty $\\lambda$. The paper also provides simulations that visualize the theoretical claims.\n\nHe, Hengzhi, Shirong Xu, and Guang Cheng. \"Golden ratio weighting prevents model collapse.\" arXiv preprint arXiv:2502.18049 (2025).", "weaknesses": "Major comments:\n\n* The model setup is somewhat too simplistic. The paper essentially assumes the following: (i) infinite supply of fresh labels (ii) the covariate matrix is fixed, and at each data point $X$, it is guaranteed to couple with (infinitely many) observations $y$. A very natural question in this setup is then: why don't we remember all new labels $y_t$, and simply taking $\\bar y_t = \\sum_{i=1}^t y_i / t$ , and solve the min-norm interpolator on the data $(X, \\bar{y}_t)$? This will completely get rid of the noise term and leave only the bias, which is strictly better than any mixing estimator with synthetic data.\n* The conclusion of model collapse is a bit narrow. It seems as long as $w > 0$, the estimator will always be reasonable (albeit, suboptimal) even if $t \\to \\infty$. This does not quite support the empirical evidence in Shumailov et al. (2024) where there is an actual degeneration of the learned distribution.\n* The major weakness of the paper is that the contributions are more technical rather than insightful: it provides limiting risks of mixing estimators in high dimensions under general $\\Sigma$ and $\\lambda$, but the setup is essentially the same as is discussed in He et al. (2025). For example in Eq. (A.5) and Appendix C, the nonasymptotic formula without making use of Marchenko-Pastur limits is sufficient to see the optimal $w$---the conclusion of optimal mixing ratio is not a high dimensional phenomenon, but only the limiting risk formulae. I feel the limiting risk formulae themselves provide little insights about model collapse. \n\nShumailov, Ilia, et al. \"The Curse of Recursion: Training on Generated Data Makes Models Forget.\" CoRR (2023).\nHe, Hengzhi, Shirong Xu, and Guang Cheng. \"Golden ratio weighting prevents model collapse.\" arXiv preprint arXiv:2502.18049 (2025).\n\n\nMinor comments:\n* It could be helpful in the statement of Theorem 3.3 showing the numerical value of $\\phi^{-1} \\approx 0.618$ so the presentation is more clear. Although it was pointed out in P5, Line 235, but I think it still helps the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9nwuP0l0Q", "reviewer": "Reviewer_LCBS", "summary": "This paper mainly studies how to prevent model collapse in a setting where in each iteration, a batch of new real data and a batch of synthetic data from the previous round are used to train the next round's generative model. The paper first studies this problem for overparametrized linear regression, which seems to be an extension of a previous paper.\n\nNext, the ridge regression is also analyzed for two popular model classes:  random-effects model and the spiked covariance model.", "questions": "Please address the issues mentioned in the weaknesses section. If at least some of them are addressed, or even the typos are fixed, I would be happy to raise my score.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is clearly written and easy to understand.\n\nMost results presented in this paper are sound.\n\nTheorems in this paper are supported by both theoretical proofs and experiments.", "weaknesses": "The first main concern for this paper is that the assumptions are too idealized, making it difficult for the results to be meaningful in practice. To be more specific, this paper mainly considers standard linear regression and ridge regression  in a **fixed design setting**. That is, the design matrix X is fixed across iterations. This setting is unrealistic in practice, as when people retrain a language model using synthetic data, they typically do not keep the prompts fixed across iterations. Also, this fixed design setting makes the ratio of the synthetic data and real data to be 1 in each iteration, which seems to be rare in practice.\n\nSecond, the contribution in this paper seems to be incremental. For example, if my understanding is correct, both this paper and the earlier paper cited by this paper(https://arxiv.org/pdf/2502.18049) aim to study how to use weighting to prevent model collapse. Theorem 3.1 in this paper aims to study this for overparametrized linear regression, but when I read the proof, it seems the weighting strategy can be derived from Lemma A.2 in the appendix directly, and from Lemma A.2 we can see that the weighting strategy for linear regression does not depend on the dimension. Therefore, it is unclear to me what important new insight can be brought by this section.\n\nAlso, there exists many typos in the paper that need to be fixed. For example, in appendix C, “Further, $w_1^{\\star 2} + (1 - w_1^{\\star})^2 = w_1^{\\star}.$”  should be \"Further, $w_1^{\\star 2} + 2(1 - w_1^{\\star})^2 = w_1^{\\star}.$\". In (C.3), the term $\\Sigma$ should not appear. \nIn the proof of Lemma B.1, (appendix page 22), the identity\n$\\frac{1}{zm(-z)} = \\frac{f(z)}{z} - 1$\nappears to be incorrect.  The term \"-1\" should be replaced by \"+1\".\nIn equation (B.4), $ \\frac{\\lambda}{2}-w$ should be corrected to $\\frac{\\lambda}{2-w}$.\n\nMoreover, in the proof for proposition 3.3, the authors seem to assume $\\lim_{ n \\to \\infty}$ and $\\lim_{t \\to \\infty}$ can be exchanged. In the main text, it takes the limit \nwith respect to $n$ first (i.e., $\\lim_{n\\to\\infty}\\lim_{t\\to\\infty}$), \nbut in the proof in the appendix the analysis is carried out in the opposite order \n(i.e., $\\lim_{t\\to\\infty}\\lim_{n\\to\\infty}$) without justification of their interchangeability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)