High-dimensional Analysis of Synthetic Data Selection
Authors: Parham Rezaei
Filip Kovačević
Francesco Locatello
Marco Mondelli
Venue: ICLR 2026 Oral (2026)
Keywords: high dimensional regression
empirical risk minimization
synthetic data
generative models
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We give a precise analysis for the problem of synthetic data selection through the lens of high-dimensional regression, and we translate the theoretical insights into a method that performs well in practice.

Abstract: Despite the progress in the development of generative models, their usefulness in creating synthetic data that improve prediction performance of classifiers has been put into question. Besides heuristic principles such as ''synthetic data should be close to the real data distribution'', it is actually not clear which specific properties affect the generalization error. Our paper addresses this question through the lens of high-dimensional regression. Theoretically, we show that, for linear models, the *covariance shift* between the target distribution and the distribution of the synthetic data affects the generalization error but, surprisingly, the mean shift does not. Furthermore, in some regimes, we prove that matching the covariance of the target distribution is optimal. Remarkably, the theoretical insights for linear models carry over to deep neural networks and generative models. We empirically demonstrate that the *covariance matching* procedure (matching the covariance of the synthetic data with that of the data coming from the target distribution) performs well against several recent approaches for synthetic data selection, across various training paradigms, datasets and generative models used for augmentation.

Published as a conference paper at ICLR 2026
OF HIGH-DIMENSIONAL ANALYSIS OF SYNTHETIC DATA SELECTION
Parham Rezaei', Filipp Kovačević", Francesco Locatello's Marco Mondelli 1 Institute of Science and Technology Austria (ISTA)
@azhamix&gmale com, filip. kovaceviciist.c....m............ ac. francesco. wwcatello@ist....oooeooo...ss..e at, marco www.deemi@ist........m........m. ac.
ABSTRACT
Despite the progress in the development of generative models, their usefulness in creating synthetic data that improve prediction performance of classifiers has been put into question. Besides heuristic principles such "synthetic data should be close to the real data distribution", it is actually not clear which specific properties affect the generalization error. Our paper addresses this question through the lens of high-dimensional regression. Theoretically, we show that, for linear models, the covariance shift between the target distribution and the distribution of the synthetic data affects the generalization eтrоr but, surprisingly, the mean shift does not. Furthermore we prove that, some settings, matching the covariance of the target distribution is optimal. Remarkably, the theoretical insights from linear models carry over to deep neural networks and generative models. We empirically demonstrate that the covariance matching procedure (matching the covariance of the synthetic data with that of the data coming from the target distribution) performs well against several recent approaches for synthetic data selection, across training paradigms, architectures, datasets and generative models used for augmentation.
INTRODUCTION
The controllable generation of arbitrary amounts of synthetic data for training machine learning models has long been considered as one of the key implications unlocked by more capable generative models (Kingma& & Welling, 2014, Goodfellow et al. 2014 Shrivastava et al.
2017 Nikolenko et al.
202I). After all, synthetic data can not only be abundant, which would already be tremendously impactful in data-scarce applications such as medicine (Esteban et al.
2017 van Breugel al. 2024), but can also address other difficulties of observational data, such as privacy (Jordon et al. 2018), imbalancedness (Parihar et al.) 2024 [Ramaswamy
al.) [021) and overall difficulty to collect, as the domain can be specific (Dunlap et al. 2023) or the task complex (Wang et al. 2023). At the same time, while generative models have progressed significantly, experimental results
still mixed. Several works are promising (Trabucco et al. 2024 He et al. 2023; Azizi et al.) 2023
Dunlap et al. 2023). ) steering and sometimes filtering the sampling by appropriately conditioning
a generative model towards the target training distribution; others outright question whether synthetic
data has any advantage over simply selecting some more data which anyway used to train the generative model (Fan et al.] 2024 Burg et al. 2023 Geng et al. 2024); some even warn that training on synthetic data may not only do worse, but also lead to unwanted effects such as model collapse (Shumailov et al. 2024) or additional bias (Wyllie et al. 2024). What emerges here is a broad challenge which consists of understanding how extra synthetic data, for example from a generative model, helps training predictors. Our paper tackles this challenge theoretically and empirically.
To do so, we assume access to a training dataset (Xe,yd with i.i.d. samples, as well as to an additional synthetic dataset (X,, v.). The samples from the synthetic dataset also i...d,, but they come from a different distribution, since they are obtained from generative model and not from the training dataset. We perform empirical risk minimization (ERM) using the augmentation ((X,, X.). (vt. ys)), and evaluate the performance on an independent test sample (Xxest, Itest) with the same distribution as (X,, )r). In this context, the challenge above leads to the following concrete question:
"Equal advising

---

Published as a conference paper at ICLR 2026
How to select the dataset (X,,ye) ye) in order to minimize the test error?
(Q)
By studying this question, we can identify which properties of the distribution of (X,, ys) improve generalization, thus guiding the selection of data obtained in practice from generative models.
Formalization of the problem. Let us first describe how we model the setting in the theoretical anal- ysis. We assume that the distributions of both the original training dataset and the additional synthetic one are mixture models. The number of mixtures corresponds to the number of classes in the datasets, with each mixture component corresponding to a single class, As common in practice (Burg et al.) 2023). the data augmentation via the synthetic dataset occurs class-by-class for problem with K classes, the number of mixtures is K and we add synthetic data of each class using a generative model.
We then address the question Q when (X,, v1) and (Xa, Vs) correspond on linear models and high-diig gigh-dimensional ridgeless regression. More precisely, we model ye = X,,5+ and ys X + gg where rows of X, are i.i.d. with mean 14 and covariance Σ,, rows of X, are ii.d. with mean 14s and covariance Σ,, and entries of Et. E, are i.i.d. with zero mean and variance g², Here, the difference between the distributions of (Xe, M/s and (X,, Us) is captured by the mean shift & ps and the covariance shift Σ, * Σ. We also consider model shift (different 3 between synthetic and real samples), deferring the details to Appendix B Our formalization deals with single class in isolation, fitting regression model to the class label and neglecting interactions between classes. While this is a strong assumption chosen for mathematical tractability, we highlight that the resulting data selection procedure is extensively tested in practical settings where it performs well against existing baselines.
single class, focusing
The surprising finding from our theoretical analysis is that, while the covari- ance shift affects the test error, the mean shift does not. This is the case as long as the training dataset (Xe, 1/)) is not too small compared to the synthetic dataset (Xg, Us), and it is especially surprising since the mean shift does affect the test error when using only synthetic data. From this insight, we the
Main contributions.
show that the problem of selecting (X., ya) can be reduced to optimization problem variance Σ, and, in some settings, matching the covariances (E, x Σx) leads to optimal performance. Most importantly, these theoretical insights are valid in practice: matching the covariance, without worrying about the mean shift, performs on par-or even outperforms several recent approaches for synthetic data selection. We summarize our contributions below:
We give a precise characterization of the test error of the min-norm least squares regression estimator, when the dimensions of 3, y/. y/ are all large and scale proportionally. Our results hold in under- parameterized (Theorem 4.1 and over- over-parameterized regimes (Theorem 4.4). showing that the test error approaches deterministic quantity that depends only the covariances E,, E, and not on the means pt, Ag. As a comparison, we also analyze training only over synthetic data, showing that in this case the test error depends on both covariances Σ, E, and means Me, Ma. see Proposition 44.2
Our characterization implies that we can select synthetic data minimizing the test error based on their covariance. We then show that, under some conditions, taking Σ Σ i.e., covariance matching. is optimal (Theorems 4.3 and 4.5) for under-parametterized and over-parameterized regimes).
We validate the effectiveness of covariance matching as a way to select synthetic data obtained from generative models in several practical scenarios. We show that this simple approach performs on par-and, and, actually, it often outperforms- variety of baselines proposed in the recent literature (He) et al. 2023] Lin et al. 2023 Hulkund et al. 2025). This conclusion consistently holds across training paradigms (training from scratch, distilling a bigger model, fine-tuning a model trained from a larger dataset), across architectures (ResNets, transformers), across datasets (CIFAR- 10, ImageNet- 100, RxRxl), and across generative models used to obtain synthetic data (SyyleGAN2-Ada, SANAI.5, PixArt-a, StableDiffusionl 44, MorphGen).
2 RELATED WORK
On the theoretical side, we focus on the high-dimensional regime in which both the number of features (i.e., dimension of (3) and the number of samples (i.e., dimensions of ya, Dt) are large and scale proportionally. This setup was considered by a line of research using random matrix theory to characterize test error and various associated phenomena (e.g.. benign overfitting (Bartlett et al. (0200
and double descent (Belkin et al. 2019)). More precisely, the test error of ridge(less) regression was
studied by Hastie al. (2022) Wu & Xu 2020): Richards et al. 2021): Cheng & Montanari 2024))
2

---

Published as a conference paper at ICLR 2026
the distribution of the ERM solution by Montanari et al. (2023), and the impact of spurious correlations by Bombari & Mondelli (002)). This motivates us to look for practical insights into synthetic data selection by performing bigh-dimensional regression analysis. Closer to our work are specific analyses involving more than one distribution, which in our case are the training/test distribution and the synthetic one used for augmentation. More precisely, the test error under distribution shift was analyzed by Patil et al. (0024); Mallinar et al. (2024). but this assumes training on one distribution and testing on the other, as opposed to training on both and testing on one. Training on surrogate data was considered by [ldiz et al. (025); Kolossov et al.
(2019): Chang al. (2021) Han & Xu
(2024); Jain et al. (2024): Iidiz al. (2025) assume that the surrogate data comes from a teacher model and study the phenomenon of weak-to-strong generalization; Kolossov et al. (2024) consider data selection given unlabeled samples plus access to a surrogate model that predicts the labels better than random guessing: Jaissing:aainnn et al. (2024) integrate surrogate and real data, but the analysis is limited to isotropic covariance. Most closely related to our theoretical setting is when training occurs on multiple data distributions and testing occurs
single one of them, which was analyzed both in 2025) and over- wwer-parameterized (Song et al. (0244) regimes. However, Yang et al.] (2025); Song al.] 2024) assume that the data distributions have zero mean, which is unrealistic in our context. In fact, centering the data would require access to the mean of the test sample, which is equivalent having access to its unknown label. Related work by El Firdoussi et al.) (2025) analyzes high- sigh-dimensional binary classification with isotropic covariance, with different objective from ours. Namely, its goal is to understand the factors influencing performance when generating synthetic data using estimated statistics from real data, pointing to degrading performance in case of bad covariance estimation. Additional related works by (Seddik et al. (2024); |Bertrand et al.) (2024) consider distribution shift of a mix of real and synthetic data through the lens of statistical approximation error, a mechanism linked to model collapse by Shumailov et al. (2024). Both [Sediik al. 2024) and [ertrand et al. 2024) examine the phenomenon within iterative retraining loops, considering a setting complementary to the one of this paper.
under- caader-prammterized Yang et al.
On the practical side, several papers studied how to incorporate synthetic data into training pre- dictors. Besides simply training better generative models, empirical work focused on upgrading the sampling process itself, under the assumption that better conditional generation would lead to more accurate predictors. More precisely, the CLIP model (Radford et al.) (021) underpins many
filtering and selection algorithms for generative data. He et al. (2023) propose using CLIP similarity to labels to prune low-quality samples from augmentations. Lin et al. (023) introduce sampling and filtering strategies based on CLIP similarity either labels or the mean representation of real data, incorporating diversity via clustering. Almost concurrently, other works argued that synthetic images underperform in scaling laws (Fan et al.) 2024) and, if the generative model pre-trained on external data, simple retrieval baselines can be hetter (Geng et al. 2024 Burg et al.) 2023). Our work can be interpreted as a more fine- mee-graieed investigation of the same problem, characterizing which properties of the generated data improve generalization. At the same time, our results do not preclude that the extra data is real data from another dataset, as tested in Figure 2 in Appendix C Closer to our solution. Hulkund et al. (025) explore the problem of data selection given fixed test set and, taking a purely empirical stance, compare several filtering methods, including an approach inspired by Gadre et al. (2023) that selects clusters of image embeddings. As heuristic, we find that this works rather well but has shortcomings, as empirically demonstrated in Table ia Appendix C
3 PRELIMINARIES
Data model. We consider data augmentation in the context of linear models. Formally, we observe two datasets (X,, y1) and (X., ys). denoting training data and augmenting synthetic data, such that (()) = X() E(i) where (()) E RT xp, 3 E R?, and E()) E RR Thus, we are given n, training samples and n. synthetic samples, all of which are p dimensional. We denote the total number of samples as n := Th. + n4. Each entry of the noise vector E(t) is sampled i.i.d. from a random variable with mean zero and variance o². The row vectors of X(o). for (i) E {t. s}. are independent random vectors with p x p population covariance matrix Egg and mean ((()- This can be written
(i) E st,
(3.1)
X() = z (E + Ln() E RR
(3.2)
where Z E RT xP, M() E IP, In, E R" is the all-ones vector, and all entries |2/20 are independent with zero mean and unit variance. By omitting subscripts, we denote by (X,y) the two
3

---

Published as a conference paper at ICLR 2026
datasets (Xes Me) and (Xa, Us) stacked, i.e., X =
X
X2
E R
y
Vt yo
R".
The vector 3 is
assumed to be the same for (Xx, y/)) and (Xx,, yo). which corresponds assuming that the conditional distribution of the labels given the features X is the same for training and synthetic data.
Note that we also consider data model in which model shift is present, i.e. training and synthetic data have different hidden parameters Bt and S,, respectively. For more details, see Appendix B
Assumptions. We make some assumptions on the data distribution which are common related
work (Yang et al. 2025 Song et al. 2024). Let 7 > 0 be a small constant. We assume that, for di > 4, ) M S r- which means that the tails the 1h-th moment of 2 upper bounded by 1/r, E| 2
do not decay too slowly. The eigenvalues of (i)). denoted as x ................................ Ap are all bounded between and -- i..., S ( <.... < X
< A(5) < T-1 which means that the covariance matrix well- conditioned (i.e., the distribution is well well-spread). Furthermore, the entries of (() E R have bounded
moments up to any order, i.e., for any lo E N, there exists constant C > 0 s.t. E [E))) | < Ck (noise is not heavy tailed). The sample sizes are comparable with the dimension p. i.e., == n/pp. Tt = nt/pu p. and T == ns/p, with S 7 I 1/T and S 7: 7s S 1/T. Lastly, let || ll2 = T(t) vP.
where r(e is a constant, with constant angle between them yp == (Ma, / (1laell2 |I
Risk and estimator. We test estimators on data sampled from the same distribution as the training dataset (X,, y1) and, given an estimator 3, its out-of- sample excess risk is defined as
Rx B[(x]] 3 x 3)² X] E
[|I B1 SetnuT |x]
where T2 has the same distribution as Z' [,, 1/2 + and | 3 = x. Mr. This definition differs from similar ones appeared in (Yang al. 2025 Song al. 2024 Hastie et al.) 2022 as the test
distribution is not zero-mean (test data cannot be centered as knowing the mean is equivalent to knowing the label). The test error is then equal to the excess risk plus the noise variance a², which corresponds to the Bayes error. Since a² is constant, minimizing excess risk and test error is the same, and we minimize the former. The excess risk decomposed into bias and variance as
Rx (3: - B)= |E| x]
Bl Tr[Cov(3) X)( :=: (x
(3.3)
Let be the min-norm least squares regression estimator of on the whole dataset available X, i.e.
arg min [116||2 b minimizes [yy -x61)) x y.
(3.4)
where (-)+ denotes the pseudo-inverse We note that gradient descent converges the interpolator which is the closest in l norm to the initialization (see Equation (33) in Bartlett et al. (2021) and, as
such, (.4) corresponds the gradient descent solution starting from 0 initialization. The results in the next section readily extend to weighted versions of objective optimized in (3.4). see Appendix A.8 for details. For this reason, we present our main findings for the unweighted objective without loss of generality. Substituting (3.4) into the excess risk decomposition (3.3) yields closed-form expressions for bias and variance:
(x(3;8) = 3 1II, + PiPT 11s and Vx( TT + +Mpt T),,
(3.5)
where E X X/n and
Ê+ (projection on the null space of X).
THEORETICAL RESULTS
We characterize the excess risk of the min-norm interpolator using both training and augmenting synthetic data. We then use the explicitly derived formulas to optimize the data selection process,
'Taking ||p|o||2 ~ vP ensures that the mean and variance of (( are of same order. In fact, || || vp would imply (ue, 8) < 1, so the mean would have a vanishing effect on the risk. Furthermore, if || Vp. then (M(), 8) >> 1, and the mean would dominate the risk entirely. In both cases the problem trivializes.
2This is technical assumption simplify the proof notation. If yp is allowed to depend on n, all results (and corresponding proofs) still hold verbatim, long either yp
for some constant 8 or yo = 1.
4

---

Published as a conference paper at ICLR 2026
in which, surprisingly, distribution means play role. We contrast this setting with having only synthetic data available, where means instead impact the excess risk. Our findings hold in both the mnder-parameterizeer and over-parameterizee regimes. For clarity, present the two regimes separately, as the precise statements and proofs rely on different technical arguments.
4.1 PES.-PAAAAMAEREE REGIME
Let us assume that 1 + y & 1/T, implying that n > p, which makes the setting under- parametrized. Thus, E = X "//X//n is full rank almost surely, which implies that 11 = - ΣイΣ =
-- -- 0. From (3.5). it follows that Bx (3: 9) 0, so the risk is only characterized by the variance Vx B). We additionally constrain the number of samples 0 <<>, 7//70 S 1/T.
7 ne. 7s 1/7 and
The following result provides a precise asymptotic characterization of the excess risk and, in doing so, it extends results by Yang et al. (2025) to non-zero centered data. Its proof is deferred to Appendix A. and we give brief sketch of the argument below.
Theorem 4.1. Let M 2
and denote the eigenvalues of M " as 2.. Xp Then, under the assumptions from Section 3 and the start of this section, it holds that, with high probability
lim 12-00
Rx 3)
o²
Tr [ M + az T2 =0,
(4.1)
where 01 and 02 are the unique positive solutions to the following two equations
a1 + 02
=1 11 01
P n
P
Aaa
ng
(4.2)
T1
A + a2
7
Proof sketch. As seen from 3.5 Rx Rx () is related to spectral properties of the sample covariance matrix Σ, dictated by its local laws. The core of argument is to connect the spectrum of E for non-centered data to its gero-centered counterpart. This is done by factoring out the means fl: M4 as rank-2 perturbation Appendix A We then apply anisotropic local laws for the zero-centered case and conclude. We finally note that this strategy gives a convergence rate of ((02p for the LHS of (4.1
random matrix with i.i.d. entries, see Propositions A..1 A.2 and A.33 in
Theorem 4. gives a deterministic equivalent of the test error obtained using training and synthetic data in the under-parameterized regime. fact, Rx 3) random quantity (the data ran- dom), while z? Tr[(an M M + a2 [p) is deterministic as it depends on properties of the data distributions. Remarkably, the deterministic equivalent depends only on the covariances E, E, (via M / ////// and does not depend on the means fe: Ps. This is highlighted in Figure la showing that the excess risk is unchanged upon varying the cosine similarity between the means. Two points are now in order, which are elaborated upon in the next two paragraphs.
(a) The independence of the test error on the mean shift is surprising, and it is in stark contrast with the setting in which we only train (X,, ys), where the performance does depend on Mas Pt-
(b) The deterministic equivalent can be optimized to find the covariance D, minimizing the error.
(a) Training only on synthetic data. We now adjust our assumption at the beginning of this section. Namely, we assume that 7/- 0, + 7. = S 1/T, which means that we are training on data from a single distribution that different from the one we are testing on.
Proposition 4.2. In the setting described above, holds that, with high probability,
2
lim 000
Rx (x
a²
7
11 pr
1/2
14 Σー M1
0. (4.3)
n
7
|| Hall2
This result (proved in Appendix A..2) extends the zero-centered expression by Hastie al. 2022 We observe consistency we disregard means (1, = Pr 0) and covariance shift (Σ,Σ, Proposition 4.2 also extends the zero-centered anisotropic setting of Yang et al. 2025 to the
5

---

Published as a conference paper at ICLR 2026
without samples from the training distribution, and consistency follows after setting μs μ4 0. The effect of the mean shift captured by / 2 113 - (14 Σ pz. // pall2)?: p,.l2)) what matters
is (i) the cosine similarity between Σ, 1/21/2 M4s and E,, 1/2 j44, and (ūi) the alignment of the principal directions of Σ., with μt,, In other words, the excess risk decreases as (i) the mean of synthetic training data aligns with the mean of test data in the directions of the symthetic covariance matrix, and (ii) the principal directions of the synthetic covariance matrix align with the test mean.
(b) Synthetic data selection.
Let us denote the deterministic quantity from as
(M = o² Tr [aa M + az a224p) "']
(4.4)
where o1 and 0r satisfy (4.2). This corresponds to the limit of the risk Rx (3; (3) due to Theorem
4.1
Note that (M) depends only the covariance matrices of the original training (E-;) and the
augmenting synthetic data (E.) via M = E. x Thus, in the under-parameteriizd setting, the
guiding question
Q posed in the introduction can be formalized
Given EG what is the optimal Σ, that minimizes R. (M))
The following theorem exactly treats this. Its proof is in Appendix A.3 and brief sketch is below. Theorem 4.3. Let M = {M E ReXP- rank(M) p, Tr[M M] =p}. Then, for Mopt E M minimizing the limit risk of Theorem 4.1 i.e., Mopt == arginf MEM (MM it holds that
Mc( Mopt)
Vi (I.....p).
(4.5)
Proof sketch. From the first equation in 4.2 (M) can be expressed in terms of a single parameter, e.g., O1. A key insight is that R.,
increasing in 01, which simplifies the optimization. Denoting with X1 2 ... 2 Ap the eigenvalues of M in decreasing order, we show that transformations of the form Ay) + (A, c, X + c) for >> can only lower a1 Thus. majorization argument allows us to conclude that the most balanced solution (namely, (4.5) is optimal.
Theorem proves that, having fixed Tr|M M). the limit R. (M) minimized when M has all eigenvalues equal. Thus, given a training covariance Σ,, choosing synthetic data with E, x E, i... matching the covariances, is optimal. This is highlighted in Figure showing that the excess risk decreases as E, aligns with Σ, Increasing the scale of Σ, also reduces the risk, i.e., for any M E TRPXP s.t. rank(M)= = p and any constant 7) > 1, it holds that (n) S R₂ (M), see Appendix A.4 for the proof and Figure Ic for an illustration. Recalling M = / 1-:55-11- 1/2 this suggests that greater diversity in synthetic data advantageous. However, as Theorem 4.1 relies on bounds on the spectra of E,, M. (see Section 3). must be of constant order, i.c., it cannot grow with and p (otherwise, the error between Rx (B; B) and (n may not vanish in (4.1b). This motivates the trace normalization (Tr[MM M p) in Theorem 4.3 While other normalizations
exist (e.g., g. on the determinant in (Yang al. 2025 ), they overly restrict the search space and make interpretation for synthetic data selection less clear.
4.2 OVER-PARAMETERIZAA REGIME
let us assume that 7 7.7,, 7 I 1/(1+ so that n < p and we are in the wer-parameteriiee regime. We sample 3 from sphere of constant radius, independently from X, K, E.. We also assume that Σ, and E, are simultaneously diagonalizable. This assumption is of technical nature and common in related work (Song et al.) 2024 Mallinar al. 2024 Ildiz al. 2025). Writing out this condition, we have the SVDs Σ. = UAU E, = UAU Let us denote by X A Af Ar, and introduce the spectral probability distributions used our claims;
As opposed to Section 4.1
A,
(000))00 Go(A')))==
Σ (B, 2 ((0)))))0)--- (4.6)
This section follows the same blueprint as Section 4.1 for the ander-parameterizedd regime. Namely, Theorem 4.4 gives deterministic equivalent of the excess risk using training and synthetic data and, in doing so, it extends results by [Song et al. (2024) to non-zer non-zero centered data. The deterministic equivalent depends only on regression coefficients 3 and covariances E,, E., and it does not depend

---

Published as a conference paper at ICLR 2026
(2e00
04
A- 5 chioo)
A- (Ra0))
5 (2h0c)
Exceas risk
0.7
0.6
E- (emplica))
C5 (enalica)
Lamplica)
Excess risk
02
m0 5 somp
5 templiccal
tempitial
Excess risk
65 perpitat perplica
0.5
04
05
00
022
04
10
Cosine similarity of means
Covariance scale
(a) Aligning the means
(b) More synthetic data
(c) Scaling the covariance
Figure I: Excess risk using training data from N (pe, Σe) and synthetic data from N (Ps Ex). where
Σ,, E, are Macc-Murrock-S5 Szegö matrices (Toeplitz matrices with geometrically decaying entries) with parameters Pt. Ps. scaled so that Tr(M M] = p. We pick | l2 = || = 2/P. P = 0.9, p= 600, n4 1 1200, n, 1200, unless varying the parameters in the plot. Each value is computed from 100 i.i.d. trials, the error band is at standard deviation, and theoretical predictions are continuous lines. Different curves correspond different values of Ps. (a) Changing the cosine similarity of the mean does not impact the risk (here, Dg is scaled by 1) == pa). (b) Larger Ps gives lower risk since Σg is closer to Σg. (c) Scaling Σ. reduces the risk.
on means μt, Ms. Then, Theorem 4.5 finds Σg that minimizes the limit risk from Theorem 4.4 when Σ, = lp. thus showing the optimality of covariance matching (Eg Σ,) with isotropic training data. The proofs of these results follow a similar argument chain as in Section 4.1 although they tend to be more technically involved. We briefly discuss differences, deferring the full arguments of Theorems and 4.5 to Appendices (n6 A.7 respectively.
Theorem 4.4. Under the assumptions from Section 3 Hand the start of this section, it holds that, with high probability,
lim n-000
Rx(3:3) V(Es, Σ.) B(E,, I1, = 0,
(4.7)
where
V(E,, Ex)= 0² X Maa3>*
1 (X,2)) BEE,, Ev, 1)=- bgd++(b4 (ba+1)2 dG, dG,(A,.)))
(a) (11
+b +1)2
and ag bi E (1,2,3,4] 4}) are the unique solutions the equations reported in Appendix A.5
We highlight two additional difficulties in the proof of Theorem 4.4 arising from the over- parameterized regime: (/) the inverse does not replace the pseudo-inverse in (3.5). and (2) the bias term does not vanish. We address the former by introducing the X-regularized ridge estimator 3x. which approximates 3 for small and admits prverse-base formulas similar to (35). Addressing the latter requires delicate control of the inverse, obtained via Woodbury formula.
Theorem 4.5. Let S {2 E REXP Tr (E) = p}. where 12Xx denotes the set of X p positive definite matrices. Recall the definitions of V(E.. E,), B(E,, Lt, ().33) from Theorem 4.4 and define RR,E,, Et, B3 V(E,, Ex) + B(E,, Ex, Then, for any E, E S,S, with high probability over the sampling of B over sphere of constant radius, it holds that
Ro(Tp. Ip. 3) S R. Ip B) + o(1)
where o1) denotes a quantity that vanishes as - cc.
Due to the complexity of the expressions for V(E,, Σ,) and BEE,, E,, the optimality of covariance matching (E, x Σx) in the wer-parameterizee regime is shown for isotropic training data (E, = 4p). At the technical level, we note that the bias generally depends on the eigenspace decomposition of the covariance matrices via Gp. as defined in (4.6). However, when E, = Lp. cancellations in the equations for b, (i E 21, 43) give that the bias B(E,, Ip, B) is close to シーツ ||11|, for any Σ,, Having obtained that, the variance is then optimized following the approach of Theorem
p
5 EXPERIMENTAL RESULTS
Theorems 45 and 4.5 show the optimality of covariance matching (E, X Σe) in both under- parameterized and over-parameterized regimes. We now extensively test the applicability of this
7

---

Published as a conference paper at ICLR 2026
Table 1: Covariance matching outperforms all baselines across three training paradigms on CIFAR-10. when the synthetic data is generated via five truncated StyleGAN2-Ada models.
Method
Scratch
Distillation
Pretrained
No synthetic
44.36 + 1.51
47.33 + 0.57
63.40 = 1.33
Center matching (He
2023
50.04 2.84
53.83 0.59
67.01 + 0.89
Center sampling (Lin
2023
50.48 + 2.03
54.91 + 1.07
67.71 = 0.90
DS3 [[ulkund et 2025
52.83 1 2.19
58.32 0.43
68.21 1 0.66
K-means LLn et al
2023
50.74 4 1.77
56,06 -: 0.68
66.50 = 1.11
Random
49.38 + 2.43
54.89 +: 0.91
67.65 + 0.77
Text matching Lin et al.
2023
50.94 1.40
55.17 15.17700.3
67.81 1 0.76
Text sampling (Lan cl al. 2023
50.28 + 1.18
54.82
+ 0.72
67.45 + 1.02
Covariance matching (ours)
54.00 1.89
59.77 + 0.61
69.20 + 0.56
Real upper bound
61.08 + 2.54
65.38 + 0.51
74.35 = 0.56
synthetic data selection criterion in range of practical settings. We consider classification problems, assume access to large pool of synthetic samples obtained from generative models, and perform the augmentation per class. We implement covariance matching via greedy algorithm: we initialize S = 2 and, until |S| nss add the from the generated pool that minimizes [E(SU(=))- - Eelle where SO() and Ex denote the sample covariance of CLIP features of the synthetic samples and real samples respectively and - || is the Frobenius norm. To accelerate the selection, we compute covariances a 32-dimensional PCA space fit on the ni real reference features. After the selection, we train classifier on the union of real and selected synthetic samples.
Experimental setup. When using CIFAR-10, we evaluate three training paradigms. (1) Scratch: train ResNet-18 (He et al.) 2016) from scratch on the available data. (2) Distillation: train a ResNet- 18 using soft targets (logits) from a ResNet-50 trained on full CIFAR-10, 10, following Hinton et al. (2015). (3) Pretrained: fine-tune an preetrannd ResNet- 18 with new classification head. We also repeat the Scratch and Distillation experiments replacing the ResNet with two transformer models (ViT and Swin-T). Unless stated otherwise, use n, 200 real images and augment with n. 800 synthetic images per class. The features for the selection algorithms are ex extracted with CLIP ViT-B, yielding a p 512-dimensional feature space, which places us in an under-parameterizee regime. We report in Table 9 in Appendix C an additional experiment for n. + n& = 400, which places us in an over- aeee-paramtterieed regime. We additionally consider ImageNet- 100 as a more diverse dataset, and RxRx1 (Sypetkowski et al.) 2023) as a specialized one. For RxRx1, we use a small subset of n, = 30 images from four common perturbations (1108, 1 124, 1137, 1138) on HUVEC cells. We consider the task of perturbation classification and augment with n, 60 samples chosen from 500 images generated by MorphGen (Demirel et al. 2025). ). Further details are in Appendix C
Baselines. We compare Covariance matching with the following baselines. (1) Center matching (e et al. )023): select the n, images nearest to the centroid of the ni, real training features. (2) Center sampling (Lin et al. 2023): sample with probability proportional to the cosine similarity to the n real training features. (3) DS3 (Hulkund et al. 2025): cluster the generated pool into 200 clusters; for each of the n, real images, retain its nearest cluster; then, sample n,, images uniformly from the retained set. (4) K-means (Lin et al. 2023): cluster the generated pool into n, clusters and choose one random representative per cluster. (5) Random: uniformly sample n. images from the generated pool. The methods "No-filtering" (Hulkund et al.) (025)). "Match-dist" (Hulkund et al. 2025). and "Match-label" (Hulkund al. 2025) are all equivalent to Random in our setting due having the same number of data for each class. (6) Text matching (Lin et al. 2023): select the n. images nearest to the class text embedding. (7) Text sampling (Lin et al. [023]: sample with probability proportional the cosine similarity to the class text embedding. We also report a baseline, No synthetic, corresponding to using only n4 samples from the training distribution (synthetic data discarded), as well as a baseline, Real upper bound, corresponding using n, + n, samples from the training distribution (synthetic data replaced by in-distribution data). All experiments are repeated over 10 random seeds (except Table 3al which is on 5 seeds), and we report the mean + 1 standard deviation.
Main findings. First, we test diversity/quality trade-offs. To do so, for each class we generate images with StyleGAN2-Ade (Karras al.) (020) under different truncations (Karras et al.) 2019): 6K images from a 0.2-truncated model with three randomized truncation centers and 4K images from a 0.6-truncated model with two randomized centers. This produces synthetic data with varying diversity
8

---

Published as a conference paper at ICLR 2026
Table 2: Covariance matching performs on par with the best baseline across three training paradigms on CIFAR-10, when the synthetic data is generated via various T2I generative models.
Method
Scratch
Distillation
Pretrained
No synthetic
44.36 + 1.51
47.33 + 0.57
63.40 = 1.33
Center matching (He
2023
53.46 1.95
57.67 + 0.58
66.52 + 01.1
Center sampling (Lin
2023
50.15 1.79
56.05 + 0.65
65.38 = 0.98
DS3 [[ulkund et 2025
54.15 2.17
59.43 1 0.73
66.00 1 0.94
K-meuns LLn et al
2023
51.63 + 1.29
56.77
+ 0.89
65.23 = 0.61
Random
51.26 + 1.96
55.27 +: 0.74
65.24 + 1.01
Text matching Lin et al
2023
51.20 1 1.82
56.08 1 0.57
65.93 1 0.59
Text sampling (Lan cl al. 2023
50.31 + 1.70
55.79
+ 0.68
64.93 + 1.12
Covariance matching (ours)
2.11 54.45
59.17 + 0.64
66.69 + 0.70
Real upper bound
61.08 + 2.54
65.38 + 0.51
74.35 = 0.56
Table 3: Covariance matching performs on par with the best baselines for two additional datasets. In (a), we train a ResNet- 18 from scratch on ImageNet-100 100 with synthetic images from StyleGAN- XL and T2I models. In (b), we train a linear model on top of an ImageNet-pretrrineed perturbation classification on a small subset of RxRx1 (Sypetkowski et al. 2023) augmented with synthetic images from MorphGen (Demirel et al.
pretrained ResNet for
2025
Method
Truncated models
T21 models
Method
MorphGen
No synthetic
10.78 1,29
No synthetic
86.83 : 2 44
Center matching (He al. 2023
53.39 1: 0.37
53.96 1: 1.06
Center mutching He et al. 2023
88.17 :: 2.35
DS3 (Halkund et al. 2025
57.47 + 0.87
53.51 + 0.31
Random
87.33 2.03
Random
51.11
0.82
19,84
1.32
K-means (Lin et al. 2023
89.00 = 1.70
Text matching Lin et al. 2023
53.39
= 0.99
53.37
+ 0.72
DS3 (Hulkund of d 2025
89.67 = 1.45
Cuvariance matching (ours)
$7.52
0.36
33.07
Center sampling Lin
2023
88.75 2.27
Real upper bound
62.67 0.65
Covariance matching (ours)
90.00 = 1.86
(a) ImageNet- 100 dataset
(b) RxRx1 dataset
and fidelity, and we note that the diversity we refer to concerns the synthetic dataset obtained from the generative model. The results of Table demonstrate that covariance matching outperforms all
baselines for all training paradigms. Table i0 Appendix C Suggesss that this superiority is partly due to selecting more diverse samples, evident from the improved Recall (Kynkäänniemi et al. 2019) FID (Heusel et al. 2017). and KID (Binkowski et al. 2018) scores guaranteed by covariance matching. We further notice that covariance matching selects 268, 245, 333 samples from the three StyleGANs with truncation 0.2, and 3692. 3462 samples from the two StyleGANs with truncation 0.6, pointing out the preference of covariance matching towards more diverse samples. Going beyond ResNets, we also demonstrate the effectiveness of covariance matching for transformer models in Table in Appendix C Second, we test text-to-ine to to-image (T2I) generative models. To do so, for each class we generate 4K SANA-1.5 [Xie et al. 2025). 4K PixArt-a (Chen et al. 2024). and 2K StableDiffusion1. 1.4 (Rombach ct al.
2022 images. Table 2 shows that covariance matching also performs well in this mixed setup.
Finally, to demonstrate the generality of our findings, we consider broader dataset from computer vision (ImageNet-100) and a specialized dataset from fluorescence microscopy (RxRx1, (Sypetkowski et al. 2023)). ). Once again, the results reported in Tables Ba 3blshow that covariance matching performs on par with the best baselines in all settings. Additionally, ly. the effectiveness is also shown for text classification the Tronic-Tweet dataset (Van Hee et al. 2018) in Table 13 in Appendix
C
Additional controlled experiments. We report additional results in Appendix C In Table 5 we consider zero-diversity generators. Specifically, for each class, we combine 2K StyleGAN2-Ada images with a total of 8K images produced by two zero-diversity generators. Each of these generators emits a single prototype per class: one near the class center of the real samples, and one near the class label's CLIP embedding. This yields high precision, but low diversity relative to the real distribution. Our results show that, again, covariance matching performs well as it avoids selecting many samples with low diversity (collapsed clusters). In contrast, not fully taking into account the diversity of selected samples, methods like DS3 perform rather poorly. In Figure 2 we consider inserting images from the target distribution into the pool of synthetic images and test the ability of different methods to select them. Specifically, we form a pool of 4K StableDiffusion! .4 images and IK images from the target distribution (different from the n4 200 images forming the training distribution), letting

---

Published as a conference paper at ICLR 2026
each method take n, 800. Our results show that covariance matching selects the highest fraction of images coming from the target distribution, whereas other selectors largely fail to do so.
Additional ablations. In Tables 6 7 we repeat the experiments of Tables 2 with DINO instead of CLIP features, demonstrating that the gains of covariance matching are not tied to particular feature extractor. In Table 8 we compare covariance matching with the direct optimization of the objective given by Theorem 4.1 As the outcomes of these two procedures are largely similar, this further justifies the covariance matching objective. In T we show that our findings replicate in wver-parameferized regime. In Table TO, examine the distribution of selections produced by each method, quantifying alignment with the test distribution and identifying which metrics best predict downstream accuracy.
In Table II we analyze the effect of varying the number of real training samples and synthetic augmentations. Table 12 reports the performance of different variants of Covariance Matching.
All these tables and figures are reported and discussed in Appendix
CONCLUSION
This paper advances understanding of the precise connection between training on mix of real and synthetic data and generalizing on real data. We start with high-dimensional linear regression analysis, where we find that only covariance shifts, and not mean shifts, affect the error. Even if our theory ignores the interactions between classes that would affect neural network training, the resulting insights transfer to realistic settings. We empirically demonstrate that matching the covariance between samples from real image classification datasets and generative models (irrespective of whether they are from GANs or diffusion model variants) improves the accuracy deep networks (ResNets and Transformers) under different training regimes (from scratch, distillation, and fine- tuning). In fact, our principled approach even performs on-par or better than existing baselines (Hulkund et al. 2025 He et al. 2023: Lin et al. 2023). Future work could extend the analysis to multiple Gaussian mixtures, which corresponds to optimizing the actual risk as opposed to modeling individual classes. We speculate that this may yield different insights when the training data have extremely imbalanced or fine-grained classes. It would also be interesting to introduce a model shift (different / between synthetic and real samples) under covariance shift. In fact, synthetic data often has small differences compared to real data, which a model may overfit on, and the phenomenon could be the cause of the collapse sometimes observed practice (Shumailov et al. [024]. Finally, we have only focused on generalization, but other quantities may be studied in this framework, including uncertainty calibration (Nixon et al., 2019), differential privacy (Dwork) (006). fairness (Barocas) et al. 2020), and validity for prediction-powered causal inference (Cadei et al. 2025).
ETHICS STATEMENT
This paper presents work whose goal is advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Our experiments in computational biology should not be interpreted as a direct clinical application.
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our results. Full implementation details are provided in Appendix C Additionally, provide the codebase as supplementary material of our submission.
ACKNOWLEDGEMENT
P.R., F.K. and M.M. are funded by the European Union (ERC, INF project number 1011613644
Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. F.L.'s 's contribution to this research was funded in part by the Austrian Science Fund (FWF) 10.55776/COE12. The authors
10

---

Published as a conference paper at ICLR 2026
thank Pragya Sur for insightful discussions and Berker Demirel for his help with the experiments cell microscopy images.
REFERENCES
Shekoofeh Azizi, Simon Komblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023.
Zhidong Bai and Jack William Silverstein. Spectral analysis of large dimensional random matrices. Springer, 2010.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning. Recommender systems handbook, 1:453 4599, 2020.
Peter L.. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 1177499:30033-30070 2020.
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint.
Acta numerica, 30:87-201, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learninn trade-off. Proceedings of the National Academy of Sciences,
practice and the classical
116632):1584--15854 2019.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data, In International Conference on Learning Representations, 2024.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Leurning Representations, 2018
Simone Bombari and Marco Mondelli. Spurious correlations high dimensional regression: The roles of regularization, simplicity bias and praaeettttttiiiionnn In International Conference Machine Learning, 2025.
Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions on Machine Learning Research, 2023.
Riccardo Cadei, llker Demirel, Piersilvio De Bartolomeis, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. Causal lifting of neural representations: Zero-shot generalization for causal inferences. arXiv preprint 2xXiv:2502.06343 2025.
Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6974-6983, 2021.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024.
Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, 52(6): 2879 - 2912, 2024.
Berker Demirel, Marco Fumero, Theofanis Karaletsos, and Francesco Locatello. Morphgen: Control- lable and morphologically plausible generative cell-imaging. ICMI. 2025 Workshop on Scaling Up Intervention Models, 2025.
Lisa Dunlap. Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion based augmentation. Advances in Neural Information Processing Systems, 36, 2023.

---

Published as a conference paper at ICLR 2026
Cynthia Dwork. Differential privacy. In International colloquium automata, languages, and programming, pp. 1-12. Springer, 2006.
Aymane EI Firdoussi, Mohamed EI Amine Seddik, Souftane Hayou, Reda Alami, Ahmed Alzubaidi, and Hakim Hacid. Maximizing the potential of synthetic data: Insights from random matrix theory. In International Conference on Learning Representations, 2025.
Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv: 700..06633. 2017.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetie images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 7382-7392, 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36: 11192-27112, 2023.
Scott Geng, Cheng- Yu Hsieh, Vivek Ramanujan, Matthew Wallingford, Chun-Liang Li, Pang Wei W Koh, and Ranjay Krishna. The unmet promise of synthetic training images: Using retrieved real images performs better. Advances Neural Information Processing Systems, 37:7902-7929, 2024.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
Qiyang Han and Xiaocong Xu. The distribution of ridgeless least squares interpolators, arXiv preprint arXiv:2307.02044: 2023.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In International Conference on Learning Representations, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531,211111 2015.
Roger A Hom and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Neha Hulkund, Alaa Maalouf, Levi Cai, Daniel Yang. Tsun-Hsuan Wang, Abigail O'Neil, Timm Haucke, Sandcep Mukherjee, Vikram Ramaswamy. Judy Hansen Shen, et al. Datas" 3: Dataset subset selection for specialization. arXiv preprint arXiv:2504 2504. 16277, 2025.
M. Emrullah Ildiz, Halil Alperen Gozeten, Fge Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In International Conference on Learning Representations, 2025.
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data. Advances Neural Information Processing Systems, 37:110246-11088) 2024.
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate- gan: Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2026
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33: 2104-121144 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards statistical theory of data selection under weak supervision. In International Conference on Learning Representations, 2024.
Yu Kuo, Yin-Hsiang Liao, Yu-Chieh Chao, Wei- Ma, and Pu-Jen Cheng. Not all LLM- generated data are equal: Rethinking data weighting in text classification. In The Thirteenth International Conference Learning Representations, 2025.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10443-10461, 2023.
Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 638 0888647,22 2023.
Neil Rohit Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-nome norm interpolation under covariate shift. In International Conference on Machine Learning. 2024.
Albert W Marshall, Ingram Olkin, and Barry C Arool.. Inequalities: theory of majorization and its applications. Springer, 1979.
Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max mar-in linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01 2019.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. International conference on machine learning, pp. 7176-7185. PMLR, 2020.
Sergey Nikolenko al. Synthetic data for deep learning, volume 174. Springer, 2021.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, June 2019.
Rishubh Parihar, Abhijnya Bhat, Abhipsa Basu, Saswat Mallick, Jogendra Nath Kundu, and R Venkatesh Babu. Balancing act: distribution-guider debiasing in diffusion models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6668-6678, 2024.
Pratik Patil, Jin-Hong Du, and Ryan Tibshirani. Optimal ridge regularization for -itto--distribution prediction. In International Conference on Machine Learning, pp.
2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8766 PMLR, 2021.
13

---

Published as a conference paper at ICLR 2026
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space biasing In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 9301-9310, 2021.
Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In International Conference on Artificial Intelligence and Statistics, pp. 3889-3897. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Mohamed Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training synthetic data? a statistical analysis of language model collapse. In Conference on Language Modeling, 2024.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2107-2116, 2017.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 63 (8122))))755577 2024.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 16857-16867, 2020.
Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpolators in transfer learning. arXiv preprint arXiv:2406. 1oXi::200..1344,, 2024.
Gilbert W Stewart and Ji-guang Sun. Matrix perturbation theory. Academic Press, 1990.
Maciej Sypetkowski, Morteza Rezanejad, Saber Saberian, Oren Kraus, John Urbanik, James Taylor, Ben Mabey, Mason Victors, Jason Yosinski, Alborz Rezazadeh Sereshkeh, al. Rxrxl: dataset for evaluating experimental batch correction methods. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4285 4294, 2023.
Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In International Conference on Learning Representations, 2024.
Boris van Breugel, Tennison Liu, Dino Oglic, and Mihaela van der Schaar. Synthetic data in biomedicine via generative artificial intelligence. Nature Reviews Bioengineering. 2(12):991-1004, 2024.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of the 12th international workshop on semantic evaluation, pp. 39-50, 2018.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self -effraeeeetted instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 13484-13508, 2023.
Denny Wu and Ji Xu. On the optimal weighted & regularization in overparameterized linear regression. In Advances in Neural Information Processing Systems, volume 33, pp. 10112-10123, 2020.
Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. Faimesss feedback loops: training synthetic data amplifies bias. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 2113-2147, 2024.
14

---

Published as a conference paper at ICLR 2026
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng YU, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of training- timenng-- and inference-ting time compute in linear diffusion transformer. In International Conference on Machine Learning, 2025.
Fan Yang, Hongyang Zhang, Sen Wu, Christopher Re, and Weijie J Su. Precise high-dimensional asymptotics for quantifying heterogeneous transfers. Journal of Machine Learning Research, 26 (188):1-88, 2025.
15

---

Published as a conference paper at ICLR 2026
A PROOFS OF THE THEORETICAL RESULTS
Additional notation. We use the shorthand [x] := {L...., n for an integer n. Given a matrix M, its operator norm is denoted by ||M |2. its i-th largest singular value by (2(M) and the corresponding i-th left-singular (resp. right-singular) vector of unit norm by u, (M) (resp. U; (MM)). Additionally, when applicable, we denote the i-th largest eigenvalue of M by (M(M). We use IR--0 to denote the set of all p x positive definite matrices, and 5 to denote a (p 1)-dimensional unit sphere. We denote by e, the i-th element of the canonical basis of IR', where the exact exponent is assumed from context. We will say that an event & happens with high probability (w.h.p.) if and only if P(E) + 1 as p, oo. Moreover, we will say that an event =\t happens with overwhelming probability if and only if, for any large constant D O, P(E) 2 1 12 for large enough Lastly, throughout this appendix, we use to denote a constant (independent of n. p) whose value may change from line to line.
For convenience, we recall some notation and definitions from Section 3 Namely, we denote by Z E Raxp random matrix with i.i.d. entries having zero mean, unit variance and bounded v/-th moment (for some き > 4). Recall M(() E IP, for (i) E {s, t). such that | = T()) / where r(a) a constant, with a constant angle between them = (1s,, (42) /[llelllll [ Also, Σ, E, RPxp be covariance matrices with bounds their spectrum as in Section Then, consider
Z 1/2 + In.4 2 1g
a data distribution
=
E TR=x and introduce its zero mean counterpart
X =
Z Z, 1/2
The corresponding sample covariance matrices are defined as E = x and
Σ.
X x
Lastly, unless stated otherwise, we work in the regime n/p = 7, where y * is
fixed constant independent of n and
A.1 PROOF OF THEOREM
We first state and prove useful results, in which we analyze the behavior of singular values of low-rank perturbation of matrices.
Proposition A.1. Let 2 ... min(npp) be the singular values of 7 = 2+1.H exists a constant c(7) > independent of n, such that, almost surely,
Then, there
vn
lim inf min(n.p) 2 c(7)).
n0000
Proof. To simplify notation we will refer min the smallest singular value of matrix. Let us choose an orthogonal matrix Q E Rax= such that Ql Vnei. Since singular values are left orthogonally invariant, we may replace Z by
Ż' = QZ Vn
+ te
Writing the rows of QZ as
Q7 =
2 7
21 E RP, Zo E (1 11xp
we have
Ž' =
Zz
For any unit vector E RR,
/ = v
1
2
+
Zest v
>
Zzz
2
2
Hence, by the variational definition of singular values, we have
Umin (2) 2 Omin
2x
n-1
Gmin
Vn-I
2,
(A.1)
16

---

Published as a conference paper at ICLR 2026
By the Bai-Yin theorem (Bai & Silverstein 2010 Theorem 5.11), for an (n 1) p random matrix
Zz with i.id entries with mean zero, unit variance and bounded fourth moments it holds
min
Zz
a.e.
1 Vp/(n
V
Therefore, applying lim inf, -0
A.1
we have
lim inf min ( lim R-+00
Zz
n-000
12
Uii
v/
lim
á-1
min
vn
32
1/2
0,
which gives the desired result as + 1.
2, + n,,44
Proposition A.2. Let N,, = X/vn -
Ruxp, Let oe 2 ... on be the
/22ii//2 + -,n,
singular values of Xn and v1 ....... be the corresponding right singular vectors. Then, as n + co, the following results hold:
1. For &p 1, we have
la. = ((VD),
((VD), and σ3 0(1);
lb.
(v).
0 1
Teall Palla
1 l12
(vr,
(v2.
O 1
|1 Mell
2. For y have
2a. 01 O(= O(1);
= 0 1 (1).))))
2b.
|eill
Proof. Let us first abuse notation and write 1, by n, zeros) and 1n, = [0, .... 0, write
[1,
1, 0, .... TJ E Rnx1 (n, ones followed
R= (ng zeros followed by n, ones). Then if
1
2,
X, = Vn
1/2
holds
Pn == 1, 1n
N, X,, + Pn:
where
(A.2)
Vn
To obtain the wanted result, will need to express the non-zero singular values and the corresponding right singular vectors of the rank-2 perturbation P,. that is oz (Pa) and U; (P.), i E [2]. Notice that
P. Pr adp.pt ofPaNT.
where 0; V and ai = V Moreover, it holds
P
(A.3)
where Q, a, a
R2pp, Note that
O = at E1lmall3 122
agar (Ho. ue)
= la b
a (Mo: pe)
of 1lleell2
b d
and it enough to analyze its SVD, since
(P,) (1)
and
((PP)) o,(P.)
2 (12 Qu
17

---

Published as a conference paper at ICLR 2026
The previous equations hold due to A.3 since d2(pp) (.())). and
(1p)pp 0
11 (Qp)
ui (p)) (Qp) uz(Qp)] (P(P,)) wo (Q, 0p (Qp))
(p()
222 (2(
This implies that, for E [2], the singular vectors 2; are in the spanf Me). Recall that the Rps.Pe)
angle between μıs and ir is fixed yp | "ulla
We first consider the case when vp < L. It holds that the eigenvalues of QrQp are
022(9) = a d + v (a d)2 + 40
(r3ag + rra - (ria
4a
(A.4)
2
(r + r)
(r ra
40 3
2
p C1.
with C₁ (1222t177)) m ( 0, since < 1. This implies that
0i oi(PP) c. vD.
for some constant
Furthermore, it almost surely holds that
(X,), XXTTX
+ 1//2 2, 2, ZT
1/2 z V (E
2,-)/
due to the convergences of the largest eigenvalue of the sample covariance matrices Z 2g Z, and Z, Z by Bai- Yin theorem (Bai & Silverstein 2010 Theorem 5.11) and the boundedness of the spectrum of E, and E, Then, from Weyl's inequality for singular values (see e.g. (Horn & Johnson 2012 Chapter 7)), we have that
(X,,, + Pr) 2 (P,) - (X(n,, for i = 1,2,
03 Xx + Pw) 03)(PP) (X,, o
which implies that 1,2 + Pa) > Vp. whereas (X Px) O1), for 2 3. For the upper bound, note that from (A.4 it holds
12 012((QT))
+ (1:03 + + C
2
implying (Px) Se. vp. Applying Weyl's inequality for singular values once more, we get
(X, + Pw) < a (X.) (P.,)) =
concluding the proof of la.
Moving onto singular vectors, let us recall the definition of spectral distance between two k- dimensional subspaces W < TRP and V < TRP, it will be used to conclude the proof. Towards this end, we first introduce principal angles 0 ....0 [0, r//2 between W and W. which are defined recursively from i
0g
{wi, W:) max Ne.WWa
COs (os(0,)=
EW
18

---

Published as a conference paper at ICLR 2026
subject to wa, Wi being orthogonal to the previous maximizers. Then, the spectral distance between W and W is defined as
d(W, W) = maxsin 0,..
sE[R]
There is an alternative way to express this spectral distance between subspaces, using their orthonor- mal basis. Namely, let W RPx and W R be such that their columns form an orthonormal basis of W and W, respectively. Then by (Stewart & Sun
1990. Chapter II, Corollary 5.4) it holds
d(W, = |1| WW )
(A.5)
2
Let us denote by V 11 (Pn) V U1 (x)) and by v the subspaces spanned by their 02 X,,
v2(Pn)
columns. Then, by Wedin's sin θ theorem, Stewart & Sun 1990. Chapter V, Theorem 4.4.) it holds that
(X,)
d(v.V)
(2 + Pa) -33x, + Pr) c. /D 0(1) = 0
As vi (Pr), 12 (Pa) E span{Me: [r} and they are linearly independent, this implics that V span(fu, }. Choosing matrices V, E IP*x and V. E Rpx2 such that their columns are orthonormal bases of V and their first column is Me Il and Il respectively, one gets that
(I VV
)
[T VV Ve
[T VV v. d(v, 0
||
vp
VVT)
Afr
-I Mee S [1 (I- M = d(V, d(v, 0o
||
vp
From this, 1b directly follows. The case & = 1 is handled analogously.
Proposition A.3. In the under-parammterizee regime, i.e., when < n, it holds that
1 Tr[E (2, + Mppl )]
1 Tr Ed + 0 -
(A.6)
Proof. We break down the LHS of A.6 into two terms
Tr
+ (Ee + pept.) )] = Ti + T2.
n
where
1 T Edl, Ti
T2 = -TPE -T[EE=+11) p.u.).
and We will deal with each of the terms individually.
n
Bounding the term Ti. It holds that
T = Tr(E+E,)
XXX-1/2
T
Xx, 2 1/2
= - Tr
nt
vn
(A.7)
1 Tr (XTx)-1
b 1
i=1 o? (X)
where X = XE, E RRxx, and k < p is the number of non-zero singular values of X. Let us prove that (x( > c for some constant c, implying that l = p. Towards this end, we write out
x X X,X,. X.,
19

---

Published as a conference paper at ICLR 2026
where X, = X, 1/2
(2,22 22,22/241,0 +1 2=. 1/2
and X, = X, 1/2
2 1e F.c 1/2
+1
v
y'
vn
v
From Proposition A.T. it follows that for large enough n, almost surely
(x(X,)
op 2c,
for some constant c, which is just c(r) from the proposition adjusted by the bound on the eigenvalues of ,, 1/2 and E 1/2 (recall that the smallest eigenvalue of Eg, De is lower bounded by 7). Plugging this in gives
(x(2)² 2 xxx(Xx)++xxx))
+ ox(xx)) 2 2c².
(A.8)
Let X X inequality for singular values, it holds that, for E (3..... ................................ 222
v and note that X is a rank-2 perturbation of X (see (A.2) Then, due to Weyl's
)42((*) S ox S ox-2
Therefore, we have
(X0)
(Xu)²
An application of the Bai-Yin theorem (Bai & Silverstein 2010 Theorem 5.11) gives that there exist constants a and b such that
0 ( < too,
for large enough n. Therefore, holds
2
0
2 <
Σ
(X0)2
i=1 oi (X0)2
3
which implies that
1 p-2 2
1
+
1m3 0i (X)
im1 8. (80)2
n2
Using the proved fact that o, (X) > c we have
1
2
+ o
(H)
(X)
oi (x)
2
Combining all the pieces, it holds that
P T
(x
n
+ + 1
(x)²
11
(X002 +0
0i
72
-1 + 0 (
x0)
1
+ 0 p
20

---

Published as a conference paper at ICLR 2026
Bounding the term T2. First, recall the shorthand X, X/ / and note that
(x(x,) = Tp( (xx 2 xx()) op (x. 2 C. where the last inequality follows from (A.8) and the bounds on the spectrum of E. Recall that n/p = y, which implies 0 ( = (t) Then, it holds that
(A.9)
1
72
11 E+M
14
M
(X,
Xxn*
H4 v/
Vn
ox()),) U/ (X (Xx))
Me Vn
(A.10)
(X())) 2
2
U2 (2( 14)
(x((x)
1
01
(X,)²
02 (2()))²
n1
1=3 (X())
< 0
1- 0
1
1
0
where the penultimate inequality follows directly from (..)) and Proposition A.2 Finally, combining the bounds on the two terms we get
Ti + Tr E, +0
proving the claim.
We conclude this appendix with the proof of Theorem 4.1
Proof of Theorem 4.1 As proved in Section
4.1 holds that (x 3) 0, from which follows
a² n Tr/E*(Ee (E, + p1p4
Rx Rx B) Vx (8:
By directly applying Proposition A.3 it holds
2²
T Exl ( [ )
T
+ mi
n
n
Plugging in the expression of a² Tr 2n] given in Yang et al. 2025 Theorem 3) gives the desired result.
where Eo x x Vn
A.2 PROOF OF PROPOSITION 4.2
Since we are in the setting where n > p, it holds that Bx (3: 3) = 0. which implies
0² T (E, + m.p )}-
Rx Rx (x
n
Note that 7 implies that X, and X X.. We also note that (A.8) still holds for X, implying that 2 is of rank p almost surely and, therefore, invertible. Thus, holds
Tr (E, + pa )} Trp + pept
To simplify exposition, we break this down into two terms
Rx 8) = V1 Vi2
with Vi 2² Tr(S-119 V2 = -² T [. and treat each of them separately.
21

---

Published as a conference paper at ICLR 2026
Bounding the term V2. Note that "/ 0 implies na will use these two values interchangeably throughout the proof. From the cyclic property of trace, we have
a² TrE pepel1
02/4 Σ Me
V2
n
Note that 14
(XTX
T 14 = 114
1tt
n
(Z,E!? + mm (Z,E!! + +mmats
-1
14
Mr
(I= Hb)
Ex 2/2
Zo + Inn
Zg + In,
(s)
E3 1// pu
n
ml E 1-1W u/:
where we use the notation p4 == Σ. 1/2 1/2 Pe. 14 = =5.==1-0- 1/2 Ms and Σ' (Z.tIn, ". (2.+ln,
" Note that due to the assumed bound on the spectrum of Dg it holds that ||4l|22 = OlvP) and ||M/l|2 = O(vp). Next, let us break down the vector μ4 into its orthogonal projection onto the subspace {u'} and {u(}+ as
Mlle = (ple p'a)) uhe |1
14 = Moje + Whis. where
phss = 14g pllls
(A.11)
Moreover, decomposition into orthogonal spaces, it holds II + ||ا(_ا11 = 11m2l1 O(p). By using this decomposition, we will shift the focus from pi phls. Namely, it holds
1/2 02 g2 E' 0² (W6le + (ILL) (Wbpp + 1bLo)
7
n
0 "ilo Sy-
1141 20
Meles
Hell
Mile sy
1
14
141l=
vn
vn
vn
vn
vn
vn
2 2 - +0 Vn
(A.12)
Vn
where the last line follows from derivations analogous to the ones around (0), this time applying
case 2. of Proposition A.2. To ease further exposition, we introduce ju,. tiee noting that |لا 112 = 0(1.
vn
In order to bound 1/2, relate E' to its zero-mean counterpart, as it is casier work with mean zero data. Towards this end, write out Σ'
Σ (7, + 1,,
(T) T
(Zg + 1 (
n
2g T 2
2g T
1L, Zo In,
Noll IT
n
n
n
Z.
Ln, M4
TT
11
Z.
+ 1igtt
20
n
22

---

Published as a conference paper at ICLR 2026
for En = Zz, All the terms above, except the first one, have rank so we Woodbury formula to take them out of the inverse when computing Σ" We introduce the following notation
"
Ze In. V
u
vn
Vn
n 1 0
U= lu 2/ E 1xx22
and C =
E R222
Under this notation it holds
Z. In, 14
11.ti.Z Zo
+ Pla T UCUT
71
n
Then, using Woodbury formula, we have
Đー = (5% + ue vu + nun") )
(2% + UCUT)
$% $1/ -UU -00
We now compute the 2 x block
22-11 11 -
1- u 1- y
-a 1-b b
- v 201 -1 11
- - v 25/%" -10
1-b b
where
a: un E01 -1 u,
b:= u S1 -1 -u ut 5% -1
d= 0 81 11
Hence
1 -n- b (C-1-
= a(n d) - (11
A b-1
Plugging back and simplifying gives the explicit formula:
Đ' E
A E6 11 ((-n d) uu T - - buw + vu v) a vu vu $ -1
which is valid whenever A * 0, i.e., whenever C-1 U 5% -U invertible.
We will now analyze the b, d terms. First, for some constants C₁, C2 >> 0 it holds almost surely that
> A1 (E) 2 (E))
0,
(A.13)
which follows from Bai-Yin theorem Bai & Silverstein 2010 Theorem 5.11), as Z, has i..d entries with mean zero, unit variance and bounded fourth moments. From this, it follows that
|a|
"
u T 10
11
Sy
12/
Vn
/n
2
1//
//
x
1159 -1122
Vn 2
2
$ c,
as well as
|a|
2 c. (A1(E))) - 2c>0, 0,
Similarly, we have
|b| = u 1 -tu|= =
1 Vn
2
<
14
13 1|12
Z 1n,
S Scpp, (A.14)
2
n
2
23

---

Published as a conference paper at ICLR 2026
2 Iiee
where the last inequality follows with high probability over the sampling of 7s since vector with i.i.d entries of mean and O(1) variance. Finally, have vn
is
d
2% -
n. 7s
2g 1n=
20 -1
Vn
Vn
2g 1n,
2 ln,
12 1 1112
vn
72
cp.
again with high probability.
We can now prove that, with high probability, A 2(p). Using "aachh--Shwaan it holds that
b² (u, v) $ ||u|| ||||| ad,
from which it follows that
A (n+ d) (1-
6)2 an 1 26 92(p),
(A.15)
since is lower bounded by a constant and 16| S c/p.
Turning back the value of interest, write
Pri. E' pels 1i.1 TiLL
1 S- 2 ((-n d) uu - buw + trul ) wvv E% PeLs
PLL.
A
7..5% iirLo + Tu.u + Tu, + To.y:
where Tu, is the summand corresponding to uu T Tu, to uo + ou and Te to vo We will prove that each of these terms, except for j4, 1.
Ey. -1 is vanishing.
First, we state useful claim, that for arbitrary deterministic unit vectors wi E IRP and w2 E RP holds with overwhelming probability
5- U2
7 (1), W2) + (n=)),
(A.16)
7-1
for some constant > 0.
Proof of claim in A.16). The result follows directly from (Yang et al.) 2025 Theorem 27). For clarity, we refer to the relevant parts of Section B.3.1 of that work. While Theorem 27 is stated in the more general anisotropic setting, specializes to our isotropic case by taking U and V from (B.3) from their work to be the identity. Substituting these choices into equation (B.6) from Yang al. 2025) for implies
(0) + a2( = p 7-1
n
Substituting this into (B.7) and applying Theorem 27 from the mentioned paper, yields with over- whelming probability
w E/ W2 w1 7 u2 In
7
for any Ci < -1/2 2/1b. Recalling that Z. has its 1hth moment bounded for eb 4, implics
We use (.16 tackle the terms Tu. and Tu, Namely, we have that
(- 9-1
Pele S/-1 u = -IlãcLo |lull2
(Teee) u) + O(n =2)
7//
14 + Onnna1))
7 1-1
vnl
O(n -o))
24

---

Published as a conference paper at ICLR 2026
with high probability. From this, it follows that
1 d
T Tu.
MeLs
20-1 "uu T5 - (21)).
(A.17)
A
Similarly,
1- b [T,ol
A Me 20 (uv T + ru ) ES - "iees
[ 2 -v)= (لك -1 (
- O(n
|| 1-b Z Ins
1 - 2
||leell22
2
= O(n
(A.18)
where the last inequality holds with high probability due (A.13) Let us denote by In,
and A.15)
Iov and turn the term Te... Notice that vn
Te
A
MiLe
-
vu
Σ
pels
-1
2
n2
Z
Zg Z.
pooee
A
n1
Vn
T1
2,
2, T
2
IT
Pels
(A.19)
Vn
11
Let us introduce matrix Q = [41 such that Φ1
gpl E RPKP, whose columns form an orthonormal basis,
Then, we have that
Z. vn
Z.
Zo
-1
Zo QQ
Z. To
Z.
-1
it
PLLo
IT n, Vn
PuLL
n
n
Z.
Z.
ql
T
(A.20)
Using (A.16) and union bound, holds with overwhelming probability that
7,5 7,5
f4Ls = 7
k = 1, (gk. + O(n = 7-1 |A + O(n-a),
Qk
n
--1
((n
k> 1.
Plugging this into (A.20) yields
-1
IT T,
Z,
Z,
Z.
P.Lo IT Z. n, Mels
7
+ O(n 9)
Σ
iT Z. n,
9k.
(A.21)
V
vn"
Let us first analyze the mean and variance of the random variable iT
Ze /n
L namely,
Z
P
iT Hi
E
Zr (I (
=0,
n
Vn
i=1
j=1
Zs
1
p
Var
iT
[ir:s)
Var
Zi (In, )i((ees))
[ || |1 = =0 (
25

---

Published as a conference paper at ICLR 2026
Therefore, using Chebyshev inequality, we have that
i T 7x vn
MeLs
0 ()
with high probability, for some constant of the random variable Ek=1 iT,
1/2 > > 0. Similarly, we calculate the mean and variance
Z, vn Qk
7
Qk
E
0, Zeo (mnD)
12
n
Z.
1
P
8
P
Var
9k
Var
Zo (in))))
Vn
vn
j=1
n.
11qul12 0(1)
Again, Chebyshev inequality implies
O(n 9)
2
IT D,
Z 0k
= 0 00 (1
k=1
with high probability. Plugging the obtained results into AA.2 and using a union bound on the probabilities, we get that
Z.
Z
Z peLe < ((n a))
Z
Zg /nl
O(n
IT
qk
+
iT
pels
n2
k=1
0 (n-c///))
with high probability. Then, we directly obtain a bound for (A.19) the form of
T (n)
(A.22)
which holds for some constant C₁ > 0 with high probability. Combining the bound in A. and the three bounds on the terms 17.17 A8.18) an (A.22). we get
pr E Pelo
7
| On
(A.23)
-
for some > 0. Using this in A.12 yields
V22 = o² Y
7-1 |1 + +(n
with high probability. Lastly, note that
MiLLL 122 11 1 (1141111 1124)1113)
1 (115_12 1/2 1el1
1/2
(
14 Σ Mss
2)
||ا 1/2
M l1
Bounding the term V₁. By following exactly the proof of the bound of the term Ti Proposi- tion A.3 one directly gets the same conclusion that
o²
Tri E.] + O (
v
Notice that
X
XTX,
1//2 Z, Z 1/2
ΣD
72
26

---

Published as a conference paper at ICLR 2026
Thus,
x²
0²
Zg
0² Tr
Z Zg
Tr
20
'Tr
1/1/
E-1/2
1/2
π
n
π
7
71
Let us write the SVD of E,"
1/2 E, 1/2
2/ " 1/2
Σ
i=1
where w; = 0i (E₁
Σ, 53 1/2).
Then it holds with overwhelming probability
Z Zg
Zg
σ²
[z Z. (Z
7 Σ
Tr
2 2
12,,,
W/.
π1
n
0²
X
1/2 E,,, -
7- (Iuu:ll+0(m 1000ll+00n00 (n--))
7 7
O(n
i=l
0²
7
A 1/2 + +((
n
7
0² 7
n y Tr (Σ,Σ,)) O
where the second line holds with overwhelming probability by using 16.16) and the union bound. The previous bound also holds with high probability, since overwhelming probability implies it. Finally, by combining the bounds on V1 and V22 one gets that, with high probability,
)
2
o² Tr (E,E,)
o²
M4 Σ 1i
Rx (B, n for some constant c 0. Taking the limit - oo on both sides yields the desired result.
|Σ 1/5
1/21/114 11l12
=0( O(nJ),
n 7 1
| 1/2 1,.11221111 Hall2
A.3 PROOF OF THEOREM
We start by removing 02 from the fixed point in as a for convenience. Plugging this into the definition of R., (M). we get
4.2 and replacing it by 1 - 0r We rename
RM(M) = Tr ( M + az Idpxp) -1 0²
a²
Xa - a
im1
where as in Theorem 4.1 we refer to d 2 2 Ao as the eigenvalues of the matrix MMT. Furthermore, the fixed point equation 4.2 can be rewritten as follows:
+ na ng
n - T2g
1)
(A.24)
Na +
2 aa
Thus, have
RM(M)
12 Th
= 0²
1)
-a--aa-- 1
B2e
n
(A.25)
p na
Now, due to the RHS of A.25 it can be seen that (M) is an increasing function of a. Let us Then, for fixed n,p, and X, we will refer to (x)) the solution to
denote by x= [AI..... Ag
the fixed point equation (A.24). Note that following Yang et al. (A2))nApennd) B.3.2] we have that
this solutions is unique and 0 < ax) a-p
Consider function f R 20 > RR
We call a function f good, if and only
Σ
(A.26)
(())(())))))))) -1 a
va(A) +1 - 2 "
-a(x))
We claim that if f is good, then
a(s(i)) < ca(t).
(A.27)
27

---

Published as a conference paper at ICLR 2026
Proof of the claim.
Consider a good function f. Then, we have
71
-1)
E (() + - -1 a (X)
xxx(X) = 1 - -a(x)
=
11 - p na(()) (X)
Furthermore, setting = 0 we get
Σ f(X); 0+1 2 =
==1
p 2hg
p
11 - ny
-1)
p
By continuity, there exists au E (()) for which
Σ
1
n2g
1)
(A)aaa + 000
71 - n200
implying a(f(x)) = ag a(x), which concludes the proof.
[p] s.t. i < j. we introduce function f/d .RP Next, for for i,j
RR defined as
A3 c
k i,
Xy +c
k= j. k * Fijj
A
where 0. We now claim that fad is good for any i,j E [p] and > 0, such that ); > M, tc.
Proof of the claim. The claim is equivalent to
1
(A, c)a((() +1 - a(t) (A) c)(((a) + - - a(x)
1
{x((a) +1- 2 - a(x) + )ya(d) +1 +1- -
For simplicity, let 8 ;= 1- 1 a(d) and = a(d). Then,
()1 c)a + + (2y + c)a +
Aa++ + Xyox +
a(4 + 21) 25
a(): +Xy) + 28 28 (Aça + o) 8)
(Na - ca + () + ca + 8)
(Ara 0)(2,a + 00 < (Aia 0a 5)(2,a + 5)
ca(dia + 8) - ca caaaaaa + 8) da² > 0
cai(), -1)) > o²a²
A \3 +
which proves the claim.
This implics that, for E (0, 1), transformations of the form
Ay, (td, (1 t)Ay, (1 t) + th2),
(A.28)
good.
28

---

Published as a conference paper at ICLR 2026
Let denote by X [],... 11. which corresponds to eigenvalues of 1p M M', that is M' = 1. E M. Pick any X" + X' that corresponds to some matrix M" E M, so it satisfies X as well --=
We recall the definition of majorization, as it will be used to conclude the proof. Namely, we say that I E is majorized by ÿ E IR whenever for all lo E [p]
k
Σ
Vi
1m1
i=
and
31
Di-
Firstly, we claim that X is majorized by X". Suppose otherwise, that for some [p]
k:
i=1
implying also that X < 1. Then, we have
p X < (p- k)
/
(p k)) + k = p.
which is contradiction.
Next, as X' is majorized by X" X can be derived from X" by a finite sequence of steps of the form in (A.28) with & E [0, 1], see (Marshall et al., 1979 Chapter 4, Proposition A.1). Since both vectors X and X" are non-increasing, the t = 0 transformation can always be omitted. Moreover, t = 1 is just the identity transformation, so it can also be omitted and we actually have t (0, 1). In formulas, we have that
X X fiii Sth (X")...)
Since each of the functions above is good, we have that a(²) < a(X"). As R., is increasing with a the smallest R, (M) is achieved for X' == [I,.... 1]. that is, Mopt = M = Ip-
A.4 PROOF OF R.(n & R.(M)
Consider the function fy Rh > Ry defined as 97 (X) = =),, for some v > 1. Note that, for all i,
Bn ())( + +1-
n7aa + 2
Aa + 2 -
Thus, (X) = n is good in the sense of A.26). From (A.27). , we obtain that a(7)) < <(()) This
implies the desired result as R2, is monotonically increasing in a from (A.25).
A.S COEFFICIENT DEFINING SYSTEM OF EQUATIONS OF THEOREM 4.4
The (a1, 422 a3, a4) is the unique solution, with 01, a2 positive, to the following system of equations:
ai + agX² 21 + a² + ( 0
f 01 + 02 +1 (
ai\a
0
(A.29)
a1 01 = 1 1/ a33' + a
di, (>*, X²), a1= 1 +2 dH,,(*,, ,\),
/ (a'
and (bı, bz, by, ba) is the unique solution, with bi, by positive, the following system of equations:
0 0-1-1 7 /
1 / b + ba
b bo +1 (x, 0 0= 7s
by + +1 - (X',
7
(A.30)
(bs b2 b2X*)
(2 - x2 bybb)
0=
di (X*,
0=
dHd (X
(X*x').
(b1 + b2 + 1)
(b, + 22 + +1)²
29

---

Published as a conference paper at ICLR 2026
A.6
PROOF OF THEOREM 4.4
Recall from 3.5 that bias and variance for non-zero centered data can be expressed as
Bx 3) - 3 [I, Mip II and (3 o²
Trp (E, + Molu, )],
π
where Σ = x X/n and - Σ+Σ (projection on the null space of X). To obtain the wanted result, we make connection to zero- zero-mean data and then use results from Song et al. (2024) to handle the zero-mean case. Unlike in the under-pant sadre-parametrizee case, the bias term does not necessarily vanish. Thus, we start off by breaking it down into two terms
Bx (3; 3) (3; B3 ();
where (3; = 3 II and (3: 18 II3. Morcover, we split the variance term
Vx (3:33) (3: (3;
with (3 B) كه E. and (3 B) = ]~ terms individually.
n TrI Me.. We will deal with each these
Bounding the term (3, Recall that X, X SVD of Σ
Then, similarly to A.10 can write the
vn"
S of
(X.)
i=1
where k min(n. = n is the number of non-zero singular values of Xn. As in (A.8), we can conclude that k = n.. Therefore, we have
I -ー Σ a (X
Σ u (Xn)))
i=++1
By definition, it holds that T = (1 E 5110 from which it follows
I1 Σ (x(,)) (us (1,(X)).
imn+1
Due to Proposition A.2
it holds almost surely that
2
(un (Xx).
(w2 (x), Maa
|oll|
|all|2
c c.p'
from which it follows
1
Σ (o i=++1
S
1 c.p
1m 11 =c
Since sampled independently from sphere of constant radius and ΠyІ is of bounded norm, is standard result that (88, Iy,)) is sub-exponential and, using Bernstein inequality, we can get that
( B) = T m = K1B, ma.) O
(A.31)
with high probability over the sampling 3.
Bounding the term ( We first introduce an object coming from a bias term of a ridge
regression estimator with coefficient A:
xA) = x
(A.32)
defined for any À > 0. is more convenient to work with (A) than Bx B approximates well B (8, B) for small A. We formalize the second claim
(8, B) and, in addition,
B 3) B3 (A) O(A)
(A.33)
proved in the same manner as (Song et al. 2024 D.82). For convenience we also carry out the proof here.
30

---

Published as a conference paper at ICLR 2026
Proof of the claim in (A.33. Let write the SVD Σ UDU Moreover, we denote by 1 D==0 and ID>0 the diagonal matrices such that
10, (LD=O)er = (1,
Dus # Dia =
[1;
Da, # 0 Da, = 0
(LDsO)i,
(0,
Then holds that
= =3'U1D=ol " _
= =1A41/10=00" B1
where we set A = EUU. Furthermore, we have
(A) -
-D3 XT) ATT
= -1
Therefore, we have
B3: V (x) & 1 - A1) (I) B112
s /3 11)
x -(A),
(2)
where the third incquality holds as ||Al|2 12 ((1) and the last inequality follows from
Proposition A.1 in the same manner as (A.9). Notice that ( = 0(1), since
[| ||aell2 = O(1) and o, (Š) > c. This finally implies
(3:3)) - B\(0| = O(A).
proving the claim.
The next step to prove the claim that, for > > 0.49 holds that
(a) 32 (Eo -E, X1) ( 2
(A.34)
Proof of the claim in A.34 Towards this end, we have
1 xxTx)
1 [ + lncit + (n,,)))(((" 1m + In,13)
ToTxx xo
Xo Ind +
Xo In,
pe pel x
n, IT X
7t Hilt
7s +
n
n
n
n
where abusing notation we write 1n, [L, zeros) and I,, = [0, .... 0, 1, ..... 1] i (n, zeros followed by n, ones).
1, ..., T E Rax1 (n, ones followed by ne
All the terms above, except the first one, have rank 1, so we use Woodbury formula to take them out of the inverse when computing (É + AT)-1, We consider the case &p * 1, as the case 3o = is analogous (it is in fact easier some steps be omitted).
31

---

Published as a conference paper at ICLR 2026
x
Mel x0
We first focus on the term (E++ and demonstrate how to handle For this purpose, we introduce the following notation
2 4e
A -=
X
1n.41
M4
T
X
Yt
Melle
n
n
1 Vn
X
Ing
(A.35)
M
vn
2t
1
and C U = ju v| E Rox2
R2x2
E
1
Under this notation it holds
X 19 + ue X
Mell UCU
Then, using Woodbury formula, we have
(1+ = A+
uv uv
+ vu T
+ n 2 =
uu
?
-A+ (CUTCUT]-
We now compute the 2 x block
c-1 U A-
-u -uTAA1 u
1 1-uTA-1 v
-a
1
T u
-n 2t - A '
1-b
-n d
where
:== T A u
b: T A u = x A A v
d: T. A A U.
(A.36)
Hence
(C-I-U-A-tU) 1
n - d b-1
A:
2 n 7
+d - -(1 (1- (A.37)
6-1
-a
Plugging back and simplifying gives the explicit formula:
(~+ A
1
A
nl d
uu (1 (1-b) (wv" T + +vu vo. A-1
A
which is valid whenever * O, i.., whenever C-1 UT A is invertible.
We will now analyze the a, b, d terms. First, recall that
T x
X 1n Ms 1T X
7s Halls + AV= = Ex + AI,
n
where ES (1* +I (x" +1+
Thus, we have
n
||^~ 1|2 -A-1
From this, follows that
|a| lal=|uTA
14
#4
<
44
AA-
11
Pt n
/n
Vn
2
2
Similarly, we have
16|
10
A
-Tul
X
/n
Vn
2
<
#4
||A-11|
X
1 n
Vn
Scx-1
VP.
32

---

Published as a conference paper at ICLR 2026
X me is
where the last inequality follows with high probability over the sampling X since vector with i.i.d entries of mean zero and O(1) variance. Finally, have
dd
0 T A
'
m1 X
X
In
A
Vn
Vn
X In
X 1 nz
||A
12
vn
v n
2
< c
-1
again with high probability.
From a slight adjustment of the second part Proposition A.2
holds for the top singular value
a (A) - o1 =Z. (0) 01
X + )n ))
2
+ + O(p).
vn
and for the corresponding right singular vector
K (1
|| 11 >> (E.),
Ms
|| 1
Me
(m
8 + In
Me
>l
--v
n
|/ l1
Note that, for < 1, it holds that (TEEITT Titellls > Cauchy-Schwarz gives
1.
Using the triangle inequality and
U1 (AA)
14
<
us
14
)
||(((()) (1(A))
us
S x 0 (
[plll2
|| ||
|| l
||
Therefore, holds that
la =
T
A 4
P4 A
pe
Vni
Vn
Σ
1
Pt. (:((),
(AA
Vn
1=1
(()(AA)
(A)
11ee
1=1
K (O((A),
2 6
(A)
[lmelll ll
1=2
Σ
(v:CA),
c
(A)
11Ase
1=2
(2) p )1)
M c 1
40 0 o
0,
since (A) = و3(Li,) + = 0(1) due to the second part of Proposition A.2 Note that, for yp 1, we not need this argument, as the μl terms are taken out of the inverse as well. In that case, we x x + A1). which immediately gives o1 (A)
take A
We can now prove that, with high probability, A = 2(p). Using Cauchh-SShwaad it holds that
b² = (u, 2)) IA S ||u||| || = ad,
from which it follows that
n 2t + d) - (1 6)2 z am 7
A
+ 2h = 12p))
7
33

---

Published as a conference paper at ICLR 2026
since is lower bounded by a constant and [6] I cx- vp v cp 0p0
At this point, we have all the necessary bounds and we work towards proving the claim. We first expand the bias term
(A) -,
-
A
where S ==
- -0-- U
A-
We now prove that is small. To do so, we decompose
- A1 '> E.
U -) UTATA A
- Λ
A-1
n ((11 +d) uu T + (1
- b) (wv" T + +ou vuu) + a m0 A-1 )
Taa Tu, + Tu,u
where Tu is the summand corresponding to un Tu, to uo vuu and To to vu Zooming in one of the terms, it holds that
T Tu = (E+ (n7/7 d)
A- uu T A
A
- -8)
(nys/ y + d)
A u (u² AuTA-1,3)
A
Note that
( X1) (ny//y + d) A-
<x² 121112 11)
11, 1122l22
(n/a/7 d)
||A- |ull2
A
sci
and TuTA~1l/2 I cA- Using this, we get that, with high probability, holds
X-2 Ta S
p
This is similar to how we obtained (A.3I, since B is sampled independently from a sphere of constant radius. With analogous passages, we have that
[Tan S c 2
A
(Taye S
holds with high probability over the sampling of (3. Putting all together, we get
A
= ($
p
Using the same argumentation applied now to - in A2 ( +
- gives
(A) = A 2
Lastly, doing all of this again to take out the terms containing иe from A, i.c., by taking
XOT 1.
x
Ā= A
7 Malte = En + AI,
we get
2 -
(A) =
A
+0
34

---

Published as a conference paper at ICLR 2026
proving the claim.
From (Song et al. 2024 D.82), follows that
D' - 2² (Eo + +1)
-1
Ee [o + )1)
-1
B = OA))
(A.38)
where 11 = I Eng Eo. Thus, by combining
A.33).
A.34
and
A..3
we conclude that
A 2
B (8,3))
- B o. = O(A) +0
= O(p -0pp111)).
(A.39)
where the last step obtained by taking p X-1/3 (this also satisfies 1 > À p which was required obtain (..44)). As xx (3, 3) = (3. () + (3, and (3, 5) O(1pp) with
high probability by A.33 we conclude that
- 8 -
Op 1/3)
(A.40)
holds with high probability over the sampling of B and X. Plugging in the expression of B 11 given in (Song et al. 2024 Theorem 4.1) yields, with high probability,
(x(3,3)= /
Ya + (b4 + 1)X² + + 1)² dG (A,, O(p-9)
where (b1, 62, b3, b4) the unique solution, with b1, positive, to A.30 Taking the limit p, n 00 gives the desired result for the bias term.
Bounding the term (2, Notice that the term V (3, 3) coincides with T2 from Proposition A.3 Moreover, we can follow the proof of the bound on T2 verbatim, only substituting for n in appropriate places (as are now in an over-parametrized setting) to get
o²
T ( n
V
(A.41)
Bounding the term V (3,8). 8). that, with high probability, holds
To make connection with zero-centered data, will first prove
o² TYE
1 TT[S + o p1/7
(8. M)
(A.42)
Similarly to the computation for (B, (), we introduce an object coming from a variance term of ridge regression estimator with coefficient A:
(* = 1 Tx -
defined for any À 0. It more convenient to work with (A) than (8 8) and, in addition, (A) approximates V (8, 3) well for small A. We formalize the second claim
(3, - (A) = O(A),
(A.43)
proved in the same manner as (Song et al. 2024 D.78). For convenience we also carry out the proof here.
Proof of claim in A.43). Let us write the SVD & = T Then it holds that
(3, TrUDDUT E.),,
(AE()= = Tr(UUDD = E.).
35

---

Published as a conference paper at ICLR 2026
Therefore, we have
V
|r
E (D=
1
1
A((D) (A(D)+ Aj²
<
>x(D)
21
- An (D)²
X
c.
= O()).
An (E)²
Here, we used the inequality x -1 (x + >) 2x S 2\/x² and the fact that E has non-zero zero singular values, each bounded below by constant, which follows from (A.8) This completes the proof of the claim.
Relying on the derivations in (ong et al.
2024
D.2) we have that
A Tr (2((++))=))
(}(() = dA
Let us denote by
V(A)= ( TT
We claim that, for any t > 0, it holds
V=((a) = ( 3)t2-3).
(A.44)
et
Proof of claim in A.44 We begin by transforming the LHS:
1 ( (A (A) (S+ (A +a+)))) - -1 )1 ")
(A + EA) - V2(A
= Tr Σx i
( ++11 -1 - (AS+1)
Tr
Σ ya
(H+++) ((++))"(((+)1 x (xi=S+1)))
(+S+1-
Σ i
n
1
Σ, (S+M) "an))""''
S (S+(A)))))))
1 Tr (5+(2+ENN))
(S+ ΣΣ
=
where the last line follows from the cyclic property of the trace and the commutativity of E,
( (v) -1 and ( + (A (A+ -1
Plugging this into the LHS of (A.44) yields
(t(A) -
1 iy
(1) (1)+++1))2(())))
1 Tr
((++x))*
-1 - (5+ ((=+xx"-((++(+=)
-($+(++EN)) " ($ + 12) -
n
ti T
((*+(++0)1)) -1 (S 11) "
|
($ ($+11) (S+()+EN)))*
tl
F - O
where the last line follows from the bound
1 Tr E 0(1) which holds due Proposition A.2
36

---

Published as a conference paper at ICLR 2026
Let us denote the zero-centered counterparts of the corresponding VX terms
3) = 72
Tr[20 |n
( (- Tr (E((B++N)=1))
(A()) = Tr[(E0 + + = dd
(A) n
Tr (E + (1)
Analogously to A.43 and (A.44). it holds that
1 (12(A+ (2) - - = 0(12-)). (A.45)
- = O(2,,
V2(A)=
The next step is to prove that, for > A A> 00.99
x-2
= =(()=)((1)) +
(A.46)
n
Proof of the claim in (A.46). Expanding the expression, want to prove that
(*(A) = Tr (II(E 1 A Tr (Ea(Eo+An + +0
2
+ in its expression, which have already analyzed in the context of 3 (B, B). Recalling the definitions of A, u, u, U, C, a, b, d, and from (A.35).
Notice that V (A) crucially contains
A.36 and A.37 we can then expand V (A) as
À
Tr (2 (
Ir (2((A+UCUT)))
71
n
X
Tr
A-
n X
Tr (E,A-1) +
n
where S == - -T TE U A --U0)-AAA--)
We now prove that $ is small. To do so, decompose
$
A Tr
Σ1 A A-
A-1
((12) n
Tt + +d) uu T + +11-1)
(1-b) b) (woJ + vuu + a ve a-1)
T Tupe + i
where Tu. is the summand corresponding to Tu Fu, to uv + vuu and Tu. to vuT. Zooming in one of the terms, it holds that
Tu,
Tr
A-1
n1
2t + dd)
wu uuTA AA-
A
d
Tr (ΣGA-1ur -uu" T A-1)
+ d
u T
A
-
A
Note that
2-2
| -
and ||u/|| 5 c. Using this, we get that, with high probability, it holds
2
Ta. <c c
37

---

Published as a conference paper at ICLR 2026
With analogous passages, we have that
x-2
[Tepl S x-2
aaael S
n
n
holds with high probability over the sampling of Z. Putting all together, we get
X Tr [ (
X Tr (E,AA--)) +
X (^=))
n
n
n
Lastly, doing all this again to take out the terms containing μ4 from A, i.c., by taking
À = A
X0 1n
14 n,
X
Ys Patt: = Eo + AI,
n
we get
X Tr (I
(I,(++ + +1)-
> Ir
Ee [o + +N) +0 X-3
proving the claim.
Finally, combining
A.43
A.44). (A.45) and A.46 for 1 > A > 0.49
and > 0, we have that
-2
3)
$ VK(08,0 33
(/ + - 1 -
(((A)) 27 (2E(A + 11) 1)) -
1
0O(A)
(2( 1
(VRG+ 11- - 18()
(V+(A+ + 12(1 1 1 (19(+1NN (2) -
+ 12()))
50(A+0
= =(4))0((1-9)+ (-
Taking = 2 and > n 1/7, get (3, - 18( (177, proving the claim from
A.42 As Vx (3:3) = (3: + (): and (2(3, = 0(1/p) by A.41 we conclude that
(x(,; 2² Tr + (E + ppl)
9²
Tr Σg
+ 0 (--1/)).
Plugging in the expression of e² Tr E Σx} given in (Song et al. high probability,
2024 Theorem 4.1) yields, with
0² Vx( = -1 /
Maass X(ax'++ax'') +aid') d1', (X", X1) +pp-), (a1 azl2 + 1)2
where (a1, 022 03 a4) is the unique solution, with a1 22 positive, to A.29 Taking the limit p, - Do gives the desired result for the variance term and concludes the prool.
A.7 PROOF OF THBBREM 4.5
For Σ, 1p and E, E xxxp it holds that
R. 4p B) V(2., 1p) + B(2., 4p. 3).
We analyze each of the two terms separately.
38

---

Published as a conference paper at ICLR 2026
Calculating B( Ip. B). Note that Σ, Ip implies X; = in all the equations in (A.30). Plugging this in, one gets that the third and fourth equation in (A.30) are satisfied for b₃ = by and by = b1 From the uniqueness of solution (61, b2, b3. b4) to the whole system of equations in (A.30). and the fact that by and by only show up in the mentioned third and fourth equation, we get that it must hold by b2 and by bj- Plugging this into the bias term get that
/
bB + (64 + 1)
BB 3)
dG,(A?, 1²)
(6, + b2 + 1)2
bi + b2 +
dG x*)
A² + b2 + 1)3
(B, u.)
b b2 +1
noting that i, E R is the eigenvector of the matrix Σg corresponding to the eigenvalue X
Recall that we have assumed in the setup of Section 4.2 that 3 is sampled from sphere of constant radius, which we will denote by rSP-1 i...., r = ||3 We now prove concentration of (( 1p, 3) over this sampling of 3. Towards this end, we introduce matrix A E R such that
1 + b2 +
kp. T A3,
A
u.u.
b1 x
Notice that first equation of A.30) yields
b + b2
1,
P
by A b2 +
which gives
Tr (A) =
b + b2 +1 = p n.
Since both b and by are positive, as stated in Theorem
4.4 it holds
1
||Al|2 (A) b1 + by +1 S 1.
Note that
(3, uz))
Egarso-18
E
by X + b2 +
1=1
P
EE u.)
b1 + by +
P
P
bn A + 02 +
p
(A.47)
Furthermore, the function 3T AB is Lipschitz over the sphere. Namely, for two vectors 31, 82 E rS it holds that
A81 ABTA81-
33_ A32l < 18T A(3: B2) + +1 A(3) - B2)1 < 2r ||A|| || B2ll, s 2r ||31 321l2
Then, due the concentration of Lipschitz functions over the sphere (Vershynin 2018 Theorem 5.1.4), we get that, with overwhelming probability,
|BT AB E,3 A = O(n
for any constant C₁ < 1/2. Plugging
A.47 gives
B(E,, Tp. 8) BT AB n2 2 0(n
p
39

---

Published as a conference paper at ICLR 2026
with overwhelming probability. We can readily calculate the bias term for Σ, Ip:
B( p: p.
{3. uz)
p- 2,2
bi b2 +1
P
Thus, for any Mg E RPXP, we have
BlIp I, Ip: 0n
(A.48)
with overwhelming probability.
Calculating 4p). Note that
V(Ds, 1p) = 02-
2
Xl(ass + axx))
dlp (A*, x²)
(a1 + a02 + 1)2
=-02_ 7 (a1> + 02 + 1)2
_ /
0z1 +a4
dHl (X*, X²)
(a1 + (1));
(A.49)
where the last equality follows from the third equation in (A.29) and the fact that X; for all i [p]. Moreover, subtracting the second from the first equation in A.22 29 yields
Q2
0 = 1
(A.50)
a1
+
11
Analyzing just the first equation
(A.29)
we get
2
a1 02
p
1,
P
a 2 +
02 +1
01 + 02 +1
which gives
a1 a2 +1 ーp-n.
i=1
Plugging this into A.50) we get that 02
1 Therefore, is the unique solution to
x + C2 = n,
(A.51)
a1
for C2 = 1 + +1>0000 0. From AA.49) we have that V(E,, Ip) only depends on E, through 01. with which it monotonically increases. To conclude this section, we will apply the majorization argument from the proof of Theorem [4.3] with slight modification. Almost all parts of the argument analogous, and we restate them mainly for convenience.
Let us denote by X= = positive solution A.55
.. "......;TT. Then, for fixed n, and A we will refer to a (x²) as the Note that from Theorem we have that this solution is unique. Consider We call function f good, if and only if
function f : !
20
R.
20-
Σ
1
(A.52)
a (X))s(()) +
a1
We claim that, if f is good, then
ai(f(X*)) (1²).
(A.53)
Proof of the claim. Consider a good function f. Then, we have
Σ
1
an (X+)
an (X+) +
= n.
02
40

---

Published as a conference paper at ICLR 2026
Furthermore, setting d1 0 get
P
= P 7 +
0- f(x²); +
I-7
n2
p p fg
> p n2
By continuity, there exists df e (0, a1 (X+)) for which
Σ
o(+)) + = =n-p
implying 01 (f((3) dj < a1 (X*), which concludes the proof.
Next, for i.j E [p] s.t. i < j. we introduce a function f IR22 - 12 defined as
x k= i,
(x)(
+c k ==j, 1 HiJ.
where e >> 0 is constant. We now claim that f is good for any i.j E [p] and c >> 0, such that
XØ 25 + c.
Proof of the claim. The claim is equivalent to
1
a1 (X))(x c) 02 a1(x+)(A; +02 ax(Ts) + ai({))23 t
For simplicity, let us denote a == a2 (A*). Then,
a(x; +2 +c)
ad, c2
adi +
a(> + 19) + 202
a(>; + X3) + (Afa 02) 02)
(Xaa ca 02 (Aja ca (2)
(Afa+ 02)+ 02) (Naa ca + (Aja ca +
ca(3ia + -2) caaaa 02) e03 >0
ca2 - X2) caa²
x X3 +c
which proves the claim.
This implies that, for & E (0, 1), transformations of the form
(A A3) + (1 + (1- () + (x3)
(A.54)
are good. Let us denote by X' = [l,... 1], which corresponds to the matrix Ip. Pick any X * Nid that corresponds to some matrix Σ, E S, so it satisfies >; 2 XØ ... 2 Xo. as well as Σ X
Firstly, we claim that x majorized by X². Suppose otherwise, that for some is [p]
k
implying also that A6 < 1. Then, we have
p =
x < - 1)
+ 12xpppp)0
(p +k
41

---

Published as a conference paper at ICLR 2026
which is contradiction.
Next, as X'd is majorized by X², And can be derived from X by a finite sequence of steps the form in (A.54) with t [0, 1], see (Marshall et al. 1979 Chapter 4, Proposition A.I). Since both vectors Xd and X are non-increasing, the 0 transformation can always be omitted. Moreover, t == is just the identity transformation, so it can also be omitted and we actually have t (0, 1). In formulas, we have that
X = fa (... Jai (x')(((()... (X*)...).
Since each of the functions above is good, we have that 01 (Xid) < aa1 <1(())) As V(Es, Ip) is increasing with 01. this directly implies that, for any Σ, E R
V(Tp: 1p) S V(E.,
Combining this with (A.48). we get
Ro B) Ro [g with overwhelming probability, which concludes the proof.
A.8 WEIGHTED OBJECTIVES UNDER DIFFERENT LABEL NOISE
Let us suppose work in the setting where the label noise E(i) differs between real and synthetic data. A natural way to account for this in training is to assign weights exactly inversely proportional the corresponding noise levels. Thus, let wi be the weight assigned the real data, and w2 the weight assigned to synthetic data. This leads to three possible regimes:
w1 < W2- Here, synthetic data effectively determines the estimator. In this context, we would be in the scenario of Proposition 4.2 As discussed there, both the mean and covariance discrepancy play a role, and thus the resulting optimality condition differs from the mixed-data case.
wi >> W2. In this regime, the estimator is driven almost entirely by the real data, reducing the classical setting studied by Hastic et al. (2022). Consequently, the choice of synthetic data has negligible impact.
W1 ~ W2. This is arguably the most interesting case. The current theory applies directly, since weighting real and synthetic observations is equivalent to scaling (E,, (46 by wi and (E,, (s)) by W2. Then, by adjusting Theorems 4.1 and 4.5 one can prove that the optimal choice becomes Σg
Thus, the qualitative conclusion remains unchanged: matching the synthetic covariance that of the real data is optimal, where the weighting reflects the appropriate scaling depending on the trust placed in the labels.
B MODEL SHIFT
In this section give a precise estimate of the excess risk of the min-norm interpolator using both training and augmenting synthetic data under model shift, assuming no covariance shift is present. Interestingly, the difference in means depending on the intensity of model shift, synthetic data may have limited or even negative effect. By characterizing this, the resulting formula suggests simple training heuristic for synthetic data training in the presence of model shift. We focus on the under- under-paramett parameterized regime, which offers technically cleaner setting for analysis. Extending these results to the over over-parament paramettrrieee regime is more involved, though we anticipate that existing techniques should make this feasible. We leave this extension for future work.
again does not affect the characterization. Furthermore,
B. PRELIMINARIES
Data model. modeled as
We consider the setup of Section with the addition of possible model shift. This is
X() + E():
(i) E (1,
(B.1)
42

---

Published as a conference paper at ICLR 2026
where B(e) E RR, and it is no longer necessary that 3, = 3r. Note that this is common modeling in the transfer learning literature, as done in prior work (Song et al. 2024 Yang et al. 2025). This setting allows us explore the impact of model shift generalization error, as the true parameter is no longer shared between training and synthetic data. Lastly, to emphasize difference in means we will assume that the synthetic data normalized, i.e., Ms 0, while keeping || rtyP.
Risk and estimator. As in Section 3 we test estimators on data sampled from the same distribution as the training data (X,;30 yt). Then, under the adjusted data model, the formula for the risk can be broken down into bias and variance term as in 3.3 Namely, it holds
Rx (3: Se B.) |aa| X] B.1l + + Tr(Cov(B | + pMr
(B.2)
:= Bx (3; Sc, 3.) + Vx (3; B.).
We are interested in the performance of the minimum norm interpolator. Its closed-form solution is unaffected by adjustment to the data model, i.c. it remains the same as in (3.4), i...
B =
arg min {1]b|[2 b minimizes ||y
xb612) (X x)*xxx y.
(B.3)
Substituting B.3 into the excess risk decomposition B. yields closed-form expressions for bias
X X
Bx Bx B.) =3, 11(2cc + pil 1118, 20, 11(2, "114)
(B. 3x)
71
(B.4)
E + Malle ) [X,X,
Xg
X3 X, (3, - Be)
+ (3, - (
7
n
and variance
d² TreS+ (E+ (E++ +ppl)) [l,
V3, 3.)
(B.5)
where E XTX/ and = - Σ (projection on the null space of X).
B.2 THEBRETICCAL RESULTS
As mentioned at the start of Appendix B aim characterize the excess risk of the interpolator under model shift, when no covariance shift is present.
Let assume that 1 + r S S. < 1/T, implying that p. which makes the setting under- parameterized. Thus, E X//nn full rank almost surely, which implies that II Σ-
I I- Sーー = O. From B.4). it follows that
X. X, Bx (.) = (8. - -3:)
X, X,
E E HPT))S
(B, - (1). (B.6)
n
We additionally constrain the number of samples as + S St, S 1/T and 0 < 75/71 < 1/T. Similarly to Section 4.2] 3g and 3, are sampled from sphere of constant radius, independently from E,, We will work in the setting where population covariances match, i.c. Σ, = E,, highlighting the influence of model shift.
The following result provides a precise asymptotic characterization of the excess risk and, in doing so, it extends results by Yang et al. (2025) non-zero centered data.
Theorem B.I. Let E, E.. Then under the assumptions from Section 3 and the start of this section it holds
Rx (x - -1>:1/2(3. -2x)
-1 -3) -1 - 0² 82 = 0,
lim 11000
where
n2(n - p) + pn $1 n² nn p)
P
p
This theorem has an interesting implication. Unlike in the under paddr-parameterized setting without model shift, the inclusion of synthetic data degrade performance in the presence of model shift.
43

---

Published as a conference paper at ICLR 2026
As discussed in the random-effects framework (Section 4.2 of (Yang et al. 2025 intensities can cause different effects from training on synthetic data. This observation suggests that when covariances synthetic dataset until the performance begins to decrease. We refer the reader to the above-cited work for more detailed discussion of these phenomena.
different shift
matched, practical training strategy would be to gradually downsample the
We now proceed to prove the main theorem of this appendix.
Proof of Theorem B.1. Firstly, note that under the assumption μa 0 holds that X, X i.e. synthetic data is centered. Next, from (B.2), we see that the risk can be broken down into the bias and variance term. We will handle each them separately. Let us start by splitting the bias term in (B.6)
into two components, i.e., Bx (3; 3.) B1 + B2, where
X X
X
X
Bi = - B2)
Σ 3-
(3, Be),
n
Xxo X
X X
B2 = - (3,-B:)
Σ
MeH4 E
--1
(B. - BB)-
Bounding the term B1. We will first prove that
X x
X T X (3, - 3.) 3.)+00000 + (
Bi (3. - (
(B.7)
"
The argument mirrors the proof of (A.34). and we include it here to point out the necessary modifica- tions.
Proof of the claim in B.7). It holds that
$
(xTx
X)
n (x Intth T((O Inette
x xo
xo +
X0 Inath
+ MelT, x
7t Meli
+
n
n
where abusing notation we write 1,, 10, 0, 1, ..... 1] 1x (n, zeros followed by ne ones). All the terms above, except the first one, have rank 1, so we use Woodbury formula to take them out of the inverse when computing Đ-1 We introduce the following notation
X x
A
ED,
n
x
L
(B.8)
Vn
In
and C = U = [u o) E Rox2
E Rx2
1
Under this notation, it holds that
x T
M.lT, x
7t Hn = UCUT
Then, using Woodbury formula, we have
i-1= A A uo + +ou' T + n 2u
= --"" A-
44

---

Published as a conference paper at ICLR 2026
We now compute the 2 x block
- A-
1 u-uTA-1
c U---
-a
v A
-n 2t - 1-1o
=
1-b b
-n 25 d
where
a:= u A u,
b: T. 1 " = T A A
d: T. 1
(B.9)
Hence
- - d b-1
(C-I-U'A~'\)-
b 2-1
A:
n + d) - -11 b)2. (B.10)
A
Plugging back and simplifying gives the explicit formula:
-ー'=Aー A-1 1 1 A
(( ((-
7t
uu - (1 - b) (wv" + vu - vui A A-1
which is valid whenever + 0, i.e., whenever C-1 A-U is invertible. We will now analyze the b, d terms. First, note that there exist constants C2 > 0 such that
0 c1 S (p( & (A) < c2.
(B.11)
By the Bai- Yin theorem (Bai Silverstein 2010 From this, it follows that
Theorem 5.) this directly implies that ||A- S
al = 1 Tu A
ul
14 A
14 vn
F4 n
||
112
14
Sc
n
vn
2
Similarly, we have
16|
10 A
- [
14
X 1nm
vn
Vn
2
x
1nn
||A-1|1
Vn
vn
< cv/p,
X
where the last inequality follows with high probability over the sampling of X", since vector with i.i.d entries of mean zero and ((1) variance. Finally, we have
is
vn
|d|
1 T A
A-''ol
I X
X
A
Vn
n
X Ln.
X TT
||A-11
la
n4
Vn
<
cpp,
again with high probability. Therefore, it holds that
a
A
Me
2 cp
(A~1)> >c>
vn
v
where last inequality follows from B.Il it holds that
We can now prove that, with high probability, A 0(p).
Using
8² = ((,, (((u,))l== 1> [|ulll ||u|| =ad,
from which it follows that
2t n + +d) - b)2 2 an 7
A
+ 2h = =((p),
7
45

---

= Published as a conference paper at ICLR 2026
since is lower bounded by constant and [6] 1 cyp. this point, we have all the necessary bounds
and we work towards proving the claim. To simplify notation we denote by E.
x2 x2
and
n
= 3, - 8. We first expand the bias term
B1 = -
= - (A-1 A-- -
= -3 S
where S == - - -
-C A-
We now prove that S is small. To do so, we decompose
- U (0-1-U - (C~--U-A-- A
3 -3 A-1
n "
+ d
uu + (1 - b) (wv T + vu ) + vv=) A-1100
TT,, Tu, + To.p
where Tu,a is the summand corresponding uu Tu. u to uo vu and Tu. to vo Zooming in on one of the terms, it holds that
Tu,a = -3 21 (n + d)
A uu A A-150
A
= -33,595-15
(n 7 + d)
A 1
u A 1- B)
A
Note that
غكو -
E E,
(nta/7 + d)
A u
s $|330
|
1
||Elll2
(nt/7 + d)
||A ||A1|12 /uul2 S
A
A
and
u A 1 $0
I c. Using this, we get that, with high probability, it holds
Tu c p
This is similar to how we obtained (A.3I), since B, and 3r are sampled independently from sphere of constant radius. With analogous passages, we have that
[Tupl S
ITal
P
holds with high probability over the sampling of 3, and B. Putting all together, we get
B = 3 Eg3 +
(1)
Using the same argumentation applied to the remaining term 5- in 3T E. - 2, 1515 gives
(a) = 3T 3 ' 150 O ())
proving the claim.
By directly applying Yang et al. 2025
Theorem 6] we obtain that
9 S ' E 3 | [/, (b)
s
0
p-/2-c
2
Finally plugging in B.7 yields
Bi - 1 - -800|1
=0
88
p1/2-c
46

---

Published as a conference paper at ICLR 2026
Bounding the term Bz. We follow the recipe devised in the proof of Proposition A.3 If we denote by Xx, /n
X
then it holds
2
1 14
Xa)
14
Σ
(x)) Mr
o(()))) i=1
(v1(X)),
1/2 (22(Xn),14
P
(((((n)) ) pe
(1(,))*
(2(,))
e ( -1),
(1 (1-0 ( (1)
_1(1)
(1)
C.T
where the penultimate inequality follows directly from the vector ju. S211
A.9 and Proposition A.2
This means that
x22 xx2
is bounded, since
E-1 n4
X
S | Σ
11.
X
X
0(1),
n
n
as an application the Bai- Yin theorem (Bai & Silverstein 2010 Theorem 5.11) gives a constant
1 x²
bound on
Since 33 and 3r are sampled independently from a sphere of constant
2 x. x
radius and p Σ sub-exponential. Using Bernstein incquality, we obtain that, for (i) E (s,
is of bounded norm, we have that
<( [x
((x2
x=x
(1 me) .(1)
2 is
2
X
x
X
x E
Mi -1
3(0
S-1
B()
=
(B.
with high probability over the sampling of Bg and B.. Therefore, we have
X X
B2
14 1T-111
(3. (1))
Bx)
where the last inequality follows by separating 3, and B, from the square and using (B.12) on each term.
Bounding the term (3; B.). Note that the variance term is the same as in the case without model shift. This means that we could simply use the already derived results from Section [ Namely,
directly from the proof of Theorem 4.1 we have that for M 2 1/2 and A
Xp
eigenvalues of M M, holds
Vx (3; Br, 3.)
Tr [alM M +2212 O(p
where dn and ag are the unique positive solutions to the following two equations
P n
n2
a1 + a2
a1
n
Aa + a2
n
As assume that Σ, E., it follows that M M Ip and
a² Tr [aan M az Ip) -1
0² p
which ultimately yields
Vx Vx Br, 9.) δ² p
= =Opp 0pp-112).
p
Finally by combining the bounds on B (B: B, 8,) follows.
B + Bg and Vx (B; Br, B.), the desired claim
47

---

Published as a conference paper at ICLR 2026
C ADDITIONAL NUMERICAL RESULTS
Setup details. We train for 200 epochs using SGD as optimizer, and we use cosine annealing: the initial learning rate is 0.1 for Scratch (0.2 for the experiment of Table and 0.01 for Distillation and Pretrained. The Distillation teacher is a ResNet-50 trained on CIFAR-10. We use an early stopping with patience 20 based on a validation subset (10% of the full training dataset). We avoid up-scaling images in the Pretrained experiments to better demonstrate the effect of synthetic data augmentation. On the generation side, to generate the images by T2I models, we use CLIP's text encoder prompt template on CIFAR-I0 and ImageNet labels. Moreover, as models like StableDiffusion I. .44 sometimes generate low quality data or images discarded by the safety checker, before applying all the algorithms, we do an initial pruning of 2% of the generated pool based on the distance to the CLIP embedding of the label. For RxRx1, we train linear classifier on frozen features from an ImageNet-pretranne ResNet. For each class, MorphGen generates a pool of 500 synthetic images; we augment the real training set (30 images/class) with 60 selected synthetic images/class and evaluate on a disjoint test set of 20 images/class. We repeat the experiment 10 times by resampling the real subset from 120 images/class. As in the main setup, CLIP features are used for the selection algorithms.
Transformer-basee models. In Table we use the same setup as Table but instead of ResNet, we train VIT and Swin-T model from scratch. We use a patch size of 4 and Adam optimizer with learning rate 0.0001 for this experiment. We observe that, in accordance with our previous findings, covariance matching surpasses other algorithms.
Table Covariance matching outperforms all baselines when fully training a transformer model on a mix of real and synthetic data.
Method
VIT
Swin-T
Scratch
Distillation
Seratcc
Distillation
No synthetic
40.11 0.59
40.32 + 1.01
40.02 + 0.70
40.84 + 0.73
Center matching He el al.
2023
43.89
0.97
43.61 +: 0.68
44.39 1 0.54
46.64 + 0.53
Center sampling (Lin a..
2025
43.89
0.95
46.29
0.80
43.94
+ 1.76
46.97
0.59
DS3 (Hulkund
2025
45.92
0.49
48.61 + 0.67
46.57
+ 0.68
49.55 + 0.72
K-means Lin
44.24 2:24+110 1.13
47.44 :: 0.97
44.71
+: 0.32
48.49 + 0.64
Random
44.07 0.82
46.50 + 0.78
44.38
+: 0.77
47.35 + 0.50
Text matching
202.
44.57 0.57
46.02
0.58
46.55
2.52
Text sampling LLn
2023
43.80 1: 0.98
46.00 :: 0.98
44.59
1: 0.93
47.62
: 0.71
Covariance matching (ours)
46.09 0.91
49.53 + 0.61
46.64
+ 0.96
50.73 + 0.44
Real upper bound
51.85 0.47
53.11 + 0.43
52.43 4: 1.39
54,80 +t 0.69
Zero-diversity generators. To assess the importance of filtering low-diversity data, we construct pool per CIFAR-10 class with 2K images from StyleGAN2-Ada and 8K images from two collapsed generators. The first collapsed model emits the image whose CLIP embedding is closest the class label; the second produces images near the mean embedding of the class's 's real subset. We sample 4K images from each collapsed generator, yielding total 10K images per class. As shown in Table most baselines over-select from the collapsed generators because they ignore the diversity of selected samples. In particular, DS3 retains the two clusters formed by the collapsed outputs and thus fails to filter them. By contrast, K-means and Covariance matching draw more from the 2K non-collapsed subset and achieve higher classification accuracy.
Leak experiment. We consider inserting ("leaking") images from the target distribution into the pool of synthetic images and test the ability of different methods to select them. We use 1K leaked CIFAR-I0 images, disjoint from the 200 (nt) real reference samples. From a pool of 4K StableDiffusion .4 images and 1K leaked images, each method selects 800 (n.)) Figure? 2 shows, for each method, the fraction of selected samples drawn from the leak. Because replacing synthetic with real augmentations yields the best accuracy (Real upper bound), an effective selector should prioritize leaked real images: covariance matching does, achieving the highest leaked fraction among all methods.
48

---

Published as a conference paper at ICLR 2026
Table 5: Covariance matching performs on par with the best baselines across three training paradigms on CIFAR-10, 10, when the synthetic data is generated via a StyleGAN2-Ada model and two zero- diversity generators.
Method
Scratch
Distillation
Pretrained
synthetic
44.36
47.33
63.40
Center matching (He al.
2023
45.33
2.43
47.50 0.55
62.96 1.26
Center sampling (Lan al.
2023
46.88
51.11
0.60
65.38
DS3 (Hulkund
53.74
59.16
1.56
69.43
K-means [Lm ef
60.20 1.35
65.03 + 0.81
72.83 + 0.48
Random
50.31 1.28
61.82
0.91
66.27 + 1.21
Text matching (Lin el al
2023
42.89 1. 1.89
47.38
1 0.76
62.82 1 1.31
Text sampling Lin el al 2023
48.13 1 1.81
50.81
1 0.77
66. 06.12-20 1.06
Covariance matching (nurs)
58.97
1.67
0.63 64.85
72.38
0.66
Real upper bound
61.08 1 2.54
65.38 1 0.51
74.35 1 0.36
Percentage of selected leaked samples
Percentage of selected leaked samples
Covuriance mutchiing
66.15%
Covariance matching
70.4%
DS3
Center matching
43.79%
DS3
K-means-
52.31%
Center matching
Center sampling
19.29%
Random
15.15%
K-meuns
52.5295
sampling
Text matching
4.77%
Random
Portion of leaks among selected samples
Portion of leaks umong selected samples
(a) based algorithms
(b) DINO-based algorithms
Figure 2: The portion of samples chosen from the set of leaked images shows that our proposed algorithm reliably selects real samples among the pool of generated examples.
Changing the feature extractor. In the main experiments, we use CLIP features for all selection methods. To test the dependence on the feature extractor, we repeat the setups of Tables [ with DINO-v2 features. As shown in Tables 6 covariance matching matches surpasses the best baseline across settings, indicating that effectiveness is not tied to specific feature extractor. We also repeat the leak experiment of Figure 2
see the bar plot in (b), showing again similar results.
Table 6: Covariance matching outperforms all baselines across three training paradigms CIFAR- 10, when the synthetic data is generated via truncated generative models and features are extracted with DINO-v2.
Method
Scratch
Distillation
Pretrained
No synthetic
44.36 1.51
47.33 0.57
63.40 1.33
Center matching (He al. 2023
50,06 1.45
54.50 + 0.62
66.23
0.72
DS3 (Hulkund 2025
52.93 1.65
58.69
0.81
68.04
K-means (Ln
51.66 1 2.10
55.97 0.58
67.00 0.884
Random
49.97 2.45
54.79 + 0.68
66.57 + 0.92
Text matching (Lin et al. Covariance matching (ours)
2023
51.52 = 1.67
65.17 + 0.57
0.57
67.13 1.13+0.00
54.97 - 2.60
59.41 + 0.81
68.87 + 0.41
Real upper bound
61.08 2.54
65.38 + 0.51
74.35 + 0.36
Optimizing the theoretical objective. We also implement greedy algorithm that, at each step. adds the sample minimizing the objective in (4.l) (Alpha matching). This method requires computing the eigenvalues of the current sample covariance and is therefore more costly than Covariance
49

---

Published as a conference paper at ICLR 2026
Table 7: Covariance matching performs on par with the best baseline across three training paradigms on CIFAR-10 10, when the synthetic data is generated via text-to-image (T2I) generative models and features are extracted with DINO-v2.
Method
Scratch
Distillation
Pretrained
synthetic
44.36 1.51
47.33 0.57
63.40 1.33
Center matching (He et al. 2025
51.75
2.01
55.67 + 0.63
66.00
0.58
DS3 (Hulkund et |
2.07 52.33
58.80
0.96
66.68
0.63
K-means (Lin el al 2025
51.14 1.90
56.93
0.46
65.71
1.7.1
Random
50.45 - 1.41
55.86 1
65.67
0.82
Text matching iL et al. 2023
51.38 1.51
55.81
+ 0.65
65.76 + 1.00
Covariance matching (ours)
32.65 = 1.47
58.78 + 0.53
67.04 0.83
Real upper bound
61.08
2.54
65.38 0.51
74.35 + 0.36
matching. As in Covariance matching. we first fit PCA on the real samples and project all features, then iteratively add the sample that yields the smallest value of (4.1). Without loss of generality, we drop the noise variance term since it scales all candidates equally. The results of Table 8 show that Alpha matching performs similarly to Covariance matching.
Table 8: Covariance matching performs on par with Alpha matching across the experiments on CIFAR-10.
Experiment
Method
Scratch
Distillation
Pretrained
Covariance matching
58.97 + 1.67
64.85 + 0.63
72.38 + 0.66
7ero-diversity models
Alpha matching
59.30 + 2.50
64.72 = 0.55
72.76 = 0.73
Covariance matching
54.00 +: 1.89
59.77 1: 0.61
69.20 1 0.56
Truncated models
Alpha matching
62.25 + 2.11
59.18 = 0.68
68.32 = 0.58
Covariance matching
54.45 1. 2.11
59.17 1 0.64
66.69 1 0.70
T2I models
Alpha matching
53.37 :: 1.85
39.03 + 0.64
66. + 0.66
Over-parameterize setting. We repeat the setup of [taking n1, 200 (instead of 800). This gives a total of n. + n, = 400 samples, which is less than the number of features p = 512, thus placing us in over aver-parameterized regime. As shown in Table the guantitative trends mirror those in the under- ander-parameterizeer case.
Table 9: Covariance matching outperforms all baselines across three training paradigms on CIFAR- 10, when the synthetic data is generated via truncated StyleGAN2-Ada models (Karras al. 2019) in the over-parameterized regime with 200 training and 200 augmenting synthetic samples.
Method
Scratch
Distillation
Pretrained
No synthetic
44.36 + 1.51
47.33 + 0.57
63.40 + 1.33
Center matching (He et al.
2023
46.45 +: 1.97
50.83 1 0.50
64.40 +: 1.11
Center sampling (Lnn ct al.
2023
47.29 + 1.33
50.89
+ 0.78
65.64 + 0.74
DS3 (Tulkund el at. 2025
48:09 1 2.04
52.65
1 0.61
66.41 1 1.35
K-means (Lin ct al. 2023
47.75 0.82
-..56
65.47
0.99
Random
47.39 1.63
0.22 50.96
65.49
Text matching LLn al.
2023
47.56 1.09
51.67
- 0.65
6.774
Text sampling Lin ét al 2025
46.93 + 1.95
50.64
63.13 1.13
Covariance matching (ours)
48.95 1: 1.28
53.28
1 0.45
66.62 1: 0.57
Real upper bound
50.79 1.70
54.66 0.91
68.97 0.88
Distribution of selected samples. Beyond accuracy, we assess how well each method's selections match the test distribution. In the CIFAR-10 setup of Table [ each method selects 800 samples per class given 200 real samples. We then calculate how well these samples match the CIFAR- training dataset. The selection obtained via Covariance matching consistently achieves lower FID/KID and covariance distance than all other baselines. Metries that couple fidelity and diversity (e.g.. FID/KID)
50

---

Published as a conference paper at ICLR 2026
show larger gains than quality metrics (e.g., Precision (Kynkäänniemi et al. 2019). Density (Naeem jet al. (0200), indicating improved distributional alignment rather than mere sample quality. The results are reported in Table I0
Table 10: Covariance matching selects samples that better match the target distribution according to various evaluation metrics.
Method
HDD J
KID
Procision 1
Rocall
Density
Coverage
Covariance Shift
K Kmmeans (Lin et al.
2023
366.52 +: 2.62
0.59
0.04
0.77 1: 0.01
0.41 0.00
0.87 a: 0.04
0.58 2: 0.01
118.91 4: 0.62
Center mitciing
2024
544.56
5.57
0.83
1993+00.06
0.78 + 0.01
0.33 0.00
0.82 + 0.03
0.48
212.55
3.03
Center sampling
F11 m1
2025
250.27
3.86
0.61 0:04
0.77 :: 0.01
0:44 0:00
0.86 2: 0.88
0.53 0.00
150.49
0.79
DS3 [uulknnd et 1mm 2026
273.59
4: 6.72
0.004
0.79 4: 0.01
0 0.01
0.84 1: 0.08
0.64 0.01
108.52
2.44
Randim
458.30
4.16
0.63
0.00
0.77
+: 0.02
0.44 0.001
0.86 + 0.05
0.53 0.00
150.66 1.08
matching
2.66
0.90
172.70
0.66
Text sampling Jm al
2024
447 53
+ 3.99
0.61 0.04
0.77
+ 0.01
0.44 0.001
0.86 + 0.03
0.53 0.01
140.58
+ 0.95
Covariance malccinng
122009
+ 1.95
0.41
0.04
0.78
0.01
0.50 0.001
0.81 : 0.05
0.08 0.01
95.55
0.58
Real and synthetic dataset sizes. Next, we analyze the behavior selection methods for different values of real dataset size and synthetic samples gathered. We use the setup of Table [ with training from scratch, and change the size of real samples n, E {200, 300} and synthetic samples ns E (600, 800, 1000) in Table 11
Table 11: An ablation on the size of the real and synthetic datasets used for training shows results consistent with those reported in Table
Method
(200)600)
(200, 800)
(200, 1000)
(300, 600)
(300, 800)
(300. 1000)
No synthetic
44.36 +: 1.51
44.36 +: 1.51
44.36 + 1,51
47.85 d: 1.21
47.85 +: 1.21
47.85
1.21
Center matching
47.51 3.04
50.04 4: 2.84
50.60 + 2.87
51.66 + 2.19
53.61 +: 3.57
54.75 + 3.44
Center sampling
50.14 +: 2.42
50.48
1: 2.03
51.40 +: 2.84
51.00 + 2 2.22
52.97
+: 3.11
55.00
3.45
DS3
50.62 + 1.25
52.83 + 2.19
53.02 + 3.18
53.33
+ 2.26
55.91
+ 2.80
55.79
+ 2.70
K-meuns
49.65 2.52
50.74
1.77
51.31 +: 3.32
52.10
2.53
52.65 3.65
54.22
3.13
Random
48.07 + 2.80
49.38 + 2.43
49.34 + 2.37
52.38 + 2.50
52.96 + 3.82
53.47
2.66
Text matching
49.90 + 2.70
50.94 + 1.40
50.72 + 2.54
52.27 + 2.52
53.04 + 2.64
52.03
10103++1 1.05
Text sampling
49.19 +: 2.30
50.28 + 1.18
49.98 + 3.40
52.20
+ 2.78
51.86
- 1.96
53.23
3.28
Covariance matching (ours)
51.91
54.00
54.64
55.38
3.27
55.63
2.70
55.80
Upper bound
57.85 1.86
61.08 + 2.54
60.99 + 2.11
18.76-2 +: 1.85
61.11 +: 2.87
02.79 2.56
Variations for covariance matching. We experiment with two other variations for covariance matching. First, incorporating more compute, we analyze look-ahead strategy. At each round, we select the top & samples for minimizing the covariance. We then exhaustively test each pair of these samples (total of (*) possibilities) and find out which pair minimizes the covariance shift. We then only add the first element of the pair. This models a ahead strategy for adding sample during each round. Second, we perform a method to find the best samples globally. To find n, data among M samples, repeat the n4, dataset and sample to get n, real data points. We then use the Hungarian algorithm to match the ny real and A synthetic datasets. After matching, we select only the samples matched from the M total synthetic data points as our selected synthetic data. This imitates global strategy to match the covariance of the data points. We use the setup from Table and report the accuracies Table 12 observing that the performance on par with covariance matching.
The look-ahead algorithm with k E (50, 100} and the Hungarian algorithm perform on par with the greedy implementation of covariance matching across three training paradigms on CIFAR-1O, when the synthetic data is generated via five truncated StyleGAN2-Ada models.
Table 12:
Method
Scratch
Distillation
Pretrained
Look-ahead (50)
54.91 1.60
59.56 + 0.71
68.10 + 0.84
Look-ahead (100)
53.84 0.95
59.79
9 0.67
68.19 + 0.74
Hungarian
53.10 1.37
58.54 1.07
68.60
1 0.87
Greedy Covariance Matching
54.00 1.80
59.77 0.61
60.20 + 0.56
51

---

Published as a conference paper at ICLR 2026
Experiment text dataset. To further demonstrate the generality of our approach beyond vision tasks, we evaluate Covariance Matching on a text classification problem. Following observations reported by Li et al. (2023), , and GPT-generated tweets from Kuo et al. (2025) as the synthetic dataset. The synthetic corpus contains 3K prompts per class. We sample 100 tweets from the real dataset and augment them with 300 synthetic samples. We extract sentence embeddings using the all-mppet-base
the Tweet Irony dataset (Van Hee et al. 2018) as the real dataset
model (Song al. 2020), and apply our selection algorithms in this feature space. A linear classifier then trained on the combined real and selected synthetic samples to classify ironic vs. non- non-ironic tweets. Table [3] shows that synthetic augmentations improve performance, and Covariance Matching outperforms all baselines, serving theory-driven heuristic for selecting synthetic data.
Table 13: Covariance matching outperforms all baselines on the Ironic-Tweet dataset.
Method
Tweet Irony
No synthetic
64.60 + 3.16
Center Matching
70.80 + 1.54
Random
67.97 + 1.20
K-means
68.85 + 1.26
DS3
69.73 + 2.51
Covariance Matching (ours)
71.49 + 1.59
52


Reviews:
[{"review_id": "9C8sCECB6V", "reviewer": "Reviewer_GUb9", "summary": "This paper presents a theoretical and empirical study regarding synthetic data selection for augmenting training datasets. It focuses on the effects of mean shift and covariance shift by theoretical analysis in high-dimensional linear regression and experiments on vision models. Their findings suggest that covariance shift rather than mean shift affects generalization error when training on a mixture of real and synthetic data, and show that the covariance matching approach for selecting synthetic data would improve model performance.", "questions": "1. How is covariance matching different from data quality measures? What makes covariance matching fundamentally different from these existing measures of data fidelity? Such as FID or other metrics. \n2. What is the relationship between covariance matching and diversity? Does matching covariance automatically mean the data are more diverse, or is diversity a broader concept that includes more than covariance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is theoretically sound by using high-dimensional linear regression scenarios and findings from experiments align with their theoretical analysis.  \n2. Although the conclusion regarding covariance shift matters is expected, the paper offers a theoretical framework that formalizes and explains this intuition. \n3. The connection between covariance matching selection and evaluation metrics such as FID and recall is quite interesting and provides insights into generative data quality.", "weaknesses": "1. In the introduction, the notation ($X_t,y_t$) is used to denote both the training and test datasets, which may confuse readers.\n2. The theoretical analysis holds strong, simplified assumptions, building on high-dimensional linear regression with Gaussian data. It doesn't account for nonlinearities, non-Gaussian feature distributions that characterize deep learning models in practice. \n3.  All experiments are conducted on vision tasks. It is unclear whether the findings hold for language tasks, which typically involve more complex architectures. \n4.  It states that training data should not be too small compared to synthetic data, but it is not formally quantified. In the experiment, the authors use 200 real and 800 synthetic samples per class, but it is unclear how varying the real/synthetic ratio affects the theoretical predictions or empirical results. Similarly, the paper mentions an upper bound on diversity scaling but does not provide a concrete characterization of when diversity stops being beneficial.\n5. The definition of diversity remains ambiguous. It is not clear whether it refers to information within training data or generated diversity beyond it. If it refers to within, does covariance matching include the concepts of diversity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8rvWeF1ipA", "reviewer": "Reviewer_1CSm", "summary": "This paper investigates how to optimally select synthetic data to minimize test error and enhance generalization performance.. The authors first analyze this problem in a linear ridgeless regression setting and derive an optimal data selection criterion that minimizes the test risk of the trained regressor. Specifically, the optimal strategy consists of selecting the set of synthetic samples whose covariance matrix is closest (in Frobenius distance) to that of the real data, a method they call covariance matching, while paying less attention to discrepancies in the first-order moment (the mean vector). Based on these theoretical insights, the authors design a practical algorithm for synthetic data selection, evaluate it on multiple classification tasks on CIFAR-10 and ImageNet, and validate their theoretical findings empirically.", "questions": "My questions are mostly related to the weaknesses discussed earlier (see **Weaknesses** section), along with few other minor remarks.\n\n1) How does your work connect to or extend previous studies that have theoretically analyzed the problem of mixing real and synthetic data, particularly the work of Firdoussi et al. (2024) [1]?\n\n2) I am somewhat skeptical about the theoretical result claiming that the mean vector does not affect generalization performance. Could this outcome stem from simplifying assumptions in your theoretical setup ? I also believe that discrepancies in the mean vectors should not pose a serious issue, since the real data mean can typically be estimated consistently, whereas the covariance matrix cannot in high dimensions (as described by the Marchenko–Pastur law).\n\n3) In Figure 1, the authors evaluated their theoretical findings using mean vectors $\\mu_s$ and $\\mu_t$ of norms in the order of $\\mathcal{O}(\\sqrt{p})$. Could you justify this choice, knowing that the dimension $p$ scales to infinity in high-dimensions ?\n\n4) Regarding the covariance matching algorithm described in the experiments section, do you think that adding multiple synthetic samples at once (rather than one at a time) would result in a different final synthetic dataset ?\n\nI would be happy to reconsider my score once my concerns have been addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper has many strengths and advantages, among which:\n\n1) The authors prove an interesting and somewhat counterintuitive result stating that the gap between the mean vector of the true data $\\mu_t$ and the synthetic one $\\mu_s$ **does not** impact the generalization error in their linear regression setting when training on a mixture of real + synthetic data. Consequently, the selection criterion needs only to consider the second-order moment (covariance).\n\n2) They provide a rigorous theoretical study of their problem through a linear regression setting both in the low (under-parametreized) and high-dimensional (overparameterized) regimes.\n\n3) The paper is overall well-written and the experiments section is extensive and well-detailed and showing promising results.", "weaknesses": "The main weaknesses of this paper are related to the novelty of their theoretical analysis and the absence of some important references in the same subject:\n\n1) **Novelty:** I am somewhat concerned about the originality of the covariance matching idea. In fact, the notion that discrepancies between the covariance matrices of real and synthetic data degrade the quality of the generated samples was already introduced and analyzed in a recent ICLR 2025 paper [1]. That work also studied training on a mixture of real and synthetic data under a theoretical high-dimensional binary classification setting. Therefore, in my view, this raises questions about the novelty of the present paper’s contributions.\n\n2) **Lack of key references:** The authors claim in the conclusion that \"they take the first step in understanding the precise connection between training on a mix of real and synthetic data and generalizing on real data\". This does not hold as previous works (that were not cited) have also tackled this same problem: [1], [2] and [3].\n\n\n[1] Aymane El Firdoussi, Mohamed El Amine Seddik, Soufiane Hayou, Reda Alami, Ahmed Alzubaidi, Hakim Hacid. Maximizing the Potential of Synthetic data: Insights from Random Matrix Theory. ICLR 2025\n\n[2] Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data, ICLR 2024 spotlight\n\n[3] Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, Merouane Debbah. How bad is training on synthetic data? a statistical analysis of language model collapse. COLM 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d73AzRmpZP", "reviewer": "Reviewer_D1Gm", "summary": "This paper investigates the incorporation of both synthetic and real datasets in a high-dimensional linear regression context to minimize the test error associated with the real data distribution. \nThe two principal conclusions are:\n(1) In the high-dimensional limit, the mean shift between the real and synthetic distributions does not impact the asymptotic test error.\n(2) An optimal synthetic data distribution has covariance proportional to that of the real distribution. \nThe authors also introduces a simple and greedy covariance-matching algorithm for data selection, and show it succeeds empirically.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is clearly written and easy to understand.\n\nThe findings are surprising to me: I do not know any previous results showing that the mean shift does not matter in combining real and synthetic data for training. Also, the optimal covariance expression makes a lot of sense.\n\nThough the theory is built on toy models, the covariance matching algorithm seems to work in more realistic problems like the imagine classification tasks in this paper.", "weaknesses": "In this paper, only direct mixing of the synthetic and real data is considered. If we assume that we are able to assign different weights to synthetic and real data, will we get different conclusions? For example, under this circumstance, can we still get the same conclusion on the optimal covariance matrix?\n\nAlso, it seems that the assumption of synthetic and real data sharing the same true parameter $\\beta$ is too strong, can you argue if this is common in practice? If no, how to relax this assumption?\n\nThe experiments are only conducted in the image dataset. Could you add more experiments on the language models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)