Why Less is More (Sometimes): A Theory of Data Curation
Authors: Elvis Dohmatob
Mohammad Pezeshki
Reyhane Askari-Hemmat
Venue: ICLR 2026 Poster (2026)
Keywords: data curation; LIMO (Less Is More); MIMO(More is More); synthetic data; beating scaling laws; mitigating model collapse; random matrix theory
Primary area: learning theory
TL;DR: We provide an exact analysis of data curation and reveal striking phenomena regarding scaling laws and also mitigating model collapse. Our results reconcile LIMO and MIMO, two different philosophies regarding data curation.

Abstract: This paper introduces a theoretical framework to resolve a central paradox in modern machine learning: When is it better to use less data? This question has become critical as classical scaling laws suggesting ``more is more'' (Sun et al., 2025) are challenged by methods like LIMO (``less is more'') and s1 (Ye et al., 2025; Muenighoff et al., 2025), which achieve superior performance with small, aggressively curated datasets. Here, we study data curation strategies where an imperfect oracle selects the training examples according to their difficulty and correctness. Our results provide exact scaling law curves for test error under both label-agnostic and label-aware curation rules, revealing when and why keeping only a subset of data can improve generalization. In contrast to classical scaling laws, we show that under certain conditions, small curated datasets can outperform full datasets, and we provide analytical conditions for this by deriving precise phase transition curves tied to data size and quality. We validate these theoretical claims with empirical results on ImageNet, confirming our predictions about when curation improves accuracy and can even mitigate model collapse. Furthermore, our framework provides a principled explanation for the contradictory curation strategies recently observed in LLM mathematical reasoning.

Published as a conference paper at ICLR 2026
WHY LESS IS MORE (SOMETIMES): A THEORY OF DATA CURATION
Elvis Dohmatob
Mohammad Pezeshki
Concordia University & Meta FAIR & Mila Institute
Meta FAIR
elvis wccmatobbecoccooeeee....
mpezeshk @meta
Reyhane Askari-Hemmat
Meta FAIR
reyhaneaskariemeta..
ABSTRACT
This paper introduces a theoretical framework to resolve central paradox in modern machine learning: When better to use less data? This question has become critical classical scaling laws suggesting "more is more" (Sun et al., 2025) are challenged by methods like LIMO ("less is more") and sl (Ye et al., 2025; Muenighoff et al., 2025), which achieve superior performance with small, aggressively curated datasets. Here, we study data curation strategies where an imperfect oracle selects the training examples according correctness. Our results provide exact scaling law curves for test error under both lahel-agnostic and label-aware curation rules, revealing when and why keeping only subset of data can improve generalization. In contrast to classical scaling laws, we show that under certain conditions, small curated datasets can outper- form full datasets, and we provide analytical conditions for this by deriving precise phase transition curves tied data size and quality. We validate these theoretical claims with empirical results on ImageNet, confirming our predictions about when curation improves accuracy and can even mitigate model collapse. Furthermore, our framework provides principled explanation for the contradictory curation strategies recently observed in LLM mathematical reasoning.
their difficulty and
INTRODUCTION
Despite remarkable advances in large language models (LLMs) and other foundation models, train- ing them remains highly inefficient, often requiring hundreds of billions of tokens. A key reason lies in how training data is used: standard loss functions treat all examples equally, regardless of their informativeness. Yet not all data points contribute equally to learning: while some accelerate progress, others are redundant or detrimental (Sorscher al., 2022). This inefficiency moti- vates the exploration of principled data curation strategies.
Recent empirical successes highlight the promise of aggressive data curation. Methods such as LIMO (Less Is More) (Ye et al., 2025) and sl (Muennighoff et al., 2025) show that curating compact sets of valid and challenging examples can dramatically improve reasoning performance, often with a fraction of the original data. These results stand in contrast to the traditional scaling law perspective (Kaplan et al., 2020; Hoffmann al., 2022), which suggests that simply increasing dataset size should monotonically improve generalization. The apparent contradiction between "less is more" and "more is more" (Sun et al., 2025) raises fundamental question: under what conditions does data curation help, and when does full-data training remain optimal?
In this work, our goal is not to propose another heuristic curation method, but rather to build a principled theoretical framework that explains why and when such strategies succeed. We analyze high-dimensiona binary classification under pruning oracles that filter examples based on difficulty and correctness. Our theory provides exact scaling laws for test error, revealing sharp phase tran- sitions tied to dataset size, label quality, and oracle reliability. These results establish conditions under which keeping only the hardest easiest examples outperforms training the full dataset.

---

Published as a conference paper at ICLR 2026
Crucially, we show how strategic curation can mitigate model collapse (Shumailov al., 2024; Dohmatob et al., 2024a), where iterative self-training on noisy or synthetic data leads to catastrophic degradation.
Main Contributions:
We develop precise theoretical framework for data curation in high-dimension learning, deriving exact scaling laws that characterize the effect of data pruning on generalization.
We demonstrate that, under realistic compute label-quality constraints, strategically pruned datasets can outperform full datasets, thereby bending classical scaling laws.
We empirically confirm theoretical predictions ImageNet and connect them recent large-scale results in LLM reasoning, providing a rigorous justification for why methods like LIMO and s succeed.
We show analytically that data curation can avert model collapse under label shift, establishing phase boundaries where uncurated training diverges while curated training remains stable.
Together, these results reframe data curation not as heuristic preprocessing step, but as a principled tool for stable and efficient learning.
2 SETUP FOR THEORETICAL ANALYSIS
To formally analyze when "less is more" versus when "more is more", we must first establish a precise mathematical setting, which is rich enough capture the complexity of the problem, but simple enough to be analytically tractable. This section defines our data generation process, the model we analyze, and, most importantly, the key quantities that will allow to distinguish between different learning regimes: the quality of the data generator and the quality of the pruning oracle.
2.1 DATA, MODEL, AND ASSUMPTIONS
Data Distributions.
Let Pu,, A denote the probability distribution on I& x IR given by:
(z, v) PaIA iff (N A), y = sign(x' Tw)) w).
The training dataset consists of n i.idd pairs (z,, from distribution Pg Pa Cg where We Rd and Cg Rdd the weights/labeling vector and the covariance matrix for the generative distribution (the "generator"). The true test data distribution is, however, P. = Pw, where w. E Rd and E E dddd are the true weights and covariance. In general, we consider wA # w, (i.e., label shift) and C * Σ (i.e., covariate shift).
The Model. Consider vector à E Rd defined as the solution to the convex optimization problem:
minimize 1 n
X /w over Rd pil(a, wiys) +
Σ
regularization parameter, and E 1) indicates if an example is kept. The downstream classifier is + (z w). The first-order condition for optimality in Eqn (2) gives the solution:
Here, {(2;y) : (2 is the squared L2 loss, x
= RX DY/n, with R == (S + Ma) and = X DX/nn
(3)
where X E B is the design matrix, Y e R² is the label vector, and D is a diagonal matrix with Du = P., indicating which examples are present.
Object of Study: High- Dimeensanlns Test Error Our goal is to characterize the classification test error, E == ((sign(x" Tw) * y), the high-dimensional proportionate scaling limit:
n, d + co, d/n + ó (0, oo).
(4)
For simplicity of presentation our main theoretical results and insights, we limit the analysis to the isotropic setting where the covariance matrices are identity matrices, i.e., Cg = Σ Ia. More general results are deferred to the appendix. Thus, our focus here is on label shift, where the labels from the generator P3 might deviate from the ground-truth labels from P..
2

---

Published as a conference paper at ICLR 2026
2.2 DATA CURATION RULES
Label-Agnostic Curation.
First, we consider a setting where an example (ri, yi) is retained based only its features T4, via pruning function q: IR + {0, } and an oracle pruning vector wa Rd:
P = g w.))
(5)
This rule the function q to select examples based on their projection onto the oracle vector wn- For instance, common strategies like "keep easy" and "keep hard" correspond to choosing q(() = 1[ 2 al retain large-margin examples (far from the decision boundary) and q(t) := 1[[t] S al to retain small-margh examples (close to the decision boundary), respectively. The notion of an example's difficulty is thus determined by the oracle W/p and the threshold a >> 0 controls the proportion of data kept. This captures the setting considered in (Sorscher et al., 2022).
Label-aware Curation. We also analyze more realistic rule where the oracle filters for the cor- rectness of the corresponding label as well. Here, an example (æi, y:) is kept if its label matches the oracle's label yf and is deemed interesting by q:
Pj =1 iff yi = vf and q(x, wo) = 1,
(6)
where yf == sig wo) is the label according the pruning oracle (not revealed to the learner!).
In the practical setting of LIMO (Ye et al., 2025) and sl (Muennighoff et al., 2025) methods, the pruning function q might capture other heuristic rules which decides if an example is sufficiently diverse or interesting to be retained in the curated dataset.
Importantly, our setup posits that the machine learner can only query the curation rule by submitting input/label pairs (24,, Wi) and obtaining bits p; E {0,1), 1), but has no access to the underlying pruning direction Wo, nor the oracle labels y = sign( (2, wo).
Desiderata:
Remark 1. The setups in Feng et al. (2025) and Firdoussi al. (2024) are a special case of Eqn (6).
This occurs when the difficulty-base praning is ignored (q 1), meaning the curation rule retains an example if and only if its label y; matches the oracle's 's label y.-
Pruning Ratio. The fraction of data retained for learning is the pruning ratio, = Pp, 1)-
Out n original examples, approximately np survive curation. small p corresponds to aggressive pruning, while p + means no data is removed.
2.3 QUANTIFYING GENERATOR AND PRUNING ORACLE QUALITY
The following constants play a crucial role in our theory. They measure the geometric alignment between the generator (the labeler of the training data, Wy); the oracle (the pruner, w.), and the ground truth (the true labeler of the test data, w..):
p Wig Cw.
w Cw=
w. C
Pg
P+ | wo wo Pg I |wollc|lwollc
Here, ||w]|c= Vw" Cwu is the Mahalanobis norm induced by the covariance matrix C.
Geometrically, p. P.. and Pg are the cosines of the angles between their respective vector pairs, while is the cotangent of the angle between the pruner (w,) and the generator (w,).
Crucially, p and pa directly quantify the performance of the generator and the pruner. Their test errors are given by the simple relationship:
Evsa (wy) (1/TT) arccos p and Ewe(w.. (1/m) arccos p..
Note that arccos has range [0, r). These constants have the following interpretation for our analysis:
Generator Quality (p): When م 1, the generator is excellent, which call strong
generator. When p < 1 corresponding to label shift, is a weak generator.
Oracle Quality (p=): When P+ + the pruning oracle is excellent and aligns well with the ground truth.
The triplet (p: Pg. Pa) will appear in analytical descriptions of the limiting test error Frest (eea(1))
3

---

Published as a conference paper at ICLR 2026
3 MAIN THEORY: WHEN TO PRUNE AND WHEN TO SCALE
We established a precise mathematical framework Section 2, defining key quantities such as the data distribution, model, and curation rules. In this section, we use this framework to develop core theory that explains when and why data pruning can improve performance by deriving exact scaling laws for test error under different data curation strategies. As we will demonstrate, our theory shows precisely how the optimal pruning strategy changes as function of
For simplicity, we present our main results for the isotropic setting where Cg La and the pruning direction wo has unit norm. General results are in the appendix.
Assumption (Symmetric Pruning Functions). q is a symmetric binary-valued measurable func- tion, i.e. q(t) = g(-t) E {0, 13 for all R. Q denotes the collection of all such functions.
This is common setup that includes rules based on the absolute value of margins, such as keeping the" "casiest" or "hardest" examples (Sorscher et al., 2022).
3.1 SETTING #1: LABEL- DATA CURATION
We first consider label-agnostic pruning, where the decision to keep an example (I4, yi) depends only on the features 7;, as in Eqn (5). For any pruning function g E Q, we define four key constants that capture its effect on the learning dynamics:
p= TE [=) 7 = T. ( 3 = 2 ((()))(TG)) 3: 2E ([(G))
(8)
where y and + are the PDF and CDF respectively of a standard Gaussian variable G ~ NN 1). Note that p p()) defined above is just the average fraction of data kept by the pruning strategy in Eqn (5).
The following theorem provides our first main result: exact analytical formula for the test error. Theorem (Exact Test Error). In the asymptotic limis Eqn (4), the test error of the model ún from Eqn (3) is given by,
Etest (w) arccos([mmo) / Vo), where
(9)
mo := wmm(-)) w-m(-\) vo = 1==pmm'(- (-X) +-((-) 20m' (-A)r(-A)
(10)
1 (mm(->)
with w = (p- PaPx),
3p.
(11)
where m, m. and are functions explicitly determined by the constants in Eqn (8). In particular, m is the Stieltjes transform of a Marchenko-Paste law, "deformed" by pruning. Details in appendix.
This theorem provides the machinery to analyze any pruning strategy q, and isolate its effect on the dynamics of the classification test error curve. This impact is entirely captured by the scalars p, 7, 3, and B. Now, we use this tool characterize the optimal choice of q.
Sketch of Proof of Theorem The full proof given in the appendix, and relies on the construction of suitable deterministic equivalents for the resolvent matrix It defined in Eqn (3) and its square R², This allows us to calculate the limiting distribution of the "margin" yæ T " at a random test point (0, Ia), and then the test error (τ)) == P(yx T à < 0). Our approach follows random matrix theory (RMT) techniques which are now prevalent in machine learning theory (Couillet & Liao, 2022; Firdoussi et al., 2024).
Optimal Pruning Strategy. In the asymptotic limit Eqn (4), let F(() be an error functional repre- senting the limiting test error for given strategy q in the data-rich, unregularized regime:
F(g)=
= lim lim 00>0 x d.n-oco d/n-+ø
lim
(w))
(12)
where き (rg n, d, X, .... is the estimator Eqn (3) fitted version of the training dataset D. pruned with the pruning strategy
The following theorem shows how the minimizer of F(g) changes based on the generator quality p.
4

---

Published as a conference paper at ICLR 2026
Theorem 2 (Optimal Pruning Strategy). Suppose that the pruning direction we has positive pro-
jection along the generator direction Wg (Pg > 0) and fix the pruning ratio E (0, 1]. Let O.p be the set of strategies that keep fraction p of the data.
(A) If the generator is excellent (p + 1) and the pruner excellent (p. - 1), then the "keep hard" ( (KH) strategy uniquely minimizes the test error F(q) over 2p-
(B) If the generator is poor (p < 1) but the pruner is excellent (p. + 1), then the "keep easy" (KE) strategy uniquely minimizes the test error F()) over Qx-
Part (A) shows that for a strong model/generator that has already mastered the task, performance is refined by focusing difficult examples-a "less is more" (Ye et al., 2025) approach. Part (B) captures the opposite scenario: for weak model/generator, the best strategy is to keep easy examples. This latter case is particularly relevant for mitigating model collapse, where model trained on its own imperfect outputs acts as a poor generator (Shumailov et al., 2024; Dohmatob al., 2025). Also Appendix C.
3.2 SETTING #2: LABEL- DATA CURATION
We now extend our analysis to the pruning rule from Eqn (6), inspired by methods like LIMO (Ye al., 2025) and sl (Muennighoff et al., 2025). Here, an example is kept only if an oracle deems its label to be correct and it satisfies the difficulty-basek rule. This requires modifying the definitions of our key constants from Equ (8), Set 2, := z Wy. 27 == x wo and fa == piys, where pt {0, 1}
is as defined in Eqn (6). The said modifications are:
af: Ozy-
Of
:== P(p = 1), = p(()))pp),, 3 ==
B
(13)
Ozg
Expectations are over the training data and derivatives are in the distribution theorretc sense. Explicit formulae for the above constants are provided in the appendix for a general pruning strategy q E Q. Theorem 3 (Test Error for Label-aware Curation). In the asymptotic limit Eqn (4), the test error Etest (w) for label-aware curation is given by the same formula as in Theorem 1, but using the modified constants from Equ (13).
Refer to the appendix for full proofs, various corollaries and their phenomenologica implications.
4 BRIDGING THEORY AND PRACTICE
Our theoretical framework provides clear principle: the optimal data curation strategy is not uni- versal but depends on the interplay between the generator's 's quality (p), the pruner's quality (p+), their alignment (py), and the amount of available data (n). In this section, we first validate our pre- dictions in a controlled synthetic environment. We then use these validated principles as a lens to interpret and unify real-world results in LLM mathematical reasoning and ImageNet classification. For comprehensive set of validations, please see Figure 4 and Appendix B.
4.1 THEORY PREDICTION: THE INTERPLAY OF GENERATOR QUALITY AND DATA SCALE
We simulate four distinct learning regimes in a 2x2 grid to characterize the test error as we vary the generator's quality (p) and the amount of available data (n)) The left column shows a strong generator (p = 1), while the right shows a poor generator (p < 1). The top row represents a small-n regime, and the bottom represents large-n regime.
In each setting, we compare a strategic 'keep hard" pruning strategy against a baseline "random" selection of the same size, where the pruner is uninformative! Figure plots the test error, showing the match between our theoretical predictions and the empirical results.
The results reveal clear pattern for when prune. In three of the four regimes, the test error is minimized when the pruning fraction = 1, confirming the "more is more" (Sun et al., 2025) principle. This holds true when:
For the "keep hard" strategy, set Ps 0.5 and pa The "random" strategy uses orthogonal pruner where p. Po
5

---

Published as a conference paper at ICLR 2026
Small-data (n=100)
Small-data (n=100)
High quality generator (p 1)
Low quality generator (p < 1)
0.50
0.50
0.45
Error Rate Rate
0.45
0.40
Theory Keep Hard
0.40
Theory Keep Hard
Empirical Keep
Empirical Keep
Theory Random
Rundom
Empirieal Ranãom
0.35
Empirical Random
10
10"
10'
Large-data (n-5000)
Large-data (n-5000)
High quality generator (p = 1)
Low quality generator (p < 1)
0.5
Thoory Keep Hard
0.6
Theory Keep Hard
0.4
Empisical Keep Haed
Empirical Keop
Theory Random
Error Rate
Theary Randem
0.3
Empirical Random
0.4
Empirical Random
0.2
0.1
0.2
0.0
10
10
10"
10
10'
Fraction of data kept
Fraction of data kept
Figure 1: Theory Prediction across four key regimes. Test error function of fraction of data kept (p 1 means keeping all the data) for "keep hard" and "random" pruning. Solid lines are theoretical predictions; dashed lines are empirical results with error bars. The plot reveals that "more is more" strategy (optimal error at p=1) is the default, holding true for small datasets (top row) or a poor generator (right column). The bottom-left quadrant shows the crucial exception: only when data is abundant and the generator is strong does the "less is more" principle apply, with aggressive pruning yielding the lowest error.
The amount of data small (top row, both poor and strong generators).
The generator is poor, even with abundant data (bottom right).
However, the bottom-left quadrant reveals the critical exception. When the data is abundant and the generator strong (p 1), the error minimized at < 1. This confirms the "less is more" principle: in this specific regime, curating small set of hard examples is the optimal strategy.
4.2 RECONCILING RECENT FINDINGS IN LLM MATH REASONING
Our framework can interpret and unify seemingly contradictory findings in LLM mathematical rea- soning. The following results are aggregated from existing literature and our theory provides a novel explanation for why different curation strategies succeed under different conditions. In this context, the generator (wy) the base LLM that produces reasoning traces, and its quality (p) reflects its proficiency specific slice of the test data.
Recent methods like LIMO and sl show that "less is more": aggressive curation of high-quality, difficult examples improves average performance paradox emerges when evaluating only on the hardest AIME questions: here, "more is more" holds true, and performance scales with the number of training examples (Table 2).
the AIME benchmark (Table 1). However,
Our theory resolves this cleanly:
For Average Performance, the base LLM is strong generator (high for the majority of problems. As predicted by our theory, the optimal strategy is to aggressively prune and "keep hard" examples refine its already strong capabilities.

---

Published as a conference paper at ICLR 2026
Table 2: AIME (Hard-Level Questions) perfor-
Table I: AIME 2024 (Average Performance) re- ported in Muennighoff et al. (2025); Ye et al. (2025).
mance reported in Sun et al. (2025).
Training Data Size
Avg@8 (%)
Training Data Size
Pass @ (%)
O (Base Qwen2.5_322)
1.0
from OpenR 1-Math
28.4
0 (Base Qwen2.5.327
16.5
2k examples
35.4
1 114k (Openthinker)
50,2
10k examples
52.1
59k (curated sl)
53.3
1 14k (Openthinker)
47.9
1k (curated from pool of 59k)
56.7
1M (Openthinker2)
64.9
For Hard Performance, same LLM is a weak generator (low p) relative to this diffi- cult data slice. In this regime, our theory correctly predicts that "more is more" approach superior, as the model needs larger dataset to build foundational skills for these novel problems.
The optimal strategy is not universal; it depends on the generator's capability relative to the target task's difficulty.
4.3 CURATION ON IMAGENET: DATA SCALE AND MODEL COLLAPSE
We demonstrate that the same principles apply to large-scale vision tasks. We use pre-trained model as both the generator (wyy and pruner (w,) to create and select from a pseudo-labeled dataset. The strength of this generator is controlled by the size (n) of its initial training set.
Optimal Strategy Depends on Data Scale. predicted, the initial data size dictates the best pruning strategy. Figure 2 shows clear crossover point:
Small n (Weak Generator): When trained on only 160K examples, the "keep easy" strategy more effective.
Large n (Strong Generator): When trained on 1.2M examples, the "keep hard" strategy becomes superior, achieving performance close model trained on ground-truth labels.
Large Data (1.2M Examples)
Small Data (160K Examples)
Error Rate vs Dataset Size
55
55
50
60
(9)
45
55
40
40
50
35
35
30
45
0.35
2-22 Data Size (x 1.2M)
Percentage of Data Kept
Percentage of Data Kept
Keep Easy
Ground Truth Label
Keep Essy
Ground Truth Label
- Keep Essy Keep Hard
Ground Truth Label
Herd
Perudo Label
Hand
Peeadoo Label
Poeudo Label
Figure 2: The optimal curation strategy depends on the data scale in ImageNet. A clear crossover point emerges as we vary the initial dataset size n, shifting the optimal strategy from "keep easy" to "keep hard" the generator model becomes stronger.
Strategic Pruning Prevents Model Collapse. This principle is vital for stability in iterative train- ing. We simulate model collapse by repeatedly re-training the model's own pseudo-labels. Figure 3 shows that while training on all data causes performance to degrade, applying the "keep hard" strategy at each step stabilizes performance and effectively prevents collapse. This demonstrates that principled curation crucial not only for one-shot efficiency but also for long-term stability in self-improvement loops.
7

---

Published as conference paper at ICLR 2026
Error Rate vs Rounds of Training
D
D
D
Error Rate (%6
f
D
40
Dataset in round
21
Pruned Dataset in round
8
2)3
1
Model trained round
Training on from previous round Round (Training examplles
30
D
truth labe
Rounds of Training
Figure 3: Strategic pruning prevents model collapse. Over multiple rounds of pseudo-labeling, training on all examples leads to performance degradation. In contrast, selectively training only hard, valid examples consistently preserves performance across rounds.
5 RELATED WORK
Beating Neural Scaling Laws. The award-winning work of Sorscher et al. (2022) show that prun- ing training set with margin-based difficulty scores bend neural scaling curves, delivering higher accuracy with fewer samples. More recent methods reasoning and program-synthe LIsks-LIMO (Ye et al., 2025) and S1 (Muennighoff al., 2025) report
drastic pic- ture: a compact set of challenging, high-quality examples drives larger gains than indiscriminate data expansion. In these pipelines the inputs (questions) are human-curated, while the oulpuly (answers or solutions) are generated by large model such as R1 (Guo et al., 2025). We provide theoretical justification for the improved scaling behavior and systematically study simpler, yet analogous, setup through controlled experiments on ImageNet (Deng et al., 2009).
In this line, let us also mention the work of Charton Kempe (2024) who have studied the effect of repeated training examples for learning mathematical tasks like computing greatest common divisors (GCD) and numerical estimation of matrix eigenvalues, and have reported improved scaling laws.
Model Collapse. Advances in generative models have led to synthetic data becoming widespread online, where it now irreversibly blends into training corpora. Recent studies have highlighted the potential for dramatic deterioration in downstream models, phenomenon known as "model col- lapse" " (Shumailov et al., 2023). Empirical studies have demonstrated this issue in various settings (Hataya et al., 2023; Martínez et al., 2023a;b; Bohacek & Farid, 2023; Briesch al., 2023). Syn- thetic data can exacerbate biases via feedback loops (Taori & Hashimoto, 2023; Wyllie et al., 2024), narrow content diversity (Padmakumar & He, 2024; Guo et al., 2023), and distort underlying distri- butions (LeBrun et al., 2021).
Theoretical analysis also examines the effects of iterative training self-generated data (Alemo- hammad et al., 2023; Bertrand et al., 2023; Dohmatoh et al., 2024a; Seddik et al., 2024). Notably, Dohmatob al. (2024b) warns that model collapse signifies a break in customary neural scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), where increasing synthesized data volume does effectively as scaling with human-generated data. As result, recent
enhance performance works have focused on avoiding correcting synthetic data to prevent model collapse. Gillman al. (2024) propose using a correction function informed by expert knowledge modify the syn- thesized data. Alemohammad et al. (2024) leverage a model trained synthetic data as negative guidance for diffusion models. Zhang et al. (2024) employ the confidence score and an Al detec- tion classifier to discard synthesized data. In contrast, propose leveraging the synthesized data through strategic selection techniques.
We also note the approach proposed by Gerstgrasser et al. (2024), which suggests accumulating multiple versions of the training dataset over time so that their union, unlike the latest version alone, retains crucial information about the ground truth distribution of the data. While this is interesting direction, we believe it may face practical limitations as both models and datasets continue to scale over time.
8

---

Published as a conference paper at ICLR 2026
Building the recent works of Feng et al. (2025); Firdoussi al. (2024) which assume pruning oracle that can only guess which examples from the training data have correct labels, we propose and analyze more general setup covering oracles which can also assess the difficulty of example.
Benefits of Synthesized Data. Synthetic data holds great potential, as it is much casier and cheaper to scale compared to human-labeled data. Numerous empirical studies have demonstrated the ben- efits of synthesized data across a wide range of settings. Common practices include cases where the downstream task slightly differs from that of the data-generating model (Cheng et al., 2024), where the generating model is significantly stronger than the consuming one (Hemmat et al., 2025), or when better prompt engineering and external information are utilized (Shin al., 2023; Hemmat et al., 2023; Nalela, 2025). Data selection is already employed in some domains, particularly in code generation and mathematics, where natural verifiers such as compilers, solutions, or heuristic verifiers exist. For instance, Haluptzok et al. (2022) generate synthesized code and filter out incor- rect samples. Ulmer et al. (2024) use conversational metrics filter synthetic dialogue data. Trinh et al. (2024) utilize symbolic deduction engine to verify correct solutions for Olympiad geometry problems. Setlur et al. (2024) apply a final answer verifier to distinguish between good and bad synthetic data. Although verifiers are used in these cases, their effects performance have not been systematically explored, especially in terms of how different types of verifiers influence outcomes.
CONCLUDING REMARKS
We put forward a principled view of aggressive data curation, demonstrating that the striking results from systems like LIMO and s1 not coincidences but follow from fundamental properties of learning with pruned data. By supplying clean theoretical lens-validated on synthetic data and ImageNet, and shown explain phenomena in LLMs-We give practitioners a clearer picture of when to discard data and why this can stabilize training and improve generalization. In doing so, we shift the focus from "more is always better" mindset toward more evidence-based. data-centric workflow.
Furthermore, our framework explains how principled curation can mitigate model collapse Shu- mailov et al. (2024), a phenomenon characterized by a shift in scaling laws Dohmatob et al. (2024b;a; 2025). By revealing the stabilizing role of strong pruning oracle, our findings also provide theoretical basis for recent empirical successes in this area Feng et al. (2025).
Limitations. While our framework provides a unifying perspective. we acknowledge its limita- tions. Our core theory assumes high- tigh-dimensional Gaussian feature model and binary classifica- tion, whereas real-world data is structured, multi-class, and often curated online. We do not address non-linear predictors, the effects of multi-epoch optimization, the interplay between pruning and active learning.
Future Directions. We see three immediate avenues for extending this work:
Analysis of non-linear models. Extending the theory to neural networks in the kernel regime, i.e., random-feature and kernel regimes-or to the infinite-width neural tangent ker- nel-would bridge the gap to practical deep learning architectures. Such analysis still be carried out using RMT ideas. Less obvious is analyzing the feature-learning regime (c.g.. SGD on moderately parametrized networks). Here, the analysis becomes significantly more difficult and we can no longer rely on classical RMT. This is an interesting future direction.
(ii) Adaptive curation loops. Incorporating iterative re-scoring and re-training would capture the feedback dynamics used in modern self-distillation and RLHF pipelines.
(iii) Broader evaluation. Testing theory-guided pruning diverse modalities (text, code, speech) and assessing its impact on fairness, privacy, and energy consumption will clarify when and how "less is more" in large-scale ML.
We hope this work provides rigorous starting point for these efforts and for the principled design of future data-centric training pipelines.

---

Published as a conference paper at ICLR 2026
REFERENCES
Sina Alemohammad, Josue -odso-Rodr,, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad. arXiv preprint arxiv:2307.01850 2023.
Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard Baraniuk. improvigg diffusion models with synthetic data, 2024. URL https://arxiv: org/abs/ 2408 16333.
Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. arXiv preprint 23xiv:2310.00420) 10.00429, 2023.
Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai ai models collapse, 2023.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, 2023.
François Charton and Julia Kempe. Emergent properties with repeated examples, 2024. URL https: arxiv.. org/ abss 2410. 07701.
Yinan Cheng, Chi-Hua Wang, Vamsi K Potluru, Tucker Balch, and Guang Cheng. Downstream task-oriented generative model selections on synthetic data training for fraud detection models. arXiv preprint arXiv:2401.000 .00974 2024.
MMPreTrrai Contributors.
Openmmlab's pre-training toolbox and benchmark. github. ctt.lb...co///// wtt.ccoooooooo open- mml ab/ hab/mppeetraing 2023.
https:/ //
Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regres- sion. arXiv preprint arXiv:2402.07712 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang. Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In Forty-first International Conference on Machine Learming, 2024b.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. In The Thirteenth International Conference on Learning Representations, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 11929, 2020.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification. In The Thirteenth International Conference on Learning Representations, 2025.
Aymane El Firdoussi, Mohamed El Amine Seddik, Soufiane Hayou, Reda Alami, Ahmed Alzubaidi, and Hakim Hacid, Maximizing the potential of synthetic data: Insights from random matrix theory, 2024.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho. and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. First Conference on Language Modeling, 2024.
10

---

Published as a conference paper at ICLR 2026
Nate Gillman, Michael Freeman, Daksh Aggarwal, HSU Chia-Hong, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcingg self consuming loops for generative model training. In Forty-first International Conference Machine Learning, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Yanzhu Guo. Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. The curious decline of linguistic diversity: Training language models on synthetic text, 2023.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach them selves to program better. The Eleventh International Conference on Learning Representations, 2022.
Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future datasets? In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 20555-20565, October 2023.
Askari Reyhane Hemmat, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriant Improving the scaling laws of synthetic data with deliberate practice. arXiv preprint arXiv:2502. 155888 2025.
Reyhane Askari Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano. Feedback-guud guided data synthesis for imbalanced classification. arXiv preprint arXiv:23/0.00/58, 2023.
Jordan Hoffmann, Sebastian Borgcaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Benjamin LeBrun, Alessandro Sordoni, and Timothy J O'Donnell. Evaluating distributional dis- tortion in neural language modeling. In International Conference on Learning Representations, 2021.
Zhenyu Liao and Michael W. Mahoney. Hessian eigenspectra of more realistic nonlinear models. In Advances in Neural Information Processing Systems, 2021.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Mare Juarez, and Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprini arxiv: 2303.01255, 2023a.
Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. arXiv preprini arxiv: 2306.061 1300, 2023b.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint 25K1:25501.139) 2025.
Polycarp Nalela. Leveraging generative ai through prompt engineering and rigorous valida- tion to create comprehensive synthetic datasets for training in healthcare. arXiv preprint arXiv:2504.20921, 2025.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In International Conference Learning Representations (ICIR), 2024.
11

---

Published as a conference paper at ICLR 2026
Mohamed El Amine Seddik, Suei- Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Ab- delkader DEBBAH. How bad is training on synthetic data? a statistical analysis of language model collapse. In First Conference Language Modeling, 2024.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. Advances in Neural Information Processing Systems, 17:43000-430331 2024.
Joonghyuk Shin, Minguk Kang, and Jaesik Park. Fill-up: Balancing long-tailes tailed data with generative models. arXiv preprint AXXiv:2306.072000 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arx1v:2305.17493, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. Ai models collapse when trained on recursively generated data. Nature, 631, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances Neural Information Processing Systems, 35:19523-19536 2022.
Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the lad- der of reasoning: What ulms can-and still can't t-solve after sft? arXiv preprint arXiv:2504. 11741, 2025.
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification dataset biases. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 33883-33920. PMIR, 23-29 Jul 2023.
Trieu Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang. Bootstrapping Ilm-based task-oriented dialogue agents via self-talk. arXiv preprint arXiv:2401.05033, 2024.
Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. Fairness feedback loops: training on synthetic data amplifies bias. The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 24, New York, NY, USA, 2024. Association for Computing Machinery.
Yixin Ye. Zhen Huang, Yang Xiao, Ethan Chern, Shijic Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
Jinghui Zhang, Dandan Qiao, Mochen Yang, and Qiang Wei. Regurgitative training: The value of real data in training large language models. arXiv preprint arXiv:2407. /2835, 2024.
12

---

Published as a conference paper at ICLR 2026
Appendix for "Why Less is More (Sometimes): A Theory of Data
Curation"
CONTENTS
Introduction
2 Setup for Theoretical Analysis
2.1
Data, Model, and Assumptions
2.2 Data Curation Rules
2.3
Quantifying Generator and Pruning Oracle Quality
3
Main Theory: When to Prune and When Scale
3.1
Setting #1: Label-Agnostic Data Curation
3.2
Setting #2: Label-aware Data Curation
4
Bridging Theory and Practice
4.1
Theory Prediction: The Interplay Generator Quality and Data Scale
4.2 Reconciling Recent Findings in LLM Math Reasoning
4.3 Curation on ImageNet: Data Scale and Model Collapse
5 Related Work
6 Concluding Remarks
A Experimental Details for ImageNet
14
A.I Dataset
14
A.2 Model Architecture
15
A.3 Training Setup
15
B Empirical Confirmation of Our Theoretical Formulae
15
B.1 Experiments for Label-Agnostic Curation Rule Eqn Eqn (5)
16
B.2 Which is Better, "Keep Easy Examples" of "Keep Hard Examples"
16
Results in the Regression Setting
17
C.I Theoretical Setup
17
C.2 Main Result for Regression
18
C.3 Optimal Pruning in Regression Setting
19
13

---

Published as a conference paper at ICLR 2026
D Main Ingredients of Proofs
21
D.I Deterministic Equivalent for the Resolvent Matrix
D.2 The Isotropic Case
22
D.3 Test Error Representation: The Classification Setting
23
Proof of Proposition
24
E,1
Asymptotics of ||
................................
24
E,2 Asymptotics of a
................................
................................
25
E,3 Asymptotics of Classification Test Error
................................
26
F Proof of Proposition
26
G Proof of Theorem 1, Theorem 3, and Corollaries
27
G.1 Proof of Theorem and Theorem
27
G.2 Proof of Corollary 1
................................
28
G.3 Proof of Theorem
29
Proof of Theorem 4 (Regression Analysis)
31
H.1 A Modified Bias- Variance Decomposition
31 ....a...........................
H.2 Proof of Theorem
................................
H.3 Proof of Corollary 2
................................
31
H.4 Proof of Proposition 3
32
I Proof of Theorem 5 (Optimal Pruning Regression Setting)
32
J Proofs of Lemmas
33
J.1 Proof of Lemma
33
K Proof of Lemma 7
34
L Proof of Lemma 3
34
L.1 Non-LIMO Case
34
L.2 LIMO Case
35
M Analytic Formulae for p(g), 7()), 3(g), and 3(g)
36
A EXPERIMENTAL DETAILS FOR IMAGENET
We now provide details for the experimental results presented in Section 4.3 of the manuscript.
A. DATASET
All experiments are conducted on the ImageNet- IK (Deng et al., 2009) dataset, which contains approximately 1.2 million training images and 50,000 validation images across 1,000 classes. For
14

---

Published as a conference paper at ICLR 2026
experiments with reduced dataset sizes, random subsampling generate smaller training sets at various fractions (e.g.. 50%, 25%, 12.5%) of the full dataset.
A.2 MODEL ARCHITECTURE
We use the Vision Transformer (ViT-B/16) (Dosovitskiy et al., 2020) our primary backbone, implemented via the MMPretrain framework (Contributors, 2023). The model uses a patch size of 16 and input resolution of 224 x 224. We apply drop path rate of 0.1 and label smoothing with a smoothing value of 0.1 the classification head. During training, we apply data augmentation techniques including Mixup (a 0.8) and CutMix (a 1.0).
A.3 TRAINING SETUP
models are trained using the Adam W optimizer. The learning rate is scaled with global batch size according to the linear scaling rule. For ViT experiments, the base learning rate is 1 1x10-1 x batch size with weight decay of 0.3,
1 and B (0.9,0.95) 0.95).
To ensure fairness across dataset sizes, we adjust the number of training epochs inversely propor- tional to the dataset fraction, that the total number of iterations remains constant.
Training is performed on nodes, each with NVIDIA H100 GPUs (total 32 GPUs), using Py- Torch's Distributed Data Parallel (DDP) via SLURM. The batch size per GPU is 128. We use synchronized batch normalization and standard augmentations including random resized crops, hor- izontal flips, RandAugment, and random erasing. Models are evaluated the standard ImageNet- IK validation set using top-1 accuracy.
B EMPIRICAL CONFIRMATION OF OUR THEORETICAL FORMULAE
We validated our framework through extensive simulations and comparison with theory, summarized
in Figure 4. Synthetic datasets were generated under the model of Section 2, with d = 200, varying sample size n, pruning fraction and generator angle p. Logistic regression with À 10-6 was trained on curated subsets, and error was measured as the angular deviation between learned and true weights.
Coverage. We tested 15 parameter settings E {500, 1000, 2000), p (0.2, 0.5, 0.8), P 70,, π/ /4], keep-easy vs. keep-hard), spanning both typical and extreme regimes.
Agreement. Theoretical and empirical results matched closely: mean relative error 1.8%, all 5%. Bland-Altman analysis showed mean difference 0.0019 with 95% limits of agreement [- 1-0.033, 0.0077].
Sweeps and Landscapes. Parameter sweeps confirmed that theory captures observed non- monotonic pruning effects, power-law scaling with n, and angular dependence. Two-dimensional landscapes (sample size x pruning fraction) showed near-identical patterns, with maximum absolute differences < 0.01.
Statistical Checks. Empirical eтror distributions (50 runs) centered tightly around theoretical pre- dictions, and theory lay within 95% confidence intervals across all tested settings.
Robustness. Agreement held across configurations, including edge cases (p 0, extreme prun- ing), indicating the framework captures the essential mechanisms.
Implication. These results establish that theory accurately predicts generalization under prun- ing in dineh-imensional linear classification, providing a reliable tool for analyzing and optimizing data curation strategies.
15

---

Published as a conference paper at ICLR 2026
Theury vs Empiniaal Ervor Rates selaiive aver: 1.85
Ervor vs Pruming Fraction
Ervur Sample Size
0.5
prt2
0.4
0.4
Emppiaa Ense
Liner Sore
Einer RaMe
Depitial prboi
Thegz
0.2
0.2
Pertict agnemeet
0.0
0.3 Enor Geverator Angle
0.2
0.5
0.7
500
2000 5003
Tazoretcol
Fruning
Size (6)
Theay Empirical by Configuratinn
0.55
Thooy
Tarptccal
0.30
Barner Nite
Unoer
0.25
Thoap 4L3
Emptrooulp-ts
Thevry prius
Thoary p-127
0.35
Empirool
100
Kxpp Bayy
Kepp Hal
Generator Angle (dagrees)
Conigumai
Figure 4: Validation of theoretical error predictions against empirical simulations. (A) Scatter plot of theory vs. empirical error across 15 configurations, with diagonal perfect agreement. (B- D) Parameter sweeps for pruning fraction, sample size, and generator angle. (E) Configuration-win comparisons, All results use logistic regression with 10 6
B. EXPERIMENTS FOR LABEL-AGNOSTI CURATION RULE EQN (5)
As promised in the main manuscript, Figure 5 presents results toy data, with curation done according to the label-aggostic nule Eqn (5).
keep hard good w.
keep easy/ good W.
keep hard good
keep easy good
0.01
0.1
0.3
32
3
35
31
3
3
keep / poor w.
keep easy poor Wo
hard easy
100
3
Samples per parameter Samples per parameter
Samples parameter Samples per parameter
(a) Beating scaling laws.
(b) Mitigating model collapse due to label shift.
Figure 5; Effeet of Label-agnostic curation rule Eqn (5) proposed in (Sorscher et al., 2022).
B.2
WHICH IS BETTER, "KEEP EASY EXAMPLES" OF "KEEP HARD EXAMPLES"?
See Figures 6 and 7.
The data is Gaussian, generated according to Eqn (1) with C La (covariance matrix of samples, under the generators distribution) and Σ Ia (ground-trutt covariance matrix). The sample size sweeps the range 10 through 10t in log-scale, while the input dimension fixed to 200. The data curation is done according to the Label-aware rule Eqn (6). The estimator to defined in Eqn (3) is
16

---

Published as a conference paper at ICLR 2026
100
n 10000
0.6
Classification emrr
0.4
0.2
keep easy
keep hard
0.0
10 1
10-?
10
10-²
10
p
P
(a) Label-agnostic curation rule Eqn (5)
= 100
= 1100
n= 10000
Classification error
0.2
keep easy
keep hard
0.0
10
10-1
10
10-2
10-1
10a
10-2
10
p
P
(h) Lahel-aware curation rule Eqn (6)
Figure 6: Beating scaling laws. Solid lines experiments; broken lines theoretical pre- dictions (Theorem and Theorem 3). For this experiment, the angle between generator labeling vector wy is perfect, i. W wa, the ground-truth. Notice the perfect agreement between theory and experiment.
computed using Scipy's 's linear algebra functions operations (from the "linalg" module therein), with regularization parameter fixed at \ 10 10-6, The classification test error Erest is defined as:
Exest (w) E foo1 (sign(æ* T v)) P(sign( 1)) * y).
(14)
The pruning direction wa in Eqn (6) chosen to make an angle (perfect pruning direction) or 0 1/10 (poor pruning direction) with the ground-truth labeling vector w. (1, ........)) 0).
For Figure 5(a) ("beating neural scaling laws"), the labeling vector wy E RR for the generator equals that of the ground-truth. Thus, the generator is taken to be perfect, setting also considered in (Sorscher et al., 2022).
For Figure 5(b) ("mitigating model collapse"), the generator is imperfect: its labeling vector wa makes an angle T/5 with the ground-truth w.. This imperfection simulates the model collapse phenomenon (Shumailov et al., 2024; Dohmatob et al., 2024a;b; Feng et al., 2025; Dohmatob al., 2025).
C RESULTS IN THE REGRESSION SETTING
C. THBORETICAL SETUP
As promised in the main paper, we now turn to the case of regression, where the label variable in the data distribution Eqn (1) now given by
= x W + 17.
(15)
where 7/ ~ \N o²) noise variable independent the covariates æ. The test error of estimator ii is now measured by
Freeg (w) B(e,))
((x T TT [w,) 2²
(16)
17

---

Published as a conference paper at ICLR 2026
100
1000
n 10000
Classification emrr
0.5
0.4
0.3
0.2
keep easy
0.1
keep hard
10-2
10 1
10-?
10
10a
10-²
10
p
p
(a) Label- curation rule Eqn (5) (proposed in (Sorscher et al., 2022))
= 100
= 1000
n= 10000
Classification error
0.5
0.4
0.3
0.2
keep easy
0.1
keep hard
10-1
10
10-2
10-1
10a
10-2
10
p
P
(h) Lahel-aware curation rule Eqn (6)
Figure 7: Mitigating model collapse. Solid lines experiments; broken lines theoretical predictions (Theorem 1 and Theorem 3). For this experiment, the angle between generator labeling vector Wy and ground-truth w, is x/20, thus simulating imperfect generator. Notice the perfect agreement between theory and experiment.
C.2 MAIN RESULT FOR REGRESSION
Define the following auxiliary quantities
we = (wi Wo)Woo ueg = Wig Wg: 6 6: Wg W., a e Tw b: cT Wy: 0² = ||e|/2. (17)
Thus, w/ is the component of Wi pointing in the direction of the pruning vector w, and wg is the perpendicular component. c² measures the disparity between the generative and the ground-truth labeling vectors Wi and w. respectively. is clear that
|| 112 = 2 1 1|2 = (1- -0,)11,,
(18)
a = ||- (p PyPt), b wmll 1|2 |
(19)
where P. Pg. and P. are defined in Eqn (8).
The following is one of our main contributions.
Theorem 4. In the limit Eqn (4), the regression test error of the model v defined in Eqn (3) is given by
Ereg (1)) + B B+ + 2A (m (mma-)aa+ + ())-))
(20)
with B = X² (m'(- + (=)) V o² om (-X).
Universality. Note that for fixed pruning rate p E (0, 1] and pruning direction wor the specific choice of pruning strategy Q used only enters the picture via y 79), defines in Eqn (8). Two pruning strategies with the same value of y induces exactly the same test error dynamics Ereg in the high-dimensional limit Eqn (4).
18

---

Published as a conference paper at ICLR 2026
Unregularized Regime. We now consider our theory in the limit À - O, in which case the esti- mator d defined in Eqn Eqn (3) reduces to the least-squares estimate for w. namely w X where (X', is the pruned training dataset, i.c the nonzero rows of (DX,DY)
Corollary 1. In the limil Eqn (4) then - 0, holds that Ereg + L, where
(A) Φ < p, then L o²6 + c².
(B) If Φ > then with == 1 - andaanda 7//
7// we have
== d²
+ (1wgg 212 /2/l12/c/aaa + + 2a b(a+b/c1) co.
Note that when 1 (corresponding pruning), the above result recovers one of the main results of Dohmatob et al. (2025), namely, their Corollary 1.
The following result is yet another important consequence.
Corollary 2. In the noiseless setting 0, the following hold:
lim lim lim (1)) = l|w, Wig w/l1² c² Ap (0,
p00 X-00
d/n-2
lim inf lim lim (w) Ilu. - w < 2,
ff flww-w'll c² < ||u. - w
p-0 PE(0.1) X-0 t.0
otherwise
d/n-s0
Thus, pruning provably mitigates model collapse, under the sufficient condition
I|w. will < Ilw. -wyll < I|w, w/
Note that if ||w_||| and
|!y 1 12, then c² ||w. wgl12 =1+ -1+ 2 2r Furthermore,
P+ (i.e wo w.).
then l|w. w/ 1|2 w| py (1 pg)
Keep if 1y xr_ w.
C.3 OPTIMAL PRUNING IN REGRESSION SETTING
Consider sub-collection of parametrized pruning strategies constructed as follows. For any p. [0, 1]. define Up.u Q by
Ip (1) 0, if a(p, < |t| b(p, u).
(21)
1, otherwise,
with alp, u) = + - ((1 (11+(11w)p/22)
= := (1-1111p))) pp//))
(22)
Thus, Up.u is the indicator function of the disjoint union of 3 intervals: [-a(p, u), alp, w)), and two "tails" (-o,, -b(p, u)) and (b(p, u), oo). Such a pruning strategy selects a mixture of "very easy" training examples (corresponding to neighborhood of 0) and "very hard" examples (corresponding to tails). The parameter controls the proportion of training data that survives pruning, i.e we have p( = p. while the parameters u controls the fraction thereof which are "very hard".
Theorem 5. For any pruning strategy
Q. there exist p, u [0, 1] such that pruning strategy qp.u induces the the same regression test error Ereg (1) for the estimator define in Eqn Eqn (3) as pruning with q. In particular, the optimal pruming strategy has the form 4p.u-
19

---

Published as a conference paper at ICLR 2026
10-3
==10-2
=
keep (per 1000)
Text etrar Englld)
0.8
10
250
0.6
0.4
0.2
10f
10 10a
104
10* 10a
104
10
10e
Original
(a) Test error vs original dataset size n. We plot the regression test error Ereg as a function of the original/unpruned datasct size d and report result for different rates of pruning (per thousand cxam- ples). Solid lines correspond to experiments while broken lines correspond to the analytic expression provided by Theorem 4. Notice the perfect match between theoretical predictions and experiment. We see that optimal is optimal consider and unregularized model (small X) and discard almost all training data!
A 10-1
A 10
== 10-1
Test arror Eng(0)
6 10-1
40-10-1
3x
(per
10
100
1000
20-1
50
250
10a
10
10a
10a
10*
Pruned dataset size np
Pruned dataset size
Pruned dataset size
(b) Test errr pruned dataset size np. We plot test emor function of the pruned dataset amount compute. Once
m actually used fit the model, the point being to control again, we see that it is optimal to discard almost all training data. However, optimal regularization is no longer zero; for nonzero X, the error might eventually increase with m.
Figure 8: Mitigating model collapse via pruning in regression setting. Different colors correspond different levels of pruning where keep only the hardest/most informative examples (x4,, vi) with the largest value of the projection of the features [ wol along the pruning direction wo-
20

---

Published as a conference paper at ICLR 2026
D MAIN INGREDIENTS OF PROOFS
D.1 DETERMINISTIC EQUIVALENT FOR THE RESOLVENT MATRIX R.
Definition (Deterministic Equivalents). Given sequence of random N N matrices (Rx) N. deterministic equivalent thereaf is a sequence of deterministic N x N matrices (Ry) N such that
Ax(RN Rry " O,
(23)
for all sequences of N x N matrices (Ax)N with bounded Frobenious norm.
Let II (resp. IT- Ia II) be the projection onto the span (resp. orthogonal complement of the span) wg- Define the following auxiliary vectors and scalars
e wo
v
V1
v1
IT-
(24)
Note that tn is (d 1) diimensional and ||i|| Vlul Henceforth make the replacement -X < 0, that the resolvent matrix R now writes
of.
R = R(z) =- (X DX/n - =laa
(25)
where recall that D is the x D diagonal matrix appearing in Eqn (3), with D₁ Ps. the prune/no prune bit for the ith training example. Let 8(z) be the unique positive solution to the fixed-point equation
-1
m(2) tr Ro(2), 8(2)= = CRo(z), Ro(2)= =
Pt 1 + pz)(z)
C zld
(26)
Note that the inner expectation evaluates to
E
Pi
0(z) =: =()(2),
1 1 (()(()
and so Ro(2) = (t(2)C - 2Ia) -1. Observe that Ro(2)(t(2)C - =Id) = Ia, and so t(2)CRo(22)
Ia + (2). We deduce that
((=)((2) n tr(())CR((z) n tr(Ta
- + =Ro(=)) の (1 2m(2)).
Thus, the equations defining m(=) and 6(z) can be rewritten as
m(=) d (r(((=)C -(aa)) P
(27)
t(2) 1+
(28)
a (1 + 2m())) = t(2)6(z) t(2)
{(2) -1 = t(2).
(29)
Solving for Φzm((2) in terms of ((z) in the last equation gives
p0(=) Φ p 1+)(2) dzm(2) + 6(2)
f(2).
Plugging this into the first equation gives the following fixed-point equation for ((z)
p- Φ (0-((2) 2n - tr(((()) -Ia)
(30)
The following result shows that It is a deterministic equivalent for R.
Proposition Recall the function t(2) as the unique positive solution to the equation Eqn (30).
Then,
R~ R. with R. C- (01/2(M(2)TT) (=)
(31)
where m(z) := t(2)
m(z) == =(z)
1
s(2) = 2t(z).
(32)
z =z
p
21

---

Published as a conference paper at ICLR 2026
D.2 THE ISOTROPIC CASE
Consider the special case where the covariance matrix is C = la- Fix an LZ-regularization parameter
and pruning rate p E [0, 1].
Lemma 1. For every = - < O. m(2) is the unique positive solution to the fixed-poine equation
Eqn (34), and is given explicitly by formula
m(z) p
2)
402
(33)
2Φz
Alternatiely,, m(2) defined in Eqn (33) unique positive solution to the fixed-point equation:
with 2 == - 1 = - 1+ om
(34)
Thus Lemma 1 shows that m(=) is the Stieltjes transform of the limiting spectral density of the resolvent matrix R appearing in Eqn (3), and has the property (among many others) that d tr R + m(2) in the limit Eqm (4). represents somewhat distorted Marchenko-Passe law; indeed, the classical MP corresponds to p + 1 (i.e. no pruning).
Furthermore, it is not hard see that
m(z) = m(z) = 6(z)/0
(35)
in this case.
Proof of Lemma 1. Indeed, observe that in the isotropic case the equation Eqn (30) reduces to t(2) = (2/(((z) 2), or equivalently
0 02 (t(z) p (t(2)
t(2)² (p o 2)t(2) + p2.
The discriminant of this quadratic equation evaluates to
(p (p-dh δ 2)² 4p= (p - + 2z)² 4pz
0 - 2)² 4z 42(p 2) 4pz
=)²² 462
and so because = < 0, the positive solution is
((x) = p - Φ +
(p ④ 2)² 40z
(36)
We deduce that
1
p 6 2 + ( Φ - 2)2 402
m(2)
t(2) -
2
2 p 0
v(p
2)² 402
(p 0 2) (p 0 2) 4
2) 2 A0z
20z
which is precisely the formula given in Eqn (34).
Spectral Functions.
Define the following auxiliary functions:
mix)) == ==),,, s(=) 1 + om)=)'
m(=
r(=) :- (2())) (())))
(37)
8(2)
where the constants 3 and 3 are as defined in Eqn (8). Notice that is (proportional to) convex combination of m and f..
We will be needing the derivatives of m', m², mí, and r'. This is the purpose of the next lemma.
22

---

Published as a conference paper at ICLR 2026
Lemma 2. We have the following identities:
(2) = 1- (1+ (1=)))))0 m²(x)) = + ( pe (o+ + 1/mlz))² pd
m(=)2
p
p
(1 (m(())) + 1)) r'(s) - (+))
vam' (3)
m = =)())²
The following result then follows directly from Proposition 1.
Corollary 3. In the isotropic setting, we have the following deterministic equivalents:
R ~ R, with R = m(()11 + s()]1I,
(38)
R² ~ m' (=)I1 (z)11
(39)
where m(=) = 1/(5(2) m((2)=11({(2)
2), s(2) = 7/ + (m(2)), and y 2 0 is as given in Eqn (8).
D.3 TEST ERROR REPRESENTATION: THE CLASSIFICATION SETTING
WLOG, suppose henceforth that iy == C2/3wy- W. == / and W. := w, are unit vectors in E.. Let u = T. and let be its completion to an orthonormal basis for the span of C and g (if th and yy are parallel, i.e if P9 = =1, we simply set = 0). Define c Id by
e := E p.y.a
(40)
for a random training data point (Ii, yi) ~ Pg and corresponding selection/no select bit pt E {0, 1} (e.g, Pt is as given Equ (5) in the case of tabel-agnostie data curation and Equ (6) in the case of Label-aware data pruning).
Also define p p(a) [0, and 7 =7(4)= 2 by
p p= Elps,, = B[(xr, wo))pp
(41)
Lemma 3. It holds that = 3, u + /2 with the Bx as given in Table 3. Also, the constants p and y defined Eqn (41) are as given in the table.
Curation p(1)
7(2)
32(4)
B1(g)
aabel-agnostic
E[g(G)]
E[g(G)
2E(g(G)p(TG)]
(E(((G))(TG)G
Label-aware
([g(G))(T]GI)
(lG(G))(FIG
Blg(G))((F))
[[G(C)
Table 3: Fundamental constants. Here, E Q is any even/symmetric pruning function and G ~
\( 1), with pdf yp and edf Φ. Recall that t = /s/ V Pg. and we use the identification 3 - 32. 3 + B1. Note that taking g = on the second row corresponds to the setup of Feng et al. (2025) and Firdoussi et al. (2024).
We are now ready to state main results, which is a generalization of Theorem and 3.
Proposition 2. Let e E RRd be as defined in Eqn (40). For a random test point (x. y) ~ P.,, we have the following dimeesiimesi representation (where G1 and G2 are iid from N (0, 1)):
yx T i & m Gr + Vv - mm22, with
(42)
mo
= -RE.,
m ~
I+ 8'
|/1/2,0111'
(43)
2c Re 1
P tr EC' H -Te-- n
(1 + 6)2
Vo
1 n
tr EC',
(44)
: E R
C' = ERRR),
='
(45)
23

---

Published as a conference paper at ICLR 2026
where 5 6(- 0 is defined by the fixed-point equations Eqn (26).
Furthermore, holds that
Erest (w) = Pyy á 0) arrcoo((mml)) vV)-
(46)
Remark 2. Note that the above result is valid for any curation strategy which maps easy training example (Ira ys) prune/no prune bit p; {0,1}. in an iid fashion. The choices Eqn (5) (label- agnostic) and Eqn (6) (Label-aware are but particular cases.
E PROOF OF PROPOSITION 2
For random test point (I, y) ~ P., we can write
yr T - = yz T 12 = sign(z [2/2w,)2 111/2
Write E aΣ w. + r, where r Σ1/2200 12 and a 0 is to be determined.
Observe that is perpendicular to 21/2w. iff r "1/2ww w Ew. ax a //1/// || 0 iff
v Ew. Ew./ ///////////
(47)
With this choice of a, one computes
yr T W ayz T T w. y2Tr.
(48)
Because is perpendicular to E1/2 we know that the above is sum of two independent random variables.
For the first summand in Eqn (48), observe that
yz / w. you T wi sign w., )a:
w. Tor,., 1x T w.rl
which has the same distribution as |G] for ~ (O, w. Ew.). For the second summand, it has distribution (o ||r] with ||||| =| 0²11221/² w.
E.1 ASYMPTOTICS oF ||
Now, one computes
w - n
1
Σ
Pell, Ra,
Σ R.M
(1 + 8)n
We deduce that
(1 ()+6)2?21))122 = Epa R. Σ
D.P.DMyEE R.ERR
$3:34
Now, observe that
1 n²
1
Σ
R_
tr(p,aoa T R.
n12
1
tr(Eppaaa I E.R
n
tr CRRRRR
ir EC'.
24

---

Published as a conference paper at ICLR 2026
For two distinct sample indices i,j [ni], we have
1/n
: = Rhj
R 12
Rijj
1+ 8
1/n
1/n Rijii R-is)
R.ER,, (R-1)
1y j R R-
1+0
1+6
1/n
1/n
= Ri-j
R ER R ب5: R-j
1 Ri R-i
1+1
1/n²
T R R-j 15 , R 3o ER ij و R_ij
+
(1 + 8)2
and so
E E.P.Psua R. = A A As + A4, where
A Elp.p,uey,r Ry En:r
A2 = 1/"
1+ viy,,
Elp.p, R ij ER Ry00 R R-ij
1/n
A3
Elp.p,yay,0 R Gfr R ER
+ 5
1/n²
A = (1 5)2 ElP.P,ueya R R R DR R-isl
By symmetry, is clear that A4 = O. order to compute A2 and Ag. we shall need the following result which can be obtained by applying Wick's identity (aka Anderson-Issen
arguments).
Lemma 4. Let x and z be iid (0,, C)) and let Rd - RR be an odd function. Define e := [[g(r)2). Then, for possibly random random d x matrices A and B independent of x and z,
(lg(z)g(2)x' Az A] = Ac, (lg(x))(=)(z Az) (x) A B] tr(BC)e T Ac 2c ACBe, (ls(x))(=)(x² Az)(x T Bz)² A, B] tr(BC)&c Act Atr(BC)e ACBc+ 2c ACBCBc. Az))(r
Applying the first part of the lemma with A RER gives A c E'c, where E' E(RER). Applying the second part of the lemma with A = Rhij ~ R and B = R ~ RER gives
Ag Ag 1 5n (tr(EC"))e T Rc + 2c RCRER()
1
1 1 tr(IC')c
c Re tr EC'.
T Rc
1+8n
1+8n n
We deduce that
|| w01
L L +c N'c-
2c T Rc
tr 2CC =.v.
(49)
(1+
n
1 8
E.2
ASYMPTOTICS OF
Mean. One computes
1
|/ 11 = End Ew. Ew.
E Pilfir T R. Ew.
1
E Elpcya R_EEwa]
1 1
E(peyaar
T B(R_,]EW
1 +
REw.
1+4 8
25

---

Published as a conference paper at ICLR 2026
Variance. On the other hand, observe that
w///Bw,ll'od = (w )2 = v Ew. w. Ev..
So, applying Eqn (49) with E replaced with the rank matrix Ew,w, w.D. E and E' replaced with W_ ER, we get
1
// II "a=
ER(i Ew. w. Eni (1 8)2 Elc REw.w! ERd
(1+ 8)2 REw.))
where we have ignored all trace terms which are now of order 1/n (negligible). The RHS of the above display coincides with the square of the estimate for /////w, |I provided earlier. We deduce that the variance of a vanishes, and so
C REW
Ea
(1 + 8)
|
w.ll
Combining with 48 and Eqn (49) completes the proof of the first part of Proposition 2, namely the convergence Eqn (42).
E.3 ASYMPTOTICS OF CLASSIFICATION TEST ERROR
In the asymptotic limit Eqn (4), one may use the representation Eqn (42) to write
lim Brest (w) lim P(yx Φ S 0)
= PPmm/Gi + Vv G² S 0)
PP
G2
S
m
G1
Vv - m²
G2
mm
PP
S
G1
Vv m²
1
+
arctan (octann-lmna (-/ml/ Vv m²)
2
arccos([ml/ ((ml//) π
arccos( /mal/ //v///)).
as claimed. Note that, we have used the fact that G2/ G1 is standard Cauchy random variable, for independent G1, G2 NO,, 1). This completes the proof Proposition 2.
F PROOF OF PROPOSITION
Using Theorem 4 of (Liao & Mahoney, 2021) (and the proof thereof) combined with some basic algebraic manipulations, we can write
R R where - /2/
(50)
Pi (II (Ta:) g2/2 =Id,
(51)
1 p+)()(
for random training example (ai, ui) ~ Pg from the generator, and corresponding prune/no prune bit pa. The matrix C is the covariance matrix of x,, Since p. is Bernoulli with mean == IP(p, 1), it is clear that
E
Pi
t(2).
1 [+p(((2)
1 1+ 0(2)
This further gives
R = 1//)
2La + C1/2 IITII/11
(52)
Pj mu
with K = IF
1 p:0(2)
where u:= E-1/2
(O, Ia) and 1
26

---

Published as a conference paper at ICLR 2026
Now, to determine the matrix K, first rewrite (uz, and v (21, where
0 T Wo
Wo
4 = | E IR
U1 ==
|wol| ||
E IR,
(53)
== II u E Rd-1
1a = II E Rd-1
(54)
The advantage this representation is that:
and are orthogonal to w.
u and u.g. are statistically independent.
u has distribution (0, 1).
has distribution (0,0, Id-1).
Combining with the fact that due to the evenness of the pruning function 9 (in Eqn (5), Eqn (6), etc.), the distribution of (z,, Vs, Ps) doesn't change if z; is replaced by -x; (so that [p,uu, = 0 for all i * j), we get:
K s(=)II + (1)TT+
where s(2) == El,,
$1(2) E[h,Gi
h; 1 + p:)(z)
Pi
(G1,
Combining with Eqn (52), we get
R- = (01/2(((=(())))) (2) Ma))))++M
(55)
where a(z) = t(2)( t(2) 8())
b(2) = s(2) t(2).
(56)
Now, using the Matrix-Inversion Lemma, one can obtain R from R-1 as follows:
(a()) a(z) la b(=)/a(z)
II.
(=)/a(>)++1
a(²) b(2) + a(2)
suffices to notice that 1/(b(2) + +a())) = 1/(s(2) - 2) = m(²) and 1/a(z) = m())) by definition, and the result follows.
G PROOF OF THEOREM 1, THEOREM 3, AND COROLLARIES
Theorem and Theorem are direct consequences of Proposition 2, where we use the deterministic equivalents provided in Corollary 3, considerably simplify the resulting formulae. Corollary is a consequence of Theorem and limiting behavior of the spectral functions given in Eqn 37.
G.1 PROOF OF THEOREM 1 AND THEOREM?
Set - Also recall that Biu Bzu, where u, v, Bi, and 32 are as in Lemma 3. Note that have the identification B B2 and = B1. We know from Proposition 1 that R R m(=)IT- + II, where IT = uu One computes
mo (w./ (ww/[lw.)) 11) Rc [ww_]] w (m(2)IT+ + m())IT) (Bru B2v),
1 ||w=||" : (81m(z)u 32m(z)o),
Moreover, on computes
w. u/'llllee ///l///w1 P. by definition, and (Wg [w] T w.) w.) ) T w. | wy w. /w. Pg Ill (w /w/l/ww/, //w
w. v
W.ell
W9 (w) wo )wol
|| 1l / /9/
p PgPa cos cos fy cos
sin 0, cos { = v1 - p2 cos & = w/B2.
/
0g
sin 0
27

---

Published as a conference paper at ICLR 2026
where we have used the identity cos 0 cos By cos f. + sin 0g sin 0. cos Ç, known as the Spherical Law of Cosines. Putting things together gives my = wm(2) + ())(2) as claimed.
Likewise, one computes
1 n
tr EC' = tr R² = tr (m'(2) + (x)II = (())) (z),
n
c Rc= = c (m(2)Il + +(())I)) c (3 B20) (m(=)[I ( ( (Biu u + B20)
+ = (+ r())).
(m)
c e = e E | e = et (m)(1) + + c m2 + ml)) r²))).
We deduce that v= = vo/ (1 8)?, where
P EC - 2c Rc Cr
v0=
n
1+8 n
L RR + (2)
2r(z) 1
tr R² pom' +z) 2r(=) (2)
1+ 8(z) n
1 + <dm(2)
the result then follows from Proposition 2.
G.2 PROOF OF COROLLARY
As usual, set == -X < 0.
(A) For Φ < p, it is easy to see from formula Eqn (33) and Lemma 2 that the limit = -> 0, one has
1
m(z) +
p 6"
m(z)= + 0,
m(=) + p/y
p 0
m m'(z) (p- Φ)3
p
m2(z) +
p 0'
m'(=) + (p 3)3
p/7²
(p(p - +)+ + 07) =
pp- ((p w/p///++////00/ + t/7),,
p
m (2)
1 + (m((z)
(p- $)²²
Furthermore, with my and vo as defined in Theorem 1, one computes
r(z) 33m(2) + mm(() B32
1
3 p/7
Fo
p- Φ
p Φ'
(()) (3) + (m) B2
(p 0)3 + B2.
p
p//²
(p(p +))
r/
(p 0)3 3
(p 3)
where and ra are as defined in the claim. We deduce that mo/ / vvo - =mg = a/ vbb - a and the result follows from Theorem
(B) Now consider the case Φ > p. Observe that my = v mg = -m 122² 2 On the other hand, from Eqn (33) we know that
-2m(2)= = Vw 2)2
40z - (p- Φ z)
(57)
26
28

---

Published as a conference paper at ICLR 2026
Combining with Lemma 2, we deduce the following limits
(2),222²m + co := 1 p/6 0,
p/o
m'
Φ - p
-=m{=),= m2)
Co
7/0 + 00
2mm
-
1 0m())
Furthermore, one computes
-2r(2) = =33 (-2m(=)) + B3 (-zm(=)) = 32200 + B8
co
coTo:
7/0 + Co
22((2) 33=²m' m2())))))))))))) + 83 m()) 8200 + 33
Co
Cu'o.
7/0 + c0
-2m = 2/ (-2m( (2 2) 2m + 2//TCO (w + w//0//////////////// + = a,
x² vo- pozz m' (z) + 2 2Φ - 2m (z)
1 dm((2) (-2r(z))
+ poco + roco - 2roco = co (pp - r0) =: b.
We deduce that
ma/ Mo/Vro- = - 2 22 a/vb.
and the result follows from Theorem 1.
G.3 PROOF OF THEOREM 2
Taking the limit Φ + in Corollary 1, we have
b- B + 82p2/22
a w w wp/7
To p. (B2 + 322/122)
p²
a/vb + w/p + 5//
(B/p) / - 2 cos C + (3/7)pp
iv 1 p cos < <+1
/B2/12 + 132/72
V 82 82/p22 122 + B2/722
2J2 +1
with =(4))= = (19),(1)) 0.
pp(g)
where we recall that w 3 ~ cos and W = 3p...
Part (A). Taking p. 1, meaning that pruning is done along the ground-truth, gives
1/V
From Corollary 1, we see that the limiting value of Eay(1), i. the functional F defined in Eqn (12), is an increasing function of the ratio j(g). The proof is completed by invoking Lemma 5 which establishes that ічкнς)) (resp. (kF())) is the unique minimizer (resp. maximizer) of the ratio j(g)
over q 2p-
Part (B), On the other hand, taking 1 gives Pg Pr. S w = 8/1- ozz We get > 0. and
a/vb - +
It is easy to show that the RHS is strictly decreasing function of j. As with part (A), the proof is completely by invoking Lemma 5 to extremize the ratio j j(q).
Lemma 5. Suppose Pg > O. For any fixed pruning strategy p (0, 1). ignoring null-sets, the unique maxinizer (resp. minimizer) of the ratio j(q) over 2, := {g E Q p(g) = p) is the "keep hard examples" pruning strategy (KH(p) (resp. the "keep easy examples" pruming strategy (kE(w))
29

---

Published as a conference paper at ICLR 2026
Proof. Clearly, there is bijective correspondence between 2o and the collection Sp of Borell subsets S S R of Gaussian measure equal to p. and verifying the symmetry condition - = S. This correspondence is simply S + 1s. the indicator function of S. Furthermore, for any S E Sp- one can write
r(ss) = 2Fo(S+), B(1s) - 2F](S)), B(1s) - 2f2(S)), with S+ = Sn (0) Fa(T)= = J [n(()p(()d
fo(t)= : e fi(t) = (20(T)) - 1)t, f2(1) = p(Tt), T: T: V Pz.
e |t 2 apt and Jp := (t E R |t| S bp) both have Gaussian measure We shall show that over the collection Tp of Borell subsets of (0, oo)
Define ap. by > 0 such that the sets Ip
:= (t -=
with Gaussian measure equal to m = p/2, the functional + F2()) is minimized (resp. maximized) by J := [dp, 00) (resp. p = [0, hp), while modulo null sets, and F1 is uniquely maximized (resp. minimized) by JØt (resp. I
Step 1: Reduction to Integration w.r.t Lebesgue Measure. For any / >> 0 and E (0, 1/2). define
((t (100, f)),
N(u) (w).
Under the change of variable t = [u), one has
(T) = (M()), where P(U)= = [ gu((u)d,, 9h == SEON, and ((T = (M(() | (ET).
Thus, the minimizers (resp. maximizers) of F over T E Tp are of the form NU) where U min- imizes (resp. maximizes) F(U) == F(U))) ing [U] Let us show that modulo null sets, F is minimized by (0, m] and maximized by (1/2 - m, 1/2) where m == p/2 E (0, 1/2).
over Borell sets U S (0, verify-
For any 0, consider the equivalent Ainar-fractions program
min (U) 20, UU) subject |U] m, Fo(U) Sr.
(58)
Step 2: Dinkelback re-Parametrization. For fixed r 0, consider the change of variable À F(())/2(U)) and define
u(A) = max Fi(U) ((2))) subject to U = m. Fo(U) Sr. (C((0,122)
(59)
The "Dinkelbach trick" tells us that X = max (U)) (2( iff u(>*) = O.
Now, the Lagrangian for the auxiliary problem is given by
C(U,A, (U) (U) + n (r Fo(U)) + (m 1U))
= [ Hu, (, d)d + m + {m, with H (u, X, n. () = gu(w) - Agz(u) - -ngo(u) - -C.
The first-order optimality conditions of U can then be expressed as
2 O, u E U, Ul,,A, () 0, otherwise.
(60)
Step 3: Shape Analysis. Now, under the assumption that Pg > 0, the functions fo and fi (therefore Go and 92) are increasing and 91 (therefore g1) is decreasing. Thus, for any X, 2 O. the function - Hu,, X, n, C) is non-increasing function, for any feasible X, S. A non-increasing function crosses zero at most once. We deduce that the optimal U must be of the form [b, 1/2), modulo a null
set. The condition [U|
forces b 1/2 m. We conclude that [1/2 m, 1/2) is the unique
minimizer of F.
Similarly, one shows that [0, m) is the unique maximizer of F.
30

---

Published as a conference paper at ICLR 2026
H PROOF OF THEOREM 4 (REGRESSION ANALYSIS)
H.1 A MODIFIED BIAS- VAR-AAA DECOMPOSITION
We start with the following general bias-variance decomposition for the regression test error.
Proposition 3. The regression test error of the estimator i defined in Eqn Eqn (3) is given exactly
by
_ tr SR -XE 10 REC)
Ereg (1) = X [wg RE g) OFE
(61)
where E = Wg W.. c² == T De, and S and R. are the random matrices defined in Eqn Eqn (3).
The first two terms in the above sum correspond to bias and variance if we had wy = i.e if we had no label-shift; the last two terms in red are a correction to take into account label shift.
H.2 PROOF OF THEOREM 4
Now, from Proposition with Σ la. we have the following deterministic equivalents:
R= m())[(= + m1))lI,
SR - la = =R 2m(2) =m(2)l-++mm(3)) =m))
R²= 0
R ml +
m'
0z
0
SR SR (m(2) 2m (=)) + (m(2) + 2m (I))TI
0z
(m(=) zm' (2)) (m)=)- m(=) + 2m (z) 2m' (2))
Furthermore, notice that because 11 is fixed-rank (in fact rank-1) matrix, so is SIE and so (1/n) - in the limit > oc. Thus, in view of using Proposition 3, one computes
E [ RERW w ! | m. m (2) ( 112 + m (=)
E SRS Ø. Bt SR a. (m<() + zm (=)) cmm((2),
E R = [w Re] e (m(() + m(()w())
Putting things together then gives
(w)) ~ x2 mm' (-)))l (m +' 1))12(6113) + 0² (-1)
+ ||]|2 - 27e T (m(->)w x) + m(())w3)
= x2. w-) 1| (o(--))] 2) (11 + 0² (mmmmm))
+ 22 (m( A)a + (n--))6) with a = Tw b = c. wg and c² := -=||,,,,
which proves Theorem 4.
H.3 PROOF OF COROLLARY
The first equation follows by taking the limit 0+ in part (A) of Corollary 1. For the second equation, note that in the limit Eqn (4) Corollary gives Ereg ~ 2² + Lo. with
0.
i P: if >
Lo = In(d.p) = coD+
where D: || 1|2 2 2a, E = | 12 26, and we recall that
Co 0= p/o, a = 7/ + co (p 7)/ == + 2app(a:) a= == -p/2).
31

---

Published as a conference paper at ICLR 2026
Now, the second branch, one computes
8- 0p =²
OLo D - - -
D
E 7 + (1b p))
Öp
Φ (Ø (p-
Φ (Ø - (p -))
One can further show the Hessian of Lo is nonnegative everywhere provided E > 0, and so every stationary point is a global minimum, provided it lies in the interval (0, Ø). Expanding to first order in p. observe that if == -D/E > 0, then we have unique stationary point po po( By the way, observe that D+c² = [.. wwy I| and E+c²² = I|,, -ug and so the condition D < 0 < F is equivalent to the condition I/w., w/yl < c² < I|,, w |?
w II?, where c² = l0. w 1l as usual,
in the statement of the result being proved. Further, one can show that for small ø.
po/s = vi/v 2 1/0, (T(p0) po)/ =
(62)
See Lemma 6. It is clear that Po < & because log 1/ > 1 for small Φ, and so Po is on the second branch of the definition of Lo p). and must therefore be a global min of Lo the interval o).
Moreover, one has (still in the limit o + 0*)
log 1/0 + xc, c(po) = 1-po/0 + 1, ei (po) = (+(((0)) + 0o. (p))))/((p0) - 0,
and so limæ-su+ L(O,po())) = =² + lim Lold, (o()) = D + 2== -ll,-- -w'll < c². O
Lemma 6. Let and po be as in the proof of Corollary 2. For o + +0, it holds that
po = vtt. 1/0, (r(p0) p0)/ = viv 2og 2og11/1// 1/6.
(63)
Proof. The idea to argue that p must be small, and so we must have a large and yy >> 0. One then considers the simplified equation D (o + (p))² + EΦ²a(p)² = O, which can be solved as function po) of & using Lambert-W W function. Finally, since ó is small ps, we can further drop the Lambert-V W function and ultimately get po = vt/ 2 1/6.
H.4 PROOF OF PROPOSITION 3
As usual, set z := -À so that = (S- =Ia)-1. Observe that the estimator given in Eqn Eqn (3) can be written as w = RSw, T.X-D// where A == Y -y-Xe Xxy, E R is the vector of epistemic label noise, which is independent of the design matrix X. and has distribution (i0,, I,). We may then decompose the regression test error of ú as follows:
Ereg(w) = [(xx の y) o² = E [(x T w x T w. ".. = E [11e0 w 1
= E |||SS + RXX D& n w. 113]
E [||SSu w. 1| + E [|] |X T DA |
= E [||Sw Wg + w w. 1 + 2 7
tr DX D
= E [| - 1 + 2
tr
SR + LA + 2E [ (SR - La)
2 [ RERw,)
SRRR + Ee 2BB [ng REel,
n
where we have used the elementary identity SR
Pa zR.
I PROOF OF THEOREM 5 (OPTIMAL PRUNING IN REGRESSION SETTING)
Note that the pruning strategy q only enters the picture via the parameter p(4) = E (a()) and 7()) == E (G)
Definition 2. Let Q be the set of all admissible pruning strategies satisfying Assumption I, and for any subset of H of Q define Spec(H) C [0, as follows:
Spec(H) == (p(g)) 74)) 1 E...
(64)
Thus, Spec(H) collects all possible values of p and y attainable by some pruning strategy q E H.
32

---

Published as a conference paper at ICLR 2026
Let Q. : (qpu (p.u) 1]2] C Q. where qp.u is defined in Equ (21). The next result gives us a tractable description of Spec Q) In particular, proves Theorem 5.
Proposition 4. We have the following analytic descriptions for Sperrppee05 Q):
Spec Q) Spec( 2.),
(65)
Spec( Q) = {p,,)) 7) S S 1, Ymmin (p) S S Ymax (p)}
(66)
where Timi (p) = 20min (p)pp(pmmmm)))) (p)): with Omin (p) = + 10000-1110 p)
(67)
Tmoo (p) ;- + 20maz (p) (p))p((amma)) (p)), with amaa (p) :- 1 1b-1( p/2).
(68)
Geometrically, Spee Q) is thus the lens-like region between graphs of the functions "Ymin and Ymaax
Proof. Recall the functions ammn (p) == Φ 10:0-~((( (00- p)/2), amaz (p) == 2 4 p/ Tmiin (p)
p 20min (p)pp (p)) and Tmaz (p) : p 20mar (p)pp( max (p) introduced the lemma.
First note that any E i is the indicator function of a disjoint union of intervals A = Uzez! such that I I iff -I E I. where -I {-t I & 17. Now, for any p E [0, the minimum (resp. maximum) feasible value for 7(9) over the surface { E Q p(g) = p} is Ymin (resp. Tmaz (p)) and it attained by taking the "keep easy" pruning strategy (t) := Sa (resp. "keep hard" pruning strategy ((t) == 41122mmo (n)). See Lemma Therefore, we must have
Spec [2 == {p(()) 7()) 1 E 2 S ((.,)) p [0, 1], (ET()))
where we recall that (p) = Mmmin (p), Ymax (p)].
We now show the other direction of the inclusion above. Given 7 E r(p), must construct E Q such that p(q) = p and 7(g) y. Indeed, for any u E [0. 1]. define E Q as the indicator function of the union of the intervals I, {t R It S a(u)} and Ju (t
[t] > b(u)),
where a(u)= Omiin ((1 - w)p) and b(u) Obmin (pu). It is easy to verify that b(u) 2 a(u). Indeed,
because 11 is non-decreasing, we know from the definition of Omar and O'mi functions that
a'aa (pr) 2 ammn (1-u)p) = 1-pu/2 2 (1+(1-u)p)) + (L+p)/2 2 = P
follows that J, and Jn are disjoint and so
(t()) = 11.J,, = 1r, + 1sa,
It is easy to verify that p(qu) pu + u1 and 7(x)) = p (2(()))(((w)) + (b()):(((())
Observe that 7(q)) increases continuously from Tiii (p) at 0 to Ymaa (p) for 1. It
follows from the Intermediate Value Theorem that there exists up E [0, 1] such that (4 = 7.
suffices to take q Qug
Finally, Spec Q) = Spec Qx) follows directly from the construction of qu.
Lemma 7. For any p E [0, 1]. we have the following.
(A) The minimum of 7(4) over all E Q is given by
Tman (p) = p Osyai (p))p((mmmn (p)), with amin (p) = ④ 15'0(11+))))) ([1+p)/2),
(69)
and is attained by setting qtt) (p|sammnn())
(B) The maximum of (44) over all q Q is given by Tmmax (p) = + ammx (p)yp(()n))))))))) (a(xaxaaxxzxx (p)), with (man( = 1-1(1 -p/2).
(70)
and is altained by setting q(() = 1 11/>=mmaz(()
J PROOFS OF LEMMAS
J.1 PROOF OF LEMMA 2
The formula for m (2) from differentiating through Eqn (34) w.n. 2, and then doing some basic al- gebraic manipulations. All the other formulae for m²)2), m(22, and (2(2) follow from the definition of the quantities and the chain rule.
33

---

Published as a conference paper at ICLR 2026
K PROOF OF LEMMA 7
(A) Every q E 2 the indicator function of some measurable S R. We wish to maximize
7(g) SA P(((t)tt over A, subject to p(q) SA (t)) p. The Lagrangian is
((A, J, P(p())tt + (p (p-Jx p(t))at) - (t2 ()1x(t)p(tddt Pd.
Since (pt) 0 for all t, it is clear that the integrand is minimized by taking
Lx(t) (..
if 0, otherwise.
Thus, by the Rearrangement inequality (for measures), it is optimal to take A (-00, VA) U (VX, oo) for some À 2 The constraint SA (t))dt = p then gives
Ví = Φ ( 2) O'minn (p).
(B) Analogous arguments.
L PROOF OF LEMMA 3
L.1 NON-LIMO CASE
Let us prove the formula for 31 and (32 given in the first row of Table 3. Consider F =
where U = mi and V := TT W. for Z ~ NN, Ia). Note that we can write =- E|F By Stein's s lemma, we have C-1/2 = anlg + bbo where
aF
oF
a: aU
b b:= E
(71)
OV
By direct computation, one has
0F
OU 2V(U)g(V),
(72)
aF
OV (V))
(73)
in the distribution-theerrticc sense. Thus, one computes
E[S(U))(V)) ((())((((V) U = 0) (V))E(((V) U = 0 = (())
=0(0) / ((ot))(()dt- p0(0) L (())(t/o)d)
1 [E(((C))((GG)
where we have used the fact that
(((()){(() 1 (0( 72
=000 (100)l////1//tt =00))((0)))
We deduce that a = (2/o))
On the other hand, for any RR one computes
Elsign(U))(( - s)] = ((sinn))) V =]
(PP(( 201 = - P(U <0 V =)).
But, conditioned on = a the distribution of U NPgs,, where = / - pg. We deduce
that IP(U 2 0 = s) = P(N(0, 2 Ppy) = PO, 0²) S Pys) = @(Ts). Like-
wise, P(U < 0 V = 5) P(A 02) < Pys) ((-rs) 1 ((TS). We deduce that
E(sign(U)8 - -s] = -(())(2((Ts)
1), and so
([sign(U)) V = s] = / d (2) ()p(s)dz = E2/(C)/2((GG- - 1))1
2 Elg((G) = 2[)(()))((T)),
34

---

Published as a conference paper at ICLR 2026
where we have used the evenness of to write (lg((G)) B[G 0. We deduce that
20 Elg(G))((GG), b (
(74)
Lets write C-1/2 و + bwg = Bu + fu, where u = wh and e is an unit-vector perpendicular to but in the plane spanned by Wi and Wig. It is easy to see that
" u
B pgu
Bg Pau
Dig P
2p + P3
We deduce that
B T v f= (w, v) oa - [[(((C) =: B2,
=
(75)
β 3= c. u b+ Pg 2)(o(())((rG) + 2+B(n(())(TGG))
(76)
To match the formulae for 81 and 82 given in Table 3, we must now show that (l
(TG)))
( =4 and conclude that 31_ To this end, write Elq (G))(TG))
Bld where ((t) == ((rG). By Stein's lemma (Gaussian integration by parts), we have
(G(( Elu(G))(+GGG) - +B(g(G)4((G)),
-((() EggG)) -TG(TG))
as claimed.
Computing and We now compute the pruning ratio by definition as == Blp..] B[((V)) = (G) and y [x[ w.) Pp] [[q(V) G2 for G (O0 1). This matches the
formulae given in the first row of Table 3.
L
L.2 LIMO CASE
Let us now prove the formula for 31 and B2 given in the second row of Table Here F = ( where H is the Heaviside step function with the convention H(O) 1/2.
Now, one computes
OF (i
(77)
OU
0F
OV
sign(U) (V) Ula())
(78)
Computing the a coefficient. One computes
((((U)((V)((UV)) = (VV)) U -= = .2(0)
[lg((V) IU
1 Blu(C))(TG))
2o
On the other hand, using the well-known identity
6(zy) = 5(y)/<=) 8(z)/bl)
one computes
Ebsjg((U)((VVVVV(V)) +1
E((1/U)((V) vo(V) + [i)))an(())((())sin(((())))
((0)) (V) (V) U 0] 0,
where the last step is because + sign(t)g(t) is an odd function, and the distribution of V condi- tioned on U is (N(0 which is symmetric around the origin. We deduce that
- [[((G)))(7TT)
(79)
35

---

Published as a conference paper at ICLR 2026
Computing the coefficient. For any E R.
(V) | =]
V s] (())p(s)) (P(U 2 0, sU | V = s) - <0, sU 2 2 = 8))
V =
Now, since the distribution of conditioned on V is (V )²), we have
if s 2 0, if < 0, PPU 2 0 V =s) = =-(Ts),
P(U O, sU V =
IP(U = 0 V = ==)) =8)= =0,
P(U < 02 = =0,
ifs 2 0, if O.
P(U < 0, sU 201 V 3)
P(U <0|V -)(-T8),
Therefore, Elsign - ( (B)ign(U)(((V)///)))
and we conclude that Ely (C) [ (ig]
with G A/(0, 1). Define h(t) ((((()) stg. It is clear that
(t() (r(t) + r((rtt) 26(x) ( Tp((t) 8(v) + TT((T).
Gaussian integration by parts then gives
(ldd) = -TKC)// (0)))(0))
But of odd (because is even), and also + Φ(r(t)) obviously even. We deduce that (UV((V = 0. Likewise, using the identity 8(UV) = + one computes
B( = ())))(00) + ( (0)))
We deduce that (1) Therefore, writing C-1/2 = Bu + Bu as before, have
o- = = 32, 3= b Elgg) =: B1,
which are precisely the formulae given in Table 3.
Computing p and ?. We now compute the pruning ratio p == Epp (UUV) and := E((x; ppl (VV) by definition of p: in Equ (6). Now, for any E R, we have
P(U <0 V s) #(-rs),
if 0,
BIH(UV) IV [ =
1/2.
if = 0,
P(U 201 = s) = =(Ts),
if > 0
[[Tss))
Integrating a with density (ps), we deduce that
p = E[((G))(r()G)) = ==E[(()))((GG))""
as claimed.
M ANALYTIC FORMULAE FOR p(g).
AND 3(q)
Note that every symmetric pruning function E Q the support function of sum T' := -SU S, where S is (up to null set) countable union of closed intervals. We consider subclass of symmetric pruning functions corresponding to finite unions, i.e
q 1r, with T S S Uنー (aj. by). 0 a1 < b 02 ... an by 80. (80)
36

---

Published as a conference paper at ICLR 2026
The "keep easy examples" (KE) and "keep hard examples" (KH) pruning functions used in (Sorscher et al., 2022) and defined defined below belong to this class
(for some a >> 0):
(KF( () = 1|4 2 a), i.e gKr(() = if [4] 2 a and gKv(()) = 0 otherwise,
(81)
(KH (1) == 1[M|] S a], i.e (()(() if [t] S a and (KH 0 otherwise,
(82)
where a > which controls the proportion p = [ppp of training data which survives the curation.
Since they correspond to taking S = [a, oo) and S = [0, a] respectively. The representation 80 also generalizes the setup of Feng et al. (2025) and Firdoussi et al. (2024) corresponds to g 1, i.c
S = [0, oo).
For any a E [0, oc]. define (a) = Jo (x()))(x))d, where the functions f& are defined by fz(z) = ((xx), f2(z) == (f(x)) fa(x) == (T())), fa(x) == (Tx)
As usual, op and + are the standard normal pdf and edf respectively.
Proposition 5. Consider symmetric pruning function q of the form Eqn (80).
(A) For label-agnostic curation Eqn (5), it holds that
p(g)= ((,) g(ag); with g(z): o(z):=24B(z) 2T((2)
(83)
j=1
7(4) = Σ [bb)) g(aa), with g(z) 2(0(2)) -2)(=)) 1,
(84)
Ø=1
B1(()
(85)
k Σ
B2(g) = 2:p(0)o
- -
(86)
1=1
(B) For Label-aware curation Eqn (6), it holds that
k p(g) 2 > -(bb)-
- I(aa):
(87)
j=1
7(1) Σ (bb Ia(a)):
(88)
j=1
(1 2 Σ Is(b,) Is(ag).
(89)
I=1
(e(g) = 2 Σ I2(b,) I2(a,).
(90)
j=1
Part (A) of the proof follows directly from Eqn (8). Part (B) of the proof is consequence of the
identity S h(z)dr= = I(b) 105-I(e I(a), where I(a) == foe h(z)dz, combined with the following lemma.
Lemma 8. For any E [0, oc). the following identities hold: [1(a) ((a) - 0; 1/2-11122 tb (0,00))))
[#2(a,
(91)
12(a) o0(0))(a/o) -
(92)
Is(a) (12(2) - (p(a) - ((0)/22, 1/00
(93)
Ta(a) Tn -oo((a))((T) p [x(( ((a))((Ta)]
(94)
The results are extended to a = co by noting that
lim T2((2) = (0(1) 2
lim aup(a)= = lim p(a)= = 0, a-
00
37


Reviews:
[{"review_id": "7rKg8moJNA", "reviewer": "Reviewer_UCRG", "summary": "This paper introduces a rigorous theoretical framework to analyze the efficacy of data curation strategies in high-dimensional learning. It addresses a central paradox in modern ML: When does aggressive data pruning (\"Less is More,\" exemplified by methods like LIMO and s1) outperform training on the full dataset (\"More is More\")?\n\nThe paper model this scenario using high-dimensional binary classification and regression with Gaussian covariates. The setup captures the interactions between a data generator ($w_g$), the ground truth ($w_*$), and a pruning oracle ($w_o$), allowing for label shift. They analyze both label-agnostic (based on difficulty/margin) and label-aware (based on difficulty and correctness) curation.\n\nUsing RMT, the paper derive exact asymptotic expressions for the test error in the proportionate scaling limit ($d/n \\to \\phi$). The central theoretical finding (Theorem 2) identifies a precise phase transition: \"Less is More\" (specifically, keeping hard examples) is optimal iff data is abundant AND the generator is strong. In all other regimes (scarce data or weak generator), using more data or keeping easy examples is superior.\n\nWhile the paper is primarily theoretical, it tries rounding itself off with some simulations and Imagenet experiments.", "questions": "- There is an inherent balance/tradeoff between optimal strategy vs model collapse mitigation. Could the authors further clarify the contradictory observation between Theorem 2(b) and Fig.3 ? To be more specific, does the optimal strategy shift from Keeping easy back to Keeping Hard when moving from label-agnostic curation to label-aware curation? Maybe an equivalent Theorem 2 for label-aware setting could help.\n\n- An important practical consideration is to identify which of the four regimes in Fig. 1 is one operating in. Practically where ground truth is scarse and strength of model is difficult to estimate, how would the paper suggest reliably diagnose this? I guess there needs to be a detector for this crossover point when KH > KE?\n\n- The RMT analysis relies on linear models, while experiments elude to ImageNet teasing the qualitative transfer, but the theoretical boundaries remain tied. Can the paper describe the mechanisms they believe enable this transfer?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Theoretical Rigor and Exactness:** The primary strength is the sophisticated mathematical analysis. By leveraging RMT, the paper derive *exact* asymptotic formulas for the generalization error (Theorems 1, 3). This allows for a precise characterization of the interplay between data quality ($\\rho$), oracle quality ($\\rho_*$), and data scale ($\\phi$), moving far beyond bounds or heuristics. The derivation of how pruning \"deforms\" the Marchenko-Pastur law governing the data spectrum is technically impressive and commendable. \n\n- **Good Resolution of the \"Less vs. More\" question:** The paper provides a clear and elegant resolution to conflicting empirical observations regarding data scaling. Fig. 1 beautifully illustrates the four key regimes of learning. The insight that the optimal strategy is fundamentally tied to the generator's strength relative to the task difficulty is a major conceptual contribution.\n\n- The framework successfully unifies several recent lines of work, including data pruning strategies and model collapse mitigation. The analysis of label-aware curation (Section 3.2) is highly relevant to modern SFT/RL*F pipelines where reward models filter policy outputs. The demonstration that strategic pruning prevents model collapse (Fig. 3) is practically significant for the stability of iterative training loops.\n\n- The authors provide some practical validation. Synthetic experiments show strong agreement with the theory (App. B, Fig. 4). Crucially, the ImageNet experiments confirm that the qualitative insights transfer to deep learning settings (ViT), and the analysis in Sec. 4.2 provides a compelling explanation for recent contradictory LLM results.", "weaknesses": "**Linear Models:** The theoretical results are derived under the assumption of linear models and Gaussian covariates. This is a known standard simplification for RMT analysis but limits the direct quantitative applicability to deep neural networks on structured data. However, the empirical results do point towards potential qualitative transfer, but would warrant a more complete exploration. \n\n**Fixed Oracle:** The framework assumes a fixed pruning oracle ($w_o$). In practice, oracles (e.g., reward models) are often learned and may evolve adaptively during training (e.g., iterative RLHF), which is not captured in the current analysis. Extending the theory to adaptive oracles would be an important future direction.\n\n\nI am **highly skeptical** that this constitutes the explanation for \"Less is More\" in modern deep learning. The assumptions are too strong, the dynamics of optimization (like SGD) and feature learning are ignored, and the empirical validation is not sufficiently robust to confirm that the mechanisms identified by the theory are the ones driving results in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84A8c8Dr05", "reviewer": "Reviewer_CgiU", "summary": "This work seeks to develop a theoretical framework for understanding data curation. In other words, they seek a theory that can predict when it is best to use less data (filtered/curated) versus all data (no filtering/curation). The authors are motivated by the fact that there are cases where small datasets have been shown to be beneficial, like LIMO and s1. Their theoretical results assume a Gaussian feature model under a binary classification setting.", "questions": "1. Why is the Gaussian feature model noted as a limitation? Wouldn't we expect gaussian distributions of features according to central limit theorem? The authors mention real-world data is \"structured\" but I find that vague.\n2. How does someone measure the quality of the generator $\\rho$ for a dataset you are given?\n3. Is it possible to know the quality of pruning? For example, suppose I curate a dataset of text by removing duplicates, how would a \"quality\" be assigned to that?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors use their theory to show that it can predict empirical results to a surprising degree (mean relative error of 1.8% according to App. B), and within these empirical results live the settings they sought to understand: when all data is needed (\"more is more\") and when curation is needed (\"less is more\")\n2. Authors consider both label-agnostic and label-aware data curation in their theory\n3. Significant extra detail for figures and proofs are provided in appendix, answering a few of my early questions", "weaknesses": "1. Arguably the most important feature of the paper is that their theory is predictive of empirical results in a very realistic dataset (ImageNet training of a ViT), but it is not clear from the writing how to use this in practice. In particular: how does someone measure the quality of the generator $\\rho$ for a dataset you are given? No commentary on this is given. A bonus would be if the authors gave a step-by-step appendix section for a practitioner on how to use their theory (willing to raise score for clarity on this)\n2. \"the base LLM is a strong generator\" based on what? how is this claim measured? \n3.  Why is $n=5000$ considered the \"large data\" setting in Figure 1? How was this number chosen? That $n$ is smaller than the CIFAR-10 training set...\n\nMinor: \n- capitalize PDF and CDF on L184", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "204fZl2qeG", "reviewer": "Reviewer_tAw9", "summary": "The paper develops a teacher–student, high-dimensional model of learning from curated data in order to reconcile two empirically observed regimes in modern LLM/vision training: \"More is more\" regime (classical scaling laws) in Kaplan et al. (2020) and Hoffmann et al. (2022), where performance improves monotonically with dataset size vs. \"Less is more\" regime (LIMO/s1–style curation), when the generator is strongly aligned with the target, then selectively keeping only the most informative/hard examples yields strictly lower test error than training on the full set. Formally, the authors setup a theoretical model with Gaussian inputs and labels generated by a linear teacher with known alignment, and the learner as a ridge (square-loss) estimator trained on a pruned sample whose selection is governed by an oracle with its own alignment parameter. Using random-matrix/deterministic-equivalent techniques, the authors derive closed-form high-dimensional limits for the test error after curation. The authors show numerical simulations of their theory and corroborated with empirical simulations.", "questions": "* My main questions are in the weaknesses section. I would especially like to see more discussions between \"KE\" and \"more is more\". \n* The paper uses deterministic equivalence to obtain risk asymptotics for the problem (2). While in general, when l is a convex loss function, using the CGMT techniques the similar asymptotics can also be obtained. See Karoui (2013). What do authors think the behavior in this setup? Would the same behavior also be expected?\n\nKaroui, Noureddine El. \"Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: rigorous results.\" arXiv preprint arXiv:1311.2445 (2013).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written and well-organized. The paper has both solid theoretical side and empirical evidence.\n* The paper has deep theoretical building blocks with high dimensional asymptotics of linear models and classification problems. Using both RMT techniques and tools from Feng et al. for classification problems to study the effects of pruning by the oracle. The conclusions are sound as far as I have checked, and the results appear novel to me.\n* The paper's conclusion provides an alternative to neural scaling laws: it is possible to perform with higher accuracy with fewer data with an expert pruner. This also gives a potential remedy to the neural collapse phenomenon where the training procedure seeks to scale with self-generated data---perhaps by scrutiny using the synthetic data (instead of training on them) and picking only harder/easier problems we can avoid model collapse from synthetic data.", "weaknesses": "Major comments:\n* I think the main inconsistency between the main message and theory is that: both \"KE\" and \"KH\" strategies are effectively \"less is more\". It is just what data should be picked. I think it is a bit of stretch to say \"KE\" is equivalent to \"more is more\", while the neural scaling laws scenario should correspond to a very small $\\phi$ instead of the \"KE\" strategy. \n* The theory relies on linear model with Gaussian covariates, and is limited to the binary classification setup with proportional asymptotics. This is, definitely limited, as is acknowledged also by the authors in the limitation discussions, a good starting point for study the pruning and data curation in large datasets. I think this is only a minor weakness.\n\nMinor comments:\n* A general style suggestion: use \\eqref instead of \\ref for equations.\n* Page 19, \"Corollary xyz\". I think this is a placeholder or typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xNVfNUJt12", "reviewer": "Reviewer_dewW", "summary": "This paper introduces a principled theoretical framework for data curation, trying to figure out when less data actually works better than more, to challenge the classical \"more is more\" scaling laws with comprehensive experiments. It derives exact scaling laws for test error under label-agnostic and label-aware curation rules, showing that strategically pruned small datasets can outperform full datasets under conditions like strong generators and abundant data, while also mitigating model collapse.", "questions": "1. The theory is framed around binary classification, but multi-class tasks are definitely more common in practice. Could you discuss whether the optimal pruning strategies could extend to multi-class settings?\n2. In the ImageNet experiment, the generator and pruner are the same pre-trained ViT, which ensures alignment, but real-world CV pipelines may use separate generators (e.g., a synthetic data model) and pruners (e.g., a human-in-the-loop verifier). Could you discuss how misalignment between generator and pruner (e.g., low ρ₊ relative to ρ) would impact the optimal curation strategy, and whether this differs from the LLM case (where generator and pruner are often separate)?\n3. I have another question for the authors, but not necessarily to be answered: LLM training encompasses pre-training, post-training, and stages such as SFT and RL within the fine-tuning process, with distinct training data used for each stage. For the LIMO method discussed in the paper, in which of these training stages is its principle more likely to be reflected?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.  The first paper tries to theoretically explain when and why LIMO happens in machine learning by developing a principled theoretical framework for data curation, enabling practitioners to design data-centric pipelines instead of blindly scaling datasets.\n2. Also provides a unifying explanation for contradictory curation strategies.\n3. The writing and presentation look perfect to me.", "weaknesses": "1. While the paper links theory to LLM math reasoning (AIME benchmark in Table 1&2), it would be better to discuss other LLM-critical tasks (e.g., code generation, natural language understanding, or multi-step reasoning benchmarks) where curation is also widely used. Additionally, the authors do not investigate how the pruning ratio or oracle quality affects performance across different model sizes (e.g., 7B vs. 70B LLMs), leaving uncertainty about whether the LIMO principle holds consistently across LLM scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)