How Text Quality Interventions Reshape Neural Scaling Laws for LLMs: Empirical Study
Authors: Newsha Ardalani
Feiyang Kang
Michael Kuchnik
Mostafa Elhoushi
Shubhabrata Sengupta
Shang-Wen Li
Carole-Jean Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Scaling law
Text quality
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present the first large-scale empirical study showing how text quality interventions reshape neural scaling laws and compute-optimal strategies for training LLMs, highlighting the need to rank data strategies using scaling law curves.

Abstract: Neural scaling laws are widely used for performance projection and resource planning, yet their sensitivity to data quality interventions remains poorly understood. We present the first large-scale empirical study of how interventions—deduplication, heuristic filtering, and LLM-guided rewriting—reshape scaling behavior in large language model training. Using QualityPajama, a suite of 23 systematically curated datasets, we train over 2,000 models (100M–8B parameters, 100M–200B tokens) to measure how text quality interventions affects
scaling-law parameters and compute-optimal design decisions. While prior studies have shown that model architecture primarily shifts coefficients, we demonstrate that data interventions shift both coefficients and exponents, fundamentally changing the fitted scaling laws in ways not anticipated by existing theory. We show that data quality ranking is scale and resource-dependent. Compute-optimal
token–to-parameter ratios vary by orders of magnitude across interventions, revealing a fundamental data quality–quantity trade-off in scaling. These findings pave the way for deeper theoretical understanding of scaling laws, establish scaling-law analysis as a principled framework for data strategy evaluation and ranking, and motivate a data-quality–aware approach to scaling next-generation LLMs.

Published as a conference paper at ICLR 2026
HOW TEXT QUALITY INTERVENTIONS RESHAPE NEU- RAL SCALING LAWS FOR LLMs: EMPIRICAL STUDY
Newsha Ardalani Feiyang Kang!2, Michael Kuchnik', Mostafa Elhoushi";
Shubho Sengupta*, Shang-Wen Li", Carole-Jean Wu!
FAIR at Meta, Virginia Tech, CCreebas Systems, AAxiom Math
ABSTRACT
Neural scaling laws are widely used for performance projection and resource planning, yet their sensitivity to data quality interventions remains poorly un- derstood. We present the first large-scale empirical study of how interven- tions- deduplication, heuristic filtering, and LLM-guided rewriting reshhp scaling behavior in large language model training. Using QualityPajama, a suite of 23 systematically curated datasets, train over 2,000 models (100M-8B pa- rameters, 100M-200B tokens) to measure how text quality interventions affects scaling-law parameters and compute-optimal design decisions. While prior stud- ies have shown that model architecture primarily shifts coefficients, we demon- strate that data interventions shift both coefficients and exponents, fundamentally changing the fitted scaling laws in ways not anticipated by existing theory. We show that data quality ranking scale and resource-dependent. Compute-optimal token-to-parameter ratios vary by orders of magnitude across interventions, re- vealing a fundamental data cuelity--uantity trade-off in scaling. These find- ings pave the way for deeper theoretical understanding of scaling laws, establish scaling-law analysis as a principled framework for data strategy evaluation and ranking, and motivate data-quality-awar approach scaling LLMss.
INTRODUCTION
While nearly all large language models trained similar sources of text reb data- the key differentiating factor among state-of-the-an models lies in the quality of their pre-training and post- training data. However, data quality itself remains elusive and context-dependent concept- what constitutes "high quality" can vary with downstream use case, compute scale, and resource constraints. This raises the question: can neural scaling laws offer a principled framework for ranking data quality across scales?
Neural scaling laws are empirical relationships that describe how model performance improves as function of resource investment typically the number of parameters and training tokens. A growing body of empirical (Hestness et al., 2017; Johnson & Nguyen, 2017: Rosenfeld et al., 2019; Kaplan et al., 2020; Hernandez et al., 2021; Ghorbani et al., 2021; Ardalani et al., 2022; Hoffmann et al., 2022; Alabdulmohsin et al., 2022: Aghajanyan et al., 2023; Isik et al., 2024; Zhang et al., 2024) and theoretical (Sharma & Kaplan, 2022; Bahri et al., 2024; Brill, 2024; Hutter, 2021; Michaud
ct al., 2023; Dohmatob et al., 2024b; Debowski, 2023; Dohmatob et al., 2024a) work has shown that pre-training loss follows pawer-law trend with respect to these axes. Neural scaling laws have been central to the development of large language models (LLMs), informing decisions about model scaling, data scaling, and compute allocation, while also serving a key tool for return- on-investment (ROI) analysis and capability forecasting (Hestness et al., 2019; Hoiem et al., 2021; Mahmood et al., 2022; Alabdulmohsin al., 2022). However, despite their widespread adoption. the impact of data quality on scaling laws remains poorly understood.
A prominent example of this uncertainty is the ongoing debate over the discrepancy between Kaplan et al. (2020)'s s and Hoffmann et al. (2022)'s prediction of the compute-on optimal ooken-to-paramedt
Work done 1 1 Correspondence Newsha Ardalani <new @ meta.com>.

---

Published as a conference paper at ICLR 2026
ratio (21 vs. 1) (Porian et al., 2024; Pearce & Song, 2024; Bi et al., 2024). Recent work specu- lates that differences in training data may have played a role in this divergence (Bi et al., 2024). While prior theoretical works (Sharma & Kaplan, 2022; Bahri al., 2024: Brill, 2024; Hutter, 2021; Michaud et al., 2023; Dohmatob et al., 2024h; Debowski, 2023; Dohmatob et al., 2024a) have linked the power-law exponents properties of the data manifold and the Zipfian distribution of input tokens, the impact of text quality interventions on these underlying structures remains poorly understood. Furthermore, prior work overlooks how data quality influences other components of the scaling --mmlamlyy, the coefficients and asymptotic loss terms-which as we will show, play a critical role in shaping loss behavior at today's 's compute scales, Moreover, most theoretical predic- tions isolate single exponent (either model or data) while holding the other in the infinite limit. As we demonstrate, understanding the joint fit essential, as the components often move opposing directions to control loss trajectory, revealing important trade-offs induced by data quality shifts. Although prior empirical work has explored the effects of synthetic noise, data source composition, and filtering algorithms in domains such as machine translation (Bansal et al., 2022) and image clas- sification (Bahri et al., 2024), to the best of our knowledge, there no systematic study examining how text-specific interventions -uch as filtering, deduplication, rephrasing and mixing synthetic and natural data-impact the components of neural scaling laws in LLM pretraining.
Our work bridges this gap by conducting a large-scale empirical analysis of diverse data quality in- terventions for pretraining large scale language models and study how they influence all components of the scaling law. We introduce benchmark of 23 curated datasets, each representing a different quality intervention, and train over 100 language models per dataset, totaling more than 2000 model training runs. This extensive experimental design enables us to disentangle the effects of data quality on scaling law components and loss behavior, and propose how to design an effective data quality strategy as we scale.
1.1 OUR CONTRIBUTIONS
QualityPajama Benchmark: We introduce QualityPujana, benchmark suite of 23 datasets designed to systematically evaluate the impact of diverse text quality interventions neural scaling behavior in LLMs. (Section 3)
Full Scaling Law Decomposition: We provide the first systematic analysis of how text- quality interventions affect all components of the joint scaling law-not only the exponents. Our results show that stronger filtering does not consistently push components toward more favorable regimes, but instead produces conflicting shifts across parameters. (Section 4)
Data-Aware Scaling Strategies: We show that designing compute-optimal scaling strate- gies requires careful accounting for data quality, as variation in quality could shift the optimal number of parameters, tokens, and their ratio by couple orders of magni- (Sec. 1)
Scale- and Pessurcee-Dependee Rankings: Data quality rankings are not uniform across scales or resource regimes. Strategies that excel at small scales may underperform at larger ones, and the optimal choice depends critically on the constraint (c.g., fixed compute vs. fixed data). Moreover, "scale" can refer to model size, dataset size, or compute budget, and the best intervention differs across these regimes. We recommend using scaling-lav curves to rank data quality strategies across different scales and resource constraints, rather than relying on small-scale experiments, which often lead to misleading conclusions. (Sec- tion 4.1)
Deduplication Efficiency: We demonstrate that deduplication yields large compute sav- ings that far exceed reductions in data volume (Section 5)
PageRank Signals: While PageRank scores correlate with improved quality, filtering based solely on PageRank does not outperform the unfiltered baseline. (Section 5)
Syathetic-Natural Data Mixing: We show that mixing synthetic and natural data consis- tently outperforms using either alone, but the optimal mixing ratio evolves as the model and compute scale. (Section 5)
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND AND RELATED WORK
The study of scaling laws in deep learning has a rich history, with numerous empirical (Hestness et al., 2017; Johnson & Nguyen, 2017; Rosenfeld et al., 2019; Kaplan et al., 2020; Hernandez al.,
2021: Ghorbani et al., 2021; Ardalani et al., 2022: Hoffmann et al., 2022; Alabdulmohsin et al.,
2022: Aghajanyan et al., 2023; Isik al., 2024; Zhang al., 2024) and theoretical Sharma & Kaplan (2022); Bahri et al. (2024); Brill (2024); Hutter (2021); Michaud et al. (2023); Dohmatob al. (2024b); Debowski (2023); Dohmatob et al. (2024a) investigations into their components. A commonly used form of the scaling law is given by:
Loss(N, D) ~ AD + BN +
where Loss typically represents cross-entropy loss, D denotes data size in tokens, N represents model size in parameters, and a, 3. A, B, and E are constants. The terms this equation cap- ture the effects of finite data, limited model capacity, and the inherent entropy of the underlying phenomenon, respectively.
Although prior work has empirically explored the impact of model architecture (Tay et al., 2022), vocabulary size and tokenizer on the components of scaling law (Hestness et al., 2017: Kaplan et al., 2020), the impact of data quality all components of scaling law remains poorly understood. Prior theoretical works on the origin of the power law and its relation to the dimensionality of data mani- fold (Sharma & Kaplan, 2022; Bahri al., 2024) and Zipfian distribution of input data (Hutter, 2021; Michaud et al., 2023) are perhaps closest to our own. usually under some simplifying assumptions like infinite data size or model size. They particularly make predictions about the exponents of power law but remain silent about other components.
Data Manifold Theory: Data manifold refers to the low-dimensional structure that higher di- mensional data lies on. Data manifold theory predicts that exponents of power law are inversely proportional to the data manifold dimension (Sharma Kaplan, 2022; Bahri et al., 2024). However, the impact of data quality on data manifold itself is poorly understood. Data quality, particularly text quality, can be characterized across various axes: diversity of topics, grammar complexity, for- matting artifacts, information density, factuality, fairness, safety, etc. While prior theoretical work do not discuss the impact of data quality explicitly, their machinery is powerful enough to make predictions. Take removing unstructured noise, like garbled text, it could ostensibly decrease the apparent dimensionality. On the other hand, deduplication could expand the data manifold. While both are different text interventions towards improving quality, one seems improve the exponent, while the other decreases.
Zipfian Distribution Theory: Zipf's law is another empirical observation that explains word fre- quencies follow a power-law in their rank. It shows up not only in word frequencies, but also in n-gram distributions (Ha et al., 2009), sentence structures, and higher-level concepts (Michaud et al., 2023). Prior work conjectures that if input data follows Zipfian distribution, the Zipf's exponent correlates with the power law exponent (Hutter, 2021; Michaud et al., 2023). However, much like data manifold theory, the impact of data quality interventions on Zipfian distribution are not quite predictable. While some data intervention techniques, like synthetic data generation cuts off the heavy tail of the input distribution, other intervention techniques like deduplication flattens the head of the curve. This implies that Zipfian slope gets steeper for synthetic data but flatter for dedupli- cated data. We will see later in Section 4, these predictions are not always consistent with empirical observations as it is not easy to predict how data quality interventions influence distribution.
Effective Tokens and Utility-Based Scaling Laws: Prior work has examined how incorporate data quality into scaling law formulations. Chang et al. (2024) focus only on the data axis, propos- ing to replace dataset size D with an effective variant, but leaving other components of the law unchanged. Muennighoff et al. (2023) extend this idea to both model size and dataset size, intro- ducing effective formulations N' and D', though their analysis is tailored to the setting of repeated epoching rather than data interventions. Goyal et al. (2024) similarly reinterpret the data exponent 33 in terms of effective utility. These approaches capture aspects of data efficiency but treat quality as primarily modifying D 3, overlooking its broader influence on parameter coefficient and ex- ponents, or irreducible loss. By contrast, we show that data interventions perturb all components of the joint scaling law fit. Most recently, Shukor al. (2025) proposed "full" scaling law for data
3

---

Published as a conference paper at ICLR 2026
mixtures, which is closest in spirit to our work. Their focus is on mixture composition the inter- vention, whereas we amalyze heuristic filtering and synthetic data rewrites, broadening the range of data-centric interventions studied under scaling laws. Overall, that text quality interventions affect all components the scaling law, not just the data dimension, providing complete picture of how quality reshapes scaling dynamics and offering practical guidance for data-centric scaling strategies.
work is the first to demonstrate
Synthetic Data Scaling Laws Fan et al. (2024) studied the impact of synthetic images on scaling laws, particularly on data exponent. Qin et al. (2025) examined how generator model size influences scaling laws on downstream tasks for L.MMs. In contrast, we study upstream loss and investigate how mixing synthetic and natural data shapes scaling behavior in LLM.
Dynamic Data Intervention and Non-Power-Law Scaling Sorscher et al. (2022) show that, with adaptive data pruning during training, it is possible to surpass standard power-law scaling and approach exponential improvements. In contrast, our work assumes interventions are applied once prior to pre-training, rather than adaptively adjusting throughout training.
Post- fest-Trainnn Data Quality Recent work has investigated the role of data quality over quantity in post-training alignment, showing that even small high-quality datasets improve performance (Zhou et al., 2023; Xia et al., 2024).
Text Quality Interventions can be characterized across multiple axes, including information en- tropy, topical diversity, grammar complexity, formatting artifacts, factuality, fairness, and safety. The exact definition of text quality typically varies by downstream usecase. In this work, we focus the impact of text quality upstream loss. Broadly. data quality can be manipulated through three strategies: filtering, which removes low-quality or undesired content using heuristics model-based approaches (Raffel et al., 2020; Lee et al., 2021; Sharma et al., 2024); mixing, which rebalances data distributions or adds high-quality subsets (Li et al., 2024; Shukor et al., 2025); and synthetic generation, which uses LLMs to clean or augment existing content (Kang et al., 2025). These approaches have informed the design of many recent LLM training corpora, including RedPa- jama (AI, 2023), Dolma (Soldaini al., 2024), Ref (Penedo et al., 2023), Fin (Penedo et al., 2024), DCLM (Li et al., 2024). Yang et al. (2023) shows how to use a small seed of syntheti- cally corrupted bad data to cluster and filter out real bad data code.
QUALITYPAJAMA
We introduce QualityPajamaa a benchmark suite of 23 datasets derived from Common Crawl, each reflecting a distinct level of data quality and intervention. The suite spans a broad spectrum of data quality techniques, including 14 filtered datasets and synthetically curated datasets, for training large language models. Table 1 summarizes the interventions used in each category. Additional details regarding dataset construction and design choices can be found in Appendix.
IMPACT OF DATA QUALITY ON SCALING LAW COMPONENTS
We aim to understand how data quality affects scaling law components, whether predictable patterns emerge under quality interventions, and how these insights can guide effective data curation.
Figure visualize the impact of text quality interventions, particularly heuristic- baase filtering and synthetic data generation, on components of neural scaling laws, namely B, A, B and E. Fach line in the radial plot represents different training set, while the radial axis displays various validation sets. It is apparent from these results that all components sensitive to training set quality well as validation set quality.
Sequential Application of Data Filters and Effeets on Scaling Components We apply series of data filters sequentially and extract intermediate datasets at each stage to conduct scaling law analysis. The order in which filters applied is indicated in the legend. Interestingly, the trajectory of changes in scaling law components does not necessarily follow the order of interventions. Take a for example: it increases after removing NSFW content (red to green), but decreases after filtering
4

---

Published as a conference paper at ICLR 2026
(higher 15 better)
(higher is better)
WAW
piiceen
A (log10-scale) (lower Is better)
(log (logl0-scale) (lower is better)
3000
E (log10-scale) (lower iS better)
alg
Perplexity Perpiexity NSFW
Perplexity NSFW Aesthetic + PageRank Perplexity Assthetic Deduplication
Perplexity NSFW Aesthetic
Figure 1: How Data Filtering Affects Scaling Law Components. Different colored lines represent different data-quality interventions in training set, while the radial axes show impact across different validation sets. While parameter estimates vary bution shift, the consistent separation between filter-specific curves across validation sets indicates that data data-quality interventions induce systematic shifts in scaling-law parameters.
evaluation domains due train-test distri-

---

Published as a conference paper at ICLR 2026
Table 1: Summary of QualityPajama dataset interventions.
Category
Description
Abbrev/Variaans
Hewristic-bused Filters (14 variants)
NSFW Filtering
Removes documents containing offensive or inappropriate content. Filters out text with undesirable patterns (e.g., "lorem ipsum", in- line code, high alphanumeric ratios >> 0.8)
nafw
Aesthetic Fillers
resthetic
PageRank Filtering
Partitions pages into /inghiunnhihhhhunnowwn based PageR- ank score. Thresholds are set to the 33rd and 67th percentiles of the score distribution of all pages in the PageRank table. (Page et al., 1999).
highLpr, med_pr,
low_pi, no_pt
Deduplication
Fuzzy deduplication using MinHashL.SH (Leskovec al., 2020) with different similarity thresholds; selects lowest perplexity doc- ument from near-duplicate clusters.
deduped_ 00
deduped_ 00.8, deduped_0. deduped_1.1
Grammar Complexity
Filters based average sentence length proxy for syntactic richness, with thresholds at tokens for short text and tokens for medium text
short_text,
med! um text, long_text
Synthetic Curation (9 variants)
High Quality Rephras-
LLM rewrites documents to clearer more coherent (Maini al., 2024). Mixtures denote the percentage of synthetic vs, natu- ral data (CC). L.M converts documents into conversational QA pairs.
110100,
ing (HQ)
H267-C033,
#C33-C067.
Question Rephrasing (QA)
Answering
QA10C,
QA67-0C33,
QA33-CC67
Textbook-style Rephrasing (TB)
Converts documents into eextbook-sylle chapters using structured prompting (inspired by Phi models (Li al., 2023; Javaheripi et al.,, 2023; Abdin et al., 2024)).
TB100,
T867-CC33, 116--0233,833--077 1B33-C067
Table 2: Can Scaling Components Reliably Rank Data Interventions? We report average Spear- man correlations across validation sets for each scaling law component. Moderate values (0.3-0.5) suggest that component-based rankings are only partially preserved across validation sets; higher values suggest reliable ordering. Results suggest that such metrics may not reliably rank natural data interventions. In contrast, rankings for synthetically curated datasets show strong consistency, suggesting scaling components are more reliable for evaluating synthetic data strategies.
Data Interventions
B
8
All heuristic filters
0.45
0.34
0.46
0.32
0.34
All synthetic data
0.81
0.91
0.76
0.91
0.54
garbled text (green to yellow). It decreases further after removing pages with low PageRank scores (yellow to blue), but then increases again after deduplication (blue to orange). As shown in Appendix A, these dynamics are not always consistent with predictions from Zipf's law or the data manifold hypothesis, showcasing the limitations of the current theory.
Component-Wii Wise Correlations We examine whether scaling law components exhibit consistent patterns under data quality changes-for example, whether improving quality increases the model exponent (a) and decreases the data exponent (3). While Figure suggests such trends qualita- tively, we quantify them via Spearman correlations (Figures 2a and 2b). The strongest, most stable correlations across all validation sets are: A x a and B x B.
Sensitivity to Validation Sets We examine whether data intervention rankings are consistent across validation sets. Table 2 reports average Spearman correlations Figure show high consistency, rankings across all 14 heuristic filters exhibit only moderate cor- relation (0.3-0.5), suggesting that filter rankings are only partially preserved across validation sets. This indicates that scaling law behavior not independent of the validation set for naturally curated datasets. On the contrary, for synthetically curated datasets, across validation sets. This indicates that scaling law behavior less sensitive to validation set for synthetically curated datasets.
component. While filters in
strong correlation

---

Published as a conference paper at ICLR 2026
<<
1.00
0.08
0.88
0.14
0.38
1.00
-0.44
0.93
-0.44
0.03
0.8
-0.19
0.96
Spearman Corelatiion
0.4
0.93
-0.36
1.00
-0.38
0.09
0.14
-0.13
1.00
0.40
-0.44
0.97
-0.38
1.00
0.17
0.0
0.38
0.28
0.28
0.40
1.00
0.03
0.24
0.17
1.00
8
A
(a) Heuristic filters
(b) Synthetically curated
Figure 2: How Scaling Law Components Co- Vary with Data Quality Intervention?. We observe strong monotonic correlations between and a, and between B and B. For synthetic data, there are also notable negative correlations between a and and between A and B.
1012
UJema (4-4058 0-15.61)
3014
Uema 3.1 (4-4058, D-15
Llema 3.1 (4-4055, B-15 (D-15.577 pwwplesspyyyyyyoocmompoo
(a-2.777)
peepesslyyyyyyy
1.04
Optimal #Paraineters
2012
acsthetic 0.318)
(2-0.29)
3014
aesthetic (2-0, 6821
(b-0.7011
parameter
assthotic (b 300.355)
(b-a=0.402)
noh_pria-0.3800
high_pr (b-0.666 (bm0.691)
2
high_pr a-0.329
3091
3821
2011
Optimal
1010
3.07
2012
10
104
Optmal
1.0
10a
100
1014
123F 1DPF
102F
10tz
30tH
1.0
1027
10a
1021
2012
2012 10P7
Compute flops
Compute flops
Compute flops
Figure 3: How Data Quality Influences Scaling Strategy? Given Nopt x C" Dopt & Co., and D Dopt/Nopt & Cb-a where a = 3/(a + 3) and = a/(a + 3). (Left) shows how optimal model size scales with compute. At today's compute budget (dashed line), the best and worst data interventions differ by over an order of magnitude optimal model size. (Right) shows the variation in token-to- parameter porermmt-er ratio, where interventions differ by up orders of magnitude at the same compute scale.
4.1 INTERPRETATIONS
Designing Compute-Optimal Scaling Strategy Requires Accounting for Data Quality: Prior work has shown that compute-optimal design decisions depend on the scaling law components: a, 3, A, and B. Since data quality influences these parameters, it directly affects compute-optimal choices. Figure 3 illustrates how the compute-optimal number of tokens, number of parameters, and their respective ratio (a proxy for sample efficiency) scale with available compute and vary with data quality intervention. Notably, today's compute scale (indicated by the dashed line), the optimal design point can differ significantly across-by by up to 14x for the number of parameters, 13x for the number of tokens, and an astonishing 182x for the token-to-paree paaramte- ratio. These results highlight the critical role of data quality in determining efficient scaling strategies, underscoring the need to account for quality variations when designing large-scat scale training runs.
Tension Among Scaling Law Components: Data interventions do not uniformly shift all com- ponents of the scaling law in direction that reduces loss. We observe that the coefficients A and B are positively correlated with their corresponding exponents a and B, respectively. This coupling creates tension in how different components influence performance. While increasing the expo-

---

Published as a conference paper at ICLR 2026
Compute-Optima Prod Test
reppperplectty
B:1537.
Bw543
ga3.23.
5-0.18.
E-0.15)
highpperplettty
A-6415, are
(-0.00
sesthetic
B-442
8.0.17
E-0.59)
costtalie
A-3222, are 39, E-0.50)
B:210
p-3.16
high pr.
Aw1159. ave 33, Ex0.30
soduped
9-157
2-3.16.
E-0.451
A-2472.
Li5n2 3.1 (l44888 0-15.67)
high perplexity (C-dam, y-t.12 [-0.25)
y=,.13,
betic high pr
IC-242e, y.. E-2.50)
(cx20ss,
y. 10, be0-26)
dedused 1.0
(C-047
ye..1 Ex3.25)
120
100
100
10
Training flops
(#tokens)
Model size (#parameters)
Figure 4: How Does the Optimal Data Quality Strategy Change with Scale and Resource Con- straints? Compute-scaling law (left), data-scaling law (middle), and scall-scaling law (right) curves show that no single data strategy remains optimal across all scales. The optimal choice shifts as the resource scale changes and also depends on which resource is constrained: model size, data size, or compute budget.
nents a and / typically leads to improved scaling and lower loss, increases in the coefficients A and B have the opposite effect, raising the loss. As result, interventions that improve one component may simultaneously degrade another.
One may argue that the trade-off between exponent and coefficient, the exponent should domi- nate, since its effect is exponential while the coefficient scales only linearly. While this may hold asymptotically at extremely large scale, Figure shows that the tension persists even at today's com- pute scale (e.g., 1021 FLOPs). This persistence may be due to the fact that the coefficients A and B vary across several orders magnitude, while the exponents a and remain relatively small, limiting their ability to compensate.
Notably, synthetically curated datasets, we observe a negative correlation between a and 3, sug- gesting that improvements in model scaling efficiency may come at the expense of data scaling efficiency. Such opposing forces highlight the complex and sometimes counteractive nature of data quality interventions on loss behavior. This underscores the need to analyze all components of the scaling law jointly, rather than relying on any single metric to assess data quality improvements.
Data Quality Rankings Vary with Scale We observe frequent crossovers between scaling curves for different data interventions (Figure 4), indicating that a dataset which minimizes loss at small scale may be outperformed by another at larger scale. This shift in relative performance highlights the risk of extrapolating small-scale experimental results to large-scale settings. Consequently, con- clusions drawn from limited-scale experiments may not generalize to high-compute regimes, and data quality strategies should be validated at or near the intended scale of deployment to ensure their effectiveness holds under real-world training budgets.
The Best Data Quality Strategy Depends on Your Resource Constraint In addition to being scale-dependent, the "best" data quality strategy depends on the specific resource constraint, as shown in Figure 4. For instance, if the goal to identify the most efficient dataset under a fixed compute budget, compute scaling provides the most relevant lens. However, if the constraint lies in model size or available training tokens, the conclusions may differ. Therefore, practitioners should be mindful of their primary resource constraint when evaluating or selecting data quality strategies, as the optimal choice inherently constraint-depeeddent.
5 DATA QUALITY INTERVENTION COMPARISONS THROUGH COMPUTE- LENS
How aggressive should deduplication be? Is there a diminishing return in compute efficiency as we dudupe more aggressively? PageRank useful signal for filtering? Is more or less compute effi-
8

---

Published as a conference paper at ICLR 2026
Compute Optimal Projected Test Li3
10
14e2211.5
8x442.22
Compute Optimal Projector Test Loss
aesthetic
(4-3211.52
9-442:27
6-6
1edures_1.2
10-
Mattpp
8-226:17
oedires_n.2
18-2465.6
B-265.90
e-3.36
1-0-174
(+0.25)
aet
(4-3001.4
B-457-31
e-0.11
P-6.16
2oduse1.9.7 18x3246.5 55 8-185. 89
p+1.30 pe0 1108 E-6.13)
locpp
(6+2477 93
5-362.39
o.6.36
P-6
152
ro.
(bx2705 588
8-258.68
ov6.35
pio
164
Ev0.231
1000
10p
10ah
10
10
1077
10th
10a0
10th
10
107
10
10
10
10
1011
10
107h
Training
Training
(a) Deduplication
PageRank Filtering
Compute- Optimal Projected Test Losa
100
sedined 1.5 14-2411 57 D-157 22
gn0.34
p-0-156
(-0.05)
Compute Optimal Projected Test Lass
cCise
(Aw3367 .55 B-4291 65 ard
39 pet 321 5-1
start text 1804230117 71
B--07 20
an0.30
pro
p-1.35)
0A186
(M-14377 .5 B-1586 d4
redlus_tect 1441845 62
B-255
gel.36
Ex6.14)
b-4795.05 (A-14293.3
test
14-3152.00
po0.337
5+1.531
Nosces Di DaIk
100
10a4
102
162
100
10
10ah
10
10s
16a
102
10P
1027
102a
10
109
16
Training flops
Training flops
(c) Sentence Length Filtering
(d) Synthetic Natural Mixing
Figure 5: Compute scaling law results for various data quality interventions.
cient train with synthetic data? Do improvements in compute-effiiiency merely reflect reductions in data volume, or can they go further? To address these questions, we analyze compute scaling laws under various data quality interventions in Figure 5.
Deduplication: Fuzzy deduplication offers substantial compute savings that far exceed reductions in dataset size. For example, exact deduplication reduces data volume to 83% of its original size yet yields 100x gain in compute efficiency. Fuzzier approaches perform requires approximately 3x less compute than dedupe_0, 10x less than exact deduplication, and 300 less than no deduplication (Figure 5a).
even better: dedupe_0
PageRank Filtering: While a higher PageRank correlates with improved quality (high_pr: > med__p pr > low_pr), filtering strictly by high PageRank does not outper- form the baseline. In contrast, including pages not found in the ranking table (no_pr) re- sults in significantly greater compute efficiency-likee due to recency effects (Figure 5b).
Synthetic-Natural Mixing: Mixing synthetic and natural data consistently outperforms using either alone, but the optimal mixing ratio evolves with compute scale (Figure 5d).
DISCUSSION
Summary: We set out to analyze the impact of text quality interventions, particularly heuristic- based filtering and LLMM-uided data rewrite, on the components of neural scaling laws in training large language models. To enable this study, we developed QualityPajama, a benchmark suite of 23 systematically constructed text datasets spanning a range of quality levels and interventions, from filtering to deduplication to paraphrasing and synthetic curation built on top of Common Crawl dataset. We found that: (1) all components of the scaling law are sensitive to data quality (2) data intervention rankings not preserved across scales; (3) the decision how to scale model size and data size with increased compute budget is heavily influenced by data quality; (4) data intervention impact compute saving goes far beyond the reduction in data volume; and (5) mixing synthetic and natural data outperforms using either alone, though the optimal ratio is scale dependent.

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck. Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.0880 2024.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for gen- erative mixed-modal language models. In International Conference on Machine Learning, pp. PMLR, 2023.
Together AL. Redpajama: Open pretraining data for llms. togethee.computer PRedaaa Data, 2023.
https://github.coo
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-2233 2022.
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Under- standing scaling laws for recommendation models. arXiv, 2208.08489, 2022. URL https: //arxiv.orgg//eeee..........ao.a org.ass//ra//////////ooo/.. 2208. 08489.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878122 2024.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning, pp. 1466-1482. PMLR, 2022.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
Xiao Bi. Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chenggi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0222 220.0424444 2024.
Matthew A Branch, Thomas F Coleman, and Yuying Li. subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems. SIAM Journal on Scientific Computing, 1-23,, 1999. doi: 10. 1111137/$10444775522289
Ari Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arXiv:2412.07942, 2024.
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, and Vikas Chandra. Scaling paramater-comstrained language models with quality data. arXiv preprint arXiv:2410.03083, 2024.
Lukasz Debowski. A simplistic model of neural scaling laws: Multiperiodie santa fe processes. arXiv preprint arXiv:2302.09044, 2023.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint urxiv:2-410004840, 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as change of scaling laws. arXiv preprint arXiv:2402.07043, 2024b.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7382-7392, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.00027. 2020.
10

---

Published as a conference paper at ICLR 2026
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021.
Sachin Goyal, Pratyush Maini, Zachary Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-das data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22702-22711, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprini arXiv:2407.21788 2024.
Le Quan Ha, Philip Hanna, Ji Ming, and Francis Jack Smith. Extending zipf's law to n-grams for large corpora. Artificial Intelligence Review, 32:101-113, 2009.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint (tti::2002.029999 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint /7 2017.
Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Compu- tational challenges in deep learning. In Proceedings of the 24th symposium principles and practice of parallel programming, pp. 1-14, 2019.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shlapentokh Shlapentokh-Rothmann Learning curves for analysis of deep networks. In International conference on machine learning, pp. 4287-4296. PMLR, 2021.
Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2102.044772 2021.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understunding of Foundation Models, 2024.
Mojan Javaheripi. Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 13):3, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //axxiv.org/abs//222......
06825.
Mark Johnson and Dat Quoc Nguyen. How much data enough? predicting how accuracy varies with training data size, 2017. httpps://weby science mq. edu. au/-mjohnson/ papers/ Johnson1 7Power pdf.
Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-War Wen Li, Ramya Raghavendra, Ruoxi Jia, and Carole-Jean Wu. Demystifying synthetic data in Ilm pre-training: A systematic study of scaling laws, benefits, and pitfalls. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp- 10750-10769, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.066 07099....9999 2021.
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive data sets. Cam- bridge university press, 2020.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muen- nighoff, Reinhard Heckel, Jean Mercat, Mayee F Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Kamal Mohamed Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Joshua P Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo. Alaaeldin El-Nouby, Hadi Pouransari. Alexander T Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-LM: In search of the next generation of training sets for language models. In The T'irtty-iihht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: httpt:hhttpeeeeeeeeeeeeee /////pppnevviieeee net/ / weerummiid-nNvee 4 IE.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allic Del Giorno, Suriya Gunasekar, and Yin Tat Lec. Textbooks are all you need ii: phi- 1.5 technical report. arXiv preprint arXiv:2309.054663 2023.
Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M Alvarez, Zhiding Yu, Sanja Fidler, and Marc T Law. How much more data do i need? estimating requirements for downstream tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 275-284--2 2022.
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephras- ing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380, 2024.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699-28722, 2023.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 06:50358-50376, 2023.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford infolab, 1999.
Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.1200 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 116,, 16, 2023. URL https: / arxiv. org/ abs 2306. 01116.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:3081 1-30849, 2024.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535-1005777 2024.
12

---

Published as a conference paper at ICLR 2026
Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R Fung, et al. Scaling laws of synthetic data for language models. arXiv preprint htth:25003.1555// 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: //jmlr. wttp://pppppppppppeeaeeeeeeeeeee Sttpperrr//1//2--11/ 200044 html.
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 12670, 2019,
Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9))1-34, 2022.
Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kushal Tirumala, Russell Howes, Hu Xu, Po-Yao Huang. Shang- Wen Li, Armen Aghajanyan, Gargi Ghosh, et al. Text quality-based pruning for efficient training of language models. arXiv preprint arXiv:2405.01 2024.
Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. arXiv preprint arXiv:2507 094404 2025.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Natham Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer Noah A. Smith, Hannanch Hajishirzi. Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprint, 2024. URL https: //axxiv.// org/ abs 2402 00159.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15: 2022.
Yi Tay, Mostafa Dehghami, Samira Abnar, Hyung Won Chung. William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 105552017.11 2022.
Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet. Olivier Teytaud, and David Lopez-Paz. Meta Lingua: A minimal PyTorch LLM training library, 2024. URL https:: /ttp://ggtttbbbt com facebookreeeerrchh...oooooo...ca /iingua.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K Jamod Millman, Nikolay Mayorov, Andrew R J Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W Moore, Jake Van- derPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E A Quintero, Charles R Harris, Anne Archibald, Antônio H Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. Seipy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272, 2020. doi: 10. 10.1088//4192-01942 152-0019--6----2-22
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint orXiv:2402.04333 2024.
13

---

Published as a conference paper at ICLR 2026
Yu Yang, Aaditya K Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozière, Carole-Jean Wu, Ari S Morcos, and Newsha Ardalami. Decoding data quality via synthetic corupttions: Embedddnggguutn -uided pruning of code data, arXiv preprint arXtv:2312.02418, 2023.
Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scout Shenker, and Ion Stoica. Spark: Cluster computing with working sets, In Proceedings of the 2nd USENIX conference on Hor topics in cloud computing. 10-10. USENIX Association, 2010.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint 12200 2023.
14

---

Published as a conference paper at ICLR 2026
Appendices
A Training Dataset
16
A.1 Baseline Dataset Choice
16
A.2 Derivative Datasets
16
Evaluation Dataset
18
C Model Design
19
C.1 List of Trained Models
19
Details on Scaling Analysis Setup
20
E Scaling Law Component Analysis
20
E.1 Heuristic Filters
20
E.2 Synthetic Data Generation
21
Limitation of Zipfian Distribution Theory
Beyond Chinchilla: Stable Scaling-Law Fits Under Train-Test Drift
29
G.I Introduction
29
G.2 Methodology
30
G.3 Coverage-Udentiffabbiitty Diagnostics in the Chinchilla Form
30
G.4 Motivation: Asymptotic Floor Collapse in the Chinchilla Form
30
G.5 Progressive Improvements via Iterative Refinement
31
G.6 Distance-Dependent E Regularization
32
G.7 Characterizing Prediction Uncertainty
33
G.8 Exploring Alternative Fitting Functions (What Did Not Work)
34
G.9
Analysis of the 7-Parameter Model
36
G.10 Conclusions
36
15

---

Published as a conference paper at ICLR 2026
A TRAINING DATASET
A. BASELINE DATASET CHOICE
We build QualityPajama on top of CommonCrawl dataset assembled by RedPajama-v2 (AI, 2023), which includes 84 Common Crawl snapshots from 2014 to 2023. RedPajama shares raw Common- Crawl dataset along with quality signals for each document but does not filter out any data from the mix. There is roughly 0.5 TB or 100 B tokens per snapshot per partition. We focus on the English subset from 34 snapshots and head partition, totaling approximately TB of data (or 3T tokens). This choice is motivated by three key considerations:
Minimal Pre-processing: To be able to evaluate the impact of data quality interventions, we require a dataset that is minimally processed. RedPajama's 's CommonCrawi is preserving much of its original form while offering clean interface.
Scale: A dataset of substantial size is necessary to support scaling law analyses across multiple orders magnitude--ver after aggressive filtering. RedPajama-v2, is well suited in terms of both volume and temporal coverage. Given that our final dataset is ss 1% of the original dataset, to enable an equal scaling range for all datasets, say upto 30B tokens, the original dataset should be in 3T tokens/ range.
URL Availability: The presence of URL for each document allows us to explore PageRank-basse filtering techniques. This is particularly useful given that crawling gorithms like Hyper Centrality (used by Common Crawl) already introduce implicit biases that we can now systematically study.
85%
PR
NST
Aesthetic
Pape
Medium
33%
High
Raw
Common
Crowl
Perp lodty
Low
70%
Tury
Medium
Deduplicatin
73%
High
Grammer Comp lodly
NQ 37N-006675
b8.9
Synthetic
HOL678-0C
Medium
OBA 33%
11%
TB 800%
TB 1967%-CC 3396
Figure 6: QualityPajama data pipeline. We show the filtering rate next to each filter. Given that our final dataset is & 1% of the original dataset in volume, to enable an equal scaling range for all datasets, say upto 30B tokens, the original dataset should be in 3T tokens/15TB range.
A.2 DERIVATIVE DATASETS
Figure 6 illustrates the pipeline used to construct the QualityPajama benchmark suite. To sup- port this, developed scalable Spark-based (Zaharia et al., 2010) data processing frame- work Paaa-aaaa that enables rapid experimentation with filtering, deduplication, and other data curation strategies.
16

---

Published as a conference paper at ICLR 2026
A.2.1 HEURISTIC-BASHR DATA QUALITY FILTERS
We carefully hand-pick a set of filters that are deemed to improve quality the extent that they are included in many data recipes used for curating well-known datasets such as C4 (Raffel et al., 2020), Dolma (Soldaini et al., 2024), RedPajama (Weber et al., 2024), RefinedWeb (Penedo et al., 2023) and FineWeb (Penedo et al., 2024). These include NSFW filtering, format-based filtering, grammar-bass based filtering, deduplication, etc. We also include some less explored filters, like PageR- anking score to study their efffcctiveness. We apply these filters sequentially and extract intermediate datasets after each stage. Heuristic filters usually are accompanied with some knobs to control their filtering degree. For instance, deduplication has similarity threshold for deeming two samples du- plicate and we are curious to understand: how does this knob controls quality? Where applicable, we experiment with multiple thresholds and retain only the "best" filtered dataset for downstream filtering. The filtering pipeline includes:
NSFW Filtering: We remove all pages containing inappropriate offensive language.
We exclude documents containing undesirable patterns such as "lorem ipsum," inline code (e.g.. "(", "javascript"), and those with a high alphanumeric character ratio (above 0.8).
Aesthetic Filters:
We partition documents into four groups low, medium, high, and not-found-based on their PageRank scores (Page et al., 1999). Since Common Crawl sampling biased towards high Hyper Centrality (correlated with PageRank), our analysis exposes implicit biases in many web-derived corpora. The thresholds are chosen to split the PageRank score distribution in our reference table into three equal parts.
PageRank Filtering:
Deduplication:
We apply deduplication at page granularity within each snapshot. For fuzzy deduplication, we use MinHashLSS (Leskovec et al., 2020) at different Jaccard sim- ilarity thresholds (0.7, 0.8, 0.9, 1.0). We build MinHash signatures on top of pre pre-processed lower-cased bi-grams with 256 permutations. We use signature to build similarity graph, from which connected components (clusters of near-duplicates) are identified. Within each cluster, the document with lowest perplexity score is retained.
Grammar Complexity: We average sentence length simple first-order proxy for syntactic complexity. Using NLTK for sentence and token segmentation, we bin documents into categories of short, medium, long, and very long sentences.
A.2.2 SYNTHETIC CURATION TECHNIQUES
While the literature on synthetic data generation is very rich, only a few have been proposed and deployed for pretraining large language models (Li et al., 2023; Javaheripi et al., 2023; Abdin al., 2024; Maini et al., 2024). Our goal here is not to generate new content but to clean up the existing content through careful prompting. We use three techniques proposed in the literature. All synthetic data was generated using a Mistral-Inssrruct- 7b-v0.. model with the following sampling parameters:
Temperature: 0.7
Top-p (nucleus sampling): 0.95
These parameters were chosen to balance creativity and coherence in the generated text.
We implemented distinct pipelines that represent leading methodologies in synthetic data generation.
We modified the prompts from the original work (if available) to promote better format-following and encourage longer, high-quality text. Generation procedures are detailed below with full prompts provided in boxes A.2.2. AA2.2.2.4
High Quality Rephrasing (HQ) Inspired by WRAP (Maini et al., 2024), we prompt LLM to rewrite source documents into clear, coherent, and well-structured text.
Question Answering Rephrasing (QA) Inspired by WRAP (Maini et al., 2024), we prompt LLM to convert source document into a conversational QA format.
Textbook-style Rephrasing (TB) Inspired by family of Phi models (Li et al., 2023; Java- heripi al., 2023; Abdin et al., 2024), first convert text into book chapter titles and then
17

---

Published as a conference paper at ICLR 2026
prompt the LLM to generate new content for each chapter, with variations in prompts for different target audiences (grade school, college, expert, general).
Light heuristic post-filtering was applied all generated synthetic datar, removing documents that were excessively short (e.g., less than 50 tokens) or excessively long relative to the target length for that generation type, such outputs occurred despite prompt length guidance. The goal of this light filtering was to remove egregious generation errors without overly sanitizing the data significantly altering its distribution.
A.2.2.1 Prompt Template HQ Rephrasing
System Prompt: additional notes.
Provide direct and detailed response to the instructions without adding
[USER]: For the following document, regardless of its original content or formatting, write a full article of the same content in high quality English language as in texts on Wikipedia: [xxxx]. Provide the rephrased article without any additional notes. Long article with full length and complete details. Rephrased article:
A.2.2.2 Prompt Template QA Rephrasing
System Prompt: additional notes.
Provide direct and detailed response to the instructions without adding
[USER]: For the following document, regardless of its original content or formatting, con- vert it inlo comprehensive list of question-answer pairs with multiple tags of "Question:" followed by "Answer:". where questions and answers original document. Document: [xxxx]. Provide the converted question-anae -nsowrr pairs without any additional notes. Question-answe answer pairs with corresponding tags ("Question:", "An- swer:"):
complete information of the
A.2.2.3 Prompt Template for Generating Textbook-style Synthetic Data: Step 1, Outline Generation
Step 1: generate an outline based on input text.
System Prompt: Provide direct and detailed response to the instructions without adding additional notes.
[USER] <4 versions>: Imagine you are prolific author tasked with writing a textbook. You working writing textbook involving the knowledge and information of the following text. Text: [xxxx]|n Your task is to write an outline for the textbook. Your target audiences are <grade school students/college students/field experts/general public>. The textbook has 10 chapters in total plus title, introduction, and appendices. Textbook outline:
A.2.2.4 Prompt Template for Generating Textbook-sty style Synthetic Data: Step 2, Chapter Generation
Step 2: generate each section based on outline.
System Prompt:
Provide direct and detailed response to the instructions without adding additional notes.
Imagine you are prolific author tasked with writing textbook. You are working writing textbook with the following outline An Outline: [xxxx] \n Your task write Chapter of the textbook. Your target audiences are grade school students. Include exercises at the end chapter to test reader's knowledge reference answers to each question.
[USER]:
chapter then provide
B EVALUATION DATASET
Unlike prior scaling law works that report training loss (Hestness et al., 2017: Hoffmann al., 2022) or test loss on a held-out validation set (Kaplan et al., 2020) from training distribution, we measure upstream loss held-out test from original CC as well as diverse set of 16 non-code/math
18

---

Published as a conference paper at ICLR 2026
Table Model configuration parameters for different scale sizes.
Model
Hidden Dim
#lyyers
#Heads
Batch Size
Grad Ace
DP
TP
#Params
100m
576
9
4
8
1
175,628. 736
200m
832
10
13
653,436, 1
298,632.256
500m
1280
16
20
1
1b
1792
22
28
8
1,317,616,388 1
2b
2240
28
35
4
8
1
2,292,740,,00
3b
2624
32
41
8
2
3,360,234,06
4b
2816
34
44
4
4,103,539,96 966
6b
3200
40
50
5.801.83,660
8b
3648
45
57
32
8,122,355, 9549
11b
4096
51
64
2
16
11,372,228,608
English text domains from The Pile (Gao et al., 2020). Because we use scaling laws for comparative analysis across data interventions, critical to assess model performance on external validation sets to enable fair and meaningful comparisons across different training datasets.
C MODEL DESIGN
C.I LIST OF TRAINED MODELS
We adopt a standard transformer-based model architecture based on LLaMA3 (Grattafiori et al., 2024) for all of our scaling analysis. In Table we list the model size and configuration of all models used in this study.
C.I.1 TRAINING AND EVALUATION HYPERPARAMETERS
We trained all models from scratch using the Meta Lingua library (Videau et al., 2024) across one multiple nodes depending model size. We AdamW (Kingma & Ba, 2014) optimizer with 81 0.9, B2 0.95, and weight decay of 0.1, paired with a cosine schedule and 10% linear warmup. All runs used a 4096-token context length, IM-token effective batch size, and the Ilama 3 Token tokenizer (128k vocab) (Grattafiori et al., 2024). Table 4 and Table list hyperparameters for training and evaluation. Table lists local batch size, gradient accumulation, data parallelism (DP) and tensor parallelism (TP) employed for each model size. These parameters are chosen such that global batch size remains at IM token across all experiments.
Table 4: Training Hyperparameters
Hyperparameter
Value
Optimizer
AdamW
Peak Learning Rate
3e
Min. LR Ratio
1e
Warmup Steps
10%
Gradient Clipping
Sequence Length
4096
Effective Batch Size
1M tokens
Prefetch Size
1024
Add BOS token
True
Add EOS token
True
Model Data Type
b
Epochs
GPU Hardware
NVIDIA A100 80GB
19

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters for Perplexity Evaluation
Hyperparameter
Value
Max Tokens Generate
1024
Generator Data Type (dtype)
6
D DETAILS ON SCALING ANALYSIS SETUP
Unless otherwise stated, all scaling results reported in the main body of the paper are obtained using a joint scaling-law fit of the following parametric form:
D) = A A + B
This section describes the methodology used throughout the paper for fitting this form. An exception Appendix G, where we adopt several engineering practices commonly used prior Chinchilla- style analyses to improve fitting stability.
We empirically estimate the parameters by fitting this function to the validation loss over 100 models, ranging from 100M to 8B (3B) parameters and trained on 100M to 40B (200B) tokens for filtering (synthetic) interventions. Each datapoint is visited only once during training, consistent with standard scaling-law methodology. This avoids confounding effects from data repetition and ensures fair comparison across datasets.
Curve-fitting and Initialization: We fit the scaling law using scipy- optimi ze curve_fit (Virtanen et al., 2020), specifically the Trust Region Re- flective (trf) optimizer (Branch et al., 1999), which supports bounded nonlinear least squares. Initial conditions are drawn from prior work (Besiroglu et al., 2024) that challenged assumptions in the original Chinchilla analysis (Hoffmann et al., 2022):
[A, B, or B, EJ [482, 2085, 0.3478, 0.3658, 1.8).
Appendix G Variant: We additionally adopt several fitting practices inspired by the Chinchilla methodology to improve fitting quality. These include fitting the amplitude parameters (A, B) in log-space, using a Huber loss to reduce sensitivity heavy-tailed residuals, detecting and removing statistical outliers prior to fitting, optimizing with L-BFGS, and estimating parameter uncertainty bootstrap resampling (1000 samples).
Parameter Count Definition: There exists inconsistency in prior work regarding whether in- clude embedding parameters in the total parameter count N. Kaplan et al. (2020)'s scaling law anal- ysis excludes embedding parameters, while the Hoffmann et al. (2022)'s 's analysis includes them. We examined both conventions and found that qualitative trends and conclusions remain consistent. For consistency, we report results using the total parameter count including embeddings.
E SCALING LAW COMPONENT ANALYSIS
In Section 4, Figure we showed the impact of a handful data quality interventions on compo- nents of scaling law. Here we show the impact of all 23 datasets from QualityPajama benchmark suite. We group the results based on the type of interventions (heuristic filters vs. synthetic). We also compare the best from each group
E. HEURISTIC FILTERS
To study the effect of heuristic-based data quality interventions scaling behavior, apply sequence commonly used filters, including NSFW removal, aesthetic filtering, PageRank- based filtering, deduplication at varying similarity thresholds, and grammar-based filtering via average sentence length. These filters are chosen based on their frequent use in high-quality dataset pipelines such C4 (Raffel al., 2020), Dolma (Soldaini al., 2024), and Feb (Penedo et al., 2024).
20

---

Published as a conference paper at ICLR 2026
For each filter, we evaluate its impact the scaling law parameters by comparing the fitted values before and after its application, as well as across different configurations (e.g., similarity thresholds for deduplication percentile cutoffs for PageRank). Detailed analyses shown in Figures 7, 8, and 9.
E.2 SYNTHETIC DATA GENERATION
To evaluate the impact of synthetic data interventions on scaling behavior, we curate datasets using three prompting strategies: high-quality rephrasing (HQ), question-answer transformation (QA), and -exthook-style rewriting (TB). These methods draw inspiration from prior work on synthetic al., 2023; Abdin et al., 2024), and
pretraining data (Maini al., 2024; Li al., 2023; Javaheripi model (Jiang et al., 2023). We mix synthetic data with natural data different ratios (e.g., 33% synthetic, 67% original). We fit scaling laws these synthetic variants to analyze how text rewriting influences parameter stability and scaling behavior. Detailed results are shown in Figure 10, 11, 12, and 13.
are applied using the Distral-Instruct-ii
F LIMITATION OF ZIPFIAN DISTRIBUTION THEORY
While prior work has suggested that token distribution characteristics-Suuts Zipfian structure could explain power law exponent's behavior, our empirical findings show that this theory may not be sufficient to explain the variation in power-law exponents. We analyzed token frequency dis- tributions across our filtered datasets (Figure 14) and found that the Zipfian exponents are weakly negatively correlated with the model size exponent a (correlation -0.37), and show little to no correlation with the data size exponent 3 (correlation = 0.0055. Table 6 shows scaling exponents across datasets. In several cases, datasets with nearly identical token distributions exhibit substan- tially different scaling behavior. This suggests that simple distributional statistics, such as Zipfian exponents, fail to capture the deeper structural or semantie properties that influence scaling dy- namics. It is possible that higher-order r--ram patterns or conceptual structures provide a more explanatory signal.
Filter
Zipf Exponent (z)
Scaling Exponent (a)
Scaling Exponent (g)
high_perplexity nsfw aesthetic
1.1820
0.3509
0.2536
1.0950
0.4341
0.1982
1,0907
0.4097
0.1946
deduped_ 0.7
1.3898
0,3633
0,2173
deduped_00 0.9
_09
1.3938
0.3392
0.2393
deduped_1.0 1.0 no_pr
1.1638
0,3499
0,2093
1.0599
0.3884
0.1855
low_pr
1.3943
0.3772
0.1805
med_pr
1.3744
0.4157
0.1982
high_p _r_pppr_____
1.2625
0.3499
0.1826
medium text
1.3541
0.3740
0.1885
long_text
1.2944
0.4106
0.1833
Table 6: Zipf exponent z and scaling law exponents a and across different data filters.
21

---

Published as a conference paper at ICLR 2026
a (higher is better)
B (higher is better)
PusteeL Coogle
Beng
Eocoy
Eocod
Oren Erine
Orens Ening
A (log 10-cale) (lower is better)
B (logae-scale] (lower is better)
Gcae beng Mics
Gute beng Pr
(log10-scale) (lower better)
Aesthetic
Ruzzy Deduplication (Jaccard Similarity 0.91
Exact Deduplication
Fuzzy Deduplication (Jaccard Similarity
Figure 7: How does deduplication affect scaling law components? The red line marks the dataset before any deduplication is applied. Other lines represent deduplicated variants using different sim- ilarity thresholds.
22

---

Published as a conference paper at ICLR 2026
a (higher better)
B (higher better)
Ooer Pebl 101
Oopr Aell lol
Pustfee Sogfe
A (logl0-scale) (lower is better)
B (log10-scale) (lower is better)
NoWel
PusMel Coyfe
Tendy
Fensy
E (log10-scale) (lower is better)
c/B
NoMel
NoWeet
Grenkes
Aesthetic Filters Hi gh PageRank Score
Low PageRank Score
PageRank Score Not Found
Madium PageRank Score
Figure 8: How does PageRank-basee filtering affect scaling law components? The red line de- notes the dataset before applying any PageRank filters. Other lines correspond to thresholds applied to the PageRank score. Low PageRank retains pages with scores below X, High PageRank retains those above Y, and Medi um PageRank keeps pages between X and Y. PageRank Not Found includes pages missing from the reference PageRank table. Thresholds X and Y the 33rd and t7th percentiles of the score distribution of pages in the PageRank table.
23

---

Published as a conference paper at ICLR 2026
(higher is better)
B (higher better)
Ening
Eniig
(logl0-scale) (lower better)
(logl0-scale) (lower better)
E (log10-scale) (lower is better)
o/
homaa
Esoo
Escod
Exact Deduplication
Medi um Length
Length
Long Length
Figure 9: How does grammar complexity (average sentence length) affect scaling law compo- nents? The red line indicates the dataset before applying any sentence length filters. Datasets filtered based on average sentence length, with thresholds set at 10 tokens for short text and 25 tokens for med nedium um text.
24

---

Published as a conference paper at ICLR 2026
a (higher is better)
B (higher is better)
(log10-scale) (lower better)
(log10-scale) (lower better)
Klijelium
Ose al
E (log10-scale) (lower is better)
a/B
1570 pacogoonts
PLAE
holeed
CC100%
HQ33%-CC675 H067%-CCBB
0A100%
QA67%-CC33%
TB33%-CO67% TB67%
H0100%
0A33%-CC67%
TB100%
Figure 10: How does synthetic data influence sealing law components? Different lines show different synthetic data generation techniques and mixing ratio, and along the radial axis have validation set.
25

---

Published as a conference paper at ICLR 2026
(higher is better)
B (higher is better)
(log10-scale) (lower better)
(log10-scale) (lower better)
Wlijelium
hostee par
Ge
Ose 0o
E (log10-scale) (lower is better)
a/B
SSHO ENCOgAAEE
holee
CC100%
H0100%
H033%-CC667%
H067%-CC33%
Figure 11: How does HQ synthetic data generation influence data quality? HQ refers high- quality rephrasing, and CC refers to the raw natural Common Crawl dataset. HQIN]-CC[M] refers to the mixture of synthetic and natural and N and M captures the percentage.
26

---

Published as a conference paper at ICLR 2026
a (higher is better)
B (higher is better)
(log10-scale) (lower better)
(log10-scale) (lower better)
Wiijelium
Noltfe Paca
Ose 00
On OS
E (log10-scale) (lower is better)
a/B
SSTO ERcOgRAEE
holee
CC100%
QA100%
QA33%-C067%
QA67%-CC33%
Figure How does QA synthetic data generation influence data quality? QA refers to Question-Answertn rephrasing, and CC refers the raw natural Common Crawl dataset. QA[N]- CC[M] refers to the mixture of synthetic and natural and and M captures the percentage.
27

---

Published as a conference paper at ICLR 2026
a (higher is better)
B (higher is better)
(log10-scale) (lower better)
(log10-scale) (lower better)
Noltfe Pack
Ge
Ose 0o
E (log10-scale) (lower is better)
a/B
SSHO EacOpAAEE
PLJE
holeed
CC100%
T8100%
T833%
TB67%-CC33%
Figure How does TB synthetic data generation influence data quality? TB refers to Textbook-style rephrasing, and CC refers the natural Common Crawl dataset. TB[N]-CCIM] refers to the mixture of synthetic and natural and N and M captures the percentage.
28

---

Published as a conference paper at ICLR 2026
0.21/ frank 23)
aechete
deduped 57
deduped 23
10
deduped, 2.0
FML
melu
Nigh pr
short Jest sturte
med um teat cnumed
long shufteed
10
Teenn Requero)
10
Figure 14: Token distribution across different Quality datasets
G BEYOND CHINCHILLA: STABLE SCALING-LAW FITS UNDER TRAIN-TEST DRIFT
We include this appendix to disentangle how much of the observed variation in scaling-law param- eters across datasets reflects true distributional differences versus instability of the scaling-law form itself. This investigation revealed that the standard Chinchilla scaling-law form fails predict performance reliably when train and test distributions differ. also exhibits severe parameter in- stability (large coefficients of variation, high CV) when the dataset lacks sufficient coverage in the large-N/large-1 D regime. We propose a new scaling-law formulation that remains identifiable with fewer datapoints, avoiding the high cost of collecting expensive large-scale measurements. Em- pirically, this formulation is substantially more robust than the traditional Chinchilla form under distribution shift and limited-data settings. A key ingredient for robust estimation is incorporating the theoretical asymptotic loss floor. Finally we show that while the standard Chinchilla form ex- hibits weak identifiability under distribution shift, stabilized fits recover the same qualitative scaling trends across data interventions, suggesting that the exponent and coefficient shifts reported in the main paper reflect true data distribution effects rather than fitting artifacts (Figure 24).
G.1 INTRODUCTION
Classical neural scaling-law formulations implicitly assume that training and evaluation data are drawn from the same underlying distribution. In practice, scaling laws are often used when train and test distributions Liffer-ssmmeinng substantially. As data curation pipelines evolve or filtering strategies become more aggressive, the training distribution can drift away from the target evaluation distributions for which we ultimately want predictions. This mismatch violates the assumptions of standard scaling-law forms and directly impacts parameter identifiability.
A second pain point arises from the need for high-compute datapoints to accurately estimate the irreducible loss floor E. When the data do not extend far enough into the large-N/lrre- D regime, the asymptotic term becomes weakly identified (Fig. 15)-and and the situation worsens when train/test distributions drift apart (Fig. 16).
29

---

Published as a conference paper at ICLR 2026
In this appendix, document step-by-step journey toward robust scaling-law form under limited datapoints: we begin by diagnosing failure modes in the baseline Chinchilla form and then describe the empirical insights that guided subsequent refinements.
G.2 METHODOLOGY
Dataset. We evaluate scaling-law forms across 299 train pairs, formed by the Cartesian prod- uct of 23 training sets from QualityPajama and 13 test sets (12 from the Pile and 1 validation set drawn from the training distribution). For each train-test pair, have 40 datapoints with approx- imately 9 x 14 N NxD coverage, with model sizes ranging from 20M to 3B parameters (~2 orders of magnitude) and data sizes from 100M to 38B tokens (~2..6 orders of magnitude).
Measure of success. We define composite quality score (0-100 points) compare scaling-law models, balancing prediction stability and predictive performance while penalizing known patho- logical solutions. We weight prediction stability (40 points) and prediction quality (40 points total) equally, reflecting the requirement that a useful scaling law must both fit the data well and yield stable predictions. The remaining 20 points penalize degenerate asymptotic behavior. The score aggregates:
1. Performance Stability (40): fraction of (rain-validation pairs where the parametric pre- diction CV (via Monte Carlo propagation of parameter uncertainty) is below 1.0.
2. E-Collapse Avoidance (20): rewards models with irreducible error satisfying E > 0.1,
penalizing pathological - solutions.
3. R² Performance (20): out-of-bag predictive accuracy on a linear scale, with n² receiving full credit.
4. RMSE Performance (20): absolute prediction error normalized between best and worst RMSE across all models.
This rubric ensures scaling-law models achieve stable prediction, avoid degenerate asymptotes, and maintain strong predictive accuracy; underperformance any dimension substantially reduces the overall score.
G.3 COVEAGG- DIAGNOSTICS IN THE CHINCHILLA FORM
In-distribution evaluation. Fig. 15 shows parameter uncertainty as a function of data coverage for the Chinchilla form under in-distribution evaluation (orig-val)). Each heatmap reports the co- efficient of variation CV = std/mean /mean for parameters when fitting subsets restricted to N < Nmax and S Dmaxx computed from 150 bootstrap samples. Dark regions correspond to stable, well- constrained estimates; bright regions indicate high uncertainty (CV < 0.1: excellent; 0.1-0.2: ac- ceptable: > 0.2: poor). Notably, the asymptotic term E exhibits sharp identifiability transition: it collapses to zero when Nmax 2B or D < 5B, and becomes non-zero and identifiable only with sufficient Daiee--llaree-D coverage. contrast, (A, A, a. B) are comparatively robust, with uncertainty decreasing smoothly coverage increases.
Distribution shift. Fig. 16 repeats the analysis under distribution shift (aesthetic-<mall Cross- domain evaluation induces severe parameter instability relative in-distribution evaluation. The asymptotic term E collapses to zero across all coverage levels. Other parameters also degrade sharply: a is unstable in 67/100 cells, B is highly unstable throughout, and shows extreme vari- ability even in the high-N/high- region; only 3 retains moderate stability. Together, these results indicate that distribution shift fundamentally destabilizes scaling- parameters: even with large models and extensive training data, parameters learned in one domain do not reliably predict loss in out-of-distributtin settings.
G.4 MOTIVATION: ASYMPTOTIC FL.OOR COLLAPSE IN THE CHINCHILLA FORM
Using the standard 5 5-parameter Chinchilla model, we observe that 59% of vrin-validation pairs produce near-zero asymptotes (F < 0.1; Fig. 18). This behavior is incompatible with theory:
30

---

Published as a conference paper at ICLR 2026
CV
CV
Mari
110
309
Figure 15: Parameter uncertainty vs. data coverage in the Chinchilla form (in-distribution; orig-vvall Each heatmap shows coefficient of variation (CV =std/moan) of parameter estimates when fitting the Chinchilla form to subsets restricted to S Nuux and D S D Panels report CVs for A, a, 2, E from 150 bootstrap samples; the bottom-right panel reports the mean esti- mated E. Numbers indicate datapoints used, Dark regions indicate stable estimates; bright regions indicate high uncertainty (CV < 0.1: excellent; 0.1-0.2: acceptable; 0.2: poor). E exhibits sharp identifiability transition, collapsing to zero when Nmax < 2B or Dmax < 5B and becoming identifiable only with sufficient Barge-N/lrrge-D coverage.
when train and test distributions diverge, we expect the asymptotic floor to be strictly positive and approximated by
E Hat + (T. Ptrain).
The collapse indicates that, without additional information, the model prefers to "explain away" the asymptote by reallocating mass into A and B. The failure is most severe when (i) train and validation distributions diverge (77% failure for high JS-divergence pairs vs. 56% for low JS-divergence pairs) and/or (ii) we lack high-compute datapoints to anchor the asymptote (82% failure with 30 points vs. 26% with > 45 points).
While JS divergence computed from unigram distributions only rough proxy for train-test drift, E-collapse rates vary sharply across validation domains. Technical and specialized domains (code, foreign languages, academic papers) exhibit E-failure rates exceeding 90%, likely reflecting limited representation in the Common Crawl-based training corpus (Fig. 17).
G.5 PROGRESSIVE IMPROVEMENTS VIA ITERATIVE REFINEMENT
Fig. 18 summarizes iterative improvements in robustness. We report out-of-bag R² and RMSE (top), catastrophic failure rates for prediction CV (> 1) and for (< 0.1) (bottom-leftt, and the aggregate quality score (bottom-right). Starting from the baseline 5-parameter Chinchilla form, we add E- regularization around a theoretical bound, and finally consider a 7-parameter extension with scale- dependent amplitude terms. Both the 5-parameter E-regularized model and the 7-parameter model substantially outperform the baseline Chinchilla formulation (71.17 and 71.84 vs. 60.40 points). Given the negligible difference (0.67 points), select the 5-parameter E-regularized model as the preferred model by Occam's razor: comparable predictive quality with fewer parameters and improved interpretability.
31

---

Published as a conference paper at ICLR 2026
CC
CV
Mach
114
319
Figure 16: Parameter uncertainty vs. data coverage in the Chinchilla form (distribution shift; aesthetic-matee Cross-domain evaluation induces severe parameter instability: E collapses to zero across all coverage levels; and become highly unstable, and A shows extreme variability even at large N and D. Only 33 remains moderately stable.
G.6
REGULARIZATION
To address F--collass in the baseline model (59.2% of pairs with E 0), introduce information-theoreti regularization strategy that (i) pulls E toward a theoretical floor and (ii) adapts regularization strength based on train-validation distribution distance.
Theoretical prior.
We set
Eprior 0.05 (Hvat + KLaiv) In(2),
where Hyal Shannon entropy of the validation set unigram distribution and KTaiv the KL di- vergence from validation to training unigram distributions. The factor 0.05 is empirical scaling constant that calibrates information-theorede quantities to the numerical range of observed neural losses: unigram cross-entropy is upper bound irreducible error, while neural models leverage richer contextual structure.
Regularization strength.
We add quadratic penalty to the Huber loss objective: RE
AE (E Eprior) 2
(3)
with distance-depennent strength
Ag 0.02 + 1.05 (1 s)3,
8 +
This design applies minimal regularization when train and validation are similar (s as 1 implies Ag 0.02), while imposing stronger regularization for distant pairs (e.g.. 0.6-0.7 yields Ag 0.12-0.17), preventing collapse.
Effect. This approach reduces E-collapse from 59.2% to 3.3%, while maintaining excellent fit quality (mean out-of-bag R² > 0.9). For brevity, we report only the final winning approach; we ex- plored ablations including hard constraints E, direct penalties for small E, regularization around alternative targets (e.g. MctM-esttmated E), and alternative distance-to-Ap mappings (fixed, lin- ear).
32

---

Published as a conference paper at ICLR 2026
E-Collapse Fallure Rate by Validation Set (Original 5-Parameter Model)
gthub
europari
sackoachange
92.3%
92.3%
youtube
69.2%
pobmedantraa
69.2%
philoapers
61.5%
gutenberg
61.9%
email
61.5%
beocs3
61.5%
wilden
upsto
46.2%
ihexporier
23.1%
23.1%
hadkamews 15.4%
books2 7.7%
High Fallure pecchecattpec peccalooos
esensubotes
-0% (Wosed)
s000%
40
60
100
Failure
Figure 17: E-collapse failure rates vary by validation domain. The baseline 5-parameter model exhibits near-total F-collapse (92-100%) for technical/speccialzzd validation sets (code, foreign lan- guage, academic papers) but maintains stability (0-23% failure) for general-text domains (movies, books, web crawls). Color coding: red 2 70%, orange 50-70%, green & 50%. Overall: 59% failure across 299 filter-validation pairs.
Winning recipe (summary).
Rg Ag .EE Eprior) 2
(5)
1
A 0.02 + 1.05 (1 8)²2
(6)
1 JS
Eprior 0.05 (Hval + KL.vv) IIn2).
G.7 CHARACTERIZING PREDICTION UNCERTAINTY
The usefulness of scaling laws forecasting tools depends on prediction uncertainty in unobserved regimes. We distinguish two sources:
ii) Functional-form uncertainty via Gaussian process diagnostics. We use Gaussian process (GP) as a diagnostic for functional misspecification. We treat the scaling law as a parametric mean function and fit GP to residuals in log-log space. The GP uses Matérn-5/2 kernel over the 2D input (log N, log D) with separate length scales, learned via maximum likelihood. Large or structured GP residual uncertainty indicates misspecification; conversely, negligible GP uncertainty (CV < 1%) suggests the parametric form sufficient and remaining uncertainty arises primarily from parameter estimation.
33

---

Published as a conference paper at ICLR 2026
Scaling Law Model Quality Comparison
1.00
Moam 008 10008: (nlgleer bettee
(Lower better
0.55
0.56
0.485
0.54
0.9195
0:9235
0.45
0.415
0.52
0.35
0.28
Meen 008
0.50
0.273
0.20
param
param
Predictive CV Falture Rtte dlowee benteed
teneline)
+.
(chanth beselin)
Fallure Itate (Loverr bostter
60
59.3%
16.4%
ichenchila beseline)
Ictencila basckne)
e.E
Quality Score better
100
Fiot fiat (60 yest
E-22 lppa
fRoee hot 125 uto
71.84
60.40
10.4
16.3
19.3
70.0
32.1
26.6
3prarn (dhnchell basdinc)
parom
parom
Figure 18: Progressive improvements in scaling-law robustness. (Top-left) out-of-bag R² and (top-right) out-of-bag RMSE quantify predictive accuracy on held-out data. (Bottom-left) catas- trophic failure rates: for predictive performance, failure is the fraction of train-test pairs with pre- diction CV > 1; for E. failure is the fraction with E < 0.1. (Bottom-right) aggregate quality score combining prediction accuracy, prediction stability, and E-collapse avoidance. The baseline 5-parameter model improves substantially with distance-dependent E-regularization; 7-parameter model with scale-dependent amplitudes yields similar overall quality. We select the 5-parameter F-regularized model by Occam's razor.
(ii) Parametric uncertainty. We estimate parameter uncertainty via bootstrap resampling (1000 samples) and propagate it into predicted loss via Monte Carlo sampling (1000 draws from the boot- strap parameter distribution).
Fig. 19 compares baseline E-rrgularized 5-parameter models across all 299 pairs. E- regularization stabilizes amplitudes (especially A) while keeping exponent uncertainty low; it re- duces median parametric prediction CV (20% - 16%). maintains negligible GP CV (< 1%), elim- inates E-collapse (59% - 3.3%), reduces catastrophic prediction failures, and improves out-of-brr of--of-ba R² for 98% of pairs.
G.8 EXPLORING ALTERNATIVE FITTING FUNCTIONS (WIAA DID NOT WORK)
This section summarizes alternative scaling-lav forms explored and abandoned because they offered comparable stability with greater complexity than the 5-parameter model with F-regularization.
34

---

Published as a conference paper at ICLR 2026
S-Farameter: No E-reg us E-reg - Confficient of Variation Compariion
Producterr
Figure 19: Prediction uncertainty breakdown: baseline 5-parameter vs. 5-parameter with E- regularization. Top: parameter-level CV for a, B, A, B. Bottom: prediction CV from parameter uncertainty, GP CV from residual functional uncertainty, and out-of-bag R², Red shading indicates catastrophic failure (CV 100%). Dashed lines indicate means/medians.
G.8.1 BAYESIAN MCMC INFERENCE (NUTS)
We implemented Bayesian MCMC (NUTS) for the 5-parameter Chinchilla form L(N, D) = A N + B DD- E, using log- log-space transformations for amplitudes to enforce positivity: log A ~ (5,,3,, log B N5,,3), log (00,
(8)
and truncated normal priors for exponents:
a ~ TruncNormal(0.25, 0.2; 0.01,0.99).
We used Student-t likelihood (v 4) on log-losses to handle heavy tails, ran 4 chains (1000 warmup 2000 samples, max tree depth 10), and monitored R and ESS. We tested both free-F and fixed-E (set to 95% of the minimum observed loss) variants to mitigate the known A-B-E degen- eracy. While MCMC eliminated E'-collapse, it catastrophically failed in predictive performance: mean out-of-bag R² = 26. 42 across 299 pairs (quality score -482/ (00/ and produced highly unstable amplitudes:
98% of pairs:
catastrophic A failures (CV(A) > 1),
41%: catastrophic B failures (CV(B) > 1),
30%: catastrophic E failures (CV(E) > 1).
G.8.2 CROSS-TERM MODULATION VIA SCALE- DEPENDENT AMPLITUDES
Motivated by 1D sliced fits within each train-validation pair-fitting L(N) at fixed 1 and T.(D) at fixed N-we observed systematic drift suggesting the Chinchilla form is too rigid over the (N, D) plane:
. A and vary consistently with D (R2 x 0.5),
and 3 vary consistently with (R² 0.3).
We explored extensions with fully scale-dependent parameters (9+ parameters) but found them un- stable given ~ 30-50 points per pair. In contrast, allowing only amplitudes to be scale-dependent was supported by the data. The resulting 7-parameter model is
L(N, D) An t - + Bg N D E,
(9)
35

---

Published as a conference paper at ICLR 2026
(High Similarity) Similarity-0.870
Vcal 13%% 01
Tearss CJ
Peurmtre (90%
Hedian prodetion
Nedior pred
Model Size
Data Size
Figure 20: Prediction uncertainty breakdown for the 7-parameter model (example pair). Left: fix D) at its median and vary N. Right: fix N at its median and vary D. Black lines show mean predictions from bootstrap-averaged parameters. Shading shows 95% intervals from parametric uncertainty (Monte Carlo over bootstrap parameters) and GP residual uncertainty (Matérn-5/2))
with
A(D) Am
B(N) Bo,
and a, B constant within each pair.
Across 299 pairs, the fitted " exhibits a striking domain-specifit pattem: pairs with specialized validation sets (math, code, foreign languages) yield >> 0, while general-text validation sets yield n < O. Interpreting the sign flip: "1 implies A(D) increases with data, suggesting that scaling dissimilar data worsen loss under strong mismatch; 71 < 0 implies A(D) decreases with data, suggesting additional data helps when domains align. This separation helps explain why constant- amplitude Chinchilla fits fail under distribution shift and motivates cross term modulation
mechanism to capture train drift.
Uncertainty breakdown (7- 77-parameter model). Fig. 20 shows a representative decomposition for high- sigh-aitll pair: GP residual uncertainty remains small and unstructured (øur 0.1), while parametric uncertainty dominates (arram ~ 3-4, ~ 45x larger). This indicates that instability arises from weak parameter identification rather than functional misspecification. While the 7-parameter model can improve stability for A and B, it often shifts instability into the modulation exponents (notably 71), which can degrade overall prediction uncertainty.
G.9 COVERAGE-STABILIT ANALYSIS OF THE 7-PARAMETER MODEL
We repeat the coverage analysis for the 7-parameter model, analogous to the baseline Chinchilla evaluation. Under in in-distribution evaluation (orig-vall), the 7-parameter model shows similarly strong stability across parameters (Fig. 21). Under distribution shift (aesthetic-matth it maintains good stability for most parameters, with notable variability in Ag. but it recovers stable, non-zero E values (Fig. 22), demonstrating improved robustness to the E + degeneracy that affeets the baseline Chinchilla form.
Across all 299 pairs, the baseline 5-parameter model show lower median CV for exponents but higher for amplitudes, though median statistics hide heavy tails of catastrophic failures. The 7- parameter model reduces certain catastrophic failures by relaxing pathological parameter coupling through ccalelmodllation terms, but can introduce instability in the modulation exponents and may worsen overall prediction uncertainty aggregate (Fig. 23).
G.10 CONCLUSIONS
We evaluated three variants: (i) the baseline 5 5-parameter Chinchilla model, (ii) a 5-parrmeter model with distance-dependent F--regularization, and (iii) 7-parameter model with scale-dependent am-
36

---

Published as a conference paper at ICLR 2026
Figure 21: Parameter uncertainty vs. coverage for the 7-parameter model (in-distribution; orig-val)
plitudes. We select the E-regularized 5-parameter model for its simplicity and strong performance: out-of-bag n22 = 0.92, 84% of pairs with prediction-CV < 1, and mean prediction-CV = 0.45.
37

---

Published as a conference paper at ICLR 2026
Figure 22: Parameter uncertainty vs. coverage for the 7-parameter model (distribution shift; cesthetic-mathhe
S-Paremater va 7-Paramater: Coufficient of Veriation Comparison (Ali 259 Pairsl
OV Predecerr
Preciatoo Performents
We Comper
Figure 23: Parameter uncertainty and predictive performance: baseline 5-parameter vs. 7- parameter model. Top: parameter CVs for a, B, 71, 72, A, B. Bottom: prediction CV from param- eter uncertainty, GP CV, out-of-bag R², and win rates. Red shading indicates catastrophic failures (CV > 100%).
38

---

Published as a conference paper at ICLR 2026
(N-scaling exponent)
(D-scaling exponent)
comaad
0.
logu((A [N-term amplitude)
log [D-term amplitude)
0g Ratio
[irreducible loss]
Perplecity NSF Amddielic Dedupication
Porplexity NSF RageRank
Perplesity
Figure 24: Scaling-Lav Parameter Shifts Persist Under Stabilized Fitting. Scaling-law param- eters (a, 3, logı A, log1a B, E) and the a/3 ratio estimated with entropy-based E regularization, the stable scaling-law recipe. While parameter estimates vary across evaluation domains due to train-tes, distribution shift, the consistent separation between filter-specffic curves across validation sets indicates that data-quality interventions induce systematic shifts in scaling-law parameters that persist even under stabilized fitting. This suggests that the observed changes in exponents and coef- ficients are not solely artifacts of scaling-law instability in the Chinchilla form, but reflect genuine differences in the effective training distribution. Amplitude parameters A and B exhibit the largest filter-driven variation across domains.
39


Reviews:
[{"review_id": "c2c71CIKSv", "reviewer": "Reviewer_1zGS", "summary": "This paper presents a large scale empirical study of how data quality interventions (deduplication, heuristic filtering and LLM-guided rewriting) affect neural scaling laws in large language model training. They use 23 systematically curated datasets to train over 2,000 models to measure how text quality interventions reshape scaling law components (coefficients A, B and exponents α, β). A key finding is that data quality interventions shift both exponents and coefficients, fundamentally changing scaling dynamics in ways not predicted by existing theory.", "questions": "1. How sensitive are the scaling law components to the curve-fitting method? Have you tested non-parametric approaches or Bayesian inference?\n2. The paper uses single-epoch training. How would results change with data repetition? Does the data manifold interpretation change?\n3. For synthetic data generation, did you experiment with different generator models or temperature settings?\n4. Can you provide theoretical intuition for why data interventions shift both exponents and coefficients rather than just one?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses a practically important question that has been largely neglected despite being central to LLM development.\n2. Over 2000 model training runs across multiple scales (100M–8B parameters) and token ranges (100M–200B) provides strong empirical support.\n3. Clear demonstration that optimal token-to-parameter ratios vary significantly  across interventions at the same compute scale.\n4. Well written paper with clear progression from dataset design → empirical findings → interpretations → implications.", "weaknesses": "1. While the paper shows that data interventions shift all scaling law components, it provides limited mechanistic understanding of WHY this happens. The discussion of Zipfian theory (Appendix G) shows weak correlations but doesn't propose alternative explanations. There is critique of existing theory (data manifold, Zipfian distribution) but it is somewhat superficial. Would have been interesting to see more discussion here.\n2. Filters are applied sequentially, but the order matters (acknowledged in Section 4). The paper doesn't fully disentangle whether observed component shifts are due to the filter itself or the interaction order.\n3. The 0.3-0.5 Spearman correlation for heuristic filters is interesting but not fully explained. Why do natural filters show lower consistency across validation sets than synthetic filters (0.91 for B)?\n4. The paper makes several claims about deduplication efficiency that appear to use different baselines or comparison points:\n  - \"Exact deduplication reduces data volume to 83% of original yet yields a 100× gain in compute efficiency\" (L431-432)\n  - \"Fuzzy deduplication (0.7) requires approximately 3× less compute than 0.9, 10× less than exact deduplication, and 300× less than no deduplication\" (L459-461)\n  - These statements are difficult to compare together. If exact dedup achieves 100× gain, is fuzzy 0.7 achieveing \"300× less” than the baseline or exact dedup? What am I missing here?\n\n5. All experiments use llama3-style transformer. Results may not generalize to other architectures (mixture-of-experts, alternative attention mechanisms, etc.). This doesn’t require more experiments, but limitation discussion could be clearer on this.\n6. The claim that \"deduplication could expand the data manifold\" is counterintuitive and under-explained. Deduplication removes redundant documents. If a document is a duplicate, removing it changes weighting of the remaining documents but how does this expand the manifold? This needs clarification or revision.\n\n## Minor\n- The paper uses \\citet in several places where \\citep would be more appropriate. Many claims about what \"prior work overlooks\" (e.g., L58 \"what prior work overlooks other components?\") lack supporting citations. Strengthen argumentative claims with explicit references.\n- L116 The statement that α, β, etc \"are constants\" is not helpful. What do these constants model/support? Provide intuitive descriptions (e.g., \"α quantifies model scaling efficiency\").\n- Consider inverting the right column of figures so that \"higher is always better\" visually. This would make the figure easier to interpret as a whole.\n- “LLM” is not abbreviated on first use in the intro. It should be defined on first use then re-used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nAw7VfKcki", "reviewer": "Reviewer_Mu49", "summary": "The paper studies the effect of data quality on LLM scaling law. The authors start from the RedPajama dataset, consider 14 Heuristic-based data quality interventions and 9 synthetic ones, and applied them in combination to induce various quality-intervened training datasets. For each such quality-invented training dataset, the authors use it to train various sized LLaMA models and evaluate their scaling law. Many findings on the specific effect of different data quality interventions on scaling laws are discussed.", "questions": "no questions", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written excellent, easy to read and follow.\n2. To my knowledge, it is the first work that analyses how specific dataset filtering methods as quality inventions affect specific coefficients in the scaling law. And the authors consider all coefficients rather than just the dataset size coefficient D.\n3. Many findings that I expect to be practically helpful in guiding training data preparation as an integral part of model training are discussed.", "weaknesses": "1. a small point for improvement is that for Figure 3, 4, 5, the legends block significant portion of the figures (especially Figure 4) and makes the figure less readable. I suggest the authors improve the presentation of these figures so that legends do not block figures.\n\n2. The paper focuses on the classic form of the scaling law and it mentions in related work that Chang et al. (2024) and Muennighoff et al. (2023) introduces the effective dataset size D'. And there might be other extended forms of the scaling laws. Could the authors consider re-run the fitting and analysis with these extended forms beside the classic one? I believe it will be very helpful because (1) it will be an ablation study strengthening the validity of the findings and (2) these extended forms might have better goodness of fit because they have more variables in the system and might fit the scaling law better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bjch2s7wY0", "reviewer": "Reviewer_TyiW", "summary": "This paper asks how interventions on data quality shape LLM scaling behavior. They introduce a benchmark named QualityPajama that consists of 23 datasets derived from CommonCrawl and train >2K models on the datasets. Analyzing the scaling laws, they find that different data quality interventions lead to different relations between parameters, and that picking the optimal (given compute) token-to-parameter ratio depends on the data quality intervention.", "questions": "Formatting notes:\n- Line 39: Citations are formatted weirdly\n- Line 87: Missing space\n- Line 123: period should be comma\n- Line 178: formatting of paragraph is weird and inconsistent", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Experiments are extensive, covering 2000 models across a large range of model sizes.\n- Some of the key findings, for example that the intervention of deduplication can yield large compute savings, have practical downstream utility for people training models with limited compute.", "weaknesses": "The paper lacks a theoretical framework through which to understand the extensive empirical results. In particular, the paper would be greatly strengthened if it introduced a new scaling law to relate data quality interventions to other parameters and to explain the empirical findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9KBkQfzVd", "reviewer": "Reviewer_922u", "summary": "training LLM's is very cost intensive, i.p. w.r.t. required number of tokens. recent pretraining dataset filters battle claiming faster downstream performance adoption. most prominent are heuristic quality filters or synthetic data augmentation. this paper is a massive evaluation on several models and datasets , claiming in fact that correlations between quality interventions and scaling law parameters can be found.", "questions": "addressing aboves weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- trained on lots of models to provide robust empirical evidence that data quality influences the exponents, not just coefficients of scaling laws\n- very relevant for future vvery costly model trainings\n- they publish a QualityPajama benchmark suite, again very relevant", "weaknesses": "W1 it empirically measures some correlation to exponents, but with little theoretical explanations or evidence. there is only that vague dismissal of zipfian theory\n\nW2 unfortunately i don't see a strong correlation in the quality filter domains and to validations sets, while it appears rather clear / obvious to hold in the synthetic LLM data augmentation. i therefore think it should be phrased that way from the very beginning? renders interpretation of the claimed findings a bit weird.\n\nW3 for the quality filters it seems strange and outdated filters are used. recently most significant are bert-style quality filters, however the authors rely on rather old pagerank?\n\n(W4 the only chosen metric the authors compare to is perplexity, but i also wouldn't know what to do otherwise in this scale.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)