Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining
Authors: Anirudh Subramanyam
Yuxin Chen
Robert L. Grossman
Venue: ICLR 2026 Poster (2026)
Keywords: quality ware scaling laws
scaling laws
data quality
LLM pretraining
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We extend the Chinchilla scaling law by introducing a data‐quality measure that predicts how noise and coverage affect loss, and validate it on machine translation and autoregressive modeling tasks.

Abstract: Scaling laws for language model training traditionally characterize how performance scales with model size and dataset volume. Prior work has explored architecture variants and data treatments such as dataset filtering and noise injection in language model pretraining; however, these studies have not formalized data quality within a principled scaling law. We introduce a dimensionless data-quality parameter Q, and propose a quality-aware scaling law extending the Chinchilla framework to predict loss as a joint function of model size, data volume, and data quality. The law is motivated by an effective-sample-size and information-theoretic view of noisy or redundant corpora, and it admits two practical estimators for Q: (i) a corruption rate proxy and (ii) a deficiency measure. Through synthetic experiments in neural machine translation and autoregressive modeling--where we systematically control data quality via multiple levels of noise injection variation--we show that loss scales predictably with data quality and that higher-quality data can substantially reduce model size and hence compute requirements. Our results demonstrate a sublinear decay of effective data with quality and robustness to moderate data corruption; out-of-sample evaluations further validate the predictive form of the law. Unlike prior empirical analyses, our work establishes an explicit, generalizable law for data quality, offering concrete guidance for balancing data curation effort and model scale in large-scale pretraining.

Published as a conference paper at ICLR 2026
SCALING LAWS REVISITED: MODELING THE ROLE OF DATA QUALITY IN LANGUAGE MODEL PRETRAINING
Anirudh Subramanyam', Yuxin Chen', Robert L. Grossman
Center for Translational Data Science
*Department of Computer Science
*Department of Medicine
University of Chicago, Chicago, IL 60637, USA
(anisub, chenyuxin, rgrossmann wechhnnneeehhhhhhceeeecee
ABSTRACT
Scaling laws for language model training traditionally characterize how perfor- mance scales with model size and dataset volume. Prior work has explored archi- tecture variants and data treatments such as dataset filtering and noise injection in language model pretraining, however, these studies have not formalized data quality within principled scaling law. We introduce a dimensionless data-quality parameter Q, and propose a quality-aware scaling law extending the Chinchilla framework predict loss data quality. The law is motivated by an dffective-sample--ee
joint function of model size, data volume, and and information- theoretic view of noisy or redundant corpora, and it admits two practical estima- tors for Q: (i) a corruption rate proxy and (ii) a deficiency measure. Through syn- thetic experiments in neural machine translation and autoregressive modeling- where we systematically control data quality via multiple levels of noise injection variation-ww we show that loss scales predictably with data quality and that higher- quality data can substantially reduce model size and hence compute requirements. Our results demonstrate a sublinear decay of effective data with quality and ro- bustness to moderate data corruption; out-of-sample evaluations further validate the predictive form of the law. Unlike prior empirical analyses, our work estab- lishes explicit, generalizable law for data quality, offering concrete guidance for balancing data curation effort and model scale in large-scale pretraining.
INTRODUCTION
is well understood in large-language model (LLM) training that both the amount of training data
and its quality influence pretraining loss. Yet most existing scaling laws are formulated solely in
terms of dataset size, remaining agnostic to quality (Henighan et al. 2020 Ghorbani et al. 2021
Ivgi al. 2022 Sorscher al. 2022] Alabdulmohsin et al.) [022]] (022) Caballero et al. 2023] Tao al.,
2024 Wu et al., 2025). This leaves a gap between common intuition-"leaane data trains better models"-aan -ndd the quantitative laws that guide large-scale training.
The importance of such quantitative framework is especially clear in specialized domains- business, scientific, medical applications-wher training corpora are limited in quantity and het- erogeneous in quality. With the same compute budget, outcomes may differ drastically depending on how clean and representative the data are. There have been quite number of studies classify data quality among multiple "dimensions" (Wang and Strong 1996 Fox et al., 1994 Sidi et al.) 2012 [SO/HC|] 2008). While the exact taxonomy varies, the core dimensions typically include: accuracy, completeness, consistency, timeliness, uniqueness, validity. The focus of these studies is usually datasets containing structured data that clean and curated for a certain task or purpose. In contrast, when collecting data to pretrain large language models, large amounts of "found" data collected and cleaned, usually by removing duplicate data and obviously "poor" quality data.
To reason about how such imperfect corpora affect performance, need simpler abstraction of quality. In this work, we introduce a single dimensionless parameter Q E (0, 1] characterizing the usable information in corpus. A value of represents fully clean and representative data,

---

Published as a conference paper at ICLR 2026
while smaller Q values reflect increasing corruption or redundancy. As discussed below (Section E) Q can be estimated via corruption rates or a more general deficiency measure, and serves as proxy for the fraction of effective samples. We incorporate this abstraction into scaling laws through term of the form (B(DDQQ) (Equation 2. linking model performance directly to both the quantity and the quality of data.
Our main contributions are:
We propose a theoretically grounded quality-awarr scaling law that augments the classical Chinchilla form by introducing dimensionless quality parameter Q. The law predicts loss as L(N, Q) A/N + B/(DDQ") + E, capturing the interplay between model size, data volume, and data quality.
We derive the scaling law from an effective sample size perspective and iheeroition-theerrtiic arguments, and provide two simple estimators for Q based on corruption rate and data deficiency. We show that under natural assumptions these estimators lead the form Def (Q) with g(Q) Q", recovering the proposed law.
We conduct controlled experiments on neural machine translation and causal language modeling to illustrate the law's utility. These results demonstrate that loss scales pre- dictably with quality parameter and that higher-quality data can compensate for smaller models. This is particularly relevant for specialized models in many real-world domains, such as business, scientific medical applications.
2 RELATED WORKS
Data quality in information systems. Data quality has long been studied in information sys- tems, where datasets are evaluated along multiple dimensions annensiyss-accuracyy completeness, consis- tency, timeliness, uniqueness and validity-to ensure that structured records support downstream tasks. Note that what call completeness is also called coverage and closely to what is usually called diversity. A consistent finding from this literature is that diverse imperfections degrade the utility of data and therefore motivate extensive curation practices (Batini et al., 2009). Large-scale pre- training, however, relies on corpora assembled from the web. Here, quality varies widely across sources, and subtle forms of corruption or deficiency persist despite deduplication or heuristic filter- ing. Nonetheless, the key insight from information systems continues to hold: the quality of training data directly influences the performance of learned models.
Data quality in large-scale pretraining. In the era of LLMs, quality has become a first-class design concern. Major corpora such as The Pile (Gao et al. 2020), RefinedWeb (Penedo et al., 2023], and Dolma (Soldaini al, 2024) highlight different philosophies of data construction, from assembling diverse sources to aggressively filtering and refining web text. Deduplication strategies (Lee et al., 2022) and contamination checks (Magar and Schwartz, 2022) have been shown to sub- stantially affect evaluation reliability. Empirical analyses further suggest that careful filtering yields performance gains comparable to scaling compute, therefore implying the nontrivial role of quality in driving LLM performance.
Classical scaling laws and their limitations. Alongside these developments, foundational studies on scaling laws formalized how performance scales with model size, dataset size, and compute. Kaplan et al.] (2020) showed that loss decreases predictably as models and datasets grow, while Hoffmann al. (2022) refined these results by introducing compute-optimal training. Despite their influence, these existing scaling laws generally assume that the underlying data is of fixed quality. In real-world corpora, noise, redundancy, or domain imbalance systematically distort scaling behavior, highlighting the need to explicitly account for quality.
Data quality and scaling laws. Several recent works have begun to address this gap. Bansal et al. (2022) study scaling laws for neural machine translation, and show that while test loss scales predictably with dataset size, architectural choices and moderate noise primarily shift the scaling curves without changing the exponent. Goyal et al. (202)) argue that data filtering is not compute- agnostic: the value of higher-quality data depends on the available compute budget, since reusing small clean datasets leads to diminishing returns. [i et al. (2024) assess data quality by comparing
2

---

Published as a conference paper at ICLR 2026
perplexity gaps between models of different sizes, theoretically linking their metric to inverse scaling laws. These studies highlight the importance of data quality but stop short of providing a unified predictive law that couples quality with model size and dataset size. By introducing dimensionless quality parameter Q into scaling laws, our work formalizes the influence of data quality and offers predictive guidance for building smaller, high accuracy models in specialized domains.
Beyond these empirical analyses, several recent studies refine scaling laws from complementary perspectives. [Chang et al. (2024) propose parameter-constreee diversity and syntheticity, revealing that effective training tokens depend on both data quantity and diversity. On the theoretical front, froot, et al. (2024) present unifying theory of neural scaling laws, distinguishing variance-liman limited and resolution limited regimes and connecting scaling exponents to generalization error. Although these contributions deepen our understanding of scaling, they do not incorporate an explicit scalar quality parameter. Our work complements this literature by modeling data quality via Q. linking it to effective sample size and generalization theory, and providing a unified scaling law that accounts for quality.
scaling laws that combine text
DATA QUALITY MEASURES
To incorporate data quality into scaling laws, we first need a formal definition of Q. this section introduce two approaches for mapping dataset to scalar Q(w) (0, 1]. with larger Q indi- cating higher quality. These definitions capture different sources of imperfection-ttok versus more general deficiency- and will later allow generalization theory. The precise choice Q is application-dependes Q degrades smoothly under corruption, providing a proxy for the usable information in a dataset.
corruption connect Q effective sample size and
the key requirement is that
3.1 DATA CORRUPTION RATE
One of the simplest measures of data quality what is sometimes called the data corruption rate, which we will denote CR. We assume 0 < CR < 1.
Definition 1. Let w be dataset with data corruption rate CR. We define the data corruption rate data quality by
Q(w)= =
As an example, we have a dataset consisting of D tokens and 10% are corrupted, then we would say that the CR 10% and the data quality Q is 90%. To simplify our formulas, assume that the data corruption rate is strictly less than 1, so that Q is strictly greater than 0. Note that standard sampling methods (e.g. Deming (1666) can be used estimate data corruption.
3.2 DATA DEFICIENCY MEASURES
We introduce a dimensionless quantity call the data deficiency, which quantifies the lack of quality in dataset W. We assume that the data deficiency has the following properties:
1. Positivity. (ww) 2 0, for all datasets
2. Continuity. A(w) a continuous function of the dataset w
3. Additivity. If W1 and W2 are independent datasets, and w is the union of W1 and wy than (ww) A (wr) A (wz).
Additive noise model. Data deficiency can be measured and quantified in many ways. One of the simplest models is model data deficiency using an additive noise model, where each data measurement E;, for 1,..., has the form
& = +
where e; is noise term. We assume that e, 0, and that the data quality measured by the data deficiency worsens as e; increases. Note that properties (1) (1)-(3) above hold when we model data quality using the additive noise model. We now add fourth assumption.
3

---

Published as a conference paper at ICLR 2026
4. Maximum quality. In the additive noise model, assume that the noise term e; captures all the noise, so that data quality is maximized when e; = O, for all
Note that the data deficiency can grow arbitrarily large with this model, so that S A(W) oo, for dataset w.
Definition 2. We define the data quality induced by data deficiency Q(w) of dataset w by
Q(w) exp (- A(w)).
4 QUALITY-AWARE SCALING LAWS
widely used empirical scaling law for LLMs (also known as the Chinchilla Scaling Law by Hoff- mann et al. 2022) is
A B
L D)
No D
where N is the number of training parameters, D is the number of training tokens, and L(N, D) is the pretraining loss. This equation was estimated empirically from experimental studies involving over 400 LLM,, with the number of parameters ranging from 70M to 16B and the number of training tokens ranging from SB to 400B. Here A, B, E, a, and / are empirically estimated constants, with E the minimal loss.
For example, one formulation studied in Hoffmann et al. (2022) is:
406.4
410.7
L(N,)) D) N70.1 + D8.28
+ 1.69.
(1)
In large-scale training, improvements such filtering out low-quality data introducing better training objectives have yielded improved performance beyond what naive scaling might predict. For instance, filtering training data remove noise has been observed to effectively improve the scaling behavior, enabling models to achieve lower loss with the same compute. This indicates that not all tokens equal: billion high-quality tokens may be far more valuable than billion noisy or redundant ones. The Chinchilla analysis already hinted that many models are undertrained given their size, pointing toward the need for either more data or better data. This motivates us extend scaling laws with explicit parameters for data quality.
4.1 EFFECTIVE SAMPLE SIZE
We now formalize when data quality Q leads to a scaling law of the form D- The key idea is that poor data quality (e.g. data corruption or large data deficiency) reduces the effective sample size the dataset. We formalize this with:
Definition 3. Let be monotone function [0, + R+ with 9(1) = 1. Then the effective
sample size of D associated with the link function g is
Dat == D (Q)))
Assumption 1 (Effeetive sample size factorization). There exists monotone function g [0. 1] + R,, with g(1) = 1 such that the expected excess loss satisfies
Ln(D.Q) RJ
D off
(D. g(Q))
where IN (D, Q) denotes the loss for given parameter size N.
This assumption is consistent with classical results: in PAC learning with random classification noise of rate < 1/2, the sample complexity inflates by factor (1 - 21))- (Angluin and Laird 1988 Kearns 1998); in regression with additive Gaussian noise of variance o², effective samples seale with the signal-to-noise ratio SNR 1/o" (Tsybakov) 2009); and in information theory, channel capacity arguments show that mutual information is reduced by a multiplicative factor depending on the corruption level (Cover and Thomas 2006). In the following two lemmas (proof deferred to the appendix), we show that the usable sample size can be written as Datt = D g(Q) with g(Q) well-approximmted by power law Q over practical ranges of Q:
4

---

Published as a conference paper at ICLR 2026
? = 1 for signal-to-nol to-soise ratio like token noise (Lemma
* y = 2 for symmetric label noise (Lemma?
Lemma (Effective sample size under additive Gaussian noise). Consider i.id. observations of the form yi [(x;) + Ei, e; NO,, a²), for I,..., Then the Fisher information contributed by each observation is proportional to 1/0², and the total Fisher information satisfies
D ID X 03'
Equivalently, the D noisy samples carry the sume information as Deft D (1/o²) noise-free samples.
We define the deficiency as
o² =
of σ²
A = In o
where oi baseline noise (i.e. e.g. noise not modeled by A). Without loss of generality, we may assume := 1. In this formulation, datasets noisier than the reference (a² > ag) have Q < 1, while "clean" datasets (a² = og) have Q 1. The effective sample size then scales as
Deft = D. Q"
and taking recovers the classical D SNR scaling.
Lemma 2 (Effective sample size under random classification noise). Consider binary classification with labels E {0,1) }} and i..dd examples (X, Y). Suppose labels are corrupted by random classi- fication noise (RCN) with flip rate 1) E [o, 1/2), iee, we observe =Y v.p. - 1j and = 1 Y w.p. n, independently of X. For any classifcation-caliiratee surrogate loss admitting unbiased noise-corrected estimator (e.g., logistic/hinge via the standard correction), the variance of the per- example corrected loss inflates by a factor proportional to 2n) Consequenly, generalization bounds and excess-risk rates scale as if the number of clean samples were
Des = (1 ?
ie., the effective sample size is reduced by 2n)² under RCN.
Lemma a formalizes the intuition that symmetric label noise reduces usable information multiplica- tively, yielding Dar = D gQ)) with g(Q) (2Q 1)2. In other words, if we define data quality via the corruption rate Q
77 (cf. Section 3. then (1 (1-200 -2n)2 (2Q 1)a. Over practical
high-quality regimes Q E (1/2, 1], the factor (2Q - 1)² is well-approximated by local power law in Q with exponent 7 so that safe model Dar D Q" with y is justified for RCN.
4.2 QUALITY-AWARD SCALING LAW
We incorporate the data quality Q and introduce the following quality-aware scaling law: Definition (Quality-Aware Scaling Law).
A
B
L(N, D.Q)
+ +E,
Na
D&Q
where Q denotes data quality and 7 is empirically measured parameter.
Note that for the highest quality data with Q = 1, this reduces to the standard Chinchilla scaling law, and as the quality of the data decreases, the loss increases, as is expected. On the other hand, as the quality of the data goes up, the amount of the data required to achieve the same loss goes down.
The following corollary formalizes how the proposed form arises under mild regularity conditions:
Corollary 1. Suppose that g(Q) is regularly varying at Q = 1, i.e. g(Q) cQ + o(1)) as Q Then the quality-aware scaling law (Definition 4 holds under Assumption 7
We next give information-theoneet perspective that leads to the same law:
5

---

Published as a conference paper at ICLR 2026
Proposition 1 intormmtion-thhorete justification). Let X denote clean tokens, X their corrupted versions produced by a memoryless channel Cq parameterized by quality Q E (0, 11, and let Z be the learned representation used for prediction. Suppose corruption reduces usable information multplicaaively,
I): = p(Q) I(X;
with p(1) =1, p(Q) monotone, and locally ({Q) = cQ as Q 1. Here T(X; Z) denotes mutual information as usual. If the data-dependent loss scales a
LD (D (D)
then the quality-waare scaling law takes the form
A
B DRQT
L(N, D,Q) =
+
+ E,
No
after reparameterization of constants.
The proof of Proposition 1 is deferred to the appendix. For a binary symmetric channel with flip H2(1) with Q V7. over practical ranges this is well-approximated by Q with = 2. In regression with additive Gaussian noise, p(Q) is proportional the signal-to-noise ratio, yielding y = Both cases recover the effective sample
rate ?, p(Q) is proportional to the capacity
size view Def = D p(Q).
4.3 COMPARISON WITH OTHER QUALITY-AWARE SCALING LAWS
Table 1: Relating different realizations of the data deficiency term A(w) to the literature
Definition of Aww)
Interpretation of Data Deficiency
Paper
N1 P2 PgG++pH
Decomposition into noise (E), inverse cov-
Ours
(equation
erage (1/F), and repeated-date measure (G), and syntheticity (S).
μ2 Dps(p)
Cluster-based density p. with Dis(p) = In(1 - p). Small p highh--uulliy;; large p quaaiitt
Chen et al. 2025
[22 DR(w)= + nr S(w)
Compression-based density (compressibil- ity as proxy for diversity) (R(w)
Chang et al. 2024
CR CR-)(w) and CR compressed size and data syntheticity S(w). Higher DR, higher S lower quality.
P3 b
Declining marginal value of data with repeated exposure k epochs; lin- ear ear-exponential saturation depending o. Models over-training on repeated data.
Goyal et al. (2024)
To relate our data dispersion to the literature, assume that the data deficiency is of the form
( nn1 F. F + MrG + M,H,
where E' quantifies the noise in the data, F quantifies the coverage/diversity of the data, G quantifies the amount of repeated data, and H quantifies the loss data quality due to synthetic data. As shown in Table by assuming particular forms for each of these terms, recover the data quality scaling laws in Chen et al. (2025).. ) Goyal et al. 22024) and Chang et al. (2024).
generalization bounds Xu and Raginsky 2017 Bu al. 2020), which relate generalization emror the mutual information between training data and learned representations weights, well by the Information Bottleneck principle (Tishby al. 1999)
'This assumption is motivated by information-thecnmm

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTAL STUDIES
To establish the quality aware scaling laws, we train decoder only language models across two tasks, neural machine translation (NMT) and causal language modeling (CLM). To capture the effects of quality and dataset size we train on 3 different dataset volumes and 7 degrees of quality. We use subsets of Paracrawl v8 (Bañón et al. (2020p) and C4 (Dodge et al. (202I)) as our training data for the translation and language tasks respectively. See Table 2 We optimize cross-entropy loss averaged over the relevant context and report the same on held out test datasets to measure the effectiveness of our quality aware scaling law for predicting the loss on out of sample data (Table 3 We use the parametric loss fitting approach described in inHHommmnnn et al. (2022) to find our parameters using two methods, Least Squares and Huber (Huber 1992). The full details of the experiments, and model training details can be found in the appendix.
5.1 TASK DESCRIPTION
NEURAL MACHINE TRANSLATION
Our first task is English to German translation using the Paracrawl v8 English to German dataset (Bañón et al. 2020). We filter the raw data with min hash deduplication, language ID filtering with 0.8 threshold and length filtering (s 258) leaving 101M sentence pairs. We use 8L GPT Neo model with a hidden size 1024 with approximately ~ 133M params. We also train our own BPE tokenizer with vocabulary size of 32000. We train 3 replicates for each combination of the 3 datasets of size 500K, IM and 2M sentence pairs and 7 quality levels for total of 63 experiment runs. We train using AdamW with maximum learning of Se-4 and cosine decay with a 20% warmup ratio. The full model configuration and training hyper-parameters are available in the appendix.
CAUSAL LANGUAGE MODELING
For the CLM objective we train our models on a subset of C4 (en) (Dodge al. 2021). We run our pretraining experiments on 8L Llama 3 (Grattafiori et al. 2024) model with hidden size of 512 and a context length of 2048. We do not use ROPE scaling. We employ random data truncation on the base dataset to sample within the context length. We also employ the pre-trained Llama-3.2 IB tokenizer. We sample datasets of three sizes; 100M 1B and 10B tokens and train for a single epoch using fused AdamW (Loshchilov and Hutter 2019) with maximum learning with cosine decay and a 10% warm-up ratio. We duplicate our experiment setting to match a total of 63 runs. The full model configuration and training hyperparameters are available the appendix.
of 1e-3
5.2 SYNTHETIC NOISE AND DATA SAMPLING STRATEGY
To simulate the different levels of data quality we add synthetic iid noise to the base dataset. The base dataset after pre-processing is assumed to have quality, Q 1.0. We vary the quality of a dataset by increasing/decrrasingg the fraction of samples that are perturbed with synthetic noise. For example, if 25% of all samples are perturbed, then the quality of that dataset is considered to be 0.75. We use two noise models, one for each task. For NMT, we randomly special tokens in selected sample to pad tokens, we do not discriminate between source and target here. For CLM, we randomly swap 50% of all non-special tokens with valid non-special tokens from the tokenizer vocabulary.
50% of all non-
5.3 DATA SAMPLING STRATEGY
We employ nested subset sampling strategy that guarantees noise and sample monotonicity. S {s1, 82, ......, sn) is the base experiment dataset with n unique samples. We draw from Š thrice using different seeds to create working datasets Sw1, Sw2 and Sus. Each working dataset Sus is then sampled to produce datasets of different sizes. Each these different sized datasets are then perturbed to various degrees to create the noised variants that dataset size.
For example, in our CLM experiments the dataset volumes are T=(0.11, 1, 10) billion (B) tokens. We first draw thrice from C4 to get our 3 working datasets S,1, Sw2 and S.3 of size 40M samples or

---

Published as a conference paper at ICLR 2026
Table 2: Estimated Parameters in Quality-Aware Scaling Law
TASK
METHOD
B
7
E
NMT
Least Squares
166.568727
0.262933
0.185135
0.146998
NMT
Huber
139.602744
0.250067
0.173161
0.066539
CLM
Least Squares
1428.225931 1441.505289
0.395142
0.388678
3.439888
CLM
Huber
0.395859
0.400657
3.439047
Table 3: Estimated Parameters on Unseen Data CLM
TASK
METHOD
B
e
E
CLM (ours)
Least Squares
1428.225931
0.395142
0.388678
3.439888
CLM (unseen)
Least Squares
1589.071797
0.396787
0.332273
4.551611
CLM (ours)
Huber
1441.505289
0.395859
0.400657
3.439047
CLM (unseen)
Huber
1427.299279
0.390546
0.336753
4.540009
2o 16B tokens each (Using the Meta Llama-3.2- tokenizer, we find that on average the "en" subset of C4 has approximately 400 tokens per sample). We ensure that for each working dataset Swi, we sample subsets of the form, i.. So. S Sloon S S10.07- At each subset level, we then compose 7 different dataset variants with = {0, 10, 20, 25, 30, 40, 50} percent noised samples (where denotes the noise percentage), meaning that quality Q = (1.0, 0.9, 0.8, 75, 0.7, 0.5.
5.4 EXPERIMENTAL SETUP
We make of multiple compute sources in this work. NMT experiments upto dataset volume of MM sentence pairs were conducted on an A100 node with 4, 80GB GPUs. All other experiments are run on 2 Hopper nodes with 4. 141GB GPUs each. We do not use multi-node or multi-GPU training. We utilise the GPU counts to parallelize experiment runs.
5.5 MAIN RESULTS
74
10.09
(a) Machine translation
(b) Next token prediction
Figure 1: Test loss as data quality varies from 0.5 to for a) machine translation and b) next token prediction.
Fitting the quality-aware scaling law. Our main results are summarized Table 2 which esti- mates B, B3 7, and F,, and Figure [ which shows the test loss varies from 0.5 to 1.0. Pre-
8

---

Published as a conference paper at ICLR 2026
dictably, find that test loss L decreases increase the volume of data D and increase the fraction of high quality samples Q. Table shows us that using our pre-trained models to evaluate loss on unseen data also follows scaling law similar to our fit in distribution data, thus show- ing evidence for scaling generalization. To further illustrate trade-offs between and Q. Figure 2 presents iso-loss contours, showing how data quality improvements dataset size at fixed model capacity.
substitute for increased
Interestingly, the estimated exponents for data quality, 3, are significantly less than one: 0.173 for NMT and f R 0.401 for CLM (with Huber estimation). This indicates that the effective dataset size decays sublinearly with quality, ie.. models are more robust to moderate corruption than pre- dicted by simple effective sample-size theories from PAC learning or channel-capacity analysis, which typically suggest 2 1. We hypothesize that this robustness arises from redundancy natu- ral language data, where even partially corrupted samples carry useful contextual information (e.g., syntax, alignment, or co-occurring clean tokens). The higher y in CLM compared NMT suggests that autoregressive language modeling is more sensitive to token corruption, whereas NMT can leverage cross-sequence redundancy to mitigate the impact of noise. Another plausible hypothesis is that in the NMT noise model (padding half the tokens in noised sample), the corrupted samples are not completely useless: alignment and context still leak enough information. In CLM (token swaps), the corruption is harsher for distributional modeling, since the noised tokens add entropy and can mislead local dependencies. Hence a larger 7.
(a) Machine translation contours
(b) Next token prediction contours
Figure Iso loss contours for machine translation (a) and next token prediction (b).
To quantify the incremental loss due to degraded quality at fixed N Isolating the quality effect. and D, define
AL = L(N, D, ((N,D1))
(4)
From the fitted law, this difference takes the form
AL x BD- -Q-
which makes clear that AL( should grow approximately linearly Q 9 1, where & is the estimated 7. We therefore directly plot AL(Q) versus Q-+ using the estimated fy from our scaling experiments.
As shown in Figure 3 the resulting plots are close to linear across the tested quality levels, validating that the multiplicative Q-7 term accurately captures the degradation due corruption. Furthermore, the plots show homogeneous linear functions, which suggest that the additive terms in our proposed scaling law, No + E, indeed do not vary with data quality Q. The stability of these plots across dataset sizes (O.1B, 1B, 10B for CLM; 0.5M, 1M, 2M pairs for NMT) (with the Huber estimation) further suggests that y is an intrinsic task- deskddepndd parameter rather than an artifact of optimization or dataset sampling.
In practice, the exponent serves as a robustness index: smaller values indicate that a model-task pair is resilient corruption, while larger values reveal greater sensitivity data quality. In

---

Published as a conference paper at ICLR 2026
1.8
200
185
0.0
1432
3.20
655
3.10
0.35
0.2
(a) NMT, 7 0.173
(b) CLM, 0.400
Figure 3: (L(Q) Q for a) NMT and b) CLM, where 9 is the estimated y parameter using the Huber estimation from the scaling experiments.
experiments, both tasks show 7 < 1. which means loss increases more slowly than linearly with
decreasing Q. In other words, the model is robust to moderate amounts of corruption one needs large drop in Q to see noticeable rise in loss.
Robustness and generalization. We have included additional validations to verify the robustness of our findings (see Appendix(Efor full details). First, we confirmed that our synthetic noise model serves valid proxy for semantic degradation by observing strictly monotonic decrease in embedding similarity with increasing noise (Figure 4. We performed learning rate sweeps and found that is stable across practical learning rate regimes (Table 9). We provide practical guide for estimating Q in Section E.3
SUMMARY AND CONCLUSION
This paper revisits scaling laws for large language model pretraining by introducing an explicit and dimensionless measure of data quality, alongside traditional factors like model size and dataset volume. We propose new quality-aware scaling law that extends the widely used Chinchilla frame- work, predicting pretraining loss joint function of model size, data volume, and data quality. By systematically injecting synthetic noise in neural machine translation and causal language modeling experiments, we demonstrate that higher data quality leads significantly lower loss for given model size and dataset. Notably, our findings show that for high-quality datasets, smaller models and less compute needed achieve strong results, which is highly relevant for domain-specific applications. The study provides a formal scaling law:
A + D&QT +
L(NDD.0) = No
where Q measures data quality and is empirically estimated. This extends earlier work that pri- marily focused on model and data size. Our experiments on both NMT and CLM tasks, using real datasets with systematically controlled levels of data corruption, confirm the predictive power of the new scaling law and offer concrete guidance for balancing data curation versus model scale. The work establishes unified framework that quantifies data quality's influence, enabling principled decisions for specialized LLM development in fields building smaller models over domain specific datasets, such those that arise in business, scientific and medical applications.
REPRODUCIBILITY STATEMENT
We include all results from our experimental runs, all model and training configurations and recipes and all steps taken to produce this work in the relevant sections in main and the appendix.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported. in part, funded by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health (NIH). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the NIH/HHLBI. We also acknowledge the University of Chicago's Research Computing Center for their support of this work.
REFERENCES
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-223122 2022.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine learning, 2(4):343-370, 1988.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, (21(27):2211178122) 2024.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel
Mikel Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, et al. Paracrawl: Web-scale ac- quisition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020). Association for Computational Linguistics (ACL), 2020.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning, pages 1466-14822 PMLR, 2022.
Carlo Batini, Cinzia Cappiello, Chiara Francalanci, and Andrea Maurino. Methodologies for data
quality assessment and improvement. ACM computing surveys (CSUR), 41(3):1-52, 2009.
Yuheng Bu, Shaofeng Zou, and Venugopal Veeravalli. Tightening mutual information- baase bounds generalization error. IEEE Journal Selected Areas in Information Theory, 1(1): 121-130, 2020.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, 2023.
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, and Vikas Chandra. Scaling constraied language models with quality data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 80-97, 2024.
Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, and Jingang Wang. Revisiting scaling laws for language models: The role of data quality and training strategies. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23881 13881-233 1-23899, 2025.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2 edition, 2006. ISBN 978-0-471-24195-99
doi: 10.1002/0471748822.
Tri Dao. Flashattention 2: Faster attention with better parallelism and work partitioning, 2023. URL
https://arxiie wwtp://////a/b///oooa//aa///// 08691
William Edwards Deming. Some theory of sampling. Courier Corporation, 1966.
Jesse Dodge, Maarten Sap, Ana Marasovié, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08754 2021.
Christopher Fox, Anany Levitin, and Thomas Redman. The notion of data and its quality dimen- sions. Information processing & management, 30(1):9-19, 1994.
11

---

Published as a conference paper at ICLR 2026
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Skyler Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.00027, 2020.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021.
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and Zico Kolter. Scaling laws for data filtering-data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22702-2271 22711, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models, 2024. htthtt http///avvvvv/va///iv/ / abs 2407.21782
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodol- ogy and distribution, pages 492-518. Springer, 1992.
ISO/IEC
Systems and software engineering - systems and software quality requirements and evaluation (square) - data quality model. Technical Report ISO/IEC 25012:2008, International Organization for Standardization, Geneva, Switzerland, 2008.
Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 7354-7371, 2022.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv: 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 2020.
Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM), 45(6):983-0066, 1998.
Katherine Lee. Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (ACL), pages 8424 8445, 2022.
Ruihang Li, Yixuan Wei, Miaosen Zhang. Nenghai Yu, Han Hu, and Houwen Peng. Scalingfilter: Assessing data quality through inverse utilization of scaling laws. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3202-3222. Association for Computational Linguistics, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv. http///bb/11111...11 05101
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.082. 08242, 2022.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
12

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ebtesam Almazrouei, Alizé Cappelli, Allal Ling, Louis Hamel, Brando Pannier, Timothée Lacroix, Hady Elsahar Chen, and Nikhil Goyal. Refinedweb: A large-scale clean web corpus for pretraining large language models. arXiv preprint arXiv:2306.011 16, 2023.
Fatimah Sidi, Payam Hassany Shariat Panahy, Lilly Suriani Affendey, Marzanah A Jabar, Hamidah Ibrahim, and Aida Mustapha. Data quality: survey of data quality dimensions. 2012 Inter- national Conference on Information Retrieval & Knowledge Management, pages 300-304. IEEE, 2012.
Luca Soldaini, Kyle Lo, Yi Luan, Eric Wallace, Michihiro Yasunaga, Jungo Kasai, Drahomira Her- mannova, Jesse Dodge, David Wadden, Quentin Lhoest, et al. Dolma: open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536 2022.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. Advances in Neural Information Processing Systems, 37:114147-1141 2024.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368-377, 1999.
Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, 2009, doi: 000100779988-0338779901111
Richard Y Wang and Diane M Strong. Beyond accuracy: What data quality means to data con- sumers. Journal of management information systems, 12(4):5-33, 1996.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ard natural language processing, 2020.
03771
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for Ilm problem-solving. In The Thirteenth International Conference on Learning Representations, 2025.
Aolin Xu and Maxim Raginsky. Information theoretic analysis of generalization capability of learn- ing algorithms. Advances in neural information processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1
PROOF OF LEMMA
Proof of Lemma / For the Gaussian location family, the log-likelihood of single observation is (()) 20 0)² + const, where 0 = f(x). Differentiating twice with respect to 0 and taking expectation over gives the Fisher information per sample:
8²
L (0) E
002 {
=
0²
By additivity of Fisher information across i.i.d. samples, In D D/o². Thus, the usable information equivalent to Deft = D. (1/o²) noise-free samples. We define the deficiency as
A = In r²
Q= of
where of baseline noise (i.e. noise for "clean" " data; wlog de == 1). this formulation, datasets
noisier than the reference (a² > og) have < 1, while "clean" datasets (a² o have Q 1.
The effective sample size then scales as
Doft D. Q²
and taking = recovers the classical D- SNR scaling.
A.2 PROOF oF LEMMA 2
Proof of Lemma For symmetric label noise with flip rate 7) the unbiased loss correction con- structs per-sample estimator (X((... Y) whose expectation matches the clean loss: E| X1
((f(X) For symmetric noise, the correction has the form i (1-1) =((()) y) ((()),1--)
1-29
Natarajan et al. 2013). This divides by (1 (1-24 2n), so the variance of inflates by factor (1 2n)* -2 relative to the clean loss. Uniform convergence or stability-based generalization bounds then inherit
/ Varle] dependence, implying excess-risk rate equivalent to having Der D (1 2n)
clean samples. Classic PAC analyses of RCN similarly show polynomial overheads governed by (1 2n) (e.g.. Angluin and Laird 1988)).
A.3 PROOF OF PROPOSITION II
Proof of Proposition By assumption (as in the Information Bottleneck principle (Tihbyy et al. 1999 ) the data-dependent loss scales as
Ln x ((x
8> 0.
Substituting the multiplicative reduction of usable information,
T(X: Z) p(Q) I(X;
obtain
LD x D-9 p(Q)~ 1 (Xx ~9
Since T(X; Z) does depend Q or D,
absorb it into the constant B. By the regular-
variation assumption, p(Q) x e Th as Q + 1, so
p(Q) P-8008
Absorbing -B into B and setting 7 Big obtain
B
LD D&Q
Adding the standard model-size contribution A/N and irreducible error floor yields
A
B
((ND,Q) ~
D&Q E.
No
which is the claimed quality-aware scaling law after reparameterizatioo of constants.
14

---

Published as a conference paper at ICLR 2026
B EXTENDED METHODS
B. TOKENIZER TRAINING
We pretrain BPE tokenizer for our experiments in Neural Machine translation from English to German. We use the HuggingFace Wolf al. (2020) tokenizers module with the following pipeline. We use NFKC normalization with Whitespace pre-tokenization with a 32000 vocabulary size and >, < /s > pad > sep the special tokens. Predictably < s > and < /s > are the bos and eos tokens respectively. < sep >> is used as the separator token to separate the source and target languages.
minimum frequeney of 2 and unk >
B.2 MODEL CONHIURATIINSS
We present the exact configurations that use for model definitions for each of the tasks. We use a GPT Neo model for NMT and Llama model for CLM. Table 4 and Table show the exact parameter definitions used for the models. We the official implementations exposed via HuggingFace transformers.
Variable
Value gelu new
activation function attention dropout attention types
0.1
[global, local]
classifier dropout
0.1
embed dropout hidden size
0.1
1024
initializer range
0.02
layer norm epsilon
1e-05
max position embeddings num heads num layers
260
8
8
resid dropout vocab size
0.1
32000
window size
256
Table 4: GPT Nco config used for NMT pretraining experiments
B.3
DATA PRE-PROCESSING
B.3.1
PARACRAWL V8
In preparing our dataset for machine translation from English German, used pretrained language detection model from FastText (Joulin et al. 2016) 1id.176. bin to filter the data as described 5.1.
B.3.2 C4
We use cleaned version of C4 by AllenAl Dodge al. (2021) available HuggingFace, they provide 5 splits, en, en. noclean,, en.noblocklist, realnewslike, and multilingual (mC4). We use the en variant in our experiments, it contains 364,868,89 samples in the train split and 364,608 samples in the validation split.
C EMPIRICAL RESULTS
C.I ADDITIONAL. EXPERIMENTAL DETAILS
Table 6 shows other hyperparameters and choices made during our training experiments. We also notably greedy dataset packing in NMT and make of aataCoollltooithlllttteeeee from
15

---

Published as a conference paper at ICLR 2026
Variable attention bias
Value
false
attention dropout head dim hidden act hidden size initializer range intermediate size max position embeddings
0.0
64
silu
512
0.02
2048
2048
mlp bias model type
false
Ilama
num attention heads
8
num hidden layers num key value heads pretraining tp rms norm eps rope scaling rope theta tie word embeddings vocab size
8
8
1
le-05
null
10000.0
true
128256
Table 5: Llama config used for CLM pretraining experiments
HuggingFace in CLM to train easily use Flash Attention 2 (Dao 2023 weight decay of 0.01 both cases.
Additionally we use
Table Training Details
TASK
D(avg)
Batch Size
Max LR
Grad Ace Steps
Warmup Ratio
NMT
3.6754244e+07
512
5e-4
8
0.2
NMT
7.3486728c+07
512
5e-4
8
0.2
NMT
1.4697256++08
1024
5e-4
8
0.2
CLM
1.0317308+008
48
le-3
0.1
CLM
L.029602c+009
48
le-3
8
0.1
CLM
1.0294779c+10
48
le-3
8
0.1
C.2 ADDITIONAL RESULTS
In reporting our performance we evaluate our trained models on test loss on heldout datasets. In NMT, during dataset prepartion we partition test set of 50,000 samples and report evaluation loss on this heldout. For CLM, we partition 40,000 samples from the validation set of C4 en and use that for evaluation. The full results shown in Table 7an Table for reference.
C.3 PARAMETRIC Loss FITTING WITH HUBER Loss
Similar to Hoffmann et al. 2022) we minimize the following objective following a regularization
introduced by Huber 1992
Huber; [SSE(b (b- Bog D, log e) log Li)
min B,e.B.y Run
where = exp (b), exp (e) and 10 using the L-BFGS B solver available through SciPy minimize. We replicate the grid initialization of parameters follows, 5 [0, 25, 5), e [0, 2, 0.5], [0.0, 0.4, 0.1] and y E [0.0, 0.4, 0.1]. We also add bounds < 3 < 1 and 0 < 7 S 1. We also report secondary fit using SciPy inbuilt function curve fitting functions using Least Squares.
16

---

Published as a conference paper at ICLR 2026
Table 7: Neural Machine Translation Results. D is the number of tokens used for training Q represents dataset quality and the test loss nats. The horizontal lines demarcate the replicate experiments and vertical sections the different dataset sizes used.
D
Q
L
D
36779001
1.00
1.853
73479862
1.00
1.565
146952084
1.00
1.351
36779001
0.90
1,859
73479862
0.90
1.596
146952084
0.90
1.370
36779001
0.80
1,914
73479862
0.80
1.629
146952084
0.80
1.377
36779001
0.75
1.949
73479862
0.75
1.662
146952084
0.75
1.393
36779001
0.70
1.964
73479862
0.70
1.672
146952084
0.70
1.399
36779001
0.60
2.018
73479862
0.60
1.715
146952084
0.60
1,434
36779001
0.50
2.117
73479862
0.50
1.799
146952084
0.50
1.468
36734209
1.00
1.842
73487922
1.00
1.572
147005794
1.00
1.358
36734209
0.90
1.866
73487922
0.90
1.602
147005794
0.90
1.359
36734209
0.80
1,941
73487922
0.80
1.631
147005794
0.80
1.384
36734209
0.75
1.952
73487922
0.75
1.655
147005794
0.75
1.386
36734209
0.70
1.951
73487922
0.70
1.688
147005794
0.70
1,403
36734209
0.60
2.025
73487922
0.60
1.722
147005794
0.60
1.432
36734209
0.50
2.087
73487922
0.50
1.791
147005794
0.50
1.469
36749529
1.00
1.853
73492417
1.00
1.576
146959769
1.00
1.360
36749529
0.90
1.891
73492417
0.90
1.607
146959769
0.90
1.362
36749529
0.80
1,927
73492417
0.80
1.626
146959769
0.80
1.388
36749529
0.75
1.955
73492417
0.75
1.662
146959769
0.75
1.394
36749529
0.70
1.954
73492417
0.70
1.673
146959769
0.70
1.412
36749529
0.60
2.024
73492417
0.60
1.715
146959769
0.60
1.437
36749529
0.50
2.082
73492417
0.50
1.791
146959769
0.50
1.462
17

---

Published as a conference paper at ICLR 2026
Table 8: Causal Language Modeling Results, D is the number of tokens used for training Q rep- resents dataset quality and the test loss nats. The horizontal lines demarcate the replicate experiments and vertical sections the different dataset sizes used.
D
L
103068758
1.00
4,401
1029888 156
1.00
3.824
10295030326
1.00
3.581
103068758
0.90
4.447
1029888156 156
0.90
3.846
10295030326
0.90
3.592
103068758
0.80
4.520
1029888155
0.80
3,867
10295030326
0.80
3.612
103068758
0.75
4.534
1029888 156
0.75
3.876
10295030326
0.75
3,621
103068758
0.70
4.566
1029888 156
0.70
3.886
10295030326
0.70
3.633
103068758
0.60
4.630
1029888 156
0.60
3.917
10295030326
0.60
3.649
103068758
0.50
4.701
1029888 156
0.50
3.957
10295030326
0.50
3.673
103036697
1.00
4.402
1029558
1.00
3.825
10292963774
1.00
3.584
103036697
0.90
4.472
1029558 108
0.90
3.841
10292963774
0.90
3.592
103036697
0.80
4.510
1029558 108
0.80
3,866
10292963774
0.80
3.607
103036697
0.75
4.550
1029558 108
0.75
3.880
10292963774
0.75
3,617
103036697
0.70
4.560
1029558 108
0.70
3.886
10292963774
0.70
3.629
103036697
0.60
4.628
1029558 108
0.60
3.921
10292963774
0.60
3.650
103036697
0.50
4.710
1029558 108
0.50
3.950
10292963774
0.50
3,668
103413788
1.00
4.407
1029362179
1.00
3.824
10296342093
1.00
3.581
103413788
0.90
4.464
1029362179
0.90
3.837
10296342093
0.90
3.589
103413788
0.80
4.502
1029362179
0.80
3,868
10296342093
0.80
3.602
103413788
0.75
4.523
1029362179
0.75
3.878
10296342093
0.75
3.614
103413788
0.70
4.557
1029362179
0.70
3.887
10296342093
0.70
3.627
103413788
0.60
4.627
1029362179
0.60
3.920
10296342093
0.60
3.646
103413788
0.50
4.693
1029362179
0.50
3.952
10296342093
0.50
3.667
18

---

Published as a conference paper at ICLR 2026
D RELATIONSHIP TO OTHER DEFINITIONS OF DATA QUALITY
In this section, we show that our quality aware scaling law recovers the scaling law in Chen et al. 2025 by assuming that the data deficiency (w)) for a dataset w has the following form:
1 A(w) = F1 + F MgG
(5)
where E is the total noise, is the coverage scaled between 0 (minimum coverage) and (maximum coverage), and G quantifies the amount of repeated data.
We assume that density as defined in Chen et al. (2025), which incorporates both coverage and repeated data, is quantified such that low density associated with high quality data and high density is associated with lower quality data.
Let p be the density of dataset defined in Chen et al. 2025). Equation with the density assumed to satisfy 0 S p < 1. Here low close to 0 corresponds to higher quality data with fewer repetitions, and high p close
corresponds to low quality data with more repetitions, leading to sub-scaling behavior Chen et al. (2025).
We define the data dispersion to be
Dis(p) = - - p).
so as approaches 1 (low quality data in the sense of et al. (2025)), Dis approaches oo, and as o approaches 0 (high quality data in the sense of Chen et (2025)), Dis approaches 1.
In the equation above, we assume that the data dispersion Dis captures the coverage 1/F of the data and density p. We also assume that the Dis captures the amount of repeated data, although will see below, this can also be modeled in more precise ways. If we fix the noise E, then the data deficiency
(w) = constant + u2 (is(ppp)
(6)
and recover the data quality scaling law in Chen et al. (002)). In particular, the density P approaches 0, the data quality factor Q approaches 1, and as the density approaches 1, the data quality degrades.
More generally, we can define a modified data dispersion
Dis (p)
In(1 p²))
Turning towards the data quality scaling law in Chang al. (2024), if instead of using the cluster based definition of density, we use the compression based definition of density, we get
CR(W) = size(w)/ /coppeesed size(w)
DR(w) CR ((R'((W)
We define dispersion as:
Dis(p) In(1 DR(w)).
With this definition of Dis(w), the rest of the argument is the same.
In similar fashion, can add term !/ S(w) the data deficiency (ww) capture the increase in data deficiency due to the negative impact of synthetic data, which we can quantify with term follows
(w = pn + 12 G3G + H
For example, Chang al. 2024 measures S(w) using the perplexity measure.
19

---

Published as a conference paper at ICLR 2026
Turning to Goyal et al. (202)), define notion of the declining value of data the number of training epochs increases (and hence the amount of repeated data increases). We can define an additive term in the data deficiency follows:
k
P3
where k is the number of epochs. With this form, for small number of epochs, the loss decreases, but as the number of epochs increases, the loss begins to grow exponentially, and then levels out, with the rate determined by "decay parameter" 7.
E ADDITIONAL VALIDATIONS AND PRACTICAL GUIDE
In this section, we present additional experiments validating our noise model and scaling law stabil- ity, along with practical guide for estimating Q.
E.1 VALIDATION SYNTHETIC NOISE MODEL
To validate that our synthetic noise injection serves meaningful proxy for real-world data degra- dation, we measured the semantic drift of the corrupted data. We computed the average cosine similarity between the embeddings of clean samples and their synthetically corrupted counterparts using a pre-traine pre-trained reference model. As shown in Figure 4 there is a strictly monotonic, near-linear increase in embedding similarity as the synthetic quality increase (noise level decrease). This con- firms that our parameter Q effectively captures the loss semantic information.
1.00
Jaarag of Samples
0.90
0.991
0.98
0.981
0.977
0.97
0.922
0.963
0.5
0.5
0.7
0.0
0.9
Dataoet Quality.
Figure Semantic drift analysis: Average cosine similarity between embeddings of clean and corrupted samples at various quality levels.
E.2 SENSITIVITY TO LEARNING RATE
We performed sensitivity analysis by retraining models at 5 different learning rates (1e-5 5e-3) across 3 dataset scales and noise levels. As shown in Table|Pand Figure the estimated is highly stable (~> 0.39) for practical learning rates near the optimal Ie-3 used our main experiments.
E.3 PRACTITIONER'S 's GUIDE TO ESTIMATING Q
We propose the following recipe for estimating Q in practice:
1. Establish baseline: Identify a small, high-quality reference subset (e.g., manually eu- rated text or a trusted domain corpus) to represent Q ~ 1.
2. Select quality proxies: Choose measurable proxies for "deficiency" relevant to the do- main.
20

---

Published as a conference paper at ICLR 2026
Table 9: Sensitivity the estimated quality exponent y learning rate.
Learning Rate
Estimated
le-5
0.037 (Undertrained)
3e-4
0.394
1e-3 (Ours)
0.396
5e-3
0.177 (Unstable)
yww ER
1.
Se-4
160
S63
0.4
(a) Y vs learning rate
(b) Test loss vs learning rate
Figure 5: Sensitivity analysis of learning rate: (a) Estimated y across different learning rates, (b) Learning rate vs test loss for different batch sizes.
3. Compute deficiency (A): Construct composite deficiency score A = E, ررمر5 where
Mj are weights determined by importance or preliminary ablation.
4. Map to Q: interval.
Calculate Q = A This maps the unbounded deficiency metric to the (0, 1]
5. Sanity check: Verify that Q correlates with downstream performance or semantic drift (as in our embedding experiment) on small validation set.
6. Fit scaling law: Use the estimated Q values for different data subsets to the parameters
(B, 7) of the scaling law L = B/DDQ**) + E (assuming N is fixed or large enough), using the robust Huber loss method described in the paper.
21


Reviews:
[{"review_id": "rHXmeroK6X", "reviewer": "Reviewer_JQ1m", "summary": "The authors use a definition of data quality to include in the chinchilla scaling laws that consider model size and data size. They motivate this data quality notion and formulation with an effective data size formulation and run experiments with an approximately 140M parameter model.\n\nThey train 3 scales with 7 corruption levels for an NMT task and a CLM task. The text corruptions are synthetically induced by either replacing 50% of non-special tokens with pad tokens or randomly swapping 50% of tokens with other random tokens from the vocab. \n\nThe authors show that they can predict test loss with this new scaling plot with additional findings on how effective data size interacts with data quality. \n\nThe paper formulates the problem well, comes up with a good extension to scaling laws that is both simple and effective. \n\nHowever the experimental setup can be better presented or even be extended to support the core claim of the paper.", "questions": "Is there a typo in definition 2? \"data deficiency data quality Q(ω)\" \n\nDo you expect your findings to generalize over your experimental setup? Particularly considering the lack of any experiments with model size how confident should we be in the estimates for the scaling laws.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents data quality scaling laws which is simple and empirically supported.\n\nThe presentation is extensive and very good.\n\nThe data subsampling scheme in the experiments as well as the high number of Q values in experiments is quite good.", "weaknesses": "The experimental setup doesn't consider scaling the N value(model parameters) the paper works with a fixed model size however the scaling law has all three components and one is actually not incorporated in the experimental setup. Yet the paper makes claims about \"smaller models and less compute is needed to achieve strong results\" \n\nThe noise injection method is unnatural. A more natural noise injection strategy could be actually replacing a certain % of documents with low quality web documents. This could create a better noise injection method and make results more relevant. \n\nThe choice to swap with pad in NMT and other tokens in CLM, as well as to use GPT type base model for one and LLama base model for the other seems interesting, particularly when keeping variables constant could help better pinpoint the effect of cross lingual redundancy. Now the effect is conflated with different noise injection strategies as well. \n\nNo sensitivity or ci's are presented. Particularly considering the difference between NMT and CLM gamma values a sensitivity analysis on the experimental design could be useful in convincing readers to the results. \n\nRevisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies (Chen et al, 2025) seems to be published online in January but the authors have not cited the work which should really be included even contrasted in their paper. The underlying question is exactly the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUB4wSA5bD", "reviewer": "Reviewer_fb9g", "summary": "This paper extends classical LLM scaling laws by incorporating data quality alongside model size and data volume. It introduces a dimensionless quality parameter $Q$ and derives a quality-aware, Chinchilla-style law from effective sample size and information-theoretic arguments. Relative to the original Chinchilla formulation, the data term $D^{\\beta}$ is additionally modulated by $Q^{\\gamma}$.\nThe authors validate the approach with controlled experiments in neural machine translation (ParaCrawl) and causal language modeling (C4), varying dataset size and injecting synthetic noise across seven quality levels. To generate noise, they pad 50% of tokens in the NMT setting and replace 50% of tokens with random tokens in the language-modeling setting. Test loss scales predictably under the proposed law.\nEmpirically, higher-quality data can substitute for larger models or greater compute, and the estimated robustness exponents are sublinear, indicating resilience to moderate corruption. Noise is less detrimental for NMT than for language modeling. The paper also provides iso-loss trade-off contours and out-of-sample validations, offering practical guidance on balancing data-curation effort against model scale—particularly relevant in specialized domains with scarce but high-quality corpora.", "questions": "It is unclear how Q should be estimated in practical settings. Could you provide a step-by-step guide for practitioners on applying the proposed method to non-standard datasets, including handling multiple noise sources?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\nThe paper proposes an original method for measuring data quality. To the best of my knowledge, the information-theoretic approach and its assumptions are new.\n\nQuality\nThe paper presents small-scale but thorough experiments. It includes carefully controlled studies in both CLM and NMT—with nested dataset sizes, seven corruption levels, and single-factor manipulations to isolate quality effects.\n\nClarity\nThe text, derivations, and presentation are clear.\n\nSignificance\nThis work can simplify practice in domains with scarce data and limited GPU budgets. It helps practitioners decide whether to invest in data quality/size or compute. It also provides a framework for working with token-level corruption in a more principled way.", "weaknesses": "Estimating Q. It is unclear how to estimate Q in practical settings for new datasets with multiple noise sources.\n\nScale of experiments. The experiments are small-scale relative to current SOTA setups: the authors use a 133M-parameter model for NMT and a small 8-layer Llama-3 variant (the parameter count is not reported in the paper).\n\nComparative evaluation. There is no direct comparison of the proposed law’s quality against competing methods. It would be helpful to include an error analysis across alternative approaches using the experimental data, or to explain why such comparisons are not applicable / are out of scope.\n\nRelated work referenced.\n\n“Scaling Laws for Data Filtering—Data Curation Cannot Be Compute-Agnostic.”\n\n“Scaling Parameter-Constrained Language Models with Quality Data.”\n\nAdditional, not previously mentioned work: https://aclanthology.org/2025.acl-long.1163/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P17n6r6jOK", "reviewer": "Reviewer_BKxM", "summary": "The paper revisits the scaling laws of large language model (LLM) pretraining by introducing an explicit, dimensionless data quality parameter Q, extending the traditional analysis of model size and data volume to a joint framework that incorporates data quality. The authors propose a quality-aware scaling law. They systematically inject synthetic noise and vary data coverage in neural machine translation and causal language modeling tasks. Experimental results show that high-quality data can significantly reduce loss for a given model size, and under high-quality data conditions, smaller models with less computational resources can achieve strong performance.", "questions": "Please refer to the relevant points in the Weaknesses section. If the authors can provide clarification and improvements, I would be very happy to raise my score.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work innovatively incorporates data quality as a single parameter into the scaling law, theoretically demonstrating a strong correlation between model performance and data quality.\n2. Controlled experiments are conducted across multiple experimental settings, validating the practical applicability of the quality-aware scaling law.", "weaknesses": "1. Although synthetic noise offers strong controllability, it may not capture the highly complex and heavy-tailed distribution of low-quality data in real training datasets. How can the current single-parameter quantification method be extended to more realistic data scenarios?\n\n2. The authors use a fixed learning rate to investigate the scaling law. During training, could different learning rates have additional effects on the conclusions regarding the data-quality scaling law?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lqUNGeOGG4", "reviewer": "Reviewer_4JDP", "summary": "This paper introduces an LLM pretraining scaling law that accounts for data quality. This takes the form of calculating a dimensionless data-quality parameter and modifying the standard Chinchilla scaling law. The paper studies two different formulations of data quality and provides experiments on neural machine translation and language modeling that empirically validate the proposed scaling law with synthetically noised datasets.", "questions": "1. Can you elaborate on the relationship between your scaling law and the  heterogenous quality scaling laws in [1]?\n\n2. The setting of Q=1 (the highest data quality) to recover the standard Chinchilla scaling law (line 250) does not make the most sense to me. I could imagine scenarios where we would want to be able to account for having \"higher quality\" data than that used for the Chinchilla law.\n\nSmall comments:\n- Line 289/290: \"(Table 2\" -> \"(Table 2)\"\n- Is there any meaning behind using $E^+$ in Tables 1, 2 and Section 5.6 instead of just E like in Equation 2?\n- Line 355: N is used for noise percent, but doesn't N already mean model size?\n- Line 471: \"Notably, their findings\" -> \"Notably, our findings\"\n- Line 477: \"Their experiments on both\" -> \"Our experiments on both\"\n\n[1]: Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic, Goyal et al., CVPR 2024.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a theoretically motivated way to incorporate a dimensionless data quality term into a Chinchilla-style scaling law.\n2. The experimental setup is sound, and the experiments show that the proposed scaling law fits LLM training in practice.\n3. The proposed scaling laws also enable studying how data quality impacts performance, as with the observation that \"models are more robust to moderate corruption than predicted by simple effective sample-size theories from PAC learning or channel-capacity analysis\" (lines 374-377).", "weaknesses": "The main weakness of the paper is that it studies limited forms of data quality that do not correspond to the kind of quality that is used in practice. In practice, pretraining data quality is often estimated with classifiers (such as estimating the degree to which a given document resembles manually curated high-quality data) or heuristics (such as removing documents with many repeated n-grams). The current experiments are limited to synthetically noised datasets. Do either of the proposed quality formulations capture this more nuanced form of data quality?\n\nI am willing to raise my score if either 1) further experiments can show that the current formulations of quality do in fact capture more practical notions of quality or 2) more reasoning is provided to show the studied kinds of noise are  practically useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)