Improving Pretraining Data Using Perplexity Correlations
Authors: Tristan Thrush
Christopher Potts
Tatsunori Hashimoto
Venue: ICLR 2025 Poster (2025)
Keywords: pretraining data; data selection; natural language processing; statistics; large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: We present new lightweight statistical methods for pretraining data selection. Our paradigm requires no LLM training, matches the best known existing method in our experiments, and has some nice theory.

Abstract: Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own.  Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier. We have now also updated this paper to include results from preregistered experiments with new pretraining data on an aggregation of 22 benchmarks up to the 1.4B scale, showing increasing improvements of our method over others with more scale. A pip package with full documentation can be found here: https://github.com/TristanThrush/perplexity-correlations.

Published as a conference paper at ICLR 2025
IMPROVING PRETRAINING DATA USING PERPLEXITY CORRELATIONS
Tristan Thrush, Christopher Potts & Tatsunori Hashimoto
Department of Computer Science
Stanford University
Stanford, CA 94305, USA
(tthrush, cgpotts, thashim)estanford.co
ABSTRACT
Quality pretraining data is often seen as the key to high-performance language models. However, progress understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based a simple observation: LLM losses on many pretraining texts are correlated with down- stream benchmark performance, and selecting high-correlation documents is effective pretraining data selection method. We build a new statistical frame- work for data selection centered around estimates of perplexit-benchhmark cor- relations and perform data selection using a sample of 90 L.LMs taken from the Open L.M Leaderboard texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 bench- marks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram clas- sifier. We have now also updated this paper to include results from preregistered experiments with new pretraining data to the 1.4B scale, showing increasing improvements of our method over others with more scale. A pip package with full documentation can be found here: http:/twwww.t...tttaaaaeeeeeeeee
aggregation of 22 benchmarks up
INTRODUCTION
Dataset curation is increasingly crucial for training high-quality large language models (LLMs). As pretraining datasets have grown, from under 200B tokens in 2020 (Raffel et al., 2020; Gao et al., 2020) to 240T tokens today (Li et al., 2024), it has become critical identify subsets of the available data that will lead to the best LLMs, and a wide range of methods have arisen to meet these needs (llyas et al., 2022; Xie al., 2023a;b; Engstrom al., 2024; Everaert & Potts, 2024; Liu al., 2024; Llama Team, 2024). However, data-driven approaches to data selection typically involve expensive model retraining steps that limit their effectiveness, and no algorithm has been reported to consistently beat or match hand-crafted classifiers for data selection (Li et al., 2024).
training new LLMs necessary for data selection? Instead of training our own models, can we use the growing collection of publicly available, high-peeformance LLMSs (Wolf et al., 2019; Beeching al., 2023) to perform data valuation and selection? This would have significant benefits: we could leverage the millions of dollars collectively spent on building these LLMs, and would have coverage over a large, heterogeneous collection of perforreform models varying in size, architectures, and pretraining data distribution.
Despite these advantages, using existing models for pretraining data selection is challenging, as the training data for these models often unknown and heterogeneous. Our key observation is that data selection can be done using two observable features all public models today: 1) all open- weight models produce causal language modeling loss for given text, and 2) all of them can be

---

Published as a conference paper at ICLR 2025
Domains
Benchmark
bbc anxiv ...
willys-hifi
SciQ
High Corr (Keep) arxiv, bbc.
Mistol
Correlations
LLMS
Libma
bbc
willys-hif
Mamba
Pytii
Low Corr (Discard) willys-hifi
logeoo
soquraay
Figure 1: We pretrain on domains where lower loss generally correlated with higher downstream performance. Our approach does this by taking public, pretrained LLMs and measuring correlations across their log-liken lieelioos (left, red matrix) and performance on a target benchmark (center, blue vector). We then perform data selection by training fastText classifier that distinguishes high cor- relation domains from others. This approach on par with the best-known data selection methods in our experiments, despite requiring no human selection of high-quality domains.
evaluated on benchmarks. Prior work has found systematic relationships between web corpus loss and benchmark performance (Wei et al., 2022; Huang et al., 2024), which suggests the possibility of using correlations between perplexity and benchmark scores as the basis for a data selection policy.
In the present paper, we pursue this possibility and find radically simple approach that is also effective: we select data via perplexity correlations (Figure 1), where we select data domains (c.g. wikipedia.org, stackoverflow.co womm, etc.) for which LLM bog-probabilities are highly correlated with downstream benchmark performance. To enable our approach, we complement our algorithm with statistical framework for correlation-base data selection and derive correlation estimators that perform well over our heterogeneous collection of LLMs.
We validate our approach using a collection of pretrained causal LLMSs on the Hugging Face Open L.M Leaderboard (Beeching et al., 2023) and find that perplexity correlations are predictive of an LLM's benchmark performance. Importantly,
find that these relationships are robust enough to enable reliable data selection that targets downstream benchmarks. In controlled pretraining experi- ments the 160M parameter scale on eight benchmarks, our approach strongly outperforms DSIR (Xie et al., 2023b) (a popular training free data selection approach based on n-gram statistics) while generally matching the performance of the best method validated scale by Li et al. (the OH-2.5 +ELI5 fastText classifier; Joulin et al. 2016) without any parameter tuning human curation. In followup experiments at the 160M to 1.4B parameter scale which we pre-registered, our approach outperforms the best Li et al. filter on the main benchmark from their paper (an aggregate of 22 benchmarks) when filtering from their base data pool, and both approaches remain close to each other when filtering from their extensively pre-filtered pool. We further find that the performance of our approach strengthens with increasing scale.
RELATED WORK
To go beyond the status quo of deduplication, perplexity filtering, and hand-curation (Laurençon al., 2022; BigScience, 2023; Marion al., 2023; Abbas al., 2023; Groeneveld al., 2024; Soldaini et al., 2024; Penedo et al., 2024; Llama Team, 2024), targeted methods have been proposed filter pretraining data so that the resulting LLM will achieve higher scores on given benchmarks. There are lightweight approaches that use n-gram overlap (Xie et al., 2023b) or embedding similarity (Everaert Potts, 2024) to select training data that is similar to data from given benchmark. There also less-scalable methods that require training proxy LLMs different data mixtures (Ilya et al., 2022; Xie et al., 2023a; Engstrom et al., 2024; Liu et al., 2024; Llama Team, 2024).
Given the high costs of proxy-based data selection methods, they have primarily been used to select among human-curated pretraining data mixtures (Llama Team, 2024; Li et al., 2024) rather than high dimensional space of mixtures. Our work takes orthogonal approach and builds upon recent observational studies that have found scaling relationships that hold across collections of uncontrolled and diverse L.LMs (Owen, 2024; Ruan et al., 2024). While these studies do not examine
2

---

Published as a conference paper at ICLR 2025
loss-to-pe -o-------rffoancee relationships or derive useful data selection methods from them, we know that losses and performance are generally highly correlated. Validation losses on samples of text corpora commonly used as a proxy for downstream performance when comparing LLMs pretrained on the same data distribution (Kaplan et al., 2020; Hoffmann et al., 2022; Wei et al., 2022), even if they have different architectures (Poli al., 2023; Peng et al., 2023; Gu & Dao, 2024).
According to a recent survey of data selection approaches by Li et al. (2024), the heavier-weight pretraining data selection methods have not shown large gains, and the current state-of-the-arr across many tasks is primitive: a fixed fastText classifier (Joulin et al., 2016) combined with an English filter as final layer after extensive deduplication and filtering. Are we missing important information that we can efficiently extract from diverse collection of already trained models, larger and more diverse than any single organization is likely to produce? We show evidence supporting this hypothesis simple doss-performane correlation coefficients are effective when used for data selection.
3 PROBLEM SETTING
Our goal is to build predictive models of how pretraining data distributions affect downstream bench- mark performance and use them to build better language models. Unfortunately, this task is challeng- ing and computationally expensive. A standard approach adopted in paradigms such datamodel- ing (Ilyas et al., 2022) is to obtain N different pretraining distributions {p: E [N], Ps E <R²²) over D >> N domains (e.g. arxiv.org, stackoverflow.com etc.), pretrain and measure model errors on a target benchmark runs, performed at a scale sufficient to obtain non-random performance on y. This can cost tens to hundreds of millions of dollars for hard benchmarks such as MMLU, where even the performance of IB parameter LLMs often does not exceed random chance (Beeching et al., 2023).
E [0. 1), and fit a model + y. This approach requires N LLM training
Instead, our work considers the following observational setting that requires no training. We obtain N pretrained, high-performance L.LMs that vary in pretraining data, tokenizer, architecture, and scale (i.e. models on Hugging Face's OpenLLM leaderboard). If we could train predictor - on these N models, we could avoid large scale model training. Unfortunately, this is impossible as the training data for these models is often proprictary, and so we have no knowledge of p.
The key observation of our work is that we can replace pi.j (the unobserved sampling probability of model i's data selection policy on document j) with an observable surrogate Ti,j. which is the nega- tive lilkiiikehodd of document j under model i.! We can then build a regression model that relates negative log-likelihood x; and benchmark error ys- Using this model, we can select pretraining data from domains j for which decreasing the loss zij is predicted to rapidly decrease error yi-
The perplexity-pefformanc performance hypothesis. We formulate the task of predicting errors y, from nega- tive log- probabilities x; as single-index model (SIM),
y = -((0", + ee)
where f R. ++ R is some unknown monotonically increasing univariate function, is zero-mean noise which is independent of x, and 0* E RRD are unknown weights over D domains.
A single index model is highly flexible (due to the arbitrary, monotone f) and has the advantage that we do not need to estimate the nonlinear function f if our goal to optimize model performance. We can see this directly from the monotonicity of f as
(0" x:) + E
x5) + = s((0", x:) +e)) < 1((0", +g))
Data selection from perplexity correlations. The weights 0* tell us which domain perplexities are correlated with downstream performance. However, this isn't sufficient for data selection. Even if we know how model likelihoods relate to model performance, we do not know how data selec- tion affects likelihoods. Even worse, this data mixture to likelihood relationship cannot be learned observationally, as we do not know the data mixture of any of our models.
'To precise, we use bits-per-byte which normalizes the sequence negative ligeliill with the number of UTF-8 bytes. This is defined in terms of the length of the string in tokens Lr, the length of the string in UTF-8 bytes T.g, and the cross entropy loss e as BPB Lg lo(2)
L
3

---

Published as a conference paper at ICLR 2025
Despite this, we show that there is a clean approach for optimizing the data mixture. Our observation is the following: if we find nonnegative 8*, sampling proportional to 0 is always a good choice. More formally, we see that this sampling distribution defines the pretraining loss such that optimizing the training loss directly optimizes the downstream task via the single index model.
Proposition 1 Suppose that 0* weights are non-negative. Then, for models with associated like- lihoods x E X C. RD, the minimizer of the pretraining loss over the 0" sampling distribution er [zy] also has the lowest expected downstream error according to the single index model:
arg min Ey [arg] arg min IE [[(00,, x) +
xEX
x
This observation follows directly from the fact that we can normalize any non-negative l into distribution (and shift the normalization constant into f) which allows us to write the inner product in the single-index model monotone function of the expected pretraining loss:
f((0", x) + e) = f(E [xy] +e).
(3)
Proposition I allows us to entirely avoid the task of finding the optimal data mixture for a target likelihood. Instead, we pick sampling distributions that make the pretraining loss monotone func- tion of the predicted downstream error. Afterward, we can rely on our ability to optimize the loss to optimize downstream performance.
This view gives us a straightforward roadmap for data selection in the remainder the paper: estimate set of domains where loss and downstream benchmark performance is highly correlated, and then constrain our 0* estimates to be a pretraining data sampling distribution.
METHODS
We now describe the details of our approach, starting by presenting the algorithm itself and the intuitions behind it, followed by a more precise and mathematical justification for the various steps.
4.1 ALGORITHM
Estimating 0*. The parameter 0 measures the relationship between log-likelihoods in domain j and downstream performance. 'Because of this, might naturally expect 07 to be related to nonlinear correlation coefficients between x and Our work uses a simple correlation measure,
7
Σ IERISS Aep
sign(yk ()arkn (zk,) (21,)
where ranky(x) the rank of x among {I1J ... NN,... This formula intuitive: when model does better than model I, what percentile is model k's log- lig-likeldhood compared to model l's? The functional form also has the benefit of being a principled estimate of 0* particular, we show in sections below that in expectation, the ranking of domains in exactly matches that of 0* (under standard high-dimensional regression assumptions; see Section 4.2 for a complete discussion). We note, though, that this is not the only correlation coefficient that performs well in our perplexity correlations toolkit (see Appendix H) and we prove that Spearman's 's rank correlation (Spearman, 1904) is principled in the same way (see Appendix C).
Selecting pretraining data. Suppose that we have an accurate estimate 7, which is nonnegative. In this case, we could use T directly as a data selection procedure and Proposition would ensure that minimizing the population pretraining loss minimizes downstream enrors. Unfortunately, "/ can be negative and the finite number of tokens per domain can make difficult to minimize the population pretraining loss. Thus, must project % onto the set of reasonable pretraining data distributions that are nonnegative and account for the per-domain token counts.
What is a good way to project set of domain rankings estimated via y into pretraining sampling distribution? Intuitively, wikipedia.org has a t = 0.5 and arxiv.org is ? = 0.9, it would be nat- ural to select tokens in order of 7,, preferring tokens from arxiv.org over tokens from wikipedia.org.
Having established the ordering of domains, the remaining question is how many tokens we take for each domain. We follow recent observations that repeating data degrades performance (Abbas al.,
4

---

Published as a conference paper at ICLR 2025
2023) to arrive at simple selection algorithm: select domains in greatest least 7, taking all the tokens in each domain once, until we exhaust our total pretraining token budget.
Full algorithm. Together, these steps result in a simple, parameter-free algorithm that calculates our rank correlation coefficient, and selects domains in order from largest to smallest coefficient. We show this process explicitly with pseudocode in Algorithm (see Appendix A), and additionally show extra step where we train fastText (Joulin et al., 2016) classifier (using standard settings and bigram features from Li et al. 2024) which distinguishes our selected documents and domains from the rest of the pool. The fastText classifier allows us to perform data selection at single- page level, and scale the selection process to larger datasets. We also found the classifier to slightly improve downstream performance over directly selecting the documents. More information on the specifics of the data selection approaches that we tested is given in Appendix G.
4.2 THEORY
We now study the approach closely and show that our choices for the correlation coefficient and projection step are extensions of the classic, high-dimensional single index model estimator of Plan et al. (2016). We describe the basic single-index model estimators first, describe our extensions, and then conclude with a discussion on how our estimator and results deviate from the theory. A discussion of other potential estimation paradigms is provided in Appendix E.
4.2.1 DIGG--IMENSIONAL ESTIMATION OF SINGLE INDEX MODELS
For our theory, we consider the standard high-dimensional regression setting of Plan et al. (2016) and Chen Banerjee (2017). Here, our goal is to estimate the unknown weights l in a single-index model = f((0", x,) +G), with x; ~ (O., for ||0*|e ll2 = 1 (assumed without loss of generality, as ||0*|e || can be absorbed by J).
Our starting point is the classic result of Plan et al. (2016), who showed
E [ = c0*
(4)
for some positive constant e and I S N. Closely related is the result of Chen & Banerjee (2017), who developed robust estimator quite similar to ours,
E [sign(y ()) x1)] 30"
for any 1 < kt, S N (where & + I) and some positive constant 3. Both of these results clearly iden- tify that for the high-ding dimenniiinii single-index model in the Gaussian setting, generalized correlation coefficients provide consistent estimates of the true regression coefficient 0".
4.2.2 DERIVING OUR ESTIMATOR
Both Plan et al. and Chen & Banerjee provide moment-matching style estimators that consistently recover 6* in high-dimensional, sparse settings. However, we found that both estimators directly the values of and this resulted in brittle estimates due to outliers in language model log- likelihoods. While outlier removal one possibility, we found that a simpler approach was to robustify the estimator of Chen & Banerjee (2017) outliers in x.
Recall that our estimate 7 is a U-statistic, defined as pairwise sums of
sign(y: y ( +xx,)),
(6)
for any < i,j < N (where * j). where Φ is the empirical CDF (effectively, normalized ranks) of the x values. This estimate is significantly less sensitive to outliers than that of Chen & Banerjee (2017), as the empirical CDF is bounded between zero and one.
We study this estimate theoretically in the Gaussian setting, where we consider the asymptotically equivalent estimator with & as the CDF of the standard Gaussian. In this case, we can show that this modified estimator is also consistent in recovering 0*
Theorem When e (0, )²), we have:
2 = sin -1
8"
E[sign(y: - (x)()(x;) - +xxx)))
(7)
π
221
5

---

Published as a conference paper at ICLR 2025
The proof is in Appendix B. Because assume ||0*|| and the expected value in Equation must be in [-1, 11. we are always in the domain of sin and can invert it. After inverting, we get:
x sin 2 [sign(ys Mj (x), +(x)))l)
as an estimate for 0", where the constant 2 x² term due noise has been dropped.
Beyond the fact that our estimator is consistent, we can show an even tighter connection to the Chen Banerjee estimator: our estimates agree when running the original estimator on tokk-tranffomedd data. More specifically, for two models x; and X, with the estimated model rankings (0, x.) >
(0, xz), the expected ranking under maantraasformation (i.e. (xx)) matches this ranking.
Corollary Suppose that θ is any vector fixed weights and (O, Then, conditioning on the event (0. x,) (0,xy), we have with probability 1 that:
0 E (x.) (0. x < (0,x,)) < (0, (0,x,) x,)
(0,x,)))
(9)
This proof follows from the same calculations as Theorem and is given in Appendix B.
4.2.3 SELECTING DATA FOR PRETRAINING
Recall that our algorithm for data selection is to constrain y to be valid sampling distribution and then sample directly from this estimate. For now, we focus constraining 0, and we will see at the end of this section that we can apply the same constraint to y directly to get the same result. The theory of constrained estimation for is simple and well-understood, with both Plan et al. (2016) and Chen & Banerjee (2017) extensively studying the problem of estimating Ô under known convex constraint set C. In particular. Plan et al. (2016) show that performing L2 projection via projj = arg mingec ||0 i la provides improved convergence rates that depend on the Gaussian mean width of C rather than the ambient dimension, and Chen & Banerjee (2017) show similar results when maximizing the linear correlation @rojj arg mingecC (0,
We take a similar approach here. We define a convex constraint set C' that forces ê to be a reasonable sampling distribution and find the best sampling distribution via the linear correlation approach.
We define C as the combination two sets of constraints. First, we must have a valid sampling distribution, so we constrain to lie in the simplex. As we noted above, it is well-known that duplicating data harms performance (Abbas et al., 2023), so we constrain 0 to avoid this by limiting the maximum weight on domains: if we will pretrain on tokens overall, we enforce 6 S Ti Vi [1, DJ. where is set so Tam is the number of tokens from the i-th domain accessible for training.
The resulting linear program has simple solution and takes the form of initializing @rmi to 0 and then iterating through the values in i from largest smallest, setting the value the corresponding index of Ooro) the maximum allowable value, until (pej sums to (see Appendix D for proof).
Theorem Suppose we want to solve:
govuj arg min (0, 0), PER
subject to:
D
Σ 0 =1
i=1
S 0, S T, Vi E D),
where Tú 0 are fixed values. Then, the solution is:
E T3
jomus 1 1 (1))) if Σ 25,,, T3 2 1 A A4 (1)))))))))))))))
TS1, $1, (10)
otherwise
where breaks ties between og and 0r for k + j; otherwise keeps ordinal relationships the same.

---

Published as a conference paper at ICLR 2025
We note that while the use of this linear program is in line with the constrained estimators proposed in Chen & Banerjee (2017), the L2 projection is arguably more natural, and does not require assum- ing that ||0||2 = for asymptotic recovery conditions. We derive similar closed-form expressions for this quadratic case in Appendix D, but do not use this approach for two separate reasons. First, the Lg projection depends on the L2 norm of ê, unlike the linear program which only depends on the ranks of the values in A. The challenge with determining the norm that the exact recovery result in Equation (7) requires knowledge of the noise level, and the trigonometric functions rely strongly on the Gaussian structure of x. Because of this, we are unlikely to be able to estimate the norm of 0 with any accuracy, and the only way to avoid this would be to treat the norm as hyperpa- rameter, which adds unnecessary complexity. The second reason is empirical (although possibly consequence of the first) - we found that the linear projection performed better across a wide range of benchmarks and conditions (see Appendix H).
We conclude by relating our theory to the full algorithm in Section 4.1. The estimation step for y the finite sample, U-estimate of the expectation in Equation (8), dropping the nonlinear transform sin and 7/2 as these two terms do not change the rankings of the domains. The data selection step directly applies our projection in Equation (10). and we make use of the fact that this projection only relies on rankings among the domains to use rather than an exact estimate for 0"
5 RESULTS
We first pretrain 160M-parameter LLMs from scratch study our primary goal of selecting pretrain- ing data to improve downstream performance, and then we present a brief overview of results on experiments which we preregistered in earlier version of this document (most experiments show a trend of even greater improvements at larger scales - up to 1.4B parameters on new benchmarks and data pools; more detail is in Appendices N and M). Finally, we present an analysis on the ability of losses to predict downstream performance. Throughout our experiments, we use single-index models trained using Algorithm 1. As shown in the algorithm, selected vs. unselected domains and use the classifier to filter the pretraining data at the page-level.
train the fastText classifier on
Input data matrix X. To build the input data matrix, X, we collected byte normalized loss values from a sample of 90 Open LLM Leaderboard (Beeching et al., 2023) LLMS that we could run without errors. Concretely, these values are defined as bits-per-byte Lg ln(2] where Lr is the token count, LB is the number of UTF-8 bytes, and é is the per-token cross-entropy (Gao et al., 2020). For our initial experiments, we collected these values on the "sample" subset" of the RedPajama V2 (RPJv2) dataset (Together Computer, 2023) for all domains with 2 25 pages in the sample. There are 9,841 domains/features. Specifics are in Appendix F. The data for the additional preregistered experiments is discussed in Appendix N. A detailed principal components analysis of X, which reveals a variety of salient embedded information in the losses, is in Appendix L.
Lrf
Target benchmark performance y. For our initial experiments, we constructed target vector, y, for LAMBADA (Paperno et al., 2016), ARC Easy (Clark et al., 2018), PIQA (Bisk et al., 2020), and SciQ (Welbl et al., 2017). These are all of the tasks reported in the Pythia scaling experiments for which a model in the 160M parameter range could meaningfully perform above chance. We also constructed target vectors for LAMBADATT LAMBADAFR. LAMBADADE, and LAMBADAES. which are subsets of LAMBADA translated into Italian, French, German, and Spanish by Black (2023). These languages match those in RPJv2, where each page is conveniently tagged as one of five languages: English, Spanish, French, German, and Italian. The correspondence between our target benchmark languages and the RPJv2 metadata allows us to easily include language filtering baselines. For the preregistered experiments, we use 22 more benchmarks (see Appendix N).
5.1 INITIAL PRETRAINING EXPERIMENTS
We begin by validating algorithm in the end-to-end task of pretraining data selection with con- trolled experiments at the 160M parameter, 3.2B token, scale. The low compute requirements of this setting allow to more extensively study replicates and ablations in Appendix H within the timeframe of a few days. While 160M models are small, this is far from an easy setting for our data

---

Published as a conference paper at ICLR 2025
Table 1: Average rankings of data selection methods (lower is better) for evals. Correlation-basee filtering beats baselines by a wide margin, and matches the current best open data filter from Li et al. (2024). Our approach significantly beats the default filter in Li et al. (2024) and loses slightly after additional manual language filtering that depends on the target task (+ manual Lang Filter).
Method
None
Lang Fill
DSTR (Nie al., 2023b)
Handerafted fastText
+ EN Lang Filker (Li al., 2024)
Handernited fae/eet
Lang Filler
Handerafied Teat
manual Lang Filler
Conclltions
Perplesity
Arg. Rank
3,750
4.000
4,500
3,750
3,250
1,375
1,750
selection algorithm. Most of the Open LLM Leaderboard models are 10 100x larger than the 160M scale, and our single index model must extrapolate substantially from 7BB scale models to our scall-scale validation setting (see Appendix K for a histogram of model sizes).
Pretraining data and setting, For the initial pretraining experiments, we used the "sample-100B" subset of RPJv2. This is much larger than the sample that we used to compute our estimate. We filtered this data so it contains only the domains used for our estimate, and then tokenized the data with the Pythia tokenizer. The vast majority of the domains from our BPB matrix were present in this larger sample of text. However, 42 (out of 9,841) were not, and so we removed them from our estimate. For every data selection method that we tested, the task was to further select 3.2B tokens for pretraining, which is Chinchilla-optimall (Hoffmann et al., 2022) for the 160M -or--maaeettrr LLM.
Baselines. We compare against several baseline data-selection methods. First, we present the results the language tags present
of uniformly sampling from the available pretraining data. Then in RPJv2 to filter only for the language matching the target task. In addition to these commonsense baselines, also run DSIR (Xie et al., 2023b), lightweight training data selection technique based on n-gram overlaps that Li et al. (2024) found to be competitive with proxy LLM-based techniques and was also validated scale (Parmar et al., 2024). Finally, we run the state-of-the-ar method for pretraining data quality filtering found by Li et al.. which is fastText classifier that beats all of the heavier-weight proxy-LLM methods tested. The classifier was trained on a manchmark-agnossee and handcrafted objective, which is to classify data as Common Crawl? (low quality) or OH2.5 (Teknium, 2023) and Reddit ELIS (Fan et al., 2019) (high quality).
combined with an English filter in Li et al.; we present results for this fastText filter with and without the English filter.
Model and hyperparameters. We use the Pythia 160M LLM configuration from Biderman et al. (2023) and optimize the hyperparameters, including learning rate, weight decay, and warmup, to minimize loss on the uniform sampling (no selection algorithm) baseline. Training hyperparameters were fixed across all methods. We provide additional training and evaluation details in Appendix G.
Results. We report average rankings over initial benchmarks in Table 1, and we find that our ap- proach significantly outperforms the basic baselines of random sampling, language filtering, and DSIR. Compared to the existing state of the art from Li et al. (2024), our approach beats the perfor- mance of the default, faglish-filtered fastText classifier, but loses slightly once we add in manual language filtering step to enable better performance on the multilingual LAMBADA datasets. For the maintext comparisons, use the optional fastText classifier from our algorithm to select pre- training data at the page levels, but we show ablations without the classifier in Appendix H.
Figure 2 shows how each data selection method affects benchmark performance in more detail. Each block of rows represents a data selection method, while an individual row represents an LLM within a method that targets a particular benchmark or set of benchmarks. Columns represent benchmarks. We see that language filtering and perplexity correlations both clearly optimize for the target bench- mark: within each block, the benchmark column matching each row typically performs best. The pattern is much less obvious for DSIR- - the heatmap looks more uniform across LLMs with different task targets. We also see that while language filtering has significant impacts on model performance, our performance significantly exceeds the impact of language filtering across all tested benchmarks. For some more analysis, including the types of domains selected, see Appendix O and J.
Finally, we note that our results are somewhat insensitive to the specifics of the cepplextt--corelattii procedure we present in Algorithm 1. We show in Appendix H that varying the projection method (linear, L2) and even using Spearman rank correlations (Spearman, 1904) often work better than the
pnttps://commonraatt
8

---

Published as a conference paper at ICLR 2025
Uniform RPJv2
None
21 30.92
00
20
19
17
23.85
011
90
95
EN Filtered
.6
57.04
64
5.78
72.92
60.80
59.09
67.10
DE Filtered
26.50
52.39
30.10
57.72
24.63
00.78
70.91
78.24
ES Filtered 28.79
Lang Filt
51.52
32.80
55.86
05.75
07.79
68.31
FR Filtered 25.00
52.50
34.00
53.07
80.29
20.30
70.19
IT Filtered
ARC
Easy*
PIQA
SciQ
15.62
24.51
32.17
24.56
LAMBADA*
LAMBADA DE
12:11
23.32
DSIR
Method
E
LAMBADA
ES*
30
23.72
24.2
LAMBADA
FR*
23.94
LAMBADA
IT*
38
08.0A
KO
fastText, EN Filt
Handcrafted
45
10
50.7
fastText
72 57
60
14.34
27.79
24.86
86
fastText
ARC Easy"
44
15
58
49
61
20
11 64
89.06
78 03
64 99
96
PIQA
44.02
38
59.00
11.69
84.22
78.76
65.10
69.84
SciQ'
40.63
58.05
03.70
12 22.36
75.96
73.09
01.24
67.77
LAMBADA
42.13
58
61 40
9:42
79 46
76
62.71
69.94
Perplexity Correlations
LAMBADA DE
42.51
50
62 00
12.94
20.06
97
21.29
24.85
LAMBADA ES'
42.07
55.77
62
15.62
23.85
28.50
22
.999
26.31
LAMBADA FR'
43.18
56.75
G3 80
12.92
26.98
22.15
LAMBADA IT
M0. 75
52 A0
22
ARC Easy
PIQA
SciQ
LAMBADA
LAMBADA DE
LAMBADA ES
LAMBADA FR
LAMBADA IT
Legend
Worst
Task
Figure 2: Pretraining results with different data selection methods. Each row is an LLM, and each column is a task. The number in the upper left indicates the ranking of the method when targeting that benchmark compared other methods (lower is better). Numbers within the heatmap denote accuracy for all benchmarks except the LAMBADA tasks for which the values are log perplexities (where lower scores better). We find that our approach appropriately optimizes data mixes for the target language and benchmark, and matches the fastTTex baseline across most benchmarks.
baselines. This suggests that the performance our approach is dependent on the precise form of the estimator, but holds broadly across perplexity-coreeatti relationships; we may also be able to prove the Appendix B and C results for many other rank correlation coefficients. Additionally, our approach performs better with the optional fastText classifier that our algorithm trains, possibly because operates at the page-level instead of the domain-level..
5.2 PREREGISTERED PRETRAINING EXPERIMENTS
For our preregistered experiments, we target aggregates of many different benchmarks: "DCLM Core", which aggregate of 22 benchmarks, and "Non-EN LAMBADA", which is aggregate of 4. We pretrain on DCLM (Li et al., 2024) data pools at chinchilla optimal (Hoffmann et al., 2022) levels going up to 1.4B parameters, and we generally see increasing perplexity correlations performance with scale, See Figure 3 and Appendix N.
5.3 PERFORMANCE RANK PREDICTIONS
Comparing model performance rankings predicted by our regression to the ground truth, we find generally accurate predictions. Figure shows 5-fold leave-out plots for PIQA, and LAMBADAre with rank predictions given by (@mmj (x)). Every point in the plot is held-out point: we estimated 0* five times, holding out different 20% of the data each time, and plotted the held-out predictions.
We find that our estimator achieves high ordinal prediction performance across all target tasks. We
include 5-fold leave-out R² scores for all tasks in Appendix I.

---

Published as a conference paper at ICLR 2025
Raw Pool
Raw Pool
Pre-filtered Pool
Non-EN LAMBADA Perf
.35
DCLM Core Perf
.30
.2
.25
.15
.20
.1
Perp. Corr. (Page)
Perp. Corr. (Domain)
.15
Handcrafted fastText
None
.10
.0
.16.41
1
1.4
.16.41
1.4
.16.41
1 1.4
Parameter Count (B)
Parameter Count (B)
Figure 3: Preregistered experiment results. We did not see benefit from using perplexity corre- lations when the dataset already extensively filtered, but saw large consistent benefits otherwise, with the benefits increasing with scale. For the pre-filtered pool, the largest correlation coefficient was .33 and the smallest was .23 with the vast majority of domains being over .29, so we could have predicted
small gains before pretraining. In the pool for DCLM Core, the largest coefficient was .32 and the smallest was .07. Pre-filtered pool results for Non-EN LAMBADA are not shown because there is only English in the pre-filtered pool. See Appendix N for more details.
PIQA
LAMBADA FR
80
Hame
Predicted Bank
60
MidraaPPo Pfo 80.
Predicted Rank
60
mamba- 14b-4
bloom Todeliam@---
50
1.46m)
50
mamma-780mm-y
40-
9pt-w
40-
mambi memba-3770m-tt
30
30
20
20
10
60
Actual Rank
Actual Rank
Figure 4: Rank predictions given by (Órroj ((x)) for PIQA and LAMBADA FR. A standard devia- tion (σ) from the ideal fit is shown in red. 2a is shown in orange. Many models outside 2a (shown in blue) are trained atypical data such multilingual data, code, GPT-4 (Brown et al., 2020) outputs. Models with atypical architectures (i.c. Mamba; Gu & Dao 2024) are shown in black. Gen- erally, estimate tightly predicts ordinal benchmark performance from web corpus losses.
Finally, discuss outliers in our prediction of model performance. Our predictions accurate for LLMs with unusual architectures (e.g. Mamha; Gu & Dao 2024), and the smallest/largest vocabulary sizes, context sizes, and parameter sizes. However, LLMs that were trained on unusual data not as well predicted by our approach (c.g. Phi; Gunasekar et al. 2023). We may require a bigger or more diverse pretraining data pool or of models find estimates that work well for these models.
CONCLUSION
Does high- geh-pperformance data selection require hand-crafted heuristics or prohibitively expensive model training runs? Our work demonsirates an alternative, viable approach: leveraging existing, public models as a source of information for data selection. Pretraining experiments suggest that simple, correlation-basee approach selecting data be effective, but more broadly, we show how to 1) use single-index models as surrogates for downstream performance and 2) build models that relate losses downstream performance and use the surrogates effectively select data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jack Spilecki for conversations on the mathematical aspects of the work. We also thank Zitong Yang. Yangjun Ruan, and Lisa Li for their helpful feedback throughout the project, Ludwig Schmidt and Samir Gadre for discussions on scaling laws involving benchmark perplexity, Rohan Pandey for conversations about scaling laws, Sung Min Park for discussions on drafts of this work, and William Held for conversations about data selection. This work supported in part by a grant from Sandia National Laboratories, and gifts from Open Philanthropy, Meta, Panasonic Research, and the Tianqiao and Chrissy Chen Institute. Any opinions, findings, and conclusions or recom- mendations expressed in this material are those of the authors and do not necessarily reflect the views of Sandia National Laboratories. Tristan Thrush is supported in part by the Stanford Graduate Fellowship.
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv, 2023.
Jason Ansel, Edward Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vetio, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael La- 208, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso. Mark Saroufim, Marcos Yukio Siraichi, Helen Suk. Michael Suo, Phil Tillet, Eikan Wang. Xiaodong Wang. William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Com- pilation. ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv, 2023.
Edward Beeching. Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf, 2023. URL https: //huggingfacce. co/ co/spaccs//oin www.giigaaaee////eeeeeeeeaeeeeee 11m leaderboard. hatt:///oopppp..aaaeeeeaarrrrrrr Open LLM Leaderboard.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- Jahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. arXiv, 2023.
BigScience. BLOOM: A 776--paramete apen-access multilingual language model. arXiv, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense natural language. AAAI, 2020.
Sid Black, 2023. URL
Multilingual LAMBADA.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv, 2020.
11

---

Published as a conference paper at ICLR 2025
Sheng Chen and Arindam Banerjee. Robust structured estimation with single-ind index models. ICML, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning chal- lenge. arXiv, 2018.
John Duchi, Shai Sawaaz,, Yoram Singer. and Tushar Chandra. Efficient projections onto the 11-ball for learning in high dimensions. ICML, 2008.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv, 2024.
Dante Everaert and Christopher Potts. GIO: Gradient information optimization for training dataset selection. ICLR, 2024.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5; long form question answering. arXiv, 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, 2023.
Xinyang Geng and Hao Liu. OaeALLaMA: An open reproduction of LLaMA, 2023. URL https: //gi thub. cww/ wwwpenmeeeeearcceeeeeeeeeeeeeaee httpchhhoppppaaaaaaaeaaaaaaaaaaa
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters. Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Worts- man, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. arXiv, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv, 2023.
Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv, 2022.
Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. COLM, 2024.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data. ICML, 2022.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv. 2016.
12

---

Published as a conference paper at ICLR 2025
Adam Tauman Kalai and Ravi Sastry. The isotron algorithm: High-dimensional isotonic regression. COLT, 2009.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv, 2020.
Nikolai E. Korotkov and Alexander N. Korotkov, Table of integrals related error function, 2019. book integrals pdf.
URL https://intra: ece. wwr.ucr...oo.. www.e///kooooooo/ppppe/////eeeee
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset. NeurlPS Datasets and Benchmarks, 2022.
Jeffrey Li, Alex Fang, Georgios Smymis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bit- ton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt. Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groen- eveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. DataComp-LM: In search of the next generation of training sets for language models. arXiv, 2024.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang. and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv, 2024.
Llama Team. The Llama 3 herd of models. arXiv, 2024.
Max Marion, Ahmet Üstün,, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining LLMs at scale. arXiv, 2023.
Edward W. Ng and Murray Geller. A table of integrals of the error functions. Journal of Research of the Natianal Bureau of Standards, Section B: Mathematical Sciences, 1968.
David Owen. How predictable is language model benchmark performance? arXiv, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. ACL, 2016.
Jupinder Parmar. Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Data, data everywhere: A guide for pretraining dataset construction. arXiv, 2024.
Karl Pearson. On lines and planes of closest fit systems of points in space. Philosophical Maga- zine, 1901.
Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale. arXiv, 2024.
13

---

Published as a conference paper at ICLR 2025
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong. Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the Transformer era. arXiv, 2023.
Yaniv Plan, Roman Vershynin, and Elena Yudovina. High-dimensional estimation with geometric constraints. Information and Inference: Journal of the IMA, 2016.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 1-67, 2020.
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv, 2024.
Shai Shalev-Shwartee and Yoram Singer. Efficient learning of label ranking by soft projections onto polyhedra. JMLR, 2006.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. arXiv,, 2024.
Charles Spearman. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 1904.
Teknium. OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants, 2023. wttpttttttstteeeinnn/eepeeeeeeee
URL
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
Together Computer, 2023, URL https: thub. com/ toge wttpehooootteee//eaaaaaaaaaaaa -Dat.. RedPajama: an Open Dataset for Training Large Language Models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using I-SNE. JMLR, 2008
14

---

Published as a conference paper at ICLR 2025
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, 2022.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. W-NUT, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's trans- formers: State-of-the-art natural language processing. arXiv, 2019.
Jeffrey M. Wooldridge. Econometric Analysis of Cross Section and Panel Data. MIT Press, 2010.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le. Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. NeurIPS, 2023a.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. NeurlPS, 2023b.
15

---

Published as a conference paper at ICLR 2025
A
MAIN ALGORITHM
Algorithm Perplexity Correlation Based Data Selection
Input: Benchmark enror vector y [0, 11.1,,. log-loss matrix normalized bits- cits-per-byte R R-Nx available tokens per domain E ND, and pretraining token target E N.
Output: Target token counts per domain t E No fastText classifier to filter pretraining data. Initialize: 7 < E RD, + [0...] No. counter 0.
Fo: ri,.... F.W <- rank (Xo: XI,...,XN] XN)
b 1. Compute the correlation coefficient
for j.j E N do
+ sign(y: mj) (r, - r,)
for EArgSort(7. descending=True) do t, f- miiii(,, counter) counter -- counter + a,
Select most corr. domains (linear projection)
if' counter 2 6 then
Break classifier = trainFastText(positivv 1eso, negative 1(=0)
Return t, classifier
B ESTIMATOR SOLUTION
B.1
LEMMA 1
Statement Lemma 1 Define the PDF of HalfNormal as f(z;0) =Ze 2a² for x > and
otherwise. Now, suppose:
G is a vector with || = 1
Zi, Ze are vectors ~ 10,
E (00,00)
NO0,)
.4 (1).
Then we have:
82
3 Z
Ziy Z2, +c>
Z'
+
V2+
σ²
where Zış is the j-th entry of Zi.
Proof: First, note:
Z1
Z₁
(Z,1(ZI-Z2,5 0
-1yl
Z1yl
Z2
-B
d
Zzyl
Z2
-B
//2 2
0²
>0,
c/o
c/o
where
denotes the vector- valued result of concatenating vectors and scalars. For readability,
Z
|
set Z.
Zz
and Be =
-3 /V2+
6/o
16

---

Published as a conference paper at ICLR 2025
Given that Be is unit-norm (by supposition, is unit-norm), and every element of Ze is ~ (O. 1) (even e/o), we can easily split a conditional random vector containing Zi, into conditionally dependent component and independent component:
Ze [Ze (..) > 0 (I- (I-3o3.) 3.Z+.
The first term orthogonal to B. and so is the part of Ze that not subject the condition. In the unconditional case, Z. (O I) and so Z" ~ (0, 1). The second term is the part of Ze that is in the direction of B.. Z HalfNormal because our dot product condition is satisfied for half of the possible non-orthegonal Ze values. Now, we focus on finding Zel Be) 0 for single index j. We have (for defined to the dimensionality of Bc):
((1 - (l-BoB..)")) + (3,2+), = = Be,)
Σ
Beg + B,2+,
ISSC
Now, note that 22 - (())) Eisiso Z BegBee is the sum of independent zero-mean Gaussians,
if)
so it itself is zero-mean Gaussian with variance:
(1 - (1-B/j)² + Σ
Beg = 2,325 + Bej
Σ
15KC ifj
15KC itj
-1 -1 + Be Σ
Be
ISiSC
= 82
where simplified the expression by recalling that Be is unit norm. So we have that Zz, is:
V B2, + Bcj = //1
83
B 7_ /2
2+
02
for 2' ~ N( 1). As corollary, can that Z2 under the same condition is given by:
B3
-B3 Z+.
6'
2 2+02
V2 72
B.2 LEMMA 2
Statement of Lemma 2 Suppose that & is the CDF of standard Gaussian, and constants, and (), Then we have:
(0 = (
a²
Proof: By the definition of the CDF of standard Gaussian, we have:
Ea c)] B[P(X s aZ c)},
where X ~ NN 1). Continuing, have:
E[ aZ - - 0)).
Now, note that X - aZ - c is the sum of independent Gaussian random variables with given mean and variance; it itself is Gaussian random variable -N(--, 2 1 To find P(X aZ - $ 0), can evaluate its CDF at 0:
) = + Va²
E
Va² +
17

---

Published as a conference paper at ICLR 2025
B.3 LEMMA 3
Statement of Lemma 3 Suppose & is the standard Gaussian CDF, Z+ HalfNormall)), and b and a are constants. Then we have:
Z
1 π
b
E
=
tan
a2
022
Proof: By the definition of expected value, we can take the following integral where fz_ is the PDF of Z.. We integrate from 0 instead of -00 because the PDF of the Standard Half Normal is in the domain below 0:
Z
zb
E
(z)dz
a
0²
+
2b
V2
+
dz
Vaz
+
TT
2b
e dz +
erf
I dz
(*).
2
22v02++
The second integral is generally non-trivial to solve, but luckily in Section 4.3 of the integral table from Ng & Geller (1968), which states:
solve it by using Equation
[ erf(cc)e- of de
VT 2d
1
tan
dvT
Where and d real and positive. We split the solution by cases: b > 0, b = O, and < 0. We find that in every case, we can manipulate our integral so that the solution is trivial the constant inside the erf(.) is positive (and so we can use the integral table). In every case, we find that the solution is tan
(Vo+tT)
Case 1: b > 0. We can use the integral table directly:
π
v
2
Va² +
(*)
tan
V2
V2
V2
π
1
1
V
+ I
Lan
2
b
Then, using the identity:
lan
x. +
lan
-
1
π i > 0,
2
we find the following:
1 + tan
2
Va
Case 2: b= = 0. Note that erf( = 0: we do not have to use the integral table:
(+) V2T
+
22
1
2
Because tan -2(0) = 0, we have:
1
(6b
tan
2
Va +1
Case 3: < O. Because orf(.)) is an odd function, we can pull the negative out:
(+) 1
2m (foe
- dz
/0 erf
z|6|
2
-
dz
v2
+1
18

---

Published as a conference paper at ICLR 2025
Now the integral table as in the b >> 0 case:
V2
Va² +1
π
tan
2T
V2
V2
V
[6]
1
Va²
1
tan
2
π
[6]
We can then use the same identity again:
1
tan
1 x + tan
if
2
to get:
1 tan
2
Because tan is an odd function, we can put the negative inside of it:
1
+
tan
2
π
Va +1
B.4
FULL PROOF
Here, prove:
0
E[sign(y1 - (2xx) D(x2))]
sin -1
+ 20
with yr, 12, (x)), (xx2), and 0" defined in the main text, for the case where ei and e2 are zero- mean Gaussian noise ~ (O, of) and ~ (0, oz). respectively.
It is easy to see that this is general version of the following theorem.
Theorem When NN o²), we have:
2
0
Elsign(w: - (xx)) - +(x;)))] sin
(7)
2
σ²
Proof: By symmetry, we have:
E[sign(y) (xxx) - (xx2))
- = xxx2) sign(vi 122 > 0] + -1) - [ 1/2) < 0j.
By increasing monotonicity of f. we have sign(vi - y2) > 0 + (xl - x2, 0-) + EA > 0, for
c = 6 ~ ~ NNO0 of + o²). So:
2 [[x(x)) - ((xe)) X2, 0*) +EA Es >
+ 2 ( ((x2)) x2, 0") + 0j.
1
Because X1 X2 and cs
d cs. the two expected values above are the same:
= Ex)(()) (xx)) X2, 0-) + +ea > 0).
By linearity of expectation:
[[ X2, 0-) + 0J - E (x] - X2, 0-) + cs > 0].
Now, we focus on finding the overall estimate for single index j. By Lemma 1, we have, for (O 1) and Z4 HalfNormal(1):
((11)) x²) c o Z4b1).
19

---

Published as a conference paper at ICLR 2025
Here, a V
(0))a
8: As corollary of Lemma 1, we can see:
and bj
2+ oz
V2+a5+a3
+ Φ< (2))](( X2. )*) + cc > 0 - (Za + b2)
"
Where b2
So for the index j, estimate is:
+0+
E Za Z4b1) - 2
| Za + c) = 7+b1]]
E Za + c)|c= Z+be]].
Using Lemma 2, have:
Z+b1
)]
Z b2
E
E
+
Va² +
Va² +
Then, using Lemma 3, we have:
1
b2
+
tan
tan -1
2
π
Va2 +1
2
π
Va²
b2
tan
tan
1
π
Va²
+
Va² +
Using the fact that tan
is an odd function and b2 = -b,
we get:
bj
tan
Va² +1
Now, we write a and b in terms of 0
03
2
v
2- +o
tan -1
(0
2
+00
2
v V4+207 +203
tan
π
1420
Using the identity sin x tan
we have:
vI-
2
0
sin
4 20 + 20
B.5 COROLLARY 1
Corollary 1 Suppose that Ô is any vector of fixed weights and (0, 1). Then, conditioning
the event (0, xx) < (5) have with probability 1 that:
(0, [(((),) (Ō xx) < (0,x,))
< < (. (0.x) x₂) < (0, x))x
(9)
To see this, we can find:
E[(((x)) {xx)) (0, x1) + > (0 x2) + [2] E( - +xx2) - -x2) + cc > 0]
Note that we have already computed this expected value in the proof above; for index j. it is:
2 sin
0
/4+ 20 + 20
Because sin-1 is an odd function, the above expression has the same sign as 0,. Because the values at every index of (x)(x))= - +(x2)] under condition and θ are the same sign, we have (E (B() (xx))), 0) O. ( E[((xi))) (0, E{+(x))))-
20

---

Published as a conference paper at ICLR 2025
C SPEARMAN RANK ESTIMATOR SOLUTION
C.I LEMMA 4
Statement of Lemma 4 Suppose + is the standard Gaussian CDF, Z is vector ~ (O..I), c [(0, o²), B is vector with || 1, and a constant. Then we have:
3,a
([ 2) -==
(1+
07
Proof: Note that
Z,1(1Z,0) +e a) d 2zر ((
3
((////) ,
=a
_ 2y1 ((
/V1+02
= =a/v
c//o : o
where
denotes the vector-valued result of concatenating vectors and scalars. For readability, we
Z e/a
3
Z
and Be
/V1+02
Given that Be is unit-norm (by supposition, 3 unit-norm), and every element of Ze is is NO, 1) (even e/o), we can easily split a conditional random vector containing Zj into conditionally de- pendent component and independent component:
Zel (2. = a/v a/v1+0²) -I-B.B.)22
B.a/ 02
The first term is orthogonal B. and so it is the part of Ze that is not subject the condition. In the unconditional case, Z. ~ (O, I) and so Z" ~ (O.
The second term comes from the part of Z. that is in the direction of Be. Solving for Ze where (Ze, Ba) a/vI+ in the where Ze and Be parallel, get constant vector: B.a/ V1+
Now, we focus on finding Ze (Ze, Go) a/V1+td for a single index j. We have (for C defined to be the dimensionality of B.):
((I B.B. )z") (B.0/ /1 1 11++ = (1,))
Σ Be Beja/Vi+ol
ISIC
Now, note that -1(113E2)
Eisiso Se و is the sum independent zero-mean Gaussians, (*)
so it itself is zero-mean Gaussian with variance:
(1 (1- (1-52 52
Σ
e = 12,3. B2
Σ
S
15150
15iS0 i$j
15
-1 =1-232 + 82
Σ
B2
1555C
=1 P2
where simplified the expression by recalling that Be is unit norm. So have that Z, is:
83
3,a
Z 1- B2, + +BB.a/VL+o'
V
1
1+
21

---

Published as a conference paper at ICLR 2025
For Z' ~ (0, 1). Now, the problem reduces to finding:
B3
Bya
E
+
7'
V
0²
1+
By Lemma 2, we have that this is:
3
+
+
(1+ 02) V 2
140
140
C.2 LEMMA 5
Statement Lemma's $ Suppose & is the standard Gaussian CDE, Y ~ NN 1 + o²), and b and are constants, Then we have:
ab
((
tan
2T
2 2V1/(2+ 202)2 + d² 22/(14 + 40 + 12 / 2
Proof: By the definition of expected value,
E {[((a))((b
(aay) fy (y)dy
1 +
erf erf
ay
1 + erf
2120 dy 2 + 2TT² V2
by
2
V2
2
Now, let's multiply the terms in the integral:
x²
2-207 dy
4 30
22T
2no²
erf
ay
1
2-20 dy
V2
22T 2TT²
by
1
erf
20 dy
V2
v 2TtT
2mo
1
a3
by
1
erf
erf
2-20 dy
2
V2T
2
The first term is an integral over the full domain of a Gaussian PDF, so it is just 1. The second and third terms are integrals over the full domains of odd functions, so they evaluate to zero. Overall, we are left with:
1 erf +
by V2
by
erf
2420 dy
V2T + 2
To solve final integral, we note that the conditions are satisfied in our case to use integral 3 in section 2.7.1 of the integral table from Korotkov & Korotkov (2019):
)rf((n=)erf((2=)) === dz
41
Lan
V
q2
+
+ 442
After applying this result, we are left with:
ab
tan
2T + 2no² /q
Vq² qa2 qbb 2
For = 24202
Substituting our variables back and simplifying, we have:
1
ab
tan
2
(2
22

---

Published as a conference paper at ICLR 2025
C.3 FULL PROOF
Here, prove:
( - (x1 6
Lan
V + +(1+ 0-2
where Φ is the empirical CDF of the values, Φx is the elementwise empirical CDF of the x values, and is zero-mean Gaussian noise ~ (O, a²). We can see that this expected value is monotonic with respect to e*, because the numerator inside tan is 0", and the 0*² in the denominator just serves increase the magnitude of the tan expression. We can also check that the overall expected value is never negative by remembering that the largest value at any index of l can be at most 1, and so 1 tan
9" can never be more than 1 6'
-4(1-o")-0-3 0-z
Now, we see that proving this theorem also shows
[((ran((yn) rank ))
monotonic with respect to 0" where rank(y)) the rank of yı among the y values, and rank(x1) is the elementwise ranks of X₃ among the x's. We can then see that Spearman's 's rank correlation, expectation, monotonic with respect to 0* (as long as the ranks are distinct), per the following equation for Spearman's 's rank correlation (Spearman, 1904):
Ys 6 V (rank((.))ran((xx))
(NN=11)
We begin our proof by finding the value for single index of x1, multiplying out the terms, and using linearity of expectation:
(E((,, Lm,., (#1,))) = (u)) 24h ) Par (x15) + D2r, (II , )²] 2IE (1) Da (x) + IE| 0l (z)
Note that E Pa,, (x1,))²) P] and E[D, (1/ (31) )1 are both
EEU /
u du 3'
where U ~ Uniform(O, 1). Now we move to finding E 2h (31 1 2 (#15)). First notice that
E (2,, = (v') (1,,,,, (m1,)),
where ((1). This is because (v1) = P(Y < un) P(f = s f-)))) =b, (v))
So, we now focus on finding E[ (1') (21,))] By the law total expectation, is:
[[ (c) 21,, (Fl = c]] , [ (#1))lV = c]]
Note that y' (0" x1) and ||0 ||2 1, so v/ NO + o²). And æı N/ 1). So, by Lemma 4, we have that this equals:
E (s) (1))))/))()))))))
where ts 0²
(1+02)
11o2
v' Also, because By (v') = P(Y' 11) Φ ( we can further simplify the expression:
(p (l4)))) (kyy)),
where p VI+62*
Now, we can write this expected value as an integral and solve it. Via Lemma 5, the solution is: pk
1
2T tan
///1/2+2007)) (6)
Now we have a solution for every component of the expected value that we hope to find. Simplifying. we are left with:
0
(((()()))) xx (x
tan
π
4(1 + 02) 802
23

---

Published as a conference paper at ICLR 2025
D OPTIMAL PROJECTED WEIGHTS SOLUTIONS
D.1 LINEAR PROJECTION
Theorem Suppose we want to solve:
@mmj arg min (0,0),
BERD
subject to:
Σ 0:
i=1
0 s 0, S T. Vi E [1, D],
where T, are fixed values. Then, the solution is:
Tk
Eir,(())2200 Tj
1
10, if 2:1 2 1 1E $1, (10)
otherwise
where breaky ties between 03 and O2 for k + j; otherwise keeps ordinal relationships the same.
Proof: We proceed by considering each of the three cases from Equation 10.
Case 1. Suppose for the sake of contradiction that the optimal solution is @prei and yet 6% < for some 0 falling under the first case of Equation 10. Now suppose that we construct O' also satisfying the projection constraints that is the same as Ooroj except in these places:
8 Oood
+ A = 7
51 2 0 8 grmi
0 Ooro
8n 2 0
for some A = 6, where 0, ... 0, are all of the 0 values which do fall under the first condition and where the corresponding (rni values are nonzero. We know that there must be some 0gt0) 0gte) from which we can subtract ,1, .., 5n (and so from which we can take the A)
because Σx r, (0,,) Tj S 1. Now, we have:
(6, goro)
0')
= 05
+ 0 + .... + -. Gpoo 0&A 0,60mmi
0 + +0pb + ... + 0,500
-oA 0, + +... 0,
s 05 +... 5n) A.
0 0.5
< 0.
At this point, the only way to avoid the contradiction result would be if d2 = 0p ... = 6.- Otherwise, the above non-strict inequality would be strict inequality. If 6 8, we know that 0g is the smallest i value satisfying condition and all of the other greater values satisfying condition must be projected their 7 threshold value (otherwise we would get the contradiction result). In this edge case can see above that rearranging the remaining weight among equal i values does not change the dot product, so all of the solutions that we can get without the contradiction result are equivalently optimal (including the solution from Equation 10).
0g. then
24

---

Published as a conference paper at ICLR 2025
Case 3. This is analogous to case 1. Suppose for the sake of contradiction that the optimal solution Ooroj and yet (0 O for some ( falling under the third case of Equation 10. Now suppose that
we construct also satisfying the projection constraints that is the same as @nmi except these places:
0 0 A 0
8 (pro)
+ 81 S Tp
0
0000)
8n S Tg
for some A -= -;= 6, where 8, .... 0g are all of the 0 values which do fall under the third condition and where the corresponding (Jroj values are not at their thresholds. By construction we know that there must be some 0g O to which we can add ... 8n. Now, we have:
(0, Ormut (0,0) 0")
OR 0p0) +
00 - - 02 0,00p0
0,00000 0, O,,On
= 0,A - 8001 -...- 0m
S 06161 + +...+ 8n) 0n
= - + A%a
s O.
At this point, the only way to avoid the contradiction result would be if òx Ó, = 0,- Otherwise, the above non non-strict inequality would be a strict inequality. 0g = 0, = ... 0g. then we know that Oz is the largest 0 value satisfying condition 3 and all of the other smaller values satisfying condition must be projected to (otherwise we would get the contradiction result). In this edge case, we can see above that rearranging the remaining weight among equal 0 values does get without the contradiction result
not change the dot product, all of the solutions that are equivalently optimal (including the solution from Equation 10).
Case 2. Above, we show that both Case 1 and Case 3 are true. So, the remaining weight must be given the single value of @oro not covered by either case.
D.2 QUADRATIC PROJECTION
D.2.1 LEMMA 4
Statement of Lemma 6 Suppose that goroj is the optimal solution to:
mmj arg min
||0 -
0ERD
subject to:
D Σ 0 =1
050, S T.
E D),
where T; > are fixed values. Then, 0g
implies that any j with 0, > 8g must have gerej O.
Proof: This is similar to Lemma 2 from Shalev-Shward & Singer (2006). Assume for the sake of contradiction Ogeo 0 and 0, 03. yet we have (700) 0.
25

---

Published as a conference paper at ICLR 2025
Now can construct another vector 0' that is the same as @rmj except in two places:
0 6
+ A
8j 0 A,
for some A satisfying 0 < A < min(0) ( )
This bound on A ensures that 0' is still
within the thresholds. We know that A can exist because min (gore) Ts , - > 0 (by supposition,
O = Ts > and 0 > 0).
Ts
Now we can compute:
0 O
2 - 0 013 = (0, - )0 2 + (0, 0 )2 (0. (Ōgro) + A])] - 100 -(0)me - A))
21 ((,, ( (0, 0 A
2 ((0, g (0, 0 min (0)ro) Te )gTr))
2 2A((0, )gTO) (0, 0g 0
22 (0. 0,)
O.
So Ozvj cannot be the optimal solution.
D.2.2 LEMMA 5
Statement of Lemma 7 Suppose that Ómvi is the optimal solution to:
ÿrmj
arg min AER
0
subject to:
D Σ
0, =1
0 5 0: S T.
Vi E [1, D),
where T; are fixed values. Then, 05Te T, implies 0 Tú for any 03 -T 0 T.-
Proof: Again, this similar to Lemma 2 from Shalev-Shwarzz Singer (2006). Assume for the sake of contradiction δg = T, and 6, - T > 0 Tss yet we have 6oros < Tj.
Now we can construct another vector 0' that the same as Qrroj, except in two places:
8¹ Aoroj A
8 0o00 + A, for some A satisfying 0- < A min( min( Tj
)
This bound A ensures that 0' is still
within the thresholds. We know that A can exist because min(0." TJ )otot) > O (by supposition,
T Orinj > O and 05mi > 0).
Now can compute:
@ @roij 2 0
0 2 (0. )roo) 2 + (0, 0 j (0. (ggro) A])2 (0,- -(ppro) + A))
((0 ) (0. mmi A
22 (0, ) (0. Orro
min (gpro) - @gmoi))
2 24( 0/500) (0. noo
(T, )GTr))
24(@0, TT) ( 0
20( Ty) (0. Tx))
O.
So @proj cannot be the optimal solution.
26

---

Published as a conference paper at ICLR 2025
D.2.3 FULL PROOF
Theorem Suppose we want to solve:
@ arg min ||0 -0
0ER
subject to:
Σ 8 =
0 o S T. Vi E [1, D],
where T; >> are fixed values. Then the solution is:
@ min(maxx(e)) X, 0), Tk).
where À is found (through e.g. bisection search) to satisfy:
D
Σ min(max (0, 0.) =1
Proof: Note that this problem is the same as the simplex projection problem from Shalev-Shwartz & Singer (2006) and Duchi et al. (2008), except here we have additional 0 s т, constraints. The Lagrangian for this problem is
N Σ
(. 110 - 0
-
(11, 0) + (S. T). 8:
2m1
To find the optimality condition with respect to single index of 0, we set the derivative to zero:
d.
0: 0: +
μ4 +
d9;
The complimentary slackness KKT condition gives us that Gi μ4 when 0 < 0, <;, so for 0; not at the boundary our constraints, we get:
0, = 0: X.
So, have that for all 0, (0, 7:), there is shared value À which subtract from 0 get the value 0, How do we know which 0; are 0 and which 0; are т; though?
Assume that we know X. By Lemma 6,
characterize the optimal solution as:
Temu max (, - X, 0),
for ) * Tk. By Lemma 7, we can characterize the optimal solution as: ف min (0, X. Tk),
for )0 0. So, we can combine these two forms to get:
min(max(@A. X, Tk)
Now recall that we have the following constraint:
D
min - A, 0), = =
Given this constraint, we can find À through search (moving the value up down). We can see this by noticing that ΣD' min (max X, 0,0).;) is strictly decreasing function of X between the setting of that makes 0, > > 0 for least one and the setting of that makes 0 - < т; for at least one i. So in this range, there is only one setting of X that satisfies this equation. We can only choose À outside of this range when El= T, 1, and in this the solution is trivial: @rooj T for all i.
Note that multiplying -! 0 2 by 1/ does not change the minimization problem and enables us to get rid of factor of 2 after taking the derivative of the Lagrangian.
27

---

Published as a conference paper at ICLR 2025
E ALTERNATIVE METHODS
Our estimator is far from the only reasonable high-dimensional. single-index model estimator. We briefly discuss some alternatives and the tradeoffs involved before moving to experimental results.
We could use classic low-dimensional methods regularized for the high-dimensional setting. This includes ordinal regression (Wooldridge, 2010) and the isotron algorithm (Kalai & Sastry, 2009). We found these methods to underperform correlation-based estimators, and tuning hyperparameters added additional complexity that was not needed in the correlation-basee approaches.
Another class of methods involve scaling laws (Kaplan al., 2020; Llama Team, 2024; Ruan et al., 2024). We could transform the values via an inverse sigmoid or power law, and fit high- dimensional linear regression methods (e.g. ridge, partial least squares, or Lasso). We initially found this approach promising, but the inverse transforms were unstable, and the combination of fitting the nonlinear transform and regularization required significant amounts of tuning.
Rank-correlation methods, including our robustified version of the estimator from Chen & Banerjee (2017), and even the standard Spearman correlation (Spearman, 1904) (see Appendix H) performed well. We believe that in general, robust per-feature correlations are likely to perform well as D >>>>>> N, and extreme levels of regularization are needed to obtain reasonable models. Sparse methods such as the Lasso (Tibshirani, 1996) are classic answer, but cannot necessarily assume that the underlying correlations 0* are sparse, and we did not find these techniques to perform well.
F LOsS MATRIX COMPUTATION SPECIFICS
For all of our initial experiments, we computed the loss matrix as follows. For efficiency purposes, sampled only 25 pages for a domain's ' bits-per-byte (BPB) computation even if domain had more than 25 pages. To get an LLM's BPB on a page, we split the page into chunks of text that were 512 tokens according to a reference tokenizer (we used the Llama 2 7B tokenizer; Touvron et al. 2023). These text chunks turned out to be small enough to fit in the context of every LLM we tested. We then averaged BPB across chunks for each page and then across pages for each domain.
G ADDITIONAL DETAILS FOR INITIAL PRETRAINING EXPERIMENTS
In this section, we specify hyperparameters and methods used for LLM pretraining and evaluation for initial LLM pretraining experiments. We also specify settings used for the data-sclection methods.
G.1 LLM PRETRAINING
We trained each LLM on 4 NVIDIA A100 GPUs At 3.2B tokens, each training run took under hours with the Hugging Face Trainer (Wolf al., 2019) and appropriate PyTorch (Ansel al., 2024) compile flags. We provide pretraining hyperparameters in Table 2. Given our per-device batch size, we found the learning rate by increasing it by a factor of 2 until we saw instability and then using the highest learning rate where no instability was observed. Refer to the Pythia paper (Biderman et al., 2023) for more information; we initialized the model from scratch using their 160M model configuration at https: / htthh/ugggggfccce woacc..o//EE wt/ElettherA/pythht.e can be assumed to be Hugging Face Trainer defaults at the time this writing.
160m. Other hyperparameters
G.2 LLM EVALUATION
At the end of the pretraining script, we used the Eleuther AI Eval Harness (Gao et al., 2023). For efficiency, we set the sample limit to 5000 examples per benchmark. Elsewhere, we used the default settings. On 4 NVIDIA A100s, it took only a few minutes per LLM to compute evaluation results for SeiQ, ARC Easy, PIQA, LAMBADA, and all of the translations of LAMBADA.
G.3 DSIR
DSIR (Xie et al., 2023b), despite its simplicity, requires some tuning. A decision must be made about how to format the bemchmark data into single piece of text per example so that it can be
28

---

Published as a conference paper at ICLR 2025
Table 2: LLM Pretraining Hyperparameters
Parameter
Value
Per-device Batch Size
128
Learning Rate
10
Warmup Ratio
0.1
Adam 31
0.9
Adam 32
0.95
Adam
10-8
Weight Decay
0.1
LR Scheduler
cosine
Max Grad Norm
1.0
BF 16
True
Distributed Backend
neel
Gradient Accumulation Steps
Table 3: Unique pretraining tokens selected per benchmark, from DSIR.
Benchmark
Tokens
ARC Easy
2,905,206,499
PIQA
2,910,486,295
SCIQ
2,920,734.022
LAMBADA
3,022,219,424
LAMBADADE
3,210,986,137
LAMBADAES
3,396,528,704
LAMBADAER
3,413,930,081
LAMBADAYT
3,384.854.845
compared with potential pretraining data in terms of n-gram overlap. The LAMBADA tasks only have one text column per example, so the decision here is trivial. Examples from the other tasks each have question, possibly context, and set of multiple choice answers choose from. We chose to concatenate all of these columns together with spaces to form one piece text per example, duplicating the same question as prefix for each different answer.
DSIR does not allow the user to specify the exact number of unique tokens desired for pretraining. only allows the specification of the number of unique pages, which can have wildly varying token counts. For every DSIR job, we set the desired number of pages to 3325589, which we found through binary search to produce slightly more than 3.2B unique tokens for LAMBADAFR. was expensive to find this number for even one bechmark, because for each iteration of the binary search, had run DSIR and then the Pythia tokenizer to know how many tokens resulted from the input page number parameter. We provide the number of unique tokens from DSIR for each task in Table We pretrained on 3.2B tokens for every LLM regardless of whether all of them were unique.
G.4 FASTTEXT
The "SOTA" fastText model from et al. (2024) available here: https: https::/huuae https://hugiingface co/ m1 A.fooundttions/asstttttthhlllll We used this model to filter data by sorting pages by the
29

---

Published as a conference paper at ICLR 2025
model's "high quality" score, including the top pages in order until we had either reached or gone slightly over 3.2B unique tokens. This aligns with the data-sclection procedure in the original paper, and is also essentially the same running the linear projection (Equation 10) at the page-level. We also applied this method when selecting data using our own fastText filter trained by our algorithm.
H ADDITIONAL INITIAL PRETRAINING RESULTS
In Figure 5, we present additional initial pretraining results for methods in our pessorerfrr correlation data selection paradigm. We find that using Spearman rank correlation (Spearman, 1904) in place of our estimator achieves comparable performance. On some tests, it performs even better than our estimator. We also find that using the quadratic projection, while perhaps more intuitive, leads to worse performance than the linear projection.
I PERFORMANCE PREDICTION SCORES
We include 5-fold leave-out R2 scores for all initial experiment tasks in Figure 6. However, com- plement these strong results with the additional observation that simply taking the mean loss across all domains is strong predictor of model performance (bottom row). The surprising effectiveness of average loss over uniformly sampled documents has been discussed extensively (Owen, 2024; Wei al., 2022: Kaplan al., 2020) and our results further suggest that regressions with correlations only slightly above the mean loss baseline can still result in effective data selection methods.
PRETRAINING LANGUAGE DISTRIBUTION
Figure 7 shows the language distributions of the projected estimates for our initial RPJv2 experi- ments. Our algorithm provides significant enrichment of the corresponding languages for the mul- tilingual benchmarks (LAMBADA_* *), but it does not exclusively select domains in one language. In contrast, for English benchmarks our approach selects nearly exclusively English data, likely due to the large quantity of high-quality English data in pretraining data pool. There are signifi- cantly fewer tokens in non-Eng annEEnlish languages in the data pool and our т constraint prevents their duplication.
Figure 8 shows what the projected estimate in our pretraining experiments would be if we had a pretraining data pool 5x large. We see here that the estimate does even better job at selecting pretraining data with the language that matches the target task.
K PARAMETER COUNT DISTRIBUTION FOR ESTIMATOR LLMS
In Figure 9, we present the parameter-coum histogram of the 90 models from the Open LLM Leader- board (Beeching et al., 2023) that we used compute our estimate for pretraining data selection. Only 8 models here are less than 160M parameters. Despite this, our estimate can be used to effec- tively pretrain 160M parameter LLMs..
L ANALYSIS OF THE MODEL-LOSS MATRIX X
What information is contained in the matrix of model losses X? Clearly, it must contain semantically meaningful information about the data, such as the language that a piece of text is in. We performed PCA (Pearson, 1901) and t-SNE (van der Maaten & Hinton, 2008) on X and plotted the first two components for each of our 9,841 RPJy2 domains. As shown in the first row of Figure 10, we found two components with relatively high singular values. The first component clearly corresponds with the language of a domain. The second component corresponds with the average bits pett-pee-byteee or entropy of domain. The t-SNE components show the same general pattern well as showing that the language clusters are very well separated. As shown in our plots, there are several salient clusters within the language clusters. Within the English cluster, we found subcluster for luxury goods, another for legal services and information, another for academic research, and even a cluster for funeral homes.
30

---

Published as a conference paper at ICLR 2025
Uniferm RP1V2
None
Uniform RPJV2
None
EN Filtered
EN Fitterdd
DE Filtered
DE Filtered
ES filtered
Lang Filt
ES Filtered
Lang Filt
FR Filtered
FR. Fitered
IT Filtared
IT Filtored
ARC Easy*
ARC Lasy
POA
PIDA
Sc10+
40.17
Sc10
57.04
59.03
2431
52.17 04.56 36.9
fastText, EN Filt
LAMBADA E5
LAMBADA FR
LAMBADA
LAMBADA
40.1
5036
41330
10.03
17
55-70
22.22
96.00
1E.C 1374 23.22 2033
52.0 19.00 20.4
2740 unen
18.00
peas
11
20.77
Handcrafted
fast FN Fi
LAMBADA DE
LAMBADA E5
LAMBADA
LAMBADA IT
53
5083
2
14.23
13
1714
22.22
2000 19.0
20.1
20.00
20.77
Handcrafted
Method
Tas:Text
feat
Fest
fastilext
ARC Easy
ARC Easy
PIOK
PIQA
LAMBADA
Sci0
Estimate Lin. Proj.
LAMBADA
SciQ
Fetimate
LAMBADA
AMRADAA
40.5
56.0
LAMBADA
LAMBADA ES
27-34
20.0
Proj.
LAMBADA
LAMBADA
bust
ARC: Eary
PIDA
LAMBADA
LAMBADA DE
LAMBADA
LANBADA
LAMBADA
ARC Easy
PIDA
LAMBADA
LAMBADA DE
LAMBADA
LAMBADA
LAMBADA
kask
Task
Task
(a) Estimate with linear projection. This is algo- rithm from the main text without training the addi- tional fext filter.
(b) Estimate with quadratic projection. Same as (a)
cept the linear projection is replaced with the quadratic projection.
Uniform RPy2
None
Uniform RF|V2 EN Filtered
None
EN Fittered
DE Filtered
Filteedd-
ES Filtered
Lang Filt
ES Filtered
Lang Filt
FR Riftared
FR Fillered
IT Ritteed
IT Filtere-..
2009
ARC Easy*
PO
40.30
24.55
PICA
Easy
1140
11:06
76:00
fastText, EN Filt
LAMBADA DE+
LAMBADA ES3
LAMBADA FA
LAMBADA T
LAMBADAY
Scio*
40.0
11:41
4017
5115
5753
14-10
53.0
10:33
17
18.00
1237
Handcrafted
fastTest, FN FM
LAMBADA DE
LAMBADA FR
LAMBADA ES
LAMBADA IT
LAMBADA
0.11
51
17.05
2014
5575
10
12
2651
22:2
29.77
2406
13.00
2010
77.51
2007
Handcrafted
DSIR
Method
fastText
festlext
fest
Text
ARC Easy
ARC Easy
FIQA!
em
PIN
2010 5133
Test
LAMBADA
LAMRADA FR
LAMBADA
LAMBADA
Sc00'
4000
sta
4:.0
$7.9
30 c
2842 1230
0.20 1653 25.99 20.0 50.0
1..0
22.1
2542
2.37
Lin, Proj.
Spearman
LAMEADA
LAMBADA ES
LAMBADA FR
LAMBADA
S:10
400
9339
2247
76.0
1112
57
2001
2072
trained on:
Lin. Proj.
Spearman
ABC. Fasy
PIO
Se10
LAWBADA
LAMBA LAMBADA DE
Es
LAMBADA m
ARC Easy
PIDA
Scl0
LOMB
LAMBADA DE
LANUADA LS
LAMGADA FR
LAM3ADA IT
LAMBADA
Task
Task
(c) Spearman rank correlation with lincar projection. Same as (a) except we replaced our estimator with the Spearman rank correlation.
(d) fast filter trained on data selected in (c). This
is the same as our algorithm the main text, replacing our estimator with the Spearman rank correlation.
Figure 5: Pretraining results for different methods within our paradigm. Overall, we that many rank-correlation pretraining data selection approaches perform well.
The second row of Figure 10 shows plots for the loss matrix when we take the principal components of the other dimension, where points correspond to the 90 LLMs. For PCA, PCI corresponds to entropy. For both cases, it is less clear what the other PCs are, but when we color the three largest families of models in our data (Pythia (Biderman et al., 2023), Qwen (Bai et al., 2023), and OpenL- lama (Geng& & Liu, 2023)), we see that model families are Clustered together in the PC graphs.
M SCALED UP EXPERIMENTS
During the ICLR review process, and before we had completed the preregistration in Appendix N, concerns about scale were raised. So, we re-ran the uniform sampling baseline and our main per- plexity correlations method to create Figure 11, analogous to Figure 2. Besides scale, the only other
31

---

Published as a conference paper at ICLR 2025
Proj. Estimate
3.6
89.713.0
78.115.0
92.611.7
89.912.4
84.324.1
92.811.8
91.8+2.0
Estimate
85.7#4.6
86.3
75.2+6.4
91.1#2.0
89.0=2.3
80.6=4.7
91.2#2.1
92.0#188
Meen Loss
86. 16.1+.4.5
3..
75.0
91.312.0
88.8
80.7
91.6+2.0
92.1=1.7
ARC Easy
PIOA
sclQ
LAMBADA
LAMBADA DE
LAMBADA ES
LAMBADA FR
LAMBADA IT
Figure 6: Held-out R² score of the average loss baseline. The 95% bootstrapped confidence intervals are wide enough that no individual comparison is significant. Across benchmarks, Qrmj has statistically significant gains over the baseline (p=0.035) as it unlikely that (aro) beats mean loss 7 times out of by chance.
correlation estimate 0, our projected estimate Øeroj and
63.17
99.01
99.02
98.88
98.59
59.16
50. 30.48
57.85
58,36
DE
10.54
0.42
0.46
0.67
0.60
15.01
11.97
5.26
14.98
ES 9.85
0.25
0.25
0.36
0.42
7.83
14.17
14.34
8.94
FR 10.41
0.32
0.27
0.09
0.39
11.46
16.89
16.89
11.25
6.03
0.00
0.00
0.00
0.00
6.53
6.48
5.65
6.48
Uniform RPJv2
ARC Easy
PIQA
SciQ
LAMBADA
LAMBADA DE
LAMBADA ES
LAMBADA FR
LAMBADA IT
Figure 7: Language distributions of pretraining data selected by perplexity correlations. The default RPJv2 distribution is given in the left column for reference. The English benchmark targets often exclusively select English but the reverse is not the case. In every case, our approach selects more data than the default from the benchmark-
language (shown as a green box in each column).
feature we changed was the number of pretraining tokens, which we 8.2B keep the ratio chinchilla-optimal (Hoffmann et al., 2022). This setting is arguably harder for perplexity correla- tions, because the uniform sampling method was allowed to sample from pool of 8.2B unique tokens, whereas for our perplexity correlations experiment, we duplicated the original 3.2B set of tokens from the 160M parameter model experiments. Despite training on data that is far more du- plicated, perplexity correlations still achieves superior performance in 7 out of 8 tasks seen in Figure/ 11.
N SCALING UP FURTHER: PREREGISTERED EXPERIMENTS
In small-scale experiments, our approach is competitive with the leading approach from et al.'s survey: fixed fastText model (Joulin et al., 2016), manually augmented with the best language fil- tering. This leading approach heuristic and hand-crafted, requiring appropriate language filtering matched the target benchmark and assumptions about what good pretraining data looks like. Our approach does not make these assumptions and could potentially improve as more public models are released and we have better data estimate 0*
While our initial results are generally positive, many past data selection methods have reported initially positive results, only later break: they may fail scale larger models or rely on specific details of their experimental setting. Our 160M-scale experiments may also raise such concerns.
We designed pre-registered scaling experiment that addresses both the concerns of scale and external validity. We used the permanence of arXiv preprints as a mechanism to preregister series of scaling experiments within the DataComp-LM framework (Li al., 2024), which testhed for data-selection techniques released with the recent survey. Pre-registering held-out scal- ing experiments committed us to reporting potentially negative results, and avoid overfitting to our chosen experimental settings. Our first arXiv submission with the preregistration is available at https://arxiv/ http:://axxiv.or//bbb... www.oog/////////ooo////.../..../ /20////bbb//....///// 05816v1.
DataComp-LM was ideal for this preregistered scaling experiment, as it standardizes the setting by providing a pool of 240 trillion tokens, pretraining code for 412M to 7B parameter models, and evaluation code for 53 benchmarks, 22 of which labelled as "core" benchmarks that scale pre- dictably. Importantly, we did not train any models on DataComp using our methods beforehand baselines, making this true held-out experiment with known higf-perf baselines.
32

---

Published as a conference paper at ICLR 2025
EN
63.17
98.39
100.00
99.94
96.69
1.09
7.64
17.27
26.35
DE
10.54
1.59
0.00
0.02
0.40
67.53
2.27
2.30
27.94
ES 9.85
0.02
0.00
0.00
1.60
0.79
36.96
33,40
4.08
FR 10.41
0.00
0.00
0.04
1.31
0.66
53.07
46,97
17.49
IT- 6.03
0.00
0.00
0.00
0.00
29.94
0.06
0.05
24.15
Uniform RPJv2
ARC Easy
PIQA
SciQ
LAMBADA
LAMBADA DE
LAMBADA ES
LAMBADA FR
LAMBADA IT
Figure 8: This figure is analogous to Figure 7, except the thresholds have been multiplied by 5.
We see that our approach selects even more relevant data when the selection pool is larger.
16
160M
14
12
10
8
4
2
0
0
2
4
6
8
Parameter Count (Billions)
Figure 9: The parameter-count histogram of the 90 models from the Open LLM Leaderboard (Beeching et al., 2023) that we used to compute our estimate for pretraining data selection. Bar widths are 160M. The smallest model in the sample has =33M parameters and the largest has R9B. The spike around 6. parameters is due to a large number of partially trained Pythia (Biderman al., 2023) checkpoints from the same training of selecting pretraining data for 160M parameter models, which is abnormally small in the set of models used compute the estimate.
that scale. Our algorithm has the hard task
We ran the best-performing approach from our paper: fastText filter trained on our correlation estimator. We defined the target benchmark for our estimator the average of the "core" DataComp- LM benchmarks and ran our estimator with perplexities from our set of 90 OpenI.M Leaderboard LLMSs on uniform subsample of the DataComp-LM pool of data. Instead of using the provided DCLM code (as planned in our preregistration), we used our own pretraining and eval code after consulting with the DCLM authors, their codebase could not be easily run outside their infra setup, and they advised us to switch to our own codebase.
code for training Pythia- aechittaccit LLMs (and the same method- ology for tuming hyperparameters as the 160M experiments before), as well as the Eleuther Eval Hamess (Gao al., 2023) for replicating the "core" benchmark evaluations using the settings ported the DCLM paper. Differences Pythia architectures and Eleuther Eval Harness implemen- tations unfortunately make our evaluation numbers not directly comparable to the original DCLM codebase numbers.
Specifically, used
We report results for the "Filtering IB-1x" track, where 1.4B parameter LLM is trained on 28.8B tokens chosen from a 1 1.64T sample of the DataComp-LMM pool of data. In the DataComp-LM paper, they apply their fixed fastText filter as final step after several complicated deduplication and filtering steps. We report results where our fastText classifier is used as direct substitute for this last step alone (filtering from the spre-filtered pool"), , as well another test which we replace the entire pipeline with one classifier (filtering from the "raw pool").
We also report results where our estimator is trained at the domain-lewe (following this paper) and where our estimator is trained at the page-level (which we had not tried before).
DDue to disk constraints, filtered starting from 10% random sample of this 1.64T sample for every method.
33

---

Published as a conference paper at ICLR 2025
Component
(a)
(b)
(c)
(d)
(c)
Pytha
Qreen
Open
Liarta
Other
-15
-100
-100
Comporentt
(I)
(h)
Figure 10: Analysis of the loss matrix. The first row treats domains examples to be projected via PCA, while the second row treats models as examples. Panels (a): eigenvalue decay for the eigende- composition of the DxD covariance matrix resulting from the loss matrix; few dominant PCs seen. (b) and (c): domains plotted by the first two PCA components showing separation of language in and entropy in (d,e) show analogous plots in t-SNE with clearer separation of language. (f): eigenvalue decay analogous to (a). (g.h): models plotted by the first two PCA components showing clustering by model family (clusters show Pythia (Biderman et al., 2023), Qwen (Bai et al., 2023), and OpenLlama (Geng & Liu, 2023) derivatives the three largest clusters in our data), and average model loss. (i.j) show analogous results under t-SNE where is normalized remove per-model entropy differences.
Uniform RPjv2
None
43.56
59.03
65.30
4.51
17:36
21.45
17.86
21.62
ARC Easy'
PIQA'
45.41
50
81
61.30
3.56
31.06
74.30
62.04
07.43
Method
SciQ'
LLM
17.64
59
56
60
5.35
74.65
69 08
59.71
66.41
Perplexity Correlations
LAMBADA'
LAMBADA DE'
42.13
56.53
60.80
5.52
15.26
20. 96
16:40
19.77
LAMBADA ES'
43.69
LAMBADA FR
65.50
50
21.08
17.19
20.23
LAMBADA IT'
42,39
18.
57,50
6.18
15.67
20.
45
15.67
20.03
ARC Easy
PIQA
SciQ
LAMBADA
LAMBADA DE
LAMBADA ES
LAMBADA FR
LAMBADA IT
Task
Analog to Figure at the 410M parameter and 8.2B token scale. For perplexity corre- lations, we duplicate the original 3.2B tokens, whereas the uniform sampling baseline is allowed to sample from 8.2B unique tokens.
Figure 11:
Finally, we report analogous results where we replace the "core" evaluation score with the average score on all of the non- -on-lih LAMBADA translations, and compare the raw fastText classifier from Li et al. (2024) to our approach, using both of these approaches in place of the full filtering pipeline from 1.64T tokens. We preregistered this additional multilingual task because "core" does not include multilingual evaluations.
Beyond our preregistered experiments, we also ran chinchilla-optimal 160M, 410M, and IB scales to complement our results. We applied the same perplexity correlations fastText filter at all data selection scales. To train it, we set the token threshold in our algorithm to 50% of the tokens used for training the estimator, meaning that approximately 50% of the domains/pages from were labeled as 'include" and the other approximate 50% were labeled as 'exclude' for the purposes of fastText training. To train estimator, used approximately the same scale of data
estimate
34

---

Published as a conference paper at ICLR 2025
initial RPJv2 BPB matrix. For the pre-filtered DCLM pool, this sample with about the same disk size as the RPJv2 estimate sample with 129.376 pages and 9,952 domains at 2 13 pages per domain (we found that the pages in this pool were about twice long average the RPJv2 pages). For the raw DCLM pool, this was a sample also of about the same disk size with 325,682 pages and 13,732 domains 2 23 pages per domain (page lengths were generally shorter in this sample).
All results can be found in Figure 3.
O
TOP CORRELATED DOMAINS PER-TASK
Here, we list the top 10 most correlated domains (before projection) found by our main rank corre- lation estimator for the pretraining experiments.
ARC Easy. popl-bridgeazzureeeee www.eeewbstes..eeeee.... superlogical. www.aacoptometry.com
www.akronchildrees..rrg www.edusofttech.com www.fredericckhurrgsoo.omm
www.latferffothhssitilrraaaa.... www.hopkinaalcchiltrens..gg
www.metropolianviioonyycccmm www.myyhearllivshere.co
PIQA. ap-.briige.auureeebbiiisseee
htwprriiiessuubb.aaiingggooooo ricardofran-
cia.com, www.aacooionnetry.comm www.akronchildrens.com www.eczemainfooluu.com
www.:p/www.cc.......oooooeeoeooo hww.://or..ooo...w.ooncmciinnnno www.medicineshoppee.cu
www.metroollianvsionyyccom
SciQ. www-bbiddge.aeee
www.bbrddd...uueeebssesesss www.eeewbsitss.neeee goodbusinesskit.com original-ideas.ne pos-
university.com, taraweddings.co thefloristic.com. www.cuemath.com www.edusofftech.com.
www.groupeproxim..cu
LAMBADA. geringas.blogspot.com boddingmarc.boosppot.comm books books.google.se chestof-
books.com, coint-rescarch-centr.. hww.:hecctttreecc.uuua..e..e snoqualmic
snoqualmie.cmmeettoiiooocccm wememtoorizon.com... twowheeledmad-
woman.blogspot.com www.ganssle.com www.sarahhague.com www.themodermnovel.om
LAMBADA DE.
biomedicalhouse..comm schariageg-
ner.worddreess.omm truthfriends.us www.aerzteblatt.de www.buddha- wwo.bddddha-bbooooiinni.... online,
www.leulschheseesmnlhiiissrrrl.. www.global2015.net www.jeedische-allgemmieeeeed
www.metropolii--erraag.ec
LAMBADA ES. archi7.net, catolico.org, cineclubdecaen.com. dbe.rah.es, www.ca-se-passe-
la-haut.fr, www.corsarios.nee, www.e-stoire.nett www.la-historiadora.com www.proverbes- francais.fr, www.vedaveda.com
LAMBADA FR. archi7.net, cineclubdecaen.com com, es.m. wtw.m.ikipedii.oom org, images.
irb-cisr.gc.ca www.ca-se-passe-laaheee www.corsarios.net, www.futura-sccieness.com
www.neurologiiccaselllonnsss www. www.eedaeddacomm
LAMBADA IT. doc.studenti.it, -essage-foryyouneee, foop.fedduuutur..co
www.getstoryshots.com com, www.global2015.nr www.peterlang com, www.scrutatio.it
www.scuolafloosofcc..com www.seuolissimacom www.storieparallele.ii
Non-EN LAMBADA (DCLM Raw Pool). digi.ub.ui--heidelleer..dd de, it ittheffrediccionary.com.
it. it.iikiquoee.om org, slideplayer.it, www.astro.com com, www.epo.org. www.kunsthaus.ch
www.logitech.com www.peterlang.com com, www.zenit.org
DCLM Core (DCLM Raw Pool). finnacce.yhho.ccm
nrich.maths.org. www..carleton.edie..e whois.epik.com. www.hom.gov.ar
www.couttycurreeyyattsccmmm www.ecdc.europa.com www.iaaf.org, www.metooffic.gov.uk
35


Reviews:
[{"review_id": "Td7wrV5Lrf", "reviewer": "Reviewer_54hW", "summary": "This paper proposes a new LLM pre-training data selection framework where no LLM training is needed. Their framework leverages publicly accessible pre-trained LLMs and is based on the correlation between downstream benchmark performance and LLMs' losses. In the paper, the authors provide theoretical background for the proposed framework and present experiments where they use 160M parameters models with a 3.2B token budget, comparing DSIR and DataComp-LM data selection. Concretely, they selected data from RedPajamas V2, and used LAMBADA, ARC-E, PIQA, and SciQ as target downstream benchmarks. In their experiments, the proposed method outperforms DSIR and shows competitive performance with DataComp-LM best data selection method.", "questions": "1. Clarification for Line 201: `select domain greatest to least \\gamma, taking all the tokens in each domain once, until we exhaust our total pretraining token budget.`\nDoes this mean that if a domain with the greatest \\gamma includes more tokens than the token budget, only tokens from that domain will be selected?\n\n2. For the proposed perplexity correlation, have you tested aggregating benchmark scores in some way to use as an overall benchmark metric? From Figure 2, I only see LLMs where data selection (ppl correlations) is optimized for each target task.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on avoiding costly pre-training ablations for LLM pre-training data selection and proposes a method that leverages public, pre-trained LLMs based on the correlation between downstream performance and perplexity. This aim is quite important for today's LLM practitioners and the proposed framework is very original.  \n\n2. Although limited (see below \"weaknesses\"), the proposed framework demonstrates promising results compared to baselines. \n\n3. Although the presentation can be improved, the paper includes a theoretical background for the proposed framework.", "weaknesses": "1. The main weakness of this paper is the scale of the experiments. The paper only includes pre-training experiments using 160M parameters LLM with a 3.2B token budget. This is very small and creates questions about the validity of the results. For example, the smallest scale for DataComp-LM paper is 400M and 8.2B and they experimented with 5 different scales which goes up to 7B parameters model trained on 276B tokens (as DCLM competition scale).\n\n2.  Experiments only include 8 evaluation benchmarks (5 of them are Lambada from different languages). This is a main limitation for both showcasing the validity of results but also limit the understanding how the proposed framework. work when there are many more benchmarks to consider for perplexity validation. Again for comparison, DataComp-LM includes an extended evaluation suite that consists of up to 53 downstream tasks. \n\n3. If the framework only optimizes data selection per task, is it not expected to have good scores on optimized tasks? Are there any experiments where authors tested aggregating benchmark scores as one overall metric?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7xxWNAnxr", "reviewer": "Reviewer_Pj1w", "summary": "This paper analyzes how to select high-quality pretraining data for large language models (LLMs) without training new LLMs. The authors find that LLM losses on certain pretraining texts correlate with downstream benchmark performance. The key idea is to select data domains where lower loss generally corresponds to higher downstream performance.\nTo do this, the authors use a collection of pre-trained LLMs and measure the log-likelihoods for texts from various domains and the performance of these LLMs on downstream benchmarks. They then calculate the correlation between these two measures for each domain. Domains with high correlations are selected for pretraining.", "questions": "- Relevant works that are not cited:\n\n[1] Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane\nTazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023\n\n[2] Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, Sara Hooker. When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale, 2023", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents a new and remarkably simple approach to data selection for LLM pretraining. It leverages readily available resources (publicly available LLMs) and a straightforward correlation-based method.  The authors ground their approach in a statistical framework, providing a theoretical basis for their correlation-based data selection. The authors demonstrate the robustness of their approach by showing its effectiveness across various benchmarks and conditions.", "weaknesses": "While the paper presents promising results at the 160M parameter scale, further validation at larger scales is necessary to assess the scalability and effectiveness of the approach for slightly more massive LLMs. The method inherits the biases present in the publicly available LLMs used for estimating correlations. This raises potential concerns about bias amplification, especially if these LLMs were trained on data with inherent biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOq6mmuJI4", "reviewer": "Reviewer_2JKP", "summary": "The authors build on a large number of existing open LMs and the observation that perplexity of an LM on a sample of text is predictive of performance on some (presumably related) benchmarks to produce a method for selecting pretraining data, aiming to optmize downstream performance.  The method doesn’t require any training of models. Specifically they greedily sample all available data, without repetition, from each domain in order of correlation until a desired token budget is met. They also filter at the  webpage level using a FastText classifier trained to discriminate the selected and non-selected domains from the previous step.\nThey apply this set up over:\n- 90 models from 33M to 9B parameters from the Open LLM leaderboard, including both intermediate and final checkpoints.\n- 4 downstream benchmarks appropriate for small models with an additional 4 evaluations created from one of the benchmarks translated into 4 different languages\n- Perplexity evaluations on 9841 internet domains (e.g. wikipedia.org) represented by a sample of 25 web pages each, derived from Common Crawl.\n- Sampling training data from a 100B token corpus, derived from Common Crawl\n- Pretraining experiments at the 160M parameter 3.2B token (“chinchilla optimal”) comparing their proposed data recipe to baselines including uniform sampling, language filtering, an n-gram overlap method (DSIR; Xie et al., 2023), and a SOTA  handcrafted FastText classifier (Li et al., 2024)\n\nWith this set up they find that:\n- Their proposed approach performs comparably to the state of the art handcrafted baseline (which was constructed with expensive pretraining experiments not required by this paper’s approach) and outperforms all other baselines.\n- Using Spearman rank correlation as opposed to the authors’ proposed correlation measure performs comparably (figure 6, appendix)\n- Ablating the webpage-level filtering using the FastText classifier trained on their selected domains, does lower performance (though the difference is smaller when using Spearman rank correlation).\n\nTheir method can also be used to predict the ranking of the 90 public models by downstream performance using only domain perplexities. They find:\n- A baseline approach that predicts the downstream rank by a model’s rank w.r.t. to  mean loss over domains already gets R^2 values from 75 to 92 (always within 5 points of the proposed approach)\n- Testing over all 8 downstream benchmarks, the proposed approach gets a statistically significantly better R^2 with p=0.035, though no individual benchmark is significantly improved.\n\nThey also provide an extensive theoretical statistical framework for using a single-index model (SIM) to represent the relationship between domain perplexity and downstream performance. They explain the use of the empirical CDF rather than raw perplexity values to better handle outliers. They also provide an approach to constrain the correlations to be a valid distribution over domains and to respect the limited number of tokens available per domain. However their empirical results “suggest that the performance of [their] approach is not dependent on the precise form of the estimator that is coupled to [their] theory results, but holds broadly across perplexity-correlation relationships” (line 470).", "questions": "1. It seems that factors such as benchmark contamination or formatting overlap between pretraining data and perplexity validation data could alter the relationship between domain perplexity and downstream performance for some observed models. This would introduce more structured noise than simple run-to-run variance due to differing training seeds. In the observed data do correlations vary dramatically for any subsets of models (i.e. are there subsets of models perhaps with systematically biased perplexity to downstream correlations)? Does your approach mitigate this possibility in any way other than by accumulating enough diverse observations that it’s unlikely a substantial number of models will be affected by the same biases?\n2. Is the projection method used in algorithm 1 when selecting data or only when predicting downstream performance ranking? Figure 6 implies that the data selection is different when using different projection methods but algorithm 1 doesn’t mention the projection step. \n3.  What does “features” refer to on line 344?\n4. Line 460: what is meant by “​​significant enrichment of the corresponding languages”\n5. Since the domain perplexities are measured over samples of 25 webpages, do the number of tokens used to estimate perplexity vary considerably from domain to domain? This may add noise to your perplexity estimates proportional to how severely some domains are subsetted with respect to tokens as observed by https://arxiv.org/abs/2312.10523 in their subsampling guideline.\n6. How are the standard deviations computed for the figure 4 ranking plots?\n\nOther comments (not questions and no response needed; wasn't sure where to put them):\n\nThe paper grounds the method in some theory by relating it to estimators for single index models.  This is not extremely important from my point of view, because the method is fairly intuitive, but it may give further intuition or justification to some readers.  The value may be in revealing the assumptions made by the model (through the lens of statistical estimation), so that they can be revised in future explorations.  I thought the discussion of the projections (4.2.3) was mostly unnecessary; I would have rather seen the experimental comparison of the different variants reported in the main paper and the derivations in the appendix, rather than the reverse.\n\nSuggestions (no response needed):\n\nIt would be interesting to see more analysis of specifically what domains are selected by the proposed approach contrasted with other baselines. Likewise how do the domains selected differ for different target downstream benchmarks (other than in terms of language)? Also are the most correlated domains also domains with few or many tokens? That is, does the final data selection only include a small subset of the positive correlation domains before satisfying the token budget or are many smaller domains selected?\n\nLine 93: There is some work which uses pretraining experiments to explore high dimensional mixtures of pretraining data (https://arxiv.org/abs/2410.15661).\n\nLine 97: There is work that directly shows the relationship between loss to downstream performance (https://arxiv.org/abs/2403.08540), though this work does not propose a data selection method.\n\nIn line 125 and 128 j switches from indexing documents and domains. Maybe use different indices to make this clearer.\n\nOn line 208 and 337 “single-page level” was confusing to me until I recalled that you are working with purely webtext and thus documents are webpages. It would be best to define this term before use to avoid possible confusion with other meanings of the word page.\n\nLine 458: avoid use of “significantly” when not discussing statistical significance.\n\nLine 526: “usual” should be “unusual” here, right?\n\nIn 4.2.1, I think the epsilons, not x, should be drawn from a standard normal.\n\nFigure 2 doesn’t clearly label methods the way they are referred to in the text.  It would be helpful to clearly mark, for example, the state-of-the-art methods from the DCLM paper as such and “our” methods (those introduced in this paper).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This approach provides a promising way to make use of “the millions of dollars collectively spent” (line 39) on the experimentation represented by all open weight models. The approach extracts information about performant pretraining data selection even when information about the pretraining data of these open-weight models is not available.\n\nTheir approach is intuitive and the core of it is quite simple (appearing to be insensitive to the trickier details suggested by theory). This means it should be straightforward for others to apply it.\n\nThe authors find that their approach achieves comparable results to a SOTA quality classifier that utilized extensive, costly pretraining experiments, while crucially the authors’ approach requires no new pretraining experiments, unlike other methods..\nThe paper provides a sound empirical validation of their approach at the 160M parameter scale. They use downstream evaluations that are appropriate for the parameter scale of their experiments (even including some basic multilingual evaluations). When optimizing for a single benchmark, the approach outperforms the state-of-the-art data selection method from DCLM if the latter is handicapped by removing the “manual filtering” step, but not if that manual filtering is included. Their set up demonstrates the utility of their approach for data with a very large number of domains (far greater than can be optimized with brute force pretraining experiments). These results should be more than enough to motivate further exploration of this technique at larger scales.\n\nI appreciated the additional analysis of outliers where the approach doesn’t do so well at predicting relative performance.  The observation that models with “unusual” data mixtures is interesting and will be useful in future work in this space.", "weaknesses": "I think the biggest weakness is that the paper’s method targets a single benchmark.  Are there obvious ways to extend the work to target some aggregation of benchmark scores?  Sure, but the paper offers no theory to suggest how we might do that, and no experiments to show whether the obvious ways of doing that would work.\nLambada, which accounts for 5 out of 8 of downstream evaluations, is a language modeling task (i.e. next word prediction) just like the perplexity measures being used as proxies in the proposed approach. This may somewhat obscure whether the proposed approach works as well for all kinds of downstream tasks which typically are not language modeling tasks. That said, the other 3 non-lambada tasks do also benefit from the proposed approach.\nIt is unclear in the empirical results of the paper what parts of the theoretical results are actually necessary for the evident success of the authors’ approach. The results in Appendix G do not conclusively favor the author’s proposed correlation measure nor their projection method. While the authors do say so themselves (line 467), it would be kinder to the reader to let the reader know this before they have waded through the dense theoretical sections of the paper. \nThe presentation of the paper also makes it unclear what theoretical results are necessary as the discussion of theory and methods are quite entangled. The paper would be much more readable if there was a simple discussion of what the proposed approach does and then only after that provide theoretical arguments for why this approach is used. I believe Algorithm 1 in the appendix provides the clearest end-to-end description of what exactly the proposed approach is. But even there I am left uncertain of details not mentioned there such as the theta projection or the use of empirical CDF. Is theta projection always used for the greedy data selection, or can the raw gamma values be used without projection since all you need is the order of the domains by the correlation?\nI believe the experiments work only at the internet domain granularity for selection.  The paper actually says very little about what a “domain” is – it seems to follow from variables defined in the redpajama data, and there are around 9000 “domains,” so the problem is of interesting size.  But I didn’t come away with a strong sense that – beyond empirical reports – there really should be a relationship between these particular “domains” and model performance on the benchmarks.  The paper would be stronger with some discussion of the qualitative assumptions that lead us to believe perplexity on specific slices of pretraining data (whether defined by internet domains or some other attributes) should be predictive of task performance. Are those assumptions grounded in any prior work?  Are they pervasive?  Are they novel?\n\nThe largest model on which the method is tested is only 160M parameters; it’s not clear whether the method will scale.  (I have no reason to believe it won’t, or that it will.)  This scale of model also implies a choice of target benchmarks that deviates pretty far from metrics most builders of language models would care about; I understand that the experiments needed to use benchmarks that would show meaningful differences at the 160M scale, but this creates even more distance between what the paper shows and what is needed in practical settings.\n\nApart from the most obvious case (based on language ID), the paper really doesn’t delve into what the predictive model itself tells us about the relationship between pretraining data composition and the benchmarks.  This is in part due to the unexciting set of benchmarks, I think.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXceAlktqn", "reviewer": "Reviewer_9vdE", "summary": "This paper proposes a new framework for pre-training data selection by leveraging large amounts of open-weight language models.  Concretely, a data domain is selected for pre-training when open-weight LLMs’ log probabilities on this domain highly correlate with performance on a target downstream task. The higher correlation a data domain demonstrates, the more weight the domain occupies the pre-training data. The proposed data selection algorithm is evaluated with small models of 160M parameters trained on a data scale of 3.2B tokens. Experimental results show that the proposed data selection algorithm, without relying on hand-crafted heuristics and expensive model training runs, approximately matches the handcrafted fastText baseline.", "questions": "- How can the proposed method be extended to selecting data for multiple downstream tasks, including unseen and/or dynamic benchmarks?\n- Will the data domains selected using models of a certain scale (e.g., < 8B) be helpful for training larger models (e.g., 70B)?  \n- How would varying the token ratio of pre-training domains affect the downstream performance? That is, mixing domains beyond \"selecting all the tokens in each domain once\" until token budget is exhausted.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper proposes a new framework for pre-training data selection without extensive manual labor by leveraging large amounts of pre-trained open-weight models.\n- The results validated on small 160M parameter models look promising.\n- The studied topic is important and relevant to many language model practitioners.", "weaknesses": "- It is unclear how well the proposed framework works for larger scale pre-training where multiple static benchmarks should be considered, and when the model size can be larger than the maximum model size (~8B) used for computing perplexity-benchmark correlation.\n- The theory section is a bit difficult to follow. In the paragraph above section 5.2 \"*This suggests that the performance of our approach is not dependent on the precise form of the estimator that is coupled to our theory results, but holds broadly across perplexity-correlation relationships.*\" If the theoretical result is less universal than empirical observation, it can be interesting to add more discussion on the more general relationship.\n- The proposed model provides the ordering of pre-training domains, however, from what I understand, the sampling ratio across domains is not determined by the benchmark-perplexity correlation, but rather “*select domains in greatest to least γ, taking all the tokens in each domain once, until we exhaust our total pretraining token budget.*” The discussion about the exact ratio to mix various domains is a little limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLK37DnVBH", "reviewer": "Reviewer_TyHg", "summary": "This paper proposes a novel approach to data selection for pretraining large language models (LLMs) using perplexity correlations. This work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark\nperformance, and selecting high-correlation documents is an effective pertaining data selection method. This approach is compute efficient and has competitive performance.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Using perplexity correlations for data selection is well-motivated and easy to implement.\n\n2. The paper provides a solid theoretical foundation for the proposed method, including a detailed discussion of the single-index model and the estimation of correlation coefficients. The mathematical derivations and proofs are rigorous and well-presented.\n\n3. In experiments, authors demonstrate that their method outperforms existing data selection techniques in terms of downstream benchmark performance.", "weaknesses": "1. While the paper demonstrates the effectiveness of the approach at the 160M parameter scale (though the data selection parameters are generated with LLM of 7B),  how the method would scale to larger models' pertaining remains unclear. The extrapolation from smaller models to larger ones could be a potential limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)